In Part 1, we dug into Pods, Deployments, and Services. In Part 2 we looked at configuration and persistence. In Part 3 we went over what’s left in terms of building your own things. In this part, we’ll talk about RBAC, Helm, and then put it all together with a capstone exercise to put it all together.

Map of the territory

RBAC

Everything we’ve done so far has been performed as a full administrator. In a real Kubernetes cluster, most users will have a more limited set of privileges (for example: different teams probably shouldn’t be able to mess with each others’ namespaces).

Core elements

  • Service Account: an identity for a non-human actor (such as a Pod, a CI pipeline, etc.)
  • Permission: a verb such as get / list / create / delete, applied to a resource such as pods, deployments.
  • Role: a set of permissions, scoped to one namespace
  • ClusterRole: a set of permissions that are not scoped to a namespace, but apply to the whole cluster (or for cluster-scoped resources like Pods and namespaces themselves)
  • RoleBinding / ClusterRoleBinding: attachment of a Role or ClusterRole to a principal (a ServiceAccount, User, or Group)

Create a restricted ServiceAccount

First, create a ServiceAccount called “readonly-sa” in a new namespace:

kubectl create namespace rbac-demo
kubectl create serviceaccount readonly-sa -n rbac-demo

As you might have guessed, we plan to make this service account a read-only account. Create a file called manifests/rbac-role.yaml with the following contents:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: rbac-demo
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: readonly-sa-binding
  namespace: rbac-demo
subjects:
  - kind: ServiceAccount
    name: readonly-sa
    namespace: rbac-demo
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

Then apply it:

kubectl apply -f manifests/rbac-role.yaml

Note we do not apply this manifest inside a namespace. Roles and RoleBindings are cluster-level objects.

Test permissions

Kubernetes has a handy “can-i” tool to let you know if you can do something without actually trying to do it:

kubectl auth can-i list pods -n rbac-demo --as=system:serviceaccount:rbac-demo:readonly-sa
kubectl auth can-i delete pods -n rbac-demo --as=system:serviceaccount:rbac-demo:readonly-sa
kubectl auth can-i list deployments -n rbac-demo --as=system:serviceaccount:rbac-demo:readonly-sa

The service account can indeed list Pods because the role we bound to it has the verb list applied to the resource pods. It cannot delete Pods because the delete verb is not present. It also cannot list deployments because we did not grant any verbs on the deployments resource.

See it live

We’ll spin up a pod using that service account and see what we can do.

Create a file called manifests/rbac-test-pod.yaml with the following contents:

apiVersion: v1
kind: Pod
metadata:
  name: rbac-test
  namespace: rbac-demo
spec:
  serviceAccountName: readonly-sa
  containers:
    - name: kubectl
      image: bitnami/kubectl
      command: ["sleep", "3600"]

And then hop into it and try things directly:

kubectl apply -f manifests/rbac-test-pod -n rbac-demo
kubectl exec -it rbac-test -n rbac-demo -- sh
# ~~inside the pod~~
kubectl get pods
kubectl delete pod rbac-test
kubectl get deployments
kubectl get pods -n default
exit

The first should succeed, and the subsequent two should fail, matching what we saw with the “can-i” tests. The last command should also fail (recall: a Role is scoped to a single namespace).

ClusterRoles

Create a file called manifests/rbac-clusterrole.yaml with the following contents:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: pod-reader-cluster
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: readonly-sa-cluster-binding
subjects:
  - kind: ServiceAccount
    name: readonly-sa
    namespace: rbac-demo
roleRef:
  kind: ClusterRole
  name: pod-reader-cluster
  apiGroup: rbac.authorization.k8s.io

Notice how this is almost identical to the Role we set up previously, except that it lacks any namespace directives. Apply it and try that last command again.

kubectl apply -f manifests/rbac-clusterrole.yaml
kubectl exec -it rbac-test -n rbac-demo -- sh
# ~~inside the pod~~
kubectl get pods -n default
exit

This time, now that we have been granted the ability to get/list/watch pods in all namespaces, this last command should succeed.

Cleanup

While Roles and RoleBindings are cluster-level objects, deleting the namespace they are bound to cascades down to them too. ClusterRoles and their bindings, however, are not bound to a namespace so must be deleted manually.

kubectl delete namespace rbac-demo
kubectl delete clusterrole pod-reader-cluster
kubectl delete clusterrolebinding readonly-sa-cluster-binding

Helm

Conveniently, even just going through these exercises we’ve felt some of the pain points that Helm alleviates. Some Helm terminology before we get too far into the weeds:

A Helm chart is a packaged set of Kubernetes manifests, along with metadata. This metadata includes things like references to other Helm charts as dependencies, post-install hooks (e.g. to populate a database you just deployed), and parametrization (e.g. allowing you to run 5 replicas in production, but only one replica in dev). A file called values.yaml contains the default parameters that a chart references, which you can override.

A Helm release is an instance of an installed chart in your cluster. You can install the same chart multiple times, side by side, with different names and configs.

Install something

Let’s install nginx via Helm:

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
helm install my-nginx bitnami/nginx --set service.type=ClusterIP
kubectl get pods,svc

When we installed the Helm chart, we got a fully working Deployment, Service, ConfigMaps, NetworkPolicies, etc.

Now that we’ve installed the chart, we can have a look at our Helm releases, and dig down into the one we just installed:

helm list
helm status my-nginx

We can also see what all it actually installed:

helm get manifest my-nginx

Upgrades and rollbacks

In Part 1 we looked at updating and rolling back a Deployment and Service. Helm has a smilar concept.

kubectl get pods
helm upgrade my-nginx bitnami/nginx --set replicaCount=3
kubectl get pods -l app.kubernetes.io/instance=my-nginx
helm history my-nginx
helm rollback my-nginx 1
helm history my-nginx

In contrast to kubectl rollout history, notice how helm history is has genuinely useful revision tracking.

Uninstalling

You can uninstall a release via its name:

helm uninstall my-nginx
kubectl get pods,svc

Build a chart

Helm has a command to automatically build out scaffolding for a chart:

helm create my-app
ls my-app

Look at my-app/templates/deployment.yaml and you’ll see something that’s a bit like a Deployment manifest, but with lots of template placeholders (i.e. {{ .Values.thing }}) sprinkled in. The actual values for these placeholders is set in my-app/values.yaml.

Edit my-app/values.yaml to set the replicaCount to 2, and the image.tag to be "1.25", then install it:

helm install test-release ./my-app
kubectl get pods -l app.kubernetes.io/instance=test-release

You can also override values at install time, without editing the chart. We’ll install a second release of this chart, with a different replicaCount:

helm install test-release-2 ./my-app --set replicaCount=5
kubectl get pods

Cleanup

helm uninstall test-release
helm uninstall test-release-2

Capstone

We’ll put together a small “notes” app. We’ll have a frontend, an API, and a Postgres database, all with proper namespacing, config, storage, health checks, ingress, and resource limits. Then we’ll package it all up as a Helm chart for good measure.

Namespace

kubectl create namespace notes-app
kubectl config set-context --current --namespace=notes-app

Database

Create and apply a manifest database.yaml to set up your database. Use it to create a Postgres StatefulSet (a single replica is fine), backed by a volumeClaimTemplate, with POSTGRES_PASSWORD sourced from a Secret. Include resource requests/limits, and a basic liveness probe. Recall that you can put multiple objects into a single manifest by separating them with a line with just three dashes (---).

When we looked at health checks before we only configured them with an httpGet property to hit an HTTP endpoint, and that’s not relevant to Postgres. Instead, we need to run a command (pg_isready -U postgres) for the liveness probe. Documentation for how to configure liveness probes with commands instead of HTTP requests is here.

Refer back as needed:

Back-end API

We’ll set up a quick Flask API to hit our database. In real life, you’d build yourself an image with your application code. But since app building isn’t our focus here, we’ll just use the base Python image and inject our code as a ConfigMap.

Create a file called backend-app.py with the following contents:

import os
import psycopg2
from flask import Flask, request, jsonify

app = Flask(__name__)

def get_db():
    return psycopg2.connect(
        host=os.environ["DB_HOST"],
        dbname=os.environ["DB_NAME"],
        user=os.environ["DB_USER"],
        password=os.environ["DB_PASSWORD"],
    )

def init_db():
    conn = get_db()
    cur = conn.cursor()
    cur.execute("CREATE TABLE IF NOT EXISTS notes (id SERIAL PRIMARY KEY, body TEXT NOT NULL)")
    conn.commit()
    cur.close()
    conn.close()

@app.route("/api/ping")
def ping():
    return jsonify(status="ok"), 200

@app.route("/api/health")
def health():
    try:
        conn = get_db()
        conn.close()
        return jsonify(status="ok"), 200
    except Exception as e:
        return jsonify(status="unavailable", error=str(e)), 503

@app.route("/api/notes", methods=["GET"])
def list_notes():
    conn = get_db(); cur = conn.cursor()
    cur.execute("SELECT id, body FROM notes ORDER BY id")
    rows = [{"id": r[0], "body": r[1]} for r in cur.fetchall()]
    cur.close(); conn.close()
    return jsonify(notes=rows)

@app.route("/api/notes", methods=["POST"])
def add_note():
    body = (request.get_json() or {}).get("body", "")
    if not body:
        return jsonify(error="body required"), 400
    conn = get_db(); cur = conn.cursor()
    cur.execute("INSERT INTO notes (body) VALUES (%s) RETURNING id", (body,))
    note_id = cur.fetchone()[0]
    conn.commit(); cur.close(); conn.close()
    return jsonify(id=note_id, body=body), 201

if __name__ == "__main__":
    init_db()
    app.run(host="0.0.0.0", port=8080)

Notice this sets up a GET and POST routes for /api/notes, representing our API, and also an /api/health and /api/ping for use with health checks. Review the implementation to help determine which is which.

Now we’ll shove this whole file into a ConfigMap so we can easily pull it into a container without having to build a custom image:

kubectl create configmap backend-code --from-file=backend-app.py -n notes-app

Note that our code relies on four environment variables (DB_HOST, DB_NAME, DB_USER, DB_PASSWORD) to connect to the database. We already have the password in a Secret from setting up the database, so create another ConfigMap called backend-config to house the three remaining. By default, the Postgres user and database both have a value of postgres.

Create and apply the manifest backend.yaml to create the Deployment and the Service.

For the Deployment:

  • use 3 replicas.
  • use the image python:3.12-slim.
  • mount the database connection parameters and secret as environment variables, and mount the code ConfigMap as a file.
  • use the container’s command list to install the dependencies (pip install flask psycopg2-binary) and run python on the app file you mounted in.
  • add the readiness and liveness probes, and a startup probe too since your start command involves installing dependencies.

For the Service, set up your selector, port, and targetPort appropriately: configure it such that when your service receives a request on port 80, it connects to your Flask app listening on port 8080.

Refer back as needed:

Frontend

We’ll set up an Nginx Deployment to serve our static frontend page, which we’ll just serve from a ConfigMap like we did the backend API python script.

Create a file index.html with the following contents:

<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  <title>Notes</title>
</head>
<body>
  <h1>Notes</h1>
  
  <form id="form">
      <input id="body" placeholder="new note" required>
      <button>Add</button>
  </form>
  
  <ul id="notes"></ul>
  
  <script>
      async function load() {
      const res = await fetch('/api/notes');
      const data = await res.json();
      document.getElementById('notes').innerHTML =
          data.notes.map(n => `<li>[${n.id}] ${n.body}</li>`).join('');
      }
  
      document.getElementById('form').onsubmit = async (e) => {
      e.preventDefault();
      const input = document.getElementById('body');
      await fetch('/api/notes', {
          method: 'POST',
          headers: { 'Content-Type': 'application/json' },
          body: JSON.stringify({ body: input.value })
      });
      input.value = '';
      load();
      };
  
      load();
  </script>
</body>
</html>

Create a configmap called frontend-html with just the above file in it.

Then create and apply a manifest frontend.yaml to set up an nginx Deployment and Service. Configure the Deployment with a single replica, and mount the ConfigMap into Nginx’s docroot (/usr/share/nginx/html).

Refer back as needed:

Ingress

Create and apply a manifest ingress.yaml to set up an Ingress which routes:

  • / to the Frontend Service
  • /api to the Backend API Service

At this point, you should be able to visit http://localhost:8080 on your host machine, and interact with the application. If so, great job! Save a couple notes.

Then delete the database pod:

kubectl delete pod db-0

and watch it come back up automatically, with no loss of data.

Refer back as needed:

Setting limits

First, set up a ResourceQuota on the notes-app namespace, sized sensibly to fit your three components’ combined requests and limits.

Next, configure RBAC for a hypothetical CI deployer: create a ServiceAccount with a Role permitting only get/list/create/update/patch on deployments/services/configmaps within the notes-app namespace.

Refer back as needed:

Package with Helm

Convert your manifests into a chart: start by running helm create notes-app, then replace the scaffolded templates with your real ones, parameterizing image tags, replica counts, and resource limits via values.yaml.

Install it helm install, verify with helm get manifest, then perform an upgrade by increasing the replica count for the backend API.

Refer back as needed:

Clean up

helm uninstall notes-app
kubectl delete namespace notes-app

Next steps

Now go build something cool :)