Skip to content

Kubernetes — core concepts for operating a cluster

Ask someone to sketch "a Kubernetes cluster" and most reach for the object model — Pods, Deployments, Services — because that's what the YAML looks like. But the object model is only what you ask for; it says nothing about who accepts the ask, who remembers it after you close your laptop, or who notices when reality drifts from it. That's the job of a small set of components most tutorials skip on the way to kubectl apply, and it's exactly where a broken Pod actually breaks.

Scope: the concepts you need to run and debug workloads on an existing cluster — the control plane, configuration and secrets, access control (RBAC), and how to read a broken Pod. The object model itself is introduced in the Foundations chapter DevOps with Kubernetes; this page goes one layer deeper, to the parts a junior is expected to defend at the gate. Cluster provisioning is the cloud's job (see the AWS module for EKS); authoring charts is the Helm course, next in this module.

TL;DR — in 30 seconds:

  • A cluster is a control plane (api-server, etcd, scheduler, controller-manager) deciding what should run, plus worker nodes (kubelet, kube-proxy, container runtime) actually running it.
  • ConfigMap holds non-secret config; Secret holds sensitive values — a Secret is base64, not encrypted, unless the cluster adds etcd encryption or an external store.
  • RBAC (Role/ClusterRole + Binding + ServiceAccount) governs who can do what; debugging a broken Pod follows one loop: get → describe → logs --previous → events.

1. What a cluster is made of

A cluster is a control plane (the brain — decides what should run) and a set of worker nodes (the muscle — actually run your containers). You talk to the control plane; it drives the nodes.

flowchart TB
  kubectl["kubectl / CI"] -->|"every read & write"| api
  subgraph CP["Control plane (managed by AWS in EKS)"]
    api["kube-apiserver<br/>the only front door"]
    etcd[("etcd<br/>cluster state = source of truth")]
    sched["kube-scheduler<br/>picks a node for each Pod"]
    ctrl["controller-manager<br/>reconcile loops"]
    api <--> etcd
    sched --> api
    ctrl --> api
  end
  subgraph N1["Worker node (data plane — you own it)"]
    kubelet["kubelet<br/>runs & health-checks Pods"]
    proxy["kube-proxy<br/>Service networking"]
    rt["container runtime<br/>(containerd)"]
    kubelet --> rt
  end
  api -->|"assigns work"| kubelet

Control-plane components

  • kube-apiserver — the single front door. Every read and write (yours, and every other component's) goes through it. It is the only thing that talks to etcd.
  • etcd — the cluster's key-value store: the source of truth for what exists and what should be running. Lose etcd without a backup and you lose the cluster's memory.
  • kube-scheduler — watches for Pods with no node yet and picks one, based on resource requests, taints/tolerations, and affinity rules.
  • controller-manager — runs the reconcile loops. A controller's whole job is observe desired vs actual, act to close the gap (e.g. the Deployment controller creates ReplicaSets; the ReplicaSet controller creates Pods to hit the replica count). This loop is the heart of how Kubernetes works.
  • cloud-controller-manager — integrates with the cloud (provisions load balancers for Service type= LoadBalancer, wires node lifecycle). In EKS this is AWS's.

Node components (on every worker)

  • kubelet — the node agent. Takes the Pods assigned to its node, tells the runtime to start their containers, and reports health back to the api-server.
  • kube-proxy — programs the node's network rules (iptables/IPVS) so a Service's virtual IP load- balances to its Pods.
  • container runtime — containerd (or CRI-O) actually runs the containers. This is the same container you met with Podman; Kubernetes just schedules it.

The control-plane / data-plane split. In a managed cluster like EKS, AWS runs and scales the control plane for you; you own the worker nodes (the data plane) that run your Pods — or hand even those to Fargate. This is the split the AWS module refers to.

2. Configuration: ConfigMap and Secret

Never bake config or credentials into an image — inject them at run time. Kubernetes has two objects for this, identical in shape, different in intent:

Object For How it's stored How a Pod consumes it
ConfigMap non-secret config (URLs, flags, files) plain text in etcd env vars, or files mounted from a volume
Secret sensitive values (tokens, passwords, TLS) base64, not encrypted by default env vars, or files mounted from a volume

Warning — a Secret is not encrypted. base64 is encoding, not encryption; anyone who can read the Secret object can decode it. A Secret buys you access control (RBAC can restrict who reads Secrets) and keeps values out of the image and manifest — not confidentiality at rest, unless the cluster enables etcd encryption (or an external store like AWS Secrets Manager / Sealed Secrets). Treat "it's a Secret" as "gate access to it," never as "it's safe in the open." (Go deeper: the upstream Encrypting Confidential Data guide — optional, not required to defend this.)

Consuming as env vars is simplest; mounting as a volume lets the app re-read a file when the value rotates without a redeploy.

3. Access control: RBAC

RBAC (Role-Based Access Control) decides who can do what to which resources. Four object types, which split cleanly into "what's allowed" and "who gets it":

flowchart LR
  subgraph who["Subject (who)"]
    sa["ServiceAccount<br/>(a Pod's identity)"]
    user["User / Group<br/>(a human)"]
  end
  subgraph grant["Binding (the grant)"]
    rb["RoleBinding<br/>(in one namespace)"]
    crb["ClusterRoleBinding<br/>(cluster-wide)"]
  end
  subgraph what["Role (what's allowed)"]
    role["Role<br/>verbs on resources,<br/>one namespace"]
    crole["ClusterRole<br/>cluster-wide /<br/>non-namespaced"]
  end
  sa --> rb --> role
  user --> crb --> crole
  • Role / ClusterRole — a set of permissions: verbs (get, list, watch, create, update, delete) on resources (pods, secrets, deployments…). A Role is scoped to one namespace; a ClusterRole is cluster-wide or covers non-namespaced things (nodes, namespaces themselves).
  • RoleBinding / ClusterRoleBinding — grants a Role to subjects: a ServiceAccount (the identity a Pod runs as), a user, or a group.
  • Least privilege — grant the narrowest Role that works. A workload that only reads one ConfigMap should not get cluster-admin.

Connects to AWS. On EKS, IRSA maps a Kubernetes ServiceAccount to an IAM role via OIDC, so a Pod gets AWS permissions without static keys — RBAC governs what it can do inside the cluster, IAM what it can do in AWS. Same least-privilege idea, two control planes. (Go deeper: the upstream Using RBAC Authorization guide — optional, not required to defend this.)

4. Debugging a broken Pod

Almost every "why isn't my app up?" is solved with the same four commands, in order:

flowchart LR
  a["kubectl get pods<br/>what STATE is it in?"] --> b["kubectl describe pod NAME<br/>read the Events at the bottom"]
  b --> c["kubectl logs NAME --previous<br/>what did the app print before it died?"]
  c --> d["kubectl get events --sort-by=.lastTimestamp<br/>cluster-level context"]
  • get tells you the state; describe shows the Events (the single most useful output — scheduling failures, image pulls, probe failures all land here); logs --previous shows what a crashed container printed before it restarted.

The states you must recognize:

Pod state What it means First thing to check
Pending not scheduled onto a node yet describe → insufficient CPU/memory, no matching node, or an unbound PersistentVolumeClaim
ContainerCreating scheduled, still pulling image / mounting volumes usually transient; if stuck, describe for a volume or secret mount error
ImagePullBackOff / ErrImagePull can't pull the image wrong image/tag, or missing registry credentials (a Secret)
CrashLoopBackOff container starts, exits, and is restarted — repeatedly, with growing backoff logs --previous: bad command, missing config/secret, or a failing liveness probe
OOMKilled container exceeded its memory limit describe → Last State; raise the limit or fix the leak
Running but 0/1 Ready container up, readiness probe not passing the probe endpoint/port, or a dependency it waits on

Probes drive much of this:

  • liveness — "is it wedged?" Fails → kubelet restarts the container (a mis-tuned liveness probe is a classic cause of CrashLoopBackOff).
  • readiness — "can it serve traffic?" Fails → the Pod is pulled out of its Service's endpoints (no restart), so traffic stops routing to it.
  • startup — a grace window for slow starters before liveness kicks in.

5. How this connects

  • The container the runtime starts is the same one you built with Podman (module 03, Containers).
  • Helm (next in this module) packages these very objects — Deployments, Services, ConfigMaps, Secrets, RBAC — into a versioned, parameterized release.
  • EKS (module 05, AWS) is this control plane, run and scaled by AWS, with IRSA bridging RBAC and IAM.

Common gotchas

  • Treating a Secret as encrypted just because it's called "Secret." Fix: it's base64 — encoding, not encryption. RBAC access control (and, if enabled, etcd encryption or an external secret store) is what actually protects it, not the encoding.
  • Jumping straight to kubectl logs when a Pod won't come up, skipping describe. Fix: for Pending/ImagePullBackOff/scheduling problems, the Events block in describe explains the failure — logs only exist once a container has actually started.
  • Granting cluster-admin (or a broad Role) to a workload's ServiceAccount because a narrow Role feels fiddly to write. Fix: least privilege — grant only the verbs/resources a workload actually needs; a ClusterRoleBinding is cluster-wide, so an overbroad grant has cluster-wide blast radius.
  • Confusing "the Pod is Running" with "the Pod is ready to serve traffic." Fix: Running describes container state; check the READY column / readiness probe status — a Running-but-not-Ready Pod is pulled from Service endpoints and gets no traffic.
Check yourself — is a Kubernetes Secret encrypted at rest by default?

No — its values are base64-encoded, not encrypted; base64 is trivially reversible by anyone who can read the object. RBAC access control, and (if enabled) etcd encryption or an external secret store, are what actually protect it — not the base64 encoding itself.

Check yourself — a Pod shows Running but 0/1 Ready. What's happening, and what's the first thing to check?

The container process started (Running), but its readiness probe is failing, so the kubelet has pulled the Pod out of its Service's endpoints — no restart happens, unlike a liveness failure. Check what the readiness probe expects (a port/path/command) and whether the app is actually up yet or blocked on a dependency.

You can defend this when you can name the control-plane components and what each does, explain why a Secret isn't encrypted, read an RBAC binding, and walk a CrashLoopBackOff from get to root cause.