Secret Management¶

Core (a load-bearing practice on every real engagement, not an optional extra). · prereq 04 (Orchestration · Kubernetes & Helm), 05 (Cloud · AWS), 15 (CI/CD) — secrets flow through all three: a Kubernetes Secret backs a running pod, AWS IAM governs who can read a cloud-stored secret, and a CI/CD pipeline is where most secrets first enter the system. · ~1 day (4 reference submodules — why secrets never live in code, secret stores, Kubernetes secrets & GitOps, and rotation & least privilege). Reality check: this is the 80/20 on-ramp; real fluency (running Vault in production, writing an External Secrets sync, designing a rotation schedule that doesn't break running services) is built on the job.

Every module up to this point has quietly assumed a password, API key, or token from somewhere — a kubeconfig, an AWS credential, a database password in a CI/CD workflow. This module is where that assumption gets a name: a secret is any value that grants access and must never be readable by anyone who shouldn't have it, and the discipline of keeping it that way — out of code, out of images, rotated, and handed out on a need-to-know basis — is what separates a toy setup from one you'd trust in production.

TL;DR — in 30 seconds:

A secret that reaches git history or a container image layer is compromised forever — deleting the file later doesn't remove it from either place.
A secret store (Vault, AWS Secrets Manager, SSM Parameter Store) holds the real value; everything else — a pod, a pipeline, a script — asks the store for it at runtime, instead of carrying a copy.
Git-friendly secrets (SOPS, Sealed Secrets) work by encrypting the value, not by hiding the file — the encrypted blob is safe to commit; only the right key can turn it back into the real secret.

Learning objectives¶

By the end, the junior can:

Explain why a secret committed to git or baked into an image layer is compromised permanently, even after the file is deleted or the image is rebuilt.
Compare a secret store (Vault, AWS Secrets Manager, SSM Parameter Store) to hardcoding a value, and explain what "ask at runtime instead of carrying a copy" actually buys you.
Explain what a Kubernetes Secret actually protects against (and doesn't — base64 is encoding, not encryption), and how SOPS or Sealed Secrets make a secret safe to commit to a GitOps repo.
Explain least-privilege access and why a short-lived, scoped credential is safer than a long-lived one, even if both are "kept secret."
Describe what secret rotation means in practice and why an un-rotated secret's risk only grows over time.

Do it with Claude

Pair, Do and Prove run in Claude Code, with the onboarding-tutor skill. Click a button to copy the exact prompt, then paste it into Claude Code in this repo.

1. Learn · acquire the concept¶

No external course — read this brief, then go deeper in the four reference pages below.

Mental model (20 minutes):

A secret's lifecycle has four concerns, and each one closes a gap the previous one leaves open:

flowchart LR
    A["Never in code<br/>keep it out of git & images"] --> B["Central store<br/>Vault · Secrets Manager · SSM"]
    B --> C["Safe to deploy<br/>K8s Secrets · SOPS · Sealed Secrets"]
    C --> D["Stays safe over time<br/>rotation · least privilege"]

Never in code — the first rule: a secret in a .env committed to git, or COPY-ed into a Dockerfile layer, is exposed the moment that history or image is readable by anyone — including a public repo fork or a leaked image, long after the file itself is "removed."
Central store — instead of copying a secret everywhere it's needed, one system holds the real value and everything else authenticates to it and asks, at runtime, for exactly what it needs.
Safe to deploy — a Kubernetes Secret gets the value into a running pod, but a raw Secret manifest is not safe to commit (base64 is not encryption); SOPS and Sealed Secrets solve that by encrypting the value itself before it ever reaches git.
Stays safe over time — a secret that's correct today but never rotated and never scoped down is a growing liability; rotation and least-privilege access are what keep "safe now" from becoming "safe forever" by accident.

Go deeper: Why secrets never live in code · Secret stores · Kubernetes Secrets & GitOps · Rotation & least privilege

Check yourself — you commit a secret, then delete it in the next commit. Is it safe now?

No. Git history still contains the old commit with the secret in it — anyone with clone access (or access to a fork made before the deletion) can still read it. The secret must be treated as compromised and rotated (replaced with a new value), not just removed from the latest commit.

Check yourself — what's the actual difference between a Kubernetes Secret and a ConfigMap?

Almost nothing at the storage layer by default — a Secret's values are base64-encoded, not encrypted, and anyone who can read the Secret object (or etcd) can decode them trivially. The real protections (encryption at rest, RBAC restricting who can get/list Secrets) have to be turned on deliberately; a Secret is a signal of intent ("treat this as sensitive"), not a guarantee by itself.

Done when: you can name, in order, the four concerns this module builds on each other (never in code → central store → safe to deploy → stays safe over time), before moving to Pair.

2. Pair · passive → active¶

Drive the onboarding-tutor:

"Give me a snippet of code with a hardcoded API key and make me identify every place that key is now exposed once it's committed and pushed."
"Quiz me on when I'd reach for Vault versus AWS Secrets Manager versus SSM Parameter Store for a given scenario, and make me justify the choice."
"Give me a Kubernetes Secret manifest and make me explain exactly what protects it and what doesn't — then show me how SOPS or Sealed Secrets would change that."
"Walk me through designing least-privilege access for a service that only needs to read one specific secret, and make me spot where an overly broad policy would have granted more."
"Quiz me on why a rotation schedule matters even for a secret that hasn't leaked — what's actually growing riskier over time?"

Common gotchas

Treating .gitignore as protection for a secret already committed. Fix: .gitignore only stops future commits from including a file — it does nothing for a secret already in history. That requires rotating the secret (and, separately, scrubbing history if the repo must stay public).
Assuming a Kubernetes Secret is encrypted because it's called "Secret." Fix: base64 is an encoding, trivially reversible — real protection comes from encryption at rest (a cluster setting) and RBAC restricting who can read Secret objects, not the name of the API object.
Committing a raw Secret manifest to a GitOps repo "just this once." Fix: a GitOps repo is read by anyone with repo access — a plaintext (or base64) Secret in it is exposed to that whole audience; always encrypt with SOPS or Sealed Secrets before it goes in git.
Granting a service broad, long-lived credentials "to be safe." Fix: broad and long-lived is the opposite of safe — it maximizes what a single leak exposes and for how long. Scope to exactly what's needed, and prefer short-lived credentials the store issues on demand over static ones.

Done when: you can explain why deleting a committed secret doesn't undo the exposure, and why a Kubernetes Secret alone isn't encryption, without notes.

3. Do · produce an artifact¶

Exercise — a secrets audit and remediation writeup:

Pick a small script, Dockerfile, or Kubernetes manifest (yours from an earlier module, or one you write for this exercise) that currently has — or plausibly could have — a hardcoded secret (an API key, password, or token).
Write secrets-audit.md documenting: where the secret currently lives (or would live) and every place it becomes exposed from there (git history, image layers, logs, a shared file).
For each exposure, write the specific fix: moving the value out of the file/image and into an environment variable sourced from a secret store at runtime; adding it to .gitignore before first commit if it must exist as a local file; or, if it was already committed, noting that it must be rotated, not just removed.
Redo the artifact "secured": the script/Dockerfile/manifest with the secret removed and replaced by a reference to where it would come from at runtime (an env var, a mounted Secret, a store lookup) — include one sentence per change on which store (Vault / Secrets Manager / SSM / K8s Secret) you'd use and why.
State one least-privilege detail: exactly what this credential should be scoped to access, and nothing more.

Done when: secrets-audit.md names every real exposure point for your chosen artifact, the "secured" version has zero hardcoded secrets, and each fix names a specific store and a specific least-privilege scope — not just "use a secret manager."

4. Prove · understanding gate¶

Mandatory. Pass bar in the Socratic gate.

Exposure: Walk through exactly how a secret committed to git, then deleted in a later commit, is still exposed — what would someone need to still read it?
Store choice: Why did you pick the specific store (Vault / Secrets Manager / SSM Parameter Store / Kubernetes Secret) for each exposure in your audit, and not one of the others?
GitOps safety: If this credential needed to live in a GitOps repo, what would make it safe to commit — and what specifically makes a raw Secret manifest not safe?
Least privilege: What's the narrowest access this credential actually needs, and what would a broader grant have exposed unnecessarily if it leaked?
Provenance: Which specific recommendation in your audit did the tutor suggest, and how did you verify it was correct — by checking documented behavior, not by trusting the explanation?

Pass bar: you can defend every fix in your audit by the specific exposure it closes — not just "it's best practice" — and show how you verified a tutor-suggested detail against real documentation. "It works but I can't say why" = back to Pair.

5. Retain · fight forgetting¶

This module has no generated Anki deck yet. Recall by re-reading the four reference pages and re-running the Do exercise's audit against a different artifact (a different script, Dockerfile, or manifest) until you can name the right store and the right least-privilege scope from memory, unprompted. Done when you can name both, cold, for any artifact someone hands you.

Key takeaways¶

A secret that reaches git history or an image layer is compromised permanently — deleting the file later removes it from the working tree, not from history or from any image already built.
A secret store (Vault, AWS Secrets Manager, SSM Parameter Store) exists so nothing has to carry a copy of a secret — it asks the store for the value at runtime, authenticated on its own terms.
A Kubernetes Secret's base64 encoding is not encryption — real protection is encryption at rest plus RBAC; SOPS and Sealed Secrets make a secret's committed value itself encrypted, which is what actually makes it safe to put in a GitOps repo.
Least privilege means scoping a credential to exactly what it needs, nothing more — the smaller the blast radius, the less a single leak costs.
Rotation treats "hasn't leaked yet" as temporary, not permanent — an un-rotated secret's risk only grows the longer it's lived and the more places it's touched.

Go deeper — curated resources¶

Hand-picked external material to take this topic further — the best books, courses, talks, and writing. Cost is tagged Free, Free online (full text/course free on the web), or Paid.

Docs · HashiCorp Vault documentation — HashiCorp. The canonical reference for Vault's secret engines, dynamic secrets, and access policies. Free.
Docs · Kubernetes Secrets — Kubernetes documentation. The authoritative explanation of what a Secret object does and doesn't protect. Free.
Tool · SOPS — Mozilla / getsops. The tool for encrypting values inside YAML/JSON/ENV files so they're safe to commit to a GitOps repo. Free.
Tool · Sealed Secrets — Bitnami. A Kubernetes controller that turns a Secret into an encrypted SealedSecret custom resource only the cluster can decrypt. Free.
Docs · AWS Secrets Manager vs. Systems Manager Parameter Store — AWS. Explains when each AWS secret-storage option fits. Free.

Gate log (mentor fills on pass)¶

Date passed:
Passed by: / checked by:
Weak spots noticed (feeds system improvement):