Kustomize Skill
Overview
Template-free, declarative Kubernetes config management via merging. kubectl apply -k <dir> and kustomize build <dir> render an overlay. Failure modes here are overwhelmingly silent (wrong output, exit 0) — always diff the kustomize build output before/after any structural change.
Core Concepts
| Term | Meaning |
| --------------- | ------------------------------------------------------------------------------------------------ |
| Base | Dir with kustomization.yaml + shared resources; never modified in place. |
| Overlay | Dir referencing a base under resources:, applying env-specific patches/generators. |
| Patch | Partial resource that modifies existing ones — strategic merge (default) or JSON 6902. |
| Component | Reusable opt-in feature bundle (kind: Component); runs against the parent's accumulated set. |
| Generator | Builds ConfigMaps/Secrets from literals/files/envs, appending a content-hash name suffix. |
| Transformer | Cross-cutting mutation: namespace, namePrefix/nameSuffix, labels, images, replicas. |
| Replacement | Copies a field value from a source resource into target field paths (successor to vars). |
Directory Layout
k8s/
├── base/ # shared, env-agnostic (kustomization.yaml + resources)
├── overlays/
│ ├── dev/ # low resources, debug
│ ├── staging/ # + monitoring component
│ └── prod/ # HA, security, digest-pinned images
└── components/ # opt-in features: monitoring, ingress, debug-tools
Keep the hierarchy shallow (base → overlay, not base → overlay1 → overlay2). Paths in resources: are relative to the kustomization.yaml's own directory.
Quick Reference
| Task | Command |
| ---------------- | -------------------------------------------------------------------------- |
| Build | kustomize build k8s/overlays/prod |
| Preview diff | kubectl diff -k k8s/overlays/prod |
| Apply | kubectl apply -k k8s/overlays/prod |
| Validate | kustomize build k8s/overlays/prod \| kubectl apply --dry-run=client -f - |
| Migrate deprecated fields | kustomize edit fix (idempotent; run per dir; see Idioms) |
| Set image tag | kustomize edit set image myapp=registry/myapp:v1.2.3 |
| Add ConfigMap | kustomize edit add configmap app-config --from-literal=KEY=value |
Generators (configMapGenerator / secretGenerator items): literals: (K=v), files: (app.properties, or alias=path), envs: (config.env). Secrets add type: (e.g. kubernetes.io/tls). behavior: = create (default) | merge | replace.
Transformers (top-level keys): namespace, namePrefix, nameSuffix, labels (use over deprecated commonLabels), commonAnnotations, images (newName/newTag/digest), replicas (- name: x count: n), replacements.
Patch Strategies
| Aspect | Strategic Merge (default) | JSON 6902 (RFC 6902) |
| ------------- | ----------------------------------------------------------- | ------------------------------------------------------ |
| Merge model | K8s-schema-aware; maps by key, lists merge on known key | op (add/remove/replace/move/copy/test) on JSON Pointer paths |
| Best for | Field/env/resource-limit tweaks, adding containers | Precise array-index ops, conditional test, CRDs |
| Whole-list replace | Unreliable — $patch: replace regressed (issue #2980, ignored/merged instead) | Reliable: op: replace on the list path |
Both go under the unified patches: field, which auto-detects type from content and whose target supports group/version/kind/name/namespace + labelSelector/annotationSelector (one patch → many resources). patchesStrategicMerge/patchesJson6902 are deprecated (v5.0.0) and gone from the v1 API.
Components
Reusable opt-in bundles for features only some environments need (monitoring, ingress, debug tools, security hardening). Use a patch instead for required, always-applied, or env-specific value tweaks.
# k8s/components/monitoring/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1alpha1 # alpha — API may change without deprecation window
kind: Component
resources:
- servicemonitor.yaml
patches:
- path: patch-metrics.yaml
target: {kind: Deployment}
labels:
- pairs: {prometheus.io/scrape: "true"}
includeSelectors: false
Reference Components under
components:, neverresources:. A Component placed underresources:is silently misapplied as a plain manifest; kustomize enforces the split (Components rejected fromresources:, Kustomizations fromcomponents:). Components run against the parent's accumulated resource set and can be nested.
Examples
Base kustomization + Deployment
# k8s/base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: myapp
# Use the labels transformer, NOT deprecated commonLabels (which mutates the
# IMMUTABLE spec.selector.matchLabels → delete/recreate on any later change).
# includeSelectors defaults false (safe on live resources); includeTemplates
# reaches pod templates. Define real selector labels directly in deployment.yaml.
labels:
- pairs: {app: myapp, managed-by: kustomize}
includeSelectors: false
includeTemplates: true
commonAnnotations: {contact: team@example.com}
resources:
- deployment.yaml
- service.yaml
- serviceaccount.yaml
configMapGenerator:
- name: app-config
literals: [LOG_LEVEL=info, MAX_CONNECTIONS=100]
images:
- name: myapp
newName: registry.example.com/myapp
newTag: latest
# k8s/base/deployment.yaml (selector label defined explicitly — not injected)
apiVersion: apps/v1
kind: Deployment
metadata: {name: myapp}
spec:
replicas: 1
selector:
matchLabels: {app: myapp}
template:
metadata:
labels: {app: myapp}
spec:
serviceAccountName: myapp
containers:
- name: app
image: myapp
ports: [{containerPort: 8080, name: http}]
envFrom: [{configMapRef: {name: app-config}}]
resources:
requests: {memory: 128Mi, cpu: 100m}
limits: {memory: 256Mi, cpu: 200m}
readinessProbe:
httpGet: {path: /ready, port: http}
initialDelaySeconds: 5
Production overlay (annotated)
# k8s/overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: myapp-prod
labels:
- pairs: {environment: prod, criticality: high}
includeSelectors: false
includeTemplates: true
commonAnnotations:
oncall: sre-team@example.com
# SINGLE resources: block. A second resources: key later in the same document
# silently wins (duplicate YAML map keys), dropping ../../base entirely.
resources:
- ../../base
- poddisruptionbudget.yaml
- horizontalpodautoscaler.yaml
- networkpolicy.yaml
patches:
- path: patch-resources.yaml # bump requests/limits
- path: patch-affinity.yaml # required podAntiAffinity + nodeAffinity
- path: patch-security.yaml # runAsNonRoot, drop ALL, RO rootfs, seccomp
configMapGenerator:
- name: app-config
behavior: merge
envs: [config-values.env]
secretGenerator:
- name: app-secrets
envs: [secrets.env] # gitignored / external-managed
images:
- name: myapp
newTag: v1.2.3
digest: sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 # pin by digest in prod
replicas:
- {name: myapp, count: 5}
components:
- ../../components/monitoring
- ../../components/ingress
Overlay patch files are minimal partial docs matched by kind/name, e.g. a security patch:
# patch-security.yaml
apiVersion: apps/v1
kind: Deployment
metadata: {name: myapp}
spec:
template:
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1000
seccompProfile: {type: RuntimeDefault}
containers:
- name: app
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: {drop: [ALL]}
JSON 6902 patch (precise array ops)
# kustomization.yaml
patches:
- path: json-patch.yaml
target: {group: apps, version: v1, kind: Deployment, name: myapp}
# json-patch.yaml — list of {op, path, value}; auto-detected as JSON 6902
- op: add # append sidecar
path: /spec/template/spec/containers/-
value: {name: log-shipper, image: fluent/fluent-bit:2.0}
- op: replace # container 0 image
path: /spec/template/spec/containers/0/image
value: registry.example.com/myapp:v1.2.3
- op: remove # env var by index
path: /spec/template/spec/containers/0/env/3
- op: test # conditional guard
path: /spec/replicas
value: 1
- op: replace
path: /spec/replicas
value: 5
Generators
configMapGenerator:
- name: app-config
behavior: merge # create (default) | replace | merge
literals: [LOG_LEVEL=debug] # overrides base key
files: [app.properties, tls.crt=certs/server.crt]
envs: [config.env]
options:
disableNameSuffixHash: true # ⚠ defeats config-driven rollouts (see Gotchas)
secretGenerator:
- name: tls-secrets
files: [tls.crt, tls.key]
type: kubernetes.io/tls
- name: app-secrets
envs: [secrets.env] # gitignored — never commit secrets
Replacements (propagate a field value)
replacements:
# Propagate hashed ConfigMap name into the Deployment volume ref
- source: {kind: ConfigMap, name: app-config, fieldPath: metadata.name}
targets:
- select: {kind: Deployment}
fieldPaths:
- spec.template.spec.volumes.[name=config].configMap.name
Remote bases
resources:
- https://github.com/org/repo//k8s/base?ref=v1.0.0 # pin to tag or full SHA (see Security)
Common Tasks
# Validate / preview
kustomize build k8s/overlays/prod | kubectl apply --dry-run=server -f -
kubectl diff -k k8s/overlays/prod
# Inspect name/hash transformations
kustomize build k8s/overlays/prod | rg '^ name:'
kustomize build k8s/overlays/prod | rg -A2 'kind: ConfigMap'
# Migrate deprecated fields in place (idempotent; wrap in a build-diff)
kustomize edit fix
# Bootstrap kustomization.yaml from existing manifests
kustomize create --autodetect # or --resources deployment.yaml,service.yaml
GitOps & CI/CD Integration
ArgoCD: Application.spec.source.path: k8s/overlays/prod, syncPolicy.automated {prune, selfHeal}.
Flux: kustomize.toolkit.fluxcd.io/v1 Kustomization with spec.path, prune: true, sourceRef (GitRepository), healthChecks.
Both bundle their own kustomize version — pin/match it (see Currency gotcha). Prefer the controller's authenticated Git source over remote bases in resources:.
# CI: render, then validate. kubeval is unmaintained — use kubeconform.
kustomize build "k8s/overlays/${OVERLAY}" > all.yaml
kubeconform -strict -summary -kubernetes-version 1.29.0 all.yaml
conftest test all.yaml # OPA policy gate
Helm inflation gotcha
helmChartsrequireskustomize build --enable-helm. Without the flag the wholehelmChartssection is silently skipped (no error), so CI can emit manifests missing entire components yet still "pass".kubectl apply -kCANNOT pass--enable-helm(unknown flag), makinghelmChartseffectively dead there — usekustomize build --enable-helm | kubectl apply -f -. It's a deliberately limited subset (no private-registry auth, no post-renderers). For anything nontrivial,helm templatethen kustomize-patch on top.
Troubleshooting
| Error (excerpt) | Cause / Fix |
| -------------------------------------------- | -------------------------------------------------------------------------- |
| evalsymlink failure ... no such file | Wrong base path; paths are relative to the kustomization.yaml's own dir. |
| no matches for OriginalId ... ConfigMap | Patched resource missing in base, or name/kind mismatch. |
| conflict: multiple matches | Patch too broad — add metadata or use JSON 6902 for precise targeting. |
| base ... refers back to overlay (cyclic) | Bases must never reference overlays. |
| unknown flag: --enable-helm | Using kubectl apply -k; switch to kustomize build --enable-helm. |
Expert Practices: Idioms, Anti-Patterns & Gotchas
High-signal guidance for production Kustomize. Most of these failure modes are silent — wrong output with exit code 0 — so always diff kustomize build output before and after any structural change.
Anti-Patterns (deprecated fields removed from the v1 API)
commonLabels → labels transformer. commonLabels (deprecated v5.0.0) injects every label into spec.selector.matchLabels in addition to metadata.labels and pod templates. Because a Deployment/StatefulSet/Service selector is immutable after first apply, adding or changing any commonLabel later forces a delete-and-recreate. The labels transformer defaults includeSelectors: false (safe on live resources); add includeTemplates: true so pod labels still propagate (Prometheus scraping, log aggregation). Reserve includeSelectors: true for stable identity labels set only at creation time — and prefer defining those directly in base manifests so the selector contract is explicit. The docs warn that changing selectors on live resources "could result in failures."
labels:
- pairs:
app.kubernetes.io/part-of: myplatform
environment: prod
includeSelectors: false # safe on live resources — does NOT touch matchLabels
includeTemplates: true # still reaches pod template labels
patchesStrategicMerge / patchesJson6902 → unified patches. Both deprecated in v5.0.0. The patches field auto-detects the type from content (resource-shaped doc = strategic merge; list of {op, path, value} = JSON 6902), and its target supports group/version/kind/name/namespace plus labelSelector and annotationSelector, so one patch can hit many resources. Migration is not behavior-neutral: patches runs after the label transformers while patchesJson6902 ran before them, so a patch that touched /metadata/labels can produce different output post-migration. Always diff.
vars → replacements. vars ($(VAR) substitution, deprecated v5.0.0) is replaced by replacements, a structured source/target model that propagates any field value (including hashed ConfigMap/Secret names). Migration is NOT 1:1: replacements works on field pointers and cannot do free-form mid-string concatenation like path: $(ROOT)/cluster/$(REVISION). It can do single-segment replacement within a string via the delimiter + index options, but for true composite strings, restructure (compose in an init container/env var) before migrating.
replacements:
- source: {kind: ConfigMap, name: app-config, fieldPath: metadata.name}
targets:
- select: {kind: Deployment}
fieldPaths:
- spec.template.spec.volumes.[name=config].configMap.name
Idioms
kustomize edit fix is the canonical, idempotent migration tool. It migrates deprecated fields in place — bases→resources, commonLabels→labels, patchesStrategicMerge/patchesJson6902→patches, and (experimentally, --vars) vars→replacements. Run it per kustomization directory; use it as a CI lint to surface remaining deprecated fields. Two things it can't guarantee semantically: the patch apply-order shift and vars mid-string interpolation — so wrap it in a build diff:
kustomize build . > before.yaml
kustomize edit fix
kustomize build . > after.yaml
diff before.yaml after.yaml # confirm semantic equivalence
Components apply against the accumulated resource set. Components use apiVersion: kustomize.config.k8s.io/v1alpha1, kind: Component (alpha — the API may change without a deprecation window). Unlike a base, a Component runs its patches/generators against the resource set the parent has accumulated so far, enabling additive opt-in feature bundles. They must be referenced under components:, never resources:, and can be nested (a Component may itself reference other Components via components:).
Design with Kustomize's deliberate constraints. The maintainers commit to NOT supporting: (1) removal directives (you cannot delete a label/patch/resource from a base — restructure/fork the base, or use a JSON 6902 op: remove for a specific path); (2) ${VAR} templating; (3) build-time side effects from CLI args or shell env vars (output must be fully determined by version-controlled files); (4) globs. The sanctioned pattern for env-driven output is to mutate the kustomization, commit, then build:
kustomize edit set image myapp=registry.example.com/myapp:${GIT_SHA}
kustomize build overlays/prod
Namespace transformer only re-homes default-named RBAC subjects. Setting top-level namespace: runs the namespace transformer in DefaultSubjectsOnly mode: it only namespaces RoleBinding/ClusterRoleBinding subjects whose name is literally default. A ServiceAccount subject with any other name gets no namespace — silently breaking RBAC — unless you configure a custom NamespaceTransformer with setRoleBindingSubjects: allServiceAccounts. Verify RBAC subjects explicitly after setting a namespace.
Gotchas (silent failures)
Strategic merge patches silently REPLACE entire list fields on CRDs. For built-in types, strategic merge knows array merge keys from the embedded OpenAPI schema (e.g. containers merges on name). Custom Resources have no such schema, so kustomize falls back to JSON merge semantics: a patched list field replaces the whole array with no error — dropping every element not in the patch. Fix with either an openapi field carrying x-kubernetes-patch-merge-key/x-kubernetes-patch-strategy, or a JSON 6902 patch (which behaves identically for built-in and custom resources):
patches:
- target: {group: argoproj.io, version: v1alpha1, kind: Application, name: my-app}
patch: |-
- op: add
path: /spec/sources/-
value:
repoURL: https://github.com/org/repo
disableNameSuffixHash: true defeats config-driven rolling updates — and a global true cannot be overridden. The content-hash suffix (app-config-k8bgk8h4t6) is what triggers a rollout on config change: the new name changes the pod spec, so pods are replaced. Disabling it gives a stable name but silently breaks this — config changes apply to the ConfigMap, but Deployments do not roll. Worse, the boolean follows "global true wins": setting it in top-level generatorOptions strips the hash from every generator, and a per-generator disableNameSuffixHash: false is ignored. Only ever disable per-generator, and only for resources referenced by a fixed external name; if you need a stable name and rolling updates, keep the hash and propagate the hashed name via replacements.
replacements list filters like [name=nginx] are unanchored regex. Since v4.5.3, spec.template.spec.containers.[name=nginx].image matches any element whose name contains nginx — my-nginx, nginx-sidecar — and all matches are targeted, silently overwriting several containers. Anchor explicitly with [name=^nginx$], or use index-based paths. (This differs from a patches target's name/namespace, which ARE auto-anchored.)
configMapGenerator behavior: merge matches by logical name AND namespace. An overlay generator with behavior: merge matches the base by pre-hash name — but if one side specifies namespace and the other omits it, kustomize treats them as different generators and creates a second ConfigMap instead of merging (no error). merge also only merges discrete keys; if both reference the same filename, the overlay file replaces the base file entirely. Keep namespace consistent (or absent) on both sides, and verify the build emits one ConfigMap, not two.
sortOptions only applies to the top-level (build-target) kustomization. sortOptions (order: legacy | fifo; v5.0.0+, replaces --reorder) controls output ordering — legacy (default) places Namespaces/ServiceAccounts/RBAC first, which is why kubectl apply -k usually works without explicit ordering. Instances in bases/components reached via resources are ignored, so set it in the overlay that is the actual build target (what ArgoCD/Flux points to), never in a shared base.
Pin a standalone kustomize version. kubectl embeds an older kustomize that does not track standalone releases (the lag has spanned major versions). Features added after the embedded version — unified patches, sortOptions, Components, newer replacements behaviors — may error or be silently ignored under kubectl apply -k while a dev's standalone v5 produces correct output. ArgoCD/Flux also bundle their own versions. Standardize on a pinned kustomize build | kubectl apply -f - in CI and match the GitOps controller's version. Check with kustomize version and kubectl version --client -o json (kustomizeVersion).
Security
Never use --load-restrictor=LoadRestrictionsNone with untrusted input. The default LoadRestrictionsRootOnly (v2.0) prevents file references (resources, configMapGenerator files) from reading outside the kustomization root — a real security boundary. With it off, a kustomization you don't fully control (e.g. a compromised remote base) can use a configMapGenerator file reference to read arbitrary host files (credentials, TLS keys, /etc/passwd). It also breaks relocatability. Only use it in fully trusted, audited build environments; the legitimate alternative for cross-directory file needs is a proper base with its own kustomization.yaml.
Pin remote bases to a full commit SHA or tag. Kustomize fetches remote bases over the network at build time. ?ref=main (or no ref, which uses default-branch HEAD) means a force-push or compromised upstream flows straight into your cluster on the next reconcile with no change to your repo. Short SHAs are unsupported — use a full fetchable SHA (strongest; tags can be force-pushed) or a versioned tag. Remote fetches also add latency and a rate-limit/availability dependency, so mirror critical bases locally. In GitOps, prefer the controller's own authenticated GitRepository/OCIRepository source.
resources:
- https://github.com/org/platform-base//k8s/base?ref=a3f8c2d1e4b5f6a7b8c9d0e1f2a3b4c5d6e7f8a9 # full SHA
- https://github.com/org/platform-base//k8s/base?ref=v2.3.1 # or versioned tag
Never commit secrets. Use sealed-secrets / external-secrets-operator / SOPS, or secretGenerator with gitignored env files. Pin production images by digest, not mutable tags. Enforce policy at admission (OPA Gatekeeper, Kyverno).
Currency
Validate with kubeconform, not kubeval. kubeval is unmaintained and its schema registry is stale, so it cannot validate recent Kubernetes API versions and silently passes manifests using newer/renamed fields. kubeconform is the maintained successor on the same JSON-schema approach, with a current registry and CRD support via -schema-location:
kustomize build k8s/overlays/prod | kubeconform -strict -summary -kubernetes-version 1.29.0
Verification Checklist
- [ ]
kustomize build <overlay>succeeds and output diffed against the prior render for unintended changes. - [ ] No deprecated fields remain (
commonLabels/patchesStrategicMerge/patchesJson6902/vars/bases) —kustomize edit fixclean. - [ ] Labels use the
labelstransformer withincludeSelectors: false; selectors defined explicitly in base manifests. - [ ] Exactly one
resources:key per file; Components undercomponents:(neverresources:). - [ ] ConfigMap/Secret hash suffix intact where rolling updates are wanted;
mergegenerators emit ONE object (consistentnamespace). - [ ] CRD list patches use JSON 6902 (or
openapimerge keys), not bare strategic merge. - [ ] Prod images pinned by digest; secrets gitignored/external; remote bases pinned to full SHA/tag.
- [ ] CI validates with
kubeconform(+ policy gate); GitOps controller kustomize version matches CI.