Introduction

Managing Kubernetes manifests across multiple environments starts simple and gets unwieldy fast. Base config, dev overrides, staging overrides, prod overrides — copying and pasting YAML across all of that is a full-time job that nobody wants. Kustomize solves this without a templating engine or new programming language.

It is built into kubectl since 1.14, so nothing to install separately. You get a base with your canonical manifests, then overlays that only contain the differences for each environment. Look at any overlay and you see exactly what changed from base — no template rendering in your head required.

This post covers the basics: base and overlay structure, patches for modifying resources, generators for ConfigMaps and Secrets, and components for reusable blocks. You will also learn when Kustomize makes sense versus Helm, and where teams run into trouble managing configuration across large deployments.

When to Use / When Not to Use

When Kustomize makes sense

Kustomize is a good fit when your team owns the application manifests directly and wants environment-specific variations without introducing a new templating layer. If your dev, staging, and production configs differ in straightforward ways (replica counts, image tags, namespace names), overlays let you express those differences clearly without Go template syntax.

GitOps workflows benefit from Kustomize because the source YAML stays readable and diffable. You do not need to mentally render a template to understand what will be deployed.

For platform teams building reusable bases that other teams consume as a starting point, Kustomize components provide composition without publishing packages.

The specific configuration differences that map cleanly to Kustomize overlays are those that act on existing fields without conditional logic: replica counts (spec.replicas), image tags (spec.containers[*].image), namespace names, common labels and annotations, and resource limits. These are field-level overrides where the overlay declares the desired end state and Kustomize merges it onto the base. If your environment differences are limited to these shapes, overlays stay readable and the kustomization file serves as a clear manifest of what differs.

GitOps workflows benefit from Kustomize because the source YAML stays readable and diffable. You do not need to mentally render a template to understand what will be deployed. A reviewer can read the overlay kustomization.yaml and immediately see that production adds five replicas and flips the image tag to 1.21.1 — no template evaluation required. This transparency reduces the feedback cycle when auditing changes in pull requests.

Platform teams building a base for other teams to fork or import benefit from components. A platform team defines a component with standard monitoring sidecars, RBAC policies, and network labels once, then publishes the path to downstream teams. Those teams import the component into their overlays without needing to understand the internals. The composition is explicit in the kustomization.yaml, and the final YAML is deterministic — run kubectl kustomize and the same output appears every time.

When to choose Helm instead

If you need to distribute a reusable package to users who should not see the underlying template logic, Helm charts are better. The extensive ecosystem of Bitnami and public charts gives you off-the-shelf solutions for databases, caches, and middleware.

When your configuration varies in complex ways that do not map cleanly to overlays and patches, Helm templating offers more expressive power. Complex configuration means conditional logic: “if the user sets high.memory in values, add a memory limit of 16Gi and also inject a sidecar container.” That kind of conditional rendering does not express cleanly as a Kustomize overlay — you end up stacking multiple overlays or fighting the patch model. Helm’s Go template syntax handles that naturally.

Complex configuration that does not map to overlays is the clearest trigger for Helm. When your values file needs conditional logic — if high.memory is set, add a 16Gi memory limit and also inject a monitoring sidecar — Kustomize overlays start to break down. You end up stacking multiple overlays, each handling one conditional branch, or you fight the patch model to express what should be a simple if-then. Helm’s Go template syntax handles this directly: {{- if eq .Values.profile "high.memory" }} with the full conditional block inline. The rendered output is predictable and the logic lives in one place.

The distribution model is the second concrete trigger. If your platform team publishes a PostgreSQL chart for application teams to install, those teams should not need to understand how the chart templates the StatefulSet, the Service, and the ConfigMap together. They provide persistence.size: 100Gi and replication.enabled: true in a values file, and Helm renders the complete package. Kustomize has no equivalent to this — you share Git paths, not installable packages with versioned releases. A chart with a version number and a values schema is a distributable artifact. A set of kustomization overlays is a directory structure that other teams must copy and maintain themselves.

The Bitnami ecosystem is a concrete benefit that rarely appears in these comparisons. Bitnami maintains Helm charts for PostgreSQL, MySQL, Redis, RabbitMQ, Kafka, and dozens of other off-the-shelf infrastructure components. You get battle-tested manifests with upgrade paths, rollback support, and sane defaults without building anything yourself. If your project needs a message queue or a cache layer, helm install bitnami/redis is a solved problem. Kustomize has no registry of equivalent community-maintained overlays for common middleware.

Kustomize Workflow Flow

flowchart TD
    A[base/<br/>kustomization.yaml] --> B[Overlays]
    B --> C[development/<br/>kustomization.yaml]
    B --> D[staging/<br/>kustomization.yaml]
    B --> E[production/<br/>kustomization.yaml]
    C --> F[kubectl kustomize<br/>./overlays/development]
    D --> G[kubectl kustomize<br/>./overlays/staging]
    E --> H[kubectl kustomize<br/>./overlays/production]
    F --> I[Transformed<br/>YAML manifests]
    G --> I
    H --> I

Kustomize Overview and kubectl Integration

Kustomize is built into kubectl since Kubernetes 1.14, so you do not need separate installation. It reads kustomization.yaml files and produces transformed Kubernetes manifests.

# Basic usage with kubectl
kubectl apply -k ./overlays/production

# Build without applying
kubectl kustomize ./base

# View diff before applying
kubectl diff -k ./overlays/production

A simple Kustomize structure:

app/
├── base/
│   ├── kustomization.yaml
│   ├── deployment.yaml
│   └── service.yaml
└── overlays/
    ├── development/
    │   └── kustomization.yaml
    └── production/
        └── kustomization.yaml

Base and Overlay Structure

The base directory contains your canonical configuration. Overlays modify the base for specific environments.

Base kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - deployment.yaml
  - service.yaml

commonLabels:
  app.kubernetes.io/part-of: myapp

images:
  - name: nginx
    newTag: "1.21"

Base deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
        - name: nginx
          image: nginx:1.21
          ports:
            - containerPort: 80

Development overlay:

# overlays/development/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

bases:
  - ../../base

namespace: myapp-dev

namePrefix: dev-

commonLabels:
  environment: development

replicas:
  - name: myapp
    count: 1

images:
  - name: nginx
    newTag: "1.21-debug"

Production overlay:

# overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

bases:
  - ../../base

namespace: myapp-prod

namePrefix: prod-

commonLabels:
  environment: production

replicas:
  - name: myapp
    count: 5

images:
  - name: nginx
    newTag: "1.21.1"

Patches and Strategic Merge

Kustomize supports two patch strategies: Strategic Merge Patches (similar to kubectl patch) and JSON 6902 patches (RFC 6902 JSON Pointer).

Strategic merge patches:

# patches/replica-change.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 10

# kustomization.yaml
patches:
  - path: patches/replica-change.yaml

JSON 6902 patches for fine-grained control:

# patches/resource-limits.json
[{
  "op": "add",
  "path": "/spec/template/spec/containers/0/resources/limits/memory",
  "value": "512Mi"
}]

# kustomization.yaml
patches:
  - target:
      kind: Deployment
      name: myapp
    path: patches/resource-limits.json

Targeted patches:

# Patch only resources matching labels
patches:
  - patch: |-
      - op: replace
        path: /spec/replicas
        value: 3
    target:
      labelSelector: tier=frontend

Generators (ConfigMap and Secret)

Kustomize can generate ConfigMaps and Secrets from files, literals, or env files.

Literal-based ConfigMap:

# kustomization.yaml
configMapGenerator:
  - name: app-config
    literals:
      - DATABASE_HOST=localhost
      - DATABASE_PORT=5432
      - LOG_LEVEL=info

File-based ConfigMap:

# config/app.properties
database.url=jdbc:postgresql://localhost:5432/mydb
cache.enabled=true

# kustomization.yaml
configMapGenerator:
  - name: app-config
    files:
      - config/app.properties

Env file ConfigMap:

# envvars.txt
PORT=8080
WORKERS=4

# kustomization.yaml
configMapGenerator:
  - name: app-env
    envs:
      - envvars.txt

Generate Secret:

configMapGenerator:
  - name: app-secret
    literals:
      - api-key=$(API_KEY)
    envs:
      - secrets.txt

Kustomize Components for Reuse

Components allow reusable configuration blocks that can be imported across multiple kustomizations.

Define a monitoring component:

# components/monitoring/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1alpha
kind: Component

resources:
  - deployment-metrics.yaml
  - service-monitor.yaml

patches:
  - patch: |-
      - op: add
        path: /spec/template/spec/containers/-
        value:
          name: prometheus
          image: prometheus:latest
          ports:
            - containerPort: 9090
    target:
      kind: Deployment

Use the component:

# overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

bases:
  - ../../base

components:
  - ../../components/monitoring

patches:
  - path: patches/production-config.yaml

Kustomize vs Helm Comparison

Both tools solve configuration management but with different philosophies:

Aspect	Kustomize	Helm
Approach	Overlay/patch	Template with values
Learning curve	Lower (no new syntax)	Higher (Go templates)
Flexibility	Limited to kustomize features	Highly flexible
Debugging	Direct YAML output	Rendered templates
Secret management	Built-in generators	External tools needed
Ecosystem	Native to Kubernetes	Large chart repository
Best for	App-centric deployments	Off-the-shelf packages

Choose Kustomize when you want direct control over your YAML and do not need to package complex application logic. Choose Helm when you want to distribute reusable packages or need the extensive ecosystem of public charts.

Production Failure Scenarios

Name prefix collisions across teams

If two teams both use namePrefix: dev- in their overlays and deploy to the same cluster, resources can collide. The dev-api Deployment from team A overwrites the dev-api Deployment from team B.

Use team-specific prefixes or deploy to isolated namespaces. The team-specific prefix pattern uses a two-part prefix: environment and team name. Instead of dev-, use dev-teamA- and dev-teamB-. This makes the collision impossible by construction — dev-teamA-api and dev-teamB-api are distinct resource names, even if both teams deploy to the same cluster and the same namespace. The naming convention lives in each team’s overlay kustomization.yaml as namePrefix: dev-teamA-.

Namespace isolation is the primary mitigation for teams that do not want to coordinate prefix naming. Each team gets its own namespace (team-a-dev, team-b-dev), and the overlay sets namespace: team-a-dev. Resources from team A and team B never share a namespace, so name collisions produce an API error rather than a silent overwrite. This approach scales better in organizations where teams do not communicate about naming conventions — the cluster API enforces isolation.

To audit for collisions before they happen, run kubectl kustomize for each overlay and grep for resource names. A CI script can extract all kind: Deployment names from each overlay’s output and compare them. If two overlays produce the same Deployment name in the same namespace, the script fails the pipeline before the apply. This check runs at the same time as the dry-run validation — it adds no additional steps to the CI workflow.

Patch targets wrong resources

Strategic merge patches apply to all resources matching the selector. If you have multiple Deployments with the same labels, a replica patch affects all of them simultaneously.

Always use precise target selectors:

patches:
  - patch: |-
      - op: replace
        path: /spec/replicas
        value: 10
    target:
      name: myapp
      kind: Deployment

ConfigMap generator creates new resources on every run

Kustomize generates ConfigMaps with content hashes in the name by default. Changing a ConfigMap literal creates a new ConfigMap with a different hash and deletes the old one. If a Deployment references the old ConfigMap name, the old ConfigMap cannot be deleted because the Deployment still needs it.

Pin ConfigMap generator outputs explicitly or use behavior: merge to update existing ConfigMaps rather than replacing them. The behavior field accepts three values: replace (the default, which deletes and recreates the ConfigMap on any content change), merge (which updates the existing ConfigMap in place and preserves its name), and retention (which keeps the old ConfigMap when a new one is generated, useful for zero-downtime transitions). Using behavior: merge is the correct choice when a Deployment references the ConfigMap by name — the Deployment keeps working while Kustomize updates the ConfigMap content in place.

The hash-based naming mechanism appends a content hash to the ConfigMap name. For example, a ConfigMap named app-config with literals LOG_LEVEL=info becomes app-config-7d8c9e5mh8. When LOG_LEVEL changes to debug, Kustomize generates app-config-3f6a1b9k2l and deletes the old one. This deterministic naming ensures that changing a ConfigMap triggers a rolling update on any Deployment that references it — the Deployment’s volume mount or environment variable reference to the old hash becomes invalid, forcing a new pod. This is the intended behavior for configuration changes that require pod restarts.

The Deployment reference problem surfaces when you use behavior: replace (the default) and the Deployment has already rolled out using the old ConfigMap hash. Kustomize deletes the old ConfigMap and creates a new one with a different hash, but the Deployment’s pod spec still references the old hash name. The new ConfigMap is created, but the Deployment cannot use it because its selector still points to the deleted resource. Using behavior: merge avoids this by keeping the ConfigMap name stable — the same name persists across content updates, so the Deployment’s reference remains valid throughout.

Overlay path not found

If you reference ../../base in an overlay and the path is wrong, Kustomize fails with a confusing error about a missing file. The error message does not clearly indicate it is a path resolution problem.

Use absolute paths or ensure your relative path references are correct before running in CI. The actual error message from Kustomize when the base path is wrong looks like this: Error: accumulating dirty './overlay': path '../base' does not exist. The word “accumulating dirty” (with a garbled character) is Kustomize’s internal term for building the resource list from multiple kustomization files, and the error does not say “cannot find base” — it says the file referenced by bases does not exist. This makes the error easy to misread as a general YAML syntax problem.

Relative paths are fragile in CI because the working directory differs from your local shell. Locally, you might run kubectl kustomize ./overlays/development from the repository root and ../../base resolves correctly. In a CI container, the working directory might be /github/workspace or a temp directory, and the same relative path breaks. The error surfaces only at apply time, after the pipeline has already committed to the build.

The absolute path solution sets bases: [/path/to/base] in the overlay kustomization.yaml. In a monorepo with a predictable layout, this resolves from any working directory. Alternatively, ensure your CI pipeline always runs kustomize from the repository root by setting workingDirectory in the pipeline config. A CI snippet that validates paths before running kustomize catches this class of error early:

# Validate base path exists before running kustomize
BASE_PATH="./base"
if [ ! -d "$BASE_PATH" ]; then
  echo "ERROR: base path $BASE_PATH does not exist"
  exit 1
fi
kubectl kustomize ./overlays/production

Common Pitfalls / Anti-Patterns

Overly deep directory nesting

Nesting overlays five levels deep — base, then environment, then region, then cluster, then tenant — makes it impossible to understand what the final manifest looks like without running kubectl kustomize. The further down the stack you go, the harder it becomes to trace a specific field back to its origin.

Two to three levels works in practice: one base, one environment overlay, and optionally one cluster or tenant overlay if you need both dimensions. Past that, the mental overhead of composing the final YAML outweighs any organizational gain.

For reuse across many overlays, use components instead of stacking more overlay layers. A component lives at the same level as base and gets imported into any overlay that needs it — one copy, no inheritance chain to trace. If you already have a deep stack, walk the directory tree and measure the longest chain. Refactoring the middle layers into components flattens the structure and makes each overlay readable on its own.

Duplicate resources across overlays

If base defines a resource and an overlay also defines the same resource without using patches or strategic merge, Kustomize produces duplicate resource errors. This is especially confusing because the error appears at apply time, not at kustomize build time.

Duplicate resources break the build in two stages. First, kubectl kustomize succeeds — it only assembles YAML, it does not validate uniqueness. The collision surfaces when kubectl apply reaches the cluster API server, which rejects two resources with the same kind and name. By then, the build pipeline has already committed to the manifests and may have pushed them to a deployment branch.

The fix is straightforward: audit your overlays regularly. A quick check runs kubectl kustomize and counts resource kinds, then compares that count against what you expect. An unexpected spike in Deployment or Service count usually means an overlay re-declared a resource that base already provides.

# Catch duplicate resources before apply
kubectl kustomize ./overlays/production > build/manifests.yaml
RESOURCE_COUNT=$(grep -c "^kind:" build/manifests.yaml)
EXPECTED_COUNT=12  # adjust for your setup
if [ "$RESOURCE_COUNT" -ne "$EXPECTED_COUNT" ]; then
  echo "WARNING: Expected $EXPECTED_COUNT resources, found $RESOURCE_COUNT"
fi

Run this in CI as part of your validation step alongside the dry-run check. It catches the class of error where someone adds a resource to an overlay without realizing base already provides it.

Not using —dry-run in CI

Deploying without testing the kustomize output first means you find out about problems after they happen in production. Always run kubectl diff -k or kubectl kustomize | kubectl apply --dry-run=server in CI before applying.

Without dry-run testing, failure is quiet until it hits production. A bad image tag stalls the Deployment at ImagePullBackOff. A malformed patch drops a field and the resource applies incomplete. A missing base reference breaks the build — but only if your CI actually runs kustomize before pushing; some pipelines skip that step entirely.

The minimum viable dry-run pipeline runs three commands:

# 1. Build and validate YAML syntax
kubectl kustomize ./overlays/production > build/manifests.yaml

# 2. Validate against cluster schema without applying
kubectl apply --dry-run=server -f build/manifests.yaml

# 3. Show what changed compared to what's running
kubectl diff -k ./overlays/production

kubectl diff -k is the most useful in pull requests. Reviewers see exactly which fields change, not just the kustomization diff, so regressions are visible before the merge. Without it, the deploy is opaque — you see what files changed, not what the cluster will actually do.

ArgoCD and Flux run their own dry-run when the controller syncs. That catches cluster-level problems. But you still want validation in CI so bad commits never reach the controller in the first place — a bad push triggering a sync in the middle of the night is avoidable with a local lint step.

Missing version pinning for kustomize

Kustomize versions differ in behavior. A kustomization that works with kustomize 3.8 may fail or produce different output with kustomize 5.0. Pin your kustomize version in CI and align it with your cluster’s kubectl version.

Version alignment matrix:

kubectl version	kustomize version	Notes
1.26	4.5.5+	Align minor versions when possible
1.27	5.0.0+	JSON 6902 patch improvements
1.28+	5.1.0+	Component API stable

Pin kustomize in CI:

# Install specific version in CI pipeline
curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash -s -- 5.1.0 /usr/local/bin

# Confirm version matches what you expect
kustomize version

Detecting version drift: When CI succeeds but your local build fails, or overlays produce different output on different machines, the kustomize version is usually the culprit. Run kustomize version everywhere and compare the output.

kubectl integration note: kubectl 1.14 bundled kustomize as a built-in, which means kubectl kustomize uses whatever version ships with your kubectl binary. This caps what you can do with the built-in. If you need a newer kustomize than your kubectl includes, install it separately and use kustomize directly instead of kubectl kustomize.

Observability Hooks

Track the health and correctness of your Kustomize deployments with these practices.

What to monitor:

• Kustomize build success/failure rate in CI
• Time taken for kubectl kustomize to complete across environments
• Number of resources generated per overlay (sudden changes indicate unintended patches)
• YAML validation errors caught in CI vs at apply time

CI/CD observability:

# Example: Kustomize build step with error capture
- name: Build Kustomize manifests
  id: kustomize-build
  run: |
    mkdir -p build
    kubectl kustomize ./overlays/production > build/manifests.yaml
    RESOURCE_COUNT=$(kubectl kustomize ./overlays/production | kubectl apply --dry-run=server -f - 2>&1 | grep -c "created\|configured")
    echo "resources::$RESOURCE_COUNT"

    # Detect significant changes
    kubectl kustomize ./overlays/production | sha256sum > build/prod-checksum.txt
  continue-on-error: false

# Track diff size
- name: Check manifest diff
  run: |
    kubectl kustomize ./overlays/production > build/new.yaml
    DIFF_COUNT=$(diff -u build/base.yaml build/new.yaml | grep -c "^[+-]")
    echo "Changed lines: $DIFF_COUNT"
    if [ "$DIFF_COUNT" -gt 100 ]; then
      echo "WARNING: Large diff detected, verify intentional changes"
    fi

Debugging commands:

# View all generated resources
kubectl kustomize ./overlays/production

# Validate against cluster without applying
kubectl apply --dry-run=server -k ./overlays/production

# Check what will change (diff mode)
kubectl diff -k ./overlays/production

# Count resources generated
kubectl kustomize ./overlays/production | kubectl apply --dry-run=server -f - 2>&1 | grep -c "resource"

# Validate YAML syntax
kubectl kustomize ./overlays/production | kubectl validate --ignore-missing-schema

# Check for deprecated APIs
kubectl kustomize ./overlays/production | kubectl-convert --dry-run=client -o yaml | head -20

Alert on deployment anomalies:

# Alert if a deployment has significantly more/fewer resources than expected
- alert: KustomizeResourceCountAnomaly
  expr: |
    (count(kustomize_generated_resources{env="production"}) by (app)
    / avg(count(kustomize_generated_resources{env="production"}) by (app))) > 1.5
  labels:
    severity: warning
  annotations:
    summary: "Kustomize resource count anomaly for {{ $labels.app }}"
    description: "Production overlay generating {{ $value }}x more resources than average. Verify intentional changes."

# Alert if kustomize builds fail repeatedly in CI
- alert: KustomizeBuildFailures
  expr: increase(kustomize_build_errors_total[10m]) > 3
  labels:
    severity: critical
  annotations:
    summary: "Multiple kustomize build failures detected"
    description: "{{ $value }} kustomize build failures in the last 10 minutes. Check CI logs."

Trade-off Summary

Aspect	Kustomize	Helm	plain YAML
Templating	No (patches/overlays)	Yes (Go templates)	No
Readability	High (plain YAML)	Medium (templates)	Highest
Reusability	Git-based	Chart repos	None
Secret handling	Sealed secrets + generators	helm-secrets	Manual
GitOps integration	Native kubectl	Flux/ArgoCD	Flux/ArgoCD
Debugging	Diff is clear	Render first	Plain kubectl
Ecosystem	Growing	Massive (Bitnami)	N/A

Interview Questions

Further Reading

• GitOps - GitOps patterns and deployment strategies
• Helm Charts - Alternative templating approach to Kubernetes configuration
• Container Registry - Managing container images at scale
• Kubernetes - Core Kubernetes component internals
• Kustomize Official Documentation
• Kubernetes SIG Kustomize

Conclusion

Key Takeaways

• Kustomize uses overlays and patches instead of templates, keeping YAML readable and diffable
• The built-in kubectl kustomize requires no separate installation (Kubernetes 1.14+)
• Name prefixes and common labels help standardize resources across environments
• Components provide reusable configuration blocks without publishing packages
• Choose Kustomize for app-centric GitOps; choose Helm for distributable packages

Kustomize Checklist

# Test kustomize output without applying
kubectl kustomize ./overlays/production

# Diff against cluster state
kubectl diff -k ./overlays/production

# Apply with dry-run to catch errors
kubectl apply -k ./overlays/production --dry-run=server

# Validate build in CI
kustomize build ./overlays/production | kubeval --strict