Introduction

GitOps extends DevOps practices by using Git as the single source of truth for declarative infrastructure and applications.

Core principles:

1. Declarative description: All infrastructure and applications defined declaratively
2. Git as source of truth: Desired state stored in Git, not in running clusters
3. Automated synchronization: Software automatically syncs cluster state to Git state
4. Pull-based updates: Operators pull changes from Git, not pushed by CI

Benefits:

• Auditable: Every change recorded in git history
• Reproducible: Environment recreation from Git is deterministic
• Fast rollback: Revert to previous commit for instant rollback
• Self-healing: Drift between Git and cluster automatically corrected
• Developer-friendly: Standard git workflows for deployments

ArgoCD Architecture and Installation

ArgoCD runs as a Kubernetes controller and continuously monitors Git repositories, comparing desired state with actual cluster state.

Architecture components:

Git Repository → ArgoCD → Kubernetes Cluster
     ↑              ↓
     ←←←←diff/sync←←←←

Installation:

# Namespace install
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Or with Helm
helm repo add argo https://argoproj.github.io/argo-helm
helm install argocd argo/argo-cd -n argocd --create-namespace

ArgoCD CLI:

# Install CLI
brew install argocd

# Login (get initial password)
argocd login --insecure --username admin --password $(kubectl get pods -n argocd -l app.kubernetes.io/name=argocd-server -o jsonpath='{.items[0].spec.containers[0].env[?(@.name=="ARGOCD_AUTH_SECRET")].value}') localhost:8080

# Add repo
argocd repo add https://github.com/myorg/manifests --username myuser --password mytoken

# Sync application
argocd app sync myapp

ArgoCD server access:

# ingress.yaml for external access
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-server-ingress
  namespace: argocd
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
spec:
  ingressClassName: nginx
  rules:
    - host: argocd.mycorp.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: argocd-server
                port:
                  number: 443

Application and ApplicationSet Resources

ArgoCD defines applications that point to Git repositories containing Kubernetes manifests.

Basic Application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/myorg/manifests.git
    targetRevision: HEAD
    path: apps/myapp/overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: myapp
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Helm-based Application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
spec:
  source:
    repoURL: https://github.com/myorg/charts.git
    chart: myapp
    targetRevision: 1.2.0
    helm:
      valueFiles:
        - values-production.yaml
      parameters:
        - name: image.tag
          value: v2.1.0
      releaseName: myapp
  destination:
    server: https://kubernetes.default.svc
    namespace: myapp

ApplicationSet for GitOps automation across clusters:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: myapp-multicluster
spec:
  generators:
    - git:
        repoURL: https://github.com/myorg/manifests.git
        revision: HEAD
        directories:
          - path: clusters/production/*
    - clusters:
        selector:
          matchLabels:
            environment: production

  template:
    metadata:
      name: myapp-{{name}}
    spec:
      project: default
      source:
        repoURL: https://github.com/myorg/manifests.git
        path: apps/myapp
        targetRevision: HEAD
      destination:
        server: "{{server}}"
        namespace: myapp
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Flux Installation and Configuration

Flux uses a different operator model with GitRepository and Kustomization resources.

Installation with Flux CLI:

# Install Flux CLI
brew install fluxcd/tap/flux

# Bootstrap to cluster
flux bootstrap github \
  --owner=myorg \
  --repository=flux-infra \
  --branch=main \
  --path=./clusters/production \
  --personal

Core Flux resources:

# GitRepository defines the source
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: myapp
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/myorg/manifests.git
  ref:
    branch: main
  secretRef:
    name: git-credentials
  ignore: |
    # Ignore documentation in subfolders
    /**/*.md

# Kustomization defines what to apply
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: myapp
  namespace: flux-system
spec:
  interval: 5m
  path: ./apps/myapp/overlays/production
  prune: true
  wait: true
  sourceRef:
    kind: GitRepository
    name: myapp
  targetNamespace: myapp
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: myapp
      namespace: myapp

Multi-environment with Flux:

# Production kustomization depends on staging
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: myapp-staging
spec:
  dependsOn:
    - name: myapp-base
  # ...

---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: myapp-production
spec:
  dependsOn:
    - name: myapp-staging # Wait for staging to be healthy
  # ...

Drift Detection and Correction

GitOps operators continuously monitor and correct drift.

ArgoCD drift detection:

# Check application health and sync status
argocd app get myapp

# Show diff between Git and cluster
argocd app diff myapp

# Manually sync after resolving drift
argocd app sync myapp

ArgoCD health assessment:

# Custom health check for custom resources
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas # Ignore replica count differences

Flux drift handling:

# Suspend reconciliation
flux suspend kustomization myapp

# Resume reconciliation
flux resume kustomization myapp

# Force reconciliation
flux reconcile kustomization myapp --with-source

GitOps with Helm and Kustomize

Both tools work well with GitOps operators.

HelmRelease with ArgoCD:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
spec:
  source:
    chart: nginx
    repoURL: https://charts.bitnami.com/bitnami
    targetRevision: 15.0.0
    helm:
      releaseName: myapp
      values:
        replicaCount: 3
        service:
          type: LoadBalancer

HelmRelease with Flux:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: myapp
spec:
  interval: 1h
  chart:
    spec:
      chart: myapp
      version: 1.0.0
      sourceRef:
        kind: HelmRepository
        name: myorg-charts
  values:
    replicaCount: 3
  install:
    crds: Create
  upgrade:
    crds: CreateReplace

Kustomize with ArgoCD:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
spec:
  source:
    repoURL: https://github.com/myorg/manifests.git
    path: apps/myapp/overlays/production
    kustomize:
      commonLabels:
        app.kubernetes.io/managed-by: argocd
      images:
        - name: myapp
          newTag: v2.1.0

When to Use / When Not to Use

When GitOps makes sense

GitOps earns its keep when you need audit trails for infrastructure changes. If your team has ever spent hours tracking down which change caused a production incident, Git history solves that by default. Every deployment is a commit, every rollback is a git revert.

Use GitOps when you manage multiple clusters. Manually keeping staging, production, and disaster-recovery clusters in sync is error-prone. GitOps operators ensure all clusters converge to the same desired state without human intervention.

GitOps also helps when you need compliance documentation. Regulated industries require evidence that production matches approved configurations. Git provides that automatically — the commit hash is the audit log entry.

The multi-cluster case is where GitOps shows the most value. When you have three clusters and you need to update an image tag across all of them, a manual process invites mistakes. One cluster gets updated, two do not, and now your environments are out of sync in a way that is hard to detect until something breaks. With GitOps, you update the Git manifest and all three clusters converge within minutes.

For single-cluster teams, the calculus is tighter. GitOps adds an operator, credentials management, and a layer of abstraction over your Kubernetes manifests. If your pipeline already deploys cleanly and your team is small enough that everyone knows what is running, the overhead may not pay off. The moment you start involving multiple teams or multiple environments, GitOps starts making sense.

When to stick with CI-driven deployments

If your team is small and your infrastructure changes infrequently, GitOps adds operational overhead without much benefit. The operator needs maintaining, credentials need rotating, and drift detection needs monitoring. For a single cluster with two engineers, a well-written CI pipeline can do the job without the extra components.

GitOps also does not play well with stateful workloads that modify their own state. A database that accepts writes directly cannot be fully GitOps-controlled because the operator cannot distinguish intentional changes from drift.

The honest case for CI-driven deployments: they are simpler to reason about when the deploy process is already working. You push code, CI runs tests, CI pushes a manifest, done. The mental model is linear and familiar. GitOps introduces operators, reconciliation loops, and the question of what happens when the operator disagrees with the cluster. For teams that are already productive with CI pipelines, switching is not a light decision.

What I see go wrong is teams adopting GitOps because it sounds sophisticated, then struggling with credential rotation, operator upgrades, and debugging why the cluster does not match Git. The operator is not magic — it is software that can break and that you need to monitor. If your infrastructure changes are infrequent enough that a CI pipeline with manual promotion gates works, stick with it.

GitOps Tool Selection Flow

flowchart TD
    A[Team needs GitOps?] --> B{Multiple clusters?}
    B -->|Yes| C{ArgoCD vs Flux?}
    B -->|No| D[Stick with CI-driven deploys]
    C -->|Need UI and ApplicationSets| E[ArgoCD]
    C -->|Need fine-grained Flux CRDs| F[Flux]
    A --> G{Need audit trail?}
    G -->|Yes| B
    G -->|No| D

ArgoCD vs Flux Comparison

Both operators implement GitOps, but they differ in philosophy and capability.

Aspect	ArgoCD	Flux
UI	Built-in web UI	CLI and external dashboards only
Multi-cluster	ApplicationSets for scale	Cluster API bootstrap
Learning curve	Simpler concepts	Flux CD v2 operators are more granular
Extension model	Plugins and config management	CRD-based operators
Helm support	Native with value overrides	HelmRelease CRD
GitHub integration	Tight with Applications	Reconciler pattern
Maturity	CNCF graduated	CNCF graduated

Choose ArgoCD when you want a UI for non-Kubernetes engineers to view deployment status. Choose Flux when you need deep integration with Kubernetes primitives or when you prefer everything as a custom resource.

Production Failure Scenarios

Common GitOps Failures

Failure	Impact	Mitigation
Git credentials expire	Operator stops syncing, drift accumulates	Use service accounts with token rotation
PR merged with bad YAML	Broken manifests deployed to production	Enable diff-before-sync, require reviews
Large manifest causes timeout	Application stuck in progressing state	Split into smaller Applications
Cluster unreachable	Sync fails, ArgoCD/Flux marks app out-of-sync	Configure retry intervals appropriately
drift detection too sensitive	Constant re-syncing, wasting resources	Configure ignoreDifferences in ArgoCD

Sync Failure Recovery Flow

flowchart TD
    A[Sync Triggered] --> B{Manifest Valid?}
    B -->|Invalid YAML| C[Sync Blocked]
    B -->|Valid| D{Resources Healthy?}
    D -->|No| E[Mark Degraded]
    D -->|Yes| F[Sync Complete]
    C --> G[Check Git commit]
    E --> H[Alert Team]
    F --> I[Monitor for Drift]
    G --> H
    H --> J[Fix and Force Sync]

Secret Rotation Without Disruption

GitOps and secrets require care. Never commit plain-text secrets to Git.

# Use sealed-secrets or external secrets operator
# Encrypt secrets before committing
kubectl create secret generic db-creds \
  --from-literal=password=supersecret \
  --dry-run=client \
  -o yaml | kubeseal --cert pub-cert.pem

# Flux external secrets example
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-creds
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: db-creds
  data:
    - secretKey: password
      remoteRef:
        key: prod/db
        property: password

Observability Hooks

Track GitOps health through sync status, drift metrics, and reconciliation times.

What to monitor:

• Sync status per application (OutOfSync, Synced, Degraded)
• Time since last successful sync
• Reconciliation duration (Flux) or sync duration (ArgoCD)
• Drift count across all managed resources
• Failed sync attempts and error messages

# ArgoCD - check sync status
argocd app get myapp --watch

# ArgoCD - list out-of-sync apps
argocd app list -o wide | grep OutOfSync

# Flux - check reconciliation status
flux get kustomizations

# Flux - check reconciliation logs
flux logs --kind=Kustomization --name=myapp

# Prometheus metrics from ArgoCD
argocd_app_sync_total{app_name="myapp", phase="success"}
argocd_app_sync_total{app_name="myapp", phase="failure"}
argocd_app_metadata{sync_token="abcd123"} # tracks commit SHA

Common Pitfalls / Anti-Patterns

Committing sensitive data to Git

This is the most common GitOps mistake. Sealed secrets, external secrets, or vault integration are mandatory. There is no acceptable reason to put passwords in Git, even in private repositories.

The temptation is psychological: private repos feel safe, so developers rationalize committing a DB_PASSWORD=secret123 alongside their Kubernetes manifests. The repo is private, after all—who else could see it? But private repos get cloned onto CI agents, developer machines, and sometimes shared across organisations without strict access controls. A single leaked git history from a compromised laptop or a misconfigured CI pipeline turns a “private” repo into a public credential dump. Credential rotation then becomes a fire drill: every system that used the old password needs updating before you can safely revoke it, and the Git history still contains the old value indefinitely unless you rewrite every commit since the leak.

Beyond the immediate exposure, there is the audit trail problem. A password in Git creates a permanent record of a secret that may never expire on its own. During security audits, investigators must assume every commit since that password was committed is potentially compromised. This balloons incident response scope dramatically—a single forgotten password can taint months of git history and trigger a cascade of mandatory rotations across downstream services.

Use a secrets operator that syncs from an external vault. Sealed Secrets (Bitnami) encrypts secrets using asymmetric keys—the cluster holds the private key, and you commit the encrypted SealedSecret resource to Git. External Secrets Operator (ESO) pulls from AWS Secrets Manager, GCP Secret Manager, or HashiCorp Vault at runtime, so the secret never lives in plain text in the repository. For ESO, you commit only the ExternalSecret CRD that references the vault path, not the secret value itself.

If you inherit a repo with leaked credentials, treat it as a security incident: rotate the credentials immediately, audit access logs, and use Git history scrubbing tools like git-filter-repo to remove the sensitive data. Remember that cloned repos and CI caches may still hold the old history.

Not using automated sync

Running ArgoCD or Flux in manual sync mode defeats the purpose. The value of GitOps is the automatic convergence to desired state. Manual sync means you have all the operational overhead of GitOps with none of the self-healing benefits.

Manual sync mode introduces a human bottleneck into every deployment. Someone must be awake, watching the CI pipeline, and manually clicking “sync” for every application on every cluster. On a Friday night rotation, that means waiting for an on-call engineer to trigger a rollout that should have happened automatically. The perceived safety of manual control—being able to review what gets deployed before it gets deployed—actually creates a false sense of security because it trains teams to ignore drift until someone notices. Drift that sits unaddressed for hours is harder to debug than drift that corrects itself in minutes.

The math is not flattering: a team running manual sync might deploy twice a day. A team running automated sync deploys every time a commit passes CI—potentially dozens of times per day. If the self-healing property of GitOps is what you are after (and it should be), manual sync gives you the worst of both worlds: all the GitOps ceremony with none of the reliability guarantees. The only valid reason to use manual sync is during active debugging of the operator itself, and that should be a temporary, deliberate decision—not a permanent configuration.

With automated sync enabled, the operator pulls every 3 minutes by default (configurable via spec.interval in Flux or sync policy in ArgoCD) and reconciles cluster state to Git state whenever a new commit appears. This gives you self-healing: if someone manually edits a deployment, the operator reverts it. If a node fails and pods reschedule, the next sync run corrects any drift.

If you need to pause reconciliation for debugging, use flux suspend kustomization <name> or ArgoCD’s argocd app set <name> --sync-policy manual — these are temporary overrides, not a permanent mode. Resume automated sync as soon as the debugging session ends.

Overly broad prune policies

With prune: true, the operator deletes resources removed from Git. On a shared cluster with multiple teams, this can cause incidents if your path patterns are too broad. Test prune behavior in staging before enabling in production.

A concrete example: say your Git repository root contains an apps/ directory and your team owns everything under apps/frontend/. Your Kustomization or Application spec.source.path is set to apps/frontend/. Another team owns apps/backend/, and their Application points to their own path. If your prune policy is set at the cluster level or your path is misconfigured to apps/ instead of apps/frontend/, removing a resource from apps/backend/ in Git causes the operator to issue a delete for that backend resource—even though it belongs to a different team and a different Application. The result is a production incident where the backend team’s deployment disappears while the frontend team is blissfully unaware they triggered it.

The failure mode is silent until it is catastrophic. The operator correctly reads its own path and correctly removes what is no longer in Git under that path. Nothing in the operator logs indicates a problem because from its perspective, it did exactly what it was configured to do. The incident only surfaces when the backend team notices their pods are missing. This is why path scoping and namespace isolation are not optional—they are the guardrails that make prune: true safe to enable.

An overly aggressive prune policy can wipe resources belonging to another team if your repository structure does not properly isolate namespaces. For example, if your Git path is apps/ and another team also writes to apps/, their resources get flagged for deletion when they remove something from their subdirectory. Configure Application-level or Kustomization-level prune settings rather than cluster-wide ones, and scope the path to only the resources your repository owns.

Flux and ArgoCD both support pruneResources with labels and namespace filters. In ArgoCD, use spec.syncPolicy.automated.prune only within a specific Application that has a clearly bounded spec.source.path. In Flux, the Kustomization spec.path defines the prune boundary. Set spec.prune to true only after confirming your path scoping is airtight.

Ignoring sync wave ordering

When deploying multiple applications that depend on each other, the order matters. ArgoCD and Flux both support dependency ordering, but it is not enabled by default. Without it, your database might try to start before the PVC is created.

The mental shift required is from “Kubernetes applies everything in parallel and it works out” to “Kubernetes applies everything in parallel unless you explicitly sequence it, and it will not work out when it matters most.” In a development environment with a single application and no dependencies, parallel application is fine. In production, where a StatefulSet with a PVC must exist before a Deployment can mount that volume, parallel application means the Deployment gets an ImagePullBackOff or a missing volume error on the first sync, and then the operator retries until it eventually lands on the right state—but only after a backoff delay that may have been minutes.

The specific failure modes cluster around storage and networking: a Deployment referencing a ConfigMap that does not yet exist, a Service pointing to a backend that has not started, an Ingress dependent on a Certificate resource that has not yet been issued. Each of these resolves eventually through retry, but the retry window is measured in minutes, not seconds. With sync waves or dependsOn explicitly declared, the operator waits for the dependency to be healthy before starting the dependent—so the Deployment waits for the Service, which waits for the backend Deployment, which waits for the database. No retry cycles, no artificial delay.

In ArgoCD, sync waves are controlled with sync-wave annotations on resources. Lower-numbered waves apply first. A typical ordering: namespace and storage classes (wave 0), then cluster-scoped resources like CustomResourceDefinitions (wave 1), then StatefulSets with PVCs (wave 2), then Deployments (wave 3). Resources in the same wave apply in parallel.

Flux uses spec.dependsOn in Kustomization resources. Each Kustomization can declare dependencies on other Kustomizations, and Flux waits for the dependency to be healthy before applying its own resources. For multi-environment setups, structure your overlays so base resources (namespaces, CRDs) are in a Kustomization that everything else depends on.

Using the same repository for everything

Monorepos with thousands of applications create performance problems. The operator scans the entire repo on every change. Split by team or by deployment boundary to keep sync times reasonable.

A GitOps monorepo with 5,000 files means every git push triggers the operator to re-evaluate all 5,000 manifests. With ArgoCD, the Application controller diffs every managed resource against live cluster state on each reconciliation cycle. If your spec.source.repoURL points to the root of a massive repo, sync latency climbs and controller CPU follows.

The two practical options are repository-per-team or repository-per-application-family. The repository-per-team model gives each team their own repo containing all the applications they own. Repository-per-application gives each application its own repo and Application resource. Both keep the operator’s workload manageable. If you must use a monorepo, use ArgoCD’s spec.source.path to limit scanning to a subdirectory, and consider enabling spec.syncPolicy.automated only for the Applications that need it.

Trade-off Analysis

Aspect	ArgoCD	Flux	Jenkins X
UI	Full UI dashboard	CLI / WebUI only	Web UI
Multi-cluster	ApplicationSets (native)	Workload partitioning	Limited
Learning curve	Moderate	Steeper	Steep
GitOps model	Pull-based	Pull-based	Pull-based
Helm support	Yes	Yes	Yes
Kustomize support	Yes	Yes (Kustomization)	Yes
Extensibility	Plugins	Go modules	Plugins
Enterprise features	Argo CD Enterprise	Weave GitOps (Enterprise)	Limited

Interview Questions

Further Reading

• ArgoCD Documentation - Official ArgoCD docs
• Flux Documentation - Official Flux CD docs
• GitOps Principles - OpenGitOps principles
• Weave GitOps - Enterprise GitOps (Flux)
• ArgoCD ApplicationSets - Multi-cluster automation
• Flux Multi-Tenant - Tenant isolation patterns

Conclusion

Key Takeaways

• GitOps makes Git the source of truth for both infrastructure and applications
• ArgoCD provides a UI and scales well with ApplicationSets
• Flux uses CRD-based operators for fine-grained control
• Always use external secrets or sealed secrets, never plain-text credentials
• Automated sync with self-heal is what separates GitOps from CI-driven deployments
• Configure ignoreDifferences to prevent alert fatigue from managed fields

GitOps Health Checklist

# Verify ArgoCD sync status
argocd app list

# Check Flux reconciliation
flux get all --namespace flux-system

# View sync history
argocd app history myapp

# Force a clean sync
argocd app sync myapp --force

# Flux: reconcile with source refresh
flux reconcile kustomization myapp --with-source

# Check for drift
argocd app diff myapp

# Verify secrets are not in Git
git log --all --full-history -S "password" -- "*.yaml"