Container Registry: Image Storage, Scanning, and Distribution

Set up and secure container registries for storing, scanning, and distributing container images across your CI/CD pipeline and clusters.

published: reading time: 34 min read author: GeekWorkBench updated: June 17, 2026
Quick Summary

Container registries sit between your build pipeline and deployment clusters, storing versioned images your CI produces and your clusters consume. Cloud-hosted options like ECR, ACR, and Artifact Registry handle scanning and IAM integration out of the box; Harbor suits self-hosted enterprise setups with replication and CVE gating. Tag images with Git SHA or semver, never :latest, and gate builds on Trivy scans for CRITICAL vulnerabilities before images get near production. Use digest references in production manifests to lock exactly which build runs. Cross-region replication cuts image pull latency for distributed clusters. Storage and egress drive registry costs—lifecycle policies that prune unused images keep bills predictable.

Introduction

When registries make sense

A container registry is essential when you are building and deploying containerized applications. If your CI/CD pipeline produces Docker images, you need somewhere to store them between the build step and the deployment step. Even for small projects, a registry gives you a versioned history of your images that you cannot get from local Docker storage.

Use a registry when you need image promotion between environments. An image built once in the CI pipeline should flow from dev to staging to production without rebuilding. The registry holds the canonical artifact at each stage.

For teams with multiple services, a shared registry lets different pipelines pull base images and intermediate layers without repeatedly downloading from public sources. This speeds up builds and reduces external dependencies.

The line is when you start deploying to anything beyond your local machine. If your pipeline produces an image and that image needs to run somewhere else — a staging server, a Kubernetes cluster, a colleague’s machine — you need a registry. Local Docker storage does not survive past the machine that built it.

For solo projects or experiments that never leave your local Docker context, you can skip a registry entirely. Run docker build && docker run and you are done. The moment you want to share an image or deploy it to a remote environment, that changes. A registry is not a luxury you add when you have many services; it is infrastructure you need as soon as you have more than one environment.

When to skip or simplify

If your application is not containerized and never will be, a container registry does not add value. Some projects are better served by plain artifact storage (S3, GCS) for JARs, binaries, or Helm charts.

For personal projects or experiments that never leave your local machine, running a local registry or skipping one entirely makes sense. You can always add a registry later when the project grows.

The clearest case for skipping a registry is non-containerized workloads. If your artifact is a compiled binary, a Helm chart, or a zip file that gets deployed via a different mechanism, a container registry is the wrong tool. Store those in S3 or GCS with lifecycle policies.

There is also the case of genuinely experimental work. A spike to validate whether a particular base image works for your application does not need a production-grade registry with replication and access controls. You can build and run locally, then push to a registry once the approach is validated. The registry is a deployment infrastructure concern, not a development infrastructure concern.

What I watch out for is the trap of thinking you will add a registry later “when the project grows.” In practice, adding a registry retroactively means changing every CI job that pushes images and every deployment that pulls them. It is easier to start with a basic registry setup and simplify it if it turns out you do not need it than to retrofit one later.

Registry Architecture Flow

flowchart LR
    A[Developer] -->|docker build| B[CI Pipeline]
    B -->|docker push| C[Container Registry]
    C -->|docker pull| D[Staging Cluster]
    C -->|docker pull| E[Production Cluster]
    F[Scanner] -->|periodic scan| C

Registry Options

Cloud-provider registries:

ProviderServiceNotes
AWSECRIntegrated with IAM, VPC endpoints
AzureACRGeo-replication, webhook integration
GCPGCRIntegrated with Cloud Build, IAM
GoogleArtifact RegistrySuccessor to GCR, multi-format

Self-hosted options:

RegistryBest For
HarborEnterprise with authentication, replication
GitLab Container RegistryGitLab-integrated deployments
Docker HubPublic images, simple needs
ChartMuseumHelm charts alongside images

AWS ECR example:

# Create ECR repository
aws ecr create-repository \
  --repository-name myapp/backend \
  --image-scanning-configuration scanOnPush=true \
  --encryption-configuration encryptionType=AES256

# Login to ECR
aws ecr get-login-password --region us-east-1 | \
  docker login --username AWS --password-stdin 123456789.dkr.ecr.us-east-1.amazonaws.com

# Push image
docker build -t myapp/backend:1.0.0 .
docker push 123456789.dkr.ecr.us-east-1.amazonaws.com/myapp/backend:1.0.0

Azure ACR example:

# Create ACR
az acr create \
  --resource-group mygroup \
  --name myregistry \
  --sku Standard \
  --location eastus

# Enable admin user
az acr update -n myregistry --admin-enabled true

# Login and push
az acr login --name myregistry
docker build -t myregistry.azurecr.io/myapp:1.0.0 .
docker push myregistry.azurecr.io/myapp:1.0.0

Image Tagging Strategies

Tags identify image versions and control which code deploys where.

Common tagging patterns:

PatternExampleUse Case
Git SHAa1b2c3dPrecise traceability
Semantic version1.2.3Releases
Date-time202603251430CI/CD timestamps
latestlatestDefault, avoid for prod
Environmentprod, stagingEnvironment promotion

Recommended strategy for production:

# Always tag with multiple identifiers
IMAGE="myregistry.azurecr.io/myapp/backend"

# SHA for exact traceability
docker build -t $IMAGE:sha-$(git rev-parse --short HEAD)

# Semantic version from tag
docker build -t $IMAGE:$(cat VERSION)

# Date-time for CI builds
docker build -t $IMAGE:$(date +%Y%m%d%H%M%S)

# Don't use :latest in production manifests
# Instead:
kubectl set image deployment/myapp backend=$IMAGE:sha-a1b2c3d

Immutable tags for releases:

# Helm values for production deployment
image:
  repository: myregistry.azurecr.io/myapp/backend
  tag: "v1.2.3" # Immutable once released
  pullPolicy: IfNotPresent # Only pulls if not present

Vulnerability Scanning

Integrate scanning into your pipeline to catch vulnerabilities before deployment.

Trivy in CI/CD:

# GitHub Actions
- name: Run Trivy vulnerability scanner
  uses: aquasecurity/trivy-action@master
  with:
    image-ref: myregistry.azurecr.io/myapp:${{ github.sha }}
    format: "sarif"
    output: "trivy-results.sarif"
    severity: "CRITICAL,HIGH"
    exit-code: "1" # Fail on critical vulnerabilities

- name: Upload Trivy results to GitHub Security tab
  uses: github/codeql-action/upload-sarif@v3
  with:
    sarif_file: "trivy-results.sarif"

Harbor scanning:

Harbor integrates Trivy and other scanners natively. Configure in /etc/harbor/harbor.yml:

# Enable CVE prevention
vulnerability:
  severity: # Reject push if severity >= this level
    - high
    - critical
  stop_uploading: true
  stop_downloading: true
  update_bulk: "1h"

Gatekeeper policy:

# OPA Gatekeeper constraint for image signatures
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredImageTag
metadata:
  name: require-tag
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
  parameters:
    namespaces: ["production"]
    exemptImages:
      - "myregistry.azurecr.io/base-image:*"

Image Promotion Between Environments

Promote images through environments with validation at each stage.

GitLab CI image promotion:

stages:
  - build
  - scan
  - staging
  - production

build:
  stage: build
  script:
    - docker build -t $IMAGE:build-$CI_COMMIT_SHA .
    - docker push $IMAGE:build-$CI_COMMIT_SHA

scan:
  stage: scan
  script:
    - trivy image --exit-code 1 --severity HIGH,CRITICAL $IMAGE:build-$CI_COMMIT_SHA
  allow_failure: false

promote:staging:
  stage: staging
  script:
    - docker tag $IMAGE:build-$CI_COMMIT_SHA $IMAGE:staging-$CI_COMMIT_SHA
    - docker push $IMAGE:staging-$CI_COMMIT_SHA
    - curl -X POST "https://argo.example.com/api/v1/apps/myapp/sync" \
      -d '{"prune":true,"dryRun":false}'
  only:
    - main

promote:production:
  stage: production
  script:
    - docker tag $IMAGE:build-$CI_COMMIT_SHA $IMAGE:production-$CI_COMMIT_SHA
    - docker push $IMAGE:production-$CI_COMMIT_SHA
    -  # Update deployment manifests or trigger ArgoCD
  when: manual
  only:
    - main

Digest-based promotion for immutability:

# Get image digest
DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' myregistry.azurecr.io/myapp:1.0.0)
echo $DIGEST
# Output: myregistry.azurecr.io/myapp@sha256:abc123...

# Deploy by digest (immutable)
docker run myregistry.azurecr.io/myapp@sha256:abc123...

Registry Caching for Faster Builds

Cache images to reduce build times and external dependencies.

GitHub Actions cache for Docker:

- uses: docker/setup-buildx-action@v3
  with:
    driver-opts: |
      image=moby/buildkit:buildx-stable
      network=host

- uses: docker/build-push-action@v5
  with:
    push: true
    tags: myregistry.azurecr.io/myapp:${{ github.sha }}
    cache-from: type=gha
    cache-to: type=gha,mode=max

Registry mirror for air-gapped environments:

# Deploy a pull-through cache
helm install registry-cache stable/docker-registry \
  --set pullThrough.enabled=true \
  --set cache.remoteURL=https://registry-1.docker.io

# Configure Docker daemon on nodes
# /etc/docker/daemon.json
{
  "registry-mirrors": ["https://cache.mycorp.example.com"]
}

Access Control and Authentication

Control who can push, pull, and manage images.

AWS ECR IAM policies:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789:role/deploy-role"
      },
      "Action": [
        "ecr:GetDownloadUrlForLayer",
        "ecr:BatchGetImage",
        "ecr:BatchCheckLayerAvailability"
      ]
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789:role/ci-service-account"
      },
      "Action": ["ecr:*"]
    }
  ]
}

Azure RBAC for ACR:

# Reader role for production pods
az role assignment create \
  --assignee-object-id $(kubectl get serviceaccount default -n production -o jsonpath='{.metadata.uid}') \
  --role AcrPull \
  --scope /subscriptions/.../resourceGroups/mygroup/providers/Microsoft.ContainerRegistry/registries/myregistry

# Contributor for CI/CD pipelines
az role assignment create \
  --assignee $CI_SERVICE_PRINCIPAL_ID \
  --role Contributor \
  --scope /subscriptions/.../resourceGroups/mygroup/providers/Microsoft.ContainerRegistry/registries/myregistry

Kubernetes image pull secrets:

# Create image pull secret
kubectl create secret docker-registry acr-secret \
  --docker-server=myregistry.azurecr.io \
  --docker-username=myuser \
  --docker-password=$(az keyvault secret show --name acr-password --vault-name myvault -o tsv --query value) \
  --docker-email=myuser@example.com

# Reference in service account
apiVersion: v1
kind: ServiceAccount
metadata:
  name: myapp-sa
  namespace: production
imagePullSecrets:
  - name: acr-secret

Production Failure Scenarios

Common Registry Failures

These failures show up most often in registry postmortems. The table captures the immediate impact and a short-term fix, but each one has deeper causes worth understanding before your pipeline hits them.

FailureImpactMitigation
Registry authentication token expiredCannot pull or push images, deployments failUse service accounts with rotating credentials
Image not found after tag deletionDeployment tries to pull deleted imageNever delete tags in production, use immutable tags
Quota exceeded on cloud registryCannot push new imagesMonitor storage usage, set lifecycle policies
Network partition to registryBuilds cannot push, deployments cannot pullUse multi-region replication, local registry cache
Vulnerability scan blocking deploysCritical CVE stops production deploymentDefine triage process for CVEs, do not scan blindly

Token expiry is the most common because it is the most invisible. Service accounts with long-lived credentials work fine for months, then fail at the worst possible moment — during a production incident when someone is already stressed and debugging a deployment. Use workload identity or OIDC federation instead, so credentials rotate automatically without any human involvement.

Tag deletion seems like good hygiene when a CI pipeline clears out old build tags to save space. The problem is that production deployments often reference those tags indirectly — a canary pointing to myapp:staging-latest or a Helm release resolving myapp:{{ .Values.imageTag }} at install time. Delete the tag, and the next deployment fails even though the image itself is still there.

Quota errors are predictable if you monitor for them. Cloud registries charge per storage byte and per API call, and both have hard limits. Set a storage alert at 80% of your quota and run cleanup before you hit the wall. Teams usually find out about quota issues the hard way — when a release pipeline stops working and they have to delete images under pressure.

Image Pull Failures

flowchart TD
    A[Pod Schedule] --> B{Pull Image}
    B -->|Success| C[Pod Starts]
    B -->|Fail| D{Error Type}
    D -->|Image Not Found| E[Check tag exists in registry]
    D -->|Auth Failed| F[Refresh pull secret]
    D -->|Quota Exceeded| G[Delete old images]
    E --> H[Redeploy]
    F --> H
    G --> H

Observability Hooks

What to monitor:

  • Image push success/failure rate
  • Pull request latency by region
  • Storage consumption growth rate
  • Vulnerability scan results trend
  • Authentication failure spikes
# AWS ECR - check repository size
aws ecr describe-repositories --query 'repositories[].{name:repositoryName,size:imageScanningConfiguration.scanOnPush}'

# Azure ACR - check usage
az acr show-usage --name myregistry

# Docker Hub - check rate limits
curl -s -o /dev/null -w "%{http_code}" https://hub.docker.com/v2/

Common Pitfalls / Anti-Patterns

Using the :latest tag in production manifests

When your manifest references myapp:latest, you have no idea which image actually deployed. If you need to roll back, you cannot because :latest keeps changing. Always use specific immutable tags (commit SHA, semantic version, or date-based).

The core problem with :latest is that it severs the link between your source code and your running deployment. Your CI pipeline builds an image and pushes it with a tag. Minutes later, someone else pushes a different image with the same :latest tag. By the time your deployment picks up that image, it could be running code from a completely different commit — or even a different branch. Rolling back becomes impossible because you cannot tell which image :latest pointed to at any given time.

Consider a real incident: a team deploys using :latest, and hours later their production is broken. They run kubectl rollout undo expecting to restore the previous version, but the previous “version” is whatever someone happened to push last. There is no version history to fall back on. The fix requires manually identifying the last known good SHA and retagging, which takes far longer than a simple git revert.

The solution is straightforward: tag every image with an immutable identifier tied to your source. Commit SHA tags (sha-abc1234) give you exact traceability — if a deployment fails, you know exactly which commit caused it. In your Kubernetes manifests, reference that SHA directly:

# In your CI pipeline, extract and embed the SHA
IMAGE="myregistry.azurecr.io/myapp/backend"
SHA=$(git rev-parse --short HEAD)
docker build -t $IMAGE:sha-$SHA .
docker push $IMAGE:sha-$SHA

# In your Helm values or kustomization overlay
image:
  tag: "sha-abc1234"  # Immutable, tied to git commit

If you must use environment-based tags, reserve those for promotion workflows (staging, production) rather than as a substitute for immutable versioning. The rule: production manifests should never reference a tag that can be overwritten.

Not scanning images before deployment

Pushing unscanned images to production means you discover vulnerabilities after they are running. Integrate Trivy or similar scanners into your CI pipeline and block pushes when critical CVEs are found.

Without scanning, a known CVE sits quietly in your base image while the CI pipeline builds and pushes your app image, the deployment rolls out, and hours later a security audit surfaces the problem — requiring an emergency patch rollout to an already-deployed fleet.

A scanning step runs after the build and push. Trivy pulls the image manifest, downloads each layer, and matches packages against CVE databases (NVD, RHEL, Debian security advisories). Severity thresholds determine when a scan fails the build.

# GitHub Actions - scan before promoting
scan:
  stage: scan
  script:
    - trivy image \
      --severity CRITICAL,HIGH \
      --exit-code 1 \
      --ignore-unfixed \
      myregistry.azurecr.io/myapp:$CI_COMMIT_SHA
  rules:
    - if: $CI_COMMIT_BRANCH == "main"

Blocking thresholds depend on your risk tolerance. A typical setup blocks on CRITICAL findings in any environment, blocks on HIGH in production, and logs MEDIUM/LOW for weekly triage. When a critical CVE has no patch, you can pin to an older base image that predates the vulnerability, apply a vendor mitigation if one exists, or document an exception with sign-off from your security team — not just the deployment team.

Existing images need rescanning too. New CVEs drop daily, and an image that was clean last week might not be today. Both Harbor and ECR support automatic rescanning when vulnerability databases update. Wire up notifications so your team hears about new critical findings before an auditor does.

Leaving unused images in the registry

Old images accumulate and consume storage. Without lifecycle policies, your registry bill grows while nobody knows which images are actually in use. Set up retention policies that delete images not referenced by any deployment.

Images pile up because every CI run produces a tag, and feature branches generate dozens per day. A repository that started with 10 images can easily have 500 within a quarter. Most of those tags never see a deployment — abandoned branches, failed runs after the push step. Every layer is stored, and the bill grows quietly until you actually look.

Lifecycle policies handle this:

AWS ECR lifecycle policies use rule-based filtering. You can target untagged images older than a certain number of days, or images with a tag prefix that no longer matches any active deployment:

# Create lifecycle policy - expire untagged images after 14 days
aws ecr put-lifecycle-policy \
  --repository-name myapp/backend \
  --lifecycle-policy-text '{
    "rules": [{
      "rulePriority": 1,
      "description": "Expire untagged images",
      "selection": {
        "tagStatus": "untagged",
        "countType": "sinceImagePushed",
        "countUnit": "days",
        "countNumber": 14
      },
      "action": {"type": "expire"}
    }]
  }'

Azure ACR lifecycle policies support similar rules with tag-based filtering. You can preserve release tags (matching v*) while expiring build tags (matching build-*):

az acr task create \
  --name image-cleanup \
  --registry myregistry \
  --cmd myregistry.azurecr.io/myapp:placeholder \
  --schedule "0 9 * * *" \
  --context /dev/null

# Purge policy via ACR task
az acr config task update \
  --name image-cleanup \
  --cmd "acr purge --filter 'myapp:build-.*' --ago 7d"

Harbor supports lifecycle rules through its UI and API, with options to retain by tag suffix, count, or days since last pull.

Auditing unused images: Before applying any cleanup policy, identify what is actually in use. Query your deployment records to find which tags or digests your Kubernetes clusters are running. Cross-reference against your registry inventory — anything not in a deployment record is a candidate for deletion.

# ECR - list untagged images with age
aws ecr list-images \
  --repository-name myapp/backend \
  --filter '{"tagStatus": "UNTAGGED"}' \
  --query 'imageIds[*].[imagePushedAt,imageDigest]' \
  --output table

# Harbor - API to list repositories and their usage
curl -s "https://harbor.mycorp.com/api/v2/projects/myapp/repositories" \
  -H "Authorization: Bearer $TOKEN" | jq '.[] | {name, tags_count}'

Beyond storage, a repository with tens of thousands of images gets slow to query — CI pipelines that call the registry for the latest tag start timing out. Cleanup keeps operations fast at scale.

Mixing image sources in one pipeline

Using Docker Hub for base images, your private registry for app images, and a third-party registry for dependencies makes it hard to track where each piece comes from. Keep a clear lineage from build to registry to deployment.

Fragmented sources add external dependencies with their own failure modes. Docker Hub goes down during a critical deploy and your pipeline fails — even though your app image is fine in ACR. A third-party registry retires an image version and your builds break silently.

Fragmented sources also kill traceability. When your pipeline pulls node:18-alpine from Docker Hub today, that tag could point to a different image next month as patches ship. Your production image changes without any code change on your end. You no longer know what is running.

The fix is straightforward: mirror public images into your private registry during infrastructure setup, not during the application pipeline. Your CI then pulls everything from one place.

# Mirror a public image into your private registry
docker pull docker.io/library/node:18-alpine
docker tag node:18-alpine myregistry.azurecr.io/base/node:18-alpine
docker push myregistry.azurecr.io/base/node:18-alpine

# Configure registry mirror in daemon for air-gapped environments
# /etc/docker/daemon.json
{
  "registry-mirrors": ["https://myregistry.azurecr.io"]
}

Tag pinning for reproducibility: Beyond keeping sources centralized, pin the full image reference — registry, repository, tag, and digest — in your build files. A node:18-alpine tag is mutable; myregistry.azurecr.io/base/node:18-alpine@sha256:abc123 is not. When you pin by digest, the exact content is guaranteed regardless of when you pull.

If a scanner tool is only available from a vendor’s registry, pull it once during infrastructure setup and store the image in your private registry. Your CI references the internal copy. The vendor’s registry is no longer on your critical path.

Forgetting about cross-region replication

If your production cluster is in us-west-2 but your registry is only in us-east-1, every image pull crosses the country. Configure replication for the regions where your clusters run.

Image pulls mean manifest fetches and layer downloads. A manifest is a few kilobytes; layers can be hundreds of megabytes. If your registry is in us-east-1 and your cluster is in us-west-2, every layer download crosses the continental US. A 500MB image with 10 layers means 5GB of cross-country traffic per deploy — multiplied across every replica. A rolling update of 20 replicas pulls 100GB across regions, plus egress charges, plus latency spikes that can timeout fast rollouts.

Provider-specific replication options:

ProviderReplication MethodConfiguration
AWS ECRMulti-region replication via same repositoryEnable in repository settings, replicate to selected regions
Azure ACRGeo-replication to paired regionsEnable during creation or via --geo-replication-enabled
GCP Artifact RegistryMulti-region repositoriesCreate repository in us-west1 and us-east1 separately
HarborReplication policies between Harbor instancesConfigure push or pull replication rules per project
# AWS ECR - enable cross-region replication
aws ecr create-repository \
  --repository-name myapp/backend \
  --image-scanning-configuration scanOnPush=true \
  --encryption-configuration encryptionType=AES256 \
  --region us-west-2  # Separate repository in target region

# Azure ACR - enable geo-replication
az acr create \
  --resource-group mygroup \
  --name myregistry \
  --sku Standard \
  --location eastus

az acr update -n myregistry --geo-replications eastus2,westus

# Verify replication status
az acr replication list -r myregistry --query '[].{region:location,status:status}'

Pull-through cache as an alternative: If multi-region replication is too expensive or complex, deploy a pull-through cache in each region. The cache mirrors images on first pull, so subsequent pulls are local. This is especially useful for base images that change infrequently — you pay the cross-region cost once, then all subsequent pulls are local.

# Harbor pull-through cache configuration
# /etc/harbor/harbor.yml
registry:
  # Pull-through cache for Docker Hub
  relativeurl: /proxy
  storage_driver: s3
 :

Replication lags. An image pushed to the primary region takes time to appear in secondaries — if your deployment automation races ahead of replication, it pulls a missing image. Monitor replication status and gate deployments on image availability in the target region.

Trade-off Analysis

RegistryHosted vs Self-HostedAuthCostBest For
Docker HubBothDocker AuthFree tier / paidOpen source images
ECRCloud-hostedIAMPay per storage + egressAWS workloads
GCR / Artifact RegistryCloud-hostedIAM + VMDFPay per storage + egressGCP workloads
ACRCloud-hostedIAM + admin keysPay per storage + egressAzure workloads
HarborSelf-hostedOIDC, LDAP, localInfrastructure costEnterprise / air-gapped
GitHub PackagesCloud-hostedGitHub tokenStorage + bandwidthGitHub-native workflows
GitLab Container RegistryCloud-hostedGitLab tokenStorage + bandwidthGitLab-native workflows

Interview Questions

1. What are the key differences between major container registry options (ECR, ACR, GCR, Harbor)?

ECR (AWS): native IAM integration, VPC endpoints for private networking, pay per storage and egress. ACR (Azure): geo-replication, webhook integration, admin user option. GCR/Artifact Registry (GCP): integrated with Cloud Build, IAM with VM metadata service. Harbor (self-hosted): enterprise-grade with OIDC/LDAP auth, replication across registries, CVE scanning. ECR/ACR/GCR are cloud-hosted managed services — no infrastructure to manage, automatic scaling. Harbor is self-hosted — full control but requires maintenance. Choose based on cloud provider, authentication integration needs, and whether you need self-hosted vs cloud-hosted.

2. How do you implement immutable tagging for production container images?

Immutable tagging strategy: 1) Never use :latest in production manifests — it changes and you lose traceability, 2) Use commit SHA tags: :sha-{git rev-parse --short HEAD} for exact traceability from code to image, 3) Use semantic version tags: :v1.2.3 for releases with clear versioning, 4) Use date-time tags: :202603251430 for CI builds that need uniqueness, 5) Tag with multiple identifiers: production image gets :v1.2.3, :sha-abc123, and environment tag, 6) Set imagePullPolicy: IfNotPresent to avoid unexpected pulls. Immutable tags mean once a tag is released, it never changes — same SHA always points to same content.

3. Describe how to integrate vulnerability scanning into a CI/CD pipeline.

Vulnerability scanning integration: 1) Use Trivy or similar scanner in CI pipeline after build and push, 2) Configure scanOnPush=true for automatic scanning on every image push, 3) Set severity thresholds — block deployment on CRITICAL vulnerabilities, warn on HIGH, 4) In GitHub Actions: use aquasecurity/trivy-action with exit-code to fail pipeline, 5) Upload results to security dashboard (GitHub Security tab, security hub), 6) For Harbor, enable CVE prevention in configuration — reject pushes when severity exceeds threshold, 7) Set up periodic rescanning of existing images for new CVEs, 8) Define triage process for handling exceptions when scanning blocks legitimate builds.

4. How do you handle cross-region replication for container registries?

Cross-region replication strategy: 1) Most cloud registries support geo-replication — enable it to replicate images across regions automatically, 2) For multi-region Kubernetes clusters, deploy images to registry regions closest to your clusters, 3) Use pull-through cache for air-gapped or remote regions — cache images locally, 4) Configure replication rules based on tags or repositories, 5) Monitor replication lag and failure rates, 6) For Harbor, use the replication feature to push to multiple Harbor instances, 7) Consider using a global DNS that routes to nearest registry region. Unreplicated registries cause image pull latency and potential availability issues if a region fails.

5. What is the difference between digest-based and tag-based image references?

Digest vs tag references: Tags are mutable — myapp:v1.2.3 can be overwritten to point to different content. Digests are immutable — myapp@sha256:abc123 always points to exactly that image content. Digest references provide true immutability — you know exactly what code is running. Use digest references in production for cryptographic certainty. Workflow: after building, inspect digest with docker inspect --format='{{index .RepoDigests 0}}', then deploy using digest. Tags are more human-readable for development and staging. Best practice: use tags for development, digest for production deployments.

6. How do you implement access control for a container registry in a CI/CD context?

Registry access control implementation: 1) Use cloud IAM roles for AWS ECR, Azure RBAC for ACR, GCP IAM for GCR — not shared passwords, 2) CI/CD pipelines get service account credentials via OIDC federation or workload identity — no long-lived secrets, 3) Production pods use image pull secrets with minimal permissions — only pull, not push, 4) Separate credentials for different environments — staging pipeline pushes to staging repo, production pipeline pushes to production repo, 5) Use least privilege: deploy role can only pull images, ci-service-account can push to specific repos, 6) Rotate credentials regularly. Kubernetes imagePullSecrets reference registry credentials stored as Kubernetes secrets.

7. What strategies exist for reducing container image build and pull times?

Image build and pull optimization: 1) Multi-stage builds — final image contains only runtime, not build tools, 2) Use slim or alpine base images to reduce size, 3) Leverage Docker layer caching — structure Dockerfile so unchanged layers are cached, 4) In CI, use cache-from and cache-to with buildx for layer caching across builds, 5) Use registry caching for dependencies — cache npm packages, Maven artifacts during build, 6) Pre-pull base images on nodes to avoid pull latency during deployment, 7) Use GitHub Actions cache for buildx, 8) For air-gapped environments, set up pull-through cache that mirrors required images. Smaller images = faster pulls, better caching = faster builds.

8. How do you handle image lifecycle management to prevent registry bloat?

Image lifecycle management: 1) Set retention policies that delete untagged images after N days, 2) Never delete tags in production — use immutable tags that are never overwritten, 3) Monitor storage growth rate and set alerts for unusual increases, 4) Use lifecycle policies based on tag patterns — keep release tags, prune CI build tags, 5) Implement cleanup for image promotion — after promoting from staging to production, delete staging tag or use automated cleanup jobs, 6) For ECR, use lifecycle policies with rule priority — stop images older than 90 days, 7) Regular audit of which images are actually in use vs just stored. Registry bloat increases costs and slows down registry operations.

9. Describe how to implement image signing and verification for supply chain security.

Image signing implementation with Cosign: 1) Sign images after push using cosign sign command with key or workload identity, 2) Store signatures in Sigstore transparency log (rekor) for auditing, 3) In Kubernetes, use OPA Gatekeeper or Kyverno admission controllers to verify signatures before allowing image pulls, 4) Configure image policy that requires all production images to be signed by specific keys, 5) Use attestation to attach metadata (SBOMs, test results) to images, 6) Verify signatures with cosign verify before deployment. Sigstore cosign provides keyless signing via OIDC — no managing signing keys manually.

10. What are the failure scenarios when image pull secrets expire or become invalid?

Image pull secret failures: 1) Pods cannot start — ImagePullBackOff with "invalid secret" error, 2) Deployment fails silently if only some replicas fail, 3) Rotating credentials requires updating Kubernetes secrets and restarting pods, 4) If using Azure admin credentials, rotation invalidates stored password. Prevention: use short-lived credentials via workload identity instead of static secrets, implement automated rotation before expiry, monitor authentication failure metrics, have runbooks for secret refresh. In ECR, use ECR credentials helper that automatically fetches short-lived tokens. In ACR, use azure-workload-identity federation to avoid secrets entirely.

11. How do you design a promotion pipeline for images moving from dev to staging to production?

Image promotion pipeline design: 1) Build once in CI — tag with commit SHA and build number, 2) Push to dev registry with dev tag, 3) Run automated tests against dev image — vulnerability scan, integration tests, 4) On success, promote to staging registry — retag with staging prefix, 5) Run E2E tests and smoke tests in staging environment, 6) On approval, promote to production — retag with production tag, use digest reference for immutable reference, 7) Track promotion history in metadata — which commit, which tests passed, who approved. Digest-based promotion ensures immutability — production always references exact SHA that passed tests.

12. What is a pull-through cache and when would you use it?

Pull-through cache: a local registry that proxies and caches images from upstream registries. Use when: you have air-gapped environments that cannot reach Docker Hub directly, you want to reduce external dependencies in CI, you have slow or unreliable internet connectivity, you want to cache base images for faster builds. Harbor and Docker Registry support pull-through cache mode. Configure nodes to use the cache as their only registry mirror. Cache automatically downloads images on first pull and serves from cache thereafter. For air-gapped Kubernetes, deploy a pull-through cache on the same network as your cluster.

13. How do you handle quota exceeded errors in cloud container registries?

Quota exceeded handling: 1) Monitor storage usage and set alerts before hitting limits, 2) Implement lifecycle policies to auto-delete old/unused images before quota is reached, 3) When quota is exceeded: delete old images, especially untagged (dangling) images, 4) Compress existing images if possible — not usually applicable since images are already compressed, 5) For ongoing prevention: implement retention policies, audit for unused images regularly, use smaller base images. Cloud providers have storage quotas and API rate limits — track both and implement cleanup before hitting limits. Emergency fix: manually delete images via cloud console or CLI.

14. What considerations exist for using Docker Hub versus private registries in production?

Docker Hub vs private registry considerations: Docker Hub has rate limits on pulls (anonymous: 100/6 hours, authenticated: 200/6 hours) that can block production deployments. Use private registries when: you hit rate limits, you need to scan images for CVEs before use, you want control over image availability and uptime, you need audit trail of who pulls which images. Private registries also provide better integration with CI/CD and Kubernetes. Public images from Docker Hub may have supply chain risks — base images could be compromised. Best practice: use private registry for all production workloads, use Docker Hub only for getting base images during build, then push to your private registry.

15. How do you troubleshoot image pull failures in Kubernetes?

Image pull troubleshooting: 1) Check pod status — ImagePullBackOff indicates auth failures, ErrImageNotFound indicates missing tag, 2) Verify image tag exists in registry: docker manifest inspect, 3) For auth failures: check imagePullSecret is correct, secret hasn't expired, service account has correct secret reference, 4) For network issues: can node reach registry? check DNS resolution, firewall rules, 5) For quota issues: check storage limits, 6) Use kubectl describe pod for detailed events, kubectl logs for container startup logs, 7) Verify imagePullPolicy — Always pulls every time, IfNotPresent uses cached. Common fixes: update image tag to correct version, create image pull secret correctly, add network policies to allow registry access.

16. What is the role of Harbor in an enterprise container strategy?

Harbor enterprise features: 1) Authentication via OIDC/LDAP for enterprise identity integration, 2) Replication across multiple Harbor instances or to cloud registries for disaster recovery, 3) Vulnerability scanning with Trivy integrated natively, 4) CVE prevention — block pushes with severity above threshold, 5) Content signing with Notary for image trust, 6) Role-based access control — different permissions for developers vs production, 7) Registry mirroring for air-gapped environments. Use Harbor when you need central management of container images across multiple teams and environments, with enterprise auth and governance requirements.

17. How do you implement webhook-based automation for container registry events?

Registry webhook automation: 1) Configure webhooks on push, pull, delete events, 2) Trigger CI/CD pipelines on image push — automatically run tests and promote images, 3) Notify Slack/Teams on security findings or quota warnings, 4) Trigger vulnerability scanning workflows on new image pushes, 5) Integrate with ArgoCD/Flux to update deployments when new image version is available. Most registries support webhook configuration: ACR webhooks, ECR event notification via SNS/SQS, Harbor webhook. Use webhook relays if internal services cannot be reached directly from registry.

18. What strategies exist for reducing cloud egress costs from container registries?

Egress cost reduction: 1) Deploy registry in same region as your clusters — images stay within same region, no egress charges, 2) Use registry replication to have images close to all clusters, 3) Implement pull-through caching so images are cached at edge locations near clusters, 4) Use smaller base images to reduce data transferred, 5) Limit unnecessary pulls — configure proper imagePullPolicy (IfNotPresent), 6) Audit who is pulling what and from where — identify unexpected cross-region pulls, 7) Consider registry egress pricing when selecting cloud provider for multi-cloud strategies. Egress can be significant at scale — monitor and optimize.

19. How do you ensure image traceability from source code to production deployment?

Image traceability implementation: 1) Tag images with git commit SHA — provides exact link to source code, 2) Store build metadata — who built, when, what source commit, what tests passed, 3) Use SBOM (Software Bill of Materials) attestation to attach provenance, 4) Implement digest references in production — exact SHA ensures you know what runs, 5) Log image references in deployment records, 6) Use image signing (Cosign) to cryptographically verify image origin, 7) In Kubernetes, annotate pods with image digest for audit. This creates audit trail from code commit → build → image → deployment.

20. What is the difference between container scanning and container signing and when would you use each?

Scanning vs signing: Container scanning analyzes images for known vulnerabilities (CVEs) — static analysis of image contents. Signing cryptographically verifies image identity and provenance — who built it, that it wasn't tampered with. Use scanning to catch security vulnerabilities before deployment — integrate into CI to block vulnerable images. Use signing to ensure you're running what you think you're running — prevents supply chain attacks where images are replaced or modified. Best practice: implement both — scan for vulnerabilities, sign for trust. Tools: Trivy/Clair for scanning, Cosign/Notary for signing.

Further Reading

Official Documentation

Vendor documentation is the ground truth when the CLI flags or UI screens described here stop matching reality. Check these links before filing an issue or assuming something is broken.

  • Docker Registry Documentation - Self-hosted registry setup, configuration options for storage backends, authentication, and notifications
  • Harbor Documentation - Enterprise registry with replication, CVE scanning via Trivy, and OIDC-based authentication
  • AWS ECR Documentation - Cloud-native image registry with IAM integration, lifecycle policies, and multi-region replication
  • Azure ACR Documentation - Container registry on Azure with geo-replication, webhook triggers, and Azure RBAC integration

The registry sits between your CI pipeline and your clusters. These guides cover the adjacent pieces — what happens before you push an image, and what happens after you pull one.

  • CI/CD Pipeline Design - Pipeline architecture patterns that complement registry setup, including multi-stage pipelines with gated promotion and artifact passing between stages
  • Deployment Strategies - Deployment patterns and rollout strategies that work with immutably-tagged images, including blue-green and canary deployments
  • Automated Testing in CI/CD - Testing strategies and quality gates, including where vulnerability scanning fits in a pipeline relative to unit and integration tests

Tools and References

Trivy and Clair both scan for CVEs but pull from different databases with different update cadences. Cosign is a different animal — it solves the trust problem, not the vulnerability problem. Knowing which tool does which keeps you from reaching for the wrong one at 2am.

  • Trivy - Vulnerability scanner for containers and infrastructure as code, used in this guide for CI integration and Harbor’s built-in scanning
  • Clair - Static vulnerability analysis for containers, originally developed by CoreOS and now maintained by Quay; different CVE database update cadence than Trivy
  • Cosign - Container image signing and verification as part of Sigstore’s keyless signing workflow; pairs with OPA Gatekeeper or Kyverno for admission control
  • OPA Gatekeeper - Policy enforcement for Kubernetes that can validate image signatures, enforce tag constraints, and require vulnerability scan results before deployment

Conclusion

Key Takeaways

A few themes show up again and again when I look at registry incidents.

First: immutability. Every other practice in this guide depends on the image your pipeline built being the same image that runs in production. That guarantee breaks when you use mutable tags, store credentials in plain text, or skip vulnerability scanning because the pipeline is already slow enough. Tags are the contract between your build and your deployment — make that contract immutable.

Second: scanning. A CVE in a base image affects every deployment that uses it. Running Trivy in CI catches some of those. Policy gates that actually block critical CVEs from reaching production are what take you from checking a box to reducing risk.

Third: credentials. Rotating long-lived passwords or managing Docker Hub credentials across a cluster is overhead that cloud IAM eliminates. Use workload identity or OIDC federation and credentials take care of themselves.

Fourth: replication. Teams that treat multi-region replication as a performance optimization usually end up learning what it is actually for — infrastructure resilience — during an outage.

Run the checklist below to see where you stand on each of these.

Registry Health Checklist

Run these before a production deployment or as part of a regular health check. Treat failures here as blockers, not warnings.

The examples use ACR and ECR syntax. Swap the CLI and flags for your provider — what each command checks stays the same.

# Verify you can pull your production image
docker pull myregistry.azurecr.io/myapp:prod && echo "Pull successful"

# Check for critical CVEs before deploying
trivy image --severity CRITICAL,HIGH myregistry.azurecr.io/myapp:v1.2.3

# List unused images (older than 90 days)
aws ecr list-images --repository-name myapp --filter '{"tagStatus": "UNTAGGED"}'

# Verify pull secret works in Kubernetes
kubectl get secret acr-secret --output jsonpath='{.data.\.dockerconfigjson}' | base64 -d

Run the pull check first when a deployment fails. If docker pull works but your pod doesn’t start, look at Kubernetes config, not the registry.

Run the Trivy scan against the exact tag you plan to deploy. Scanning :latest or a staging tag while deploying something different doesn’t verify anything.

Unused image audits catch accumulation before it hits your quota. Run monthly, and watch your storage growth rate — a CloudWatch alarm or equivalent keeps you from getting surprised.

The Kubernetes pull secret check catches expired credentials before they show up as ImagePullBackOff in production. If the decoded secret looks stale, refresh it before your next deployment window.

Category

Related Posts

Container Images: Building, Optimizing, and Distributing

Learn how Docker container images work, layer caching strategies, image optimization techniques, and how to publish your own images to container registries.

#docker #containers #devops

Docker Networking: From Bridge to Overlay

Master Docker's networking models—bridge, host, overlay, and macvlan—for connecting containers across hosts and distributed applications.

#docker #networking #containers

Docker Fundamentals

Learn Docker containerization fundamentals: images, containers, volumes, networking, and best practices for building and deploying applications.

#docker #containers #devops