Metrics, Monitoring, and Alerting: From SLIs to Production Alerts

Monitoring is how you know your system is healthy. Without it, you are blind to failures, degradation, and capacity issues until users report them. This guide covers the theory and practice of building monitoring systems that actually help.

We assume you have basic familiarity with logs and metrics collection. If you need a logging refresher, our Logging Best Practices guide covers structured logging first.

Introduction

There are three established methodologies for defining what to measure: RED, USE, and Google’s Four Golden Signals. Each serves different purposes.

RED Method

RED focuses on request-driven services, particularly APIs:

• Rate: Requests per second
• Errors: Error rate (usually percentage of requests resulting in errors)
• Duration: Response time distribution (p50, p95, p99)

# Request rate
sum(rate(http_requests_total[5m]))

# Error rate
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

# Duration percentiles
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

USE Method

USE focuses on resource utilization:

• Utilization: How busy is the resource
• Saturation: How much work is queued beyond what the resource can handle
• Errors: Internal errors preventing correct operation

# CPU utilization
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory saturation
node_memory_Active_bytes / node_memory_MemTotal_bytes * 100

# Disk I/O errors
rate(node_disk_io_time_seconds_total{ mode != "idle" }[5m])

Four Golden Signals

Google’s SRE book defines four signals every service should monitor:

• Latency: How long operations take
• Traffic: How much demand exists
• Errors: How often requests fail
• Saturation: How full the system is

For most web services, these four signals cover what matters most.

Core Concepts

SLIs are the actual metrics you measure. They define what “good” looks like for your service.

Common SLIs for Web Services

An SLI is a measurement definition: it pinpoints exactly what you are tracking, how you are measuring it, what outcome counts as good, and what range is acceptable. Getting this definition wrong creates bad incentives. If you measure “requests that return any response,” you end up with a service that returns 500 errors as fast as possible to look healthy. If you measure “requests that return a successful response within 500ms,” your definition already encodes both correctness and performance. The table below gives you a practical starting point for web services, but you should adjust thresholds based on what your users actually experience rather than these generic values.

SLI	Definition	Good	Acceptable
Availability	Percentage of requests that get a successful response	99.9%	99.5%
Latency	p95 response time under normal conditions	< 200ms	< 500ms
Throughput	Requests handled per second	Varies	Above baseline
Error Rate	Percentage of errors (5xx, timeouts)	< 0.1%	< 1%

Think carefully about the “good” column. A p95 latency target of 200ms is meaningless if your p50 is already 180ms — it means half your users are experiencing worse-than-target latency. Plot your actual latency distribution before setting targets. Similarly, error rate thresholds depend on your dependency reliability: a service that calls five downstream APIs, each with 99.9% availability, already has a mathematical floor of approximately 99.5% end-to-end availability due to compounding failures, regardless of how well your own code performs.

Measuring SLIs

Define SLIs precisely so they can be measured consistently:

# sli-config.yaml
service: api-gateway
environment: production

slis:
  - name: request_success_rate
    description: "Percentage of requests returning non-5xx responses"
    query: |
      sum(rate(http_requests_total{service="api-gateway",status!~"5.."}[5m]))
      /
      sum(rate(http_requests_total{service="api-gateway"}[5m]))

  - name: p95_latency
    description: "95th percentile request duration"
    query: |
      histogram_quantile(0.95,
        sum(rate(http_request_duration_seconds_bucket{service="api-gateway"}[5m])) by (le)
      )

  - name: error_rate
    description: "Percentage of requests returning 5xx errors"
    query: |
      sum(rate(http_requests_total{service="api-gateway",status=~"5.."}[5m]))
      /
      sum(rate(http_requests_total{service="api-gateway"}[5m]))

Service Level Objectives

SLOs are the targets you want to achieve. They transform SLIs into goals your team commits to.

Defining SLOs

SLOs combine an SLI, a threshold, and a time window:

# SLO configuration
objectives:
  - display_name: "API Availability"
    sli: request_success_rate
    target: 99.9
    window: 30d
    description: "API should be available 99.9% of the time"

  - display_name: "API Latency"
    sli: p95_latency
    target: 99.0
    threshold_ms: 200
    window: 30d
    description: "95% of requests should complete within 200ms"

  - display_name: "API Error Rate"
    sli: error_rate
    target: 99.5
    threshold_percent: 0.1
    window: 30d
    description: "Error rate should stay below 0.1%"

Error Budgets

Error budgets convert SLO compliance into actionable information. If your availability SLO is 99.9% over 30 days, you have 43 minutes of allowed downtime in that period:

# Calculate error budget
def error_budget(slo_target, window_days=30):
    window_seconds = window_days * 24 * 60 * 60
    allowed_downtime = window_seconds * (1 - slo_target)
    return allowed_downtime

# 99.9% over 30 days = 43.2 minutes
budget = error_budget(0.999)
print(f"Monthly error budget: {budget / 60:.1f} minutes")
# Monthly error budget: 43.2 minutes

Error budgets tell you when you can be aggressive with releases (large budget remaining) and when you need to be careful (budget nearly exhausted).

flowchart TB
    subgraph "Error Budget Lifecycle"
        A["SLO Target\n99.9% = 43.2 min/month"]
        B["Error Budget\n= Allowed Downtime"]
        C["Budget Remaining\n= Good (deploy freely)"]
        D["Budget Burning\n= Slow Burn (> 6x)"]
        E["Budget Critical\n= Fast Burn (14.4x)"]
        F["Budget Exhausted\n= SLA Breach"]
    end
    A --> B
    B --> C
    B --> D
    D --> E
    E --> F

SLO Alerting

Set up alerting before you burn through the budget:

# Alert when 10% of budget is burned in 1 hour
- alert: SLOErrorBudgetBurning
  expr: |
    (
      sum(rate(http_requests_total{service="api-gateway",status=~"5.."}[1h]))
      /
      sum(rate(http_requests_total{service="api-gateway"}[1h]))
    )
    > (1 - 0.999) * 0.1
  labels:
    severity: warning
  annotations:
    summary: "Error budget burning fast"
    description: "More than 10% of the 30-day error budget has been burned in the last hour"

Multi-Window Burn-Rate Alerting

Standard threshold alerts catch sudden spikes but miss slow leaks. Multi-window burn-rate alerting detects both:

Window	Burn Rate Multiplier	Budget Burned	Use Case
1 hour	14.4x	1% per hour	Fast burn (page immediately)
6 hours	6x	10% per 6 hours	Medium burn (warning)
3 days	3x	10% per 3 days	Slow leak (investigate)
30 days	1x	100% per 30 days	Budget exhausted (review)

Fast Burn Alert (1-Hour Window)

# Multi-window burn-rate alerting rules
groups:
  - name: slo-burn-rate
    interval: 30s
    rules:
      # 1-hour window: Page immediately if burning 14.4x sustainable rate
      # At 99.9% SLO, sustainable error rate = 0.001
      # 14.4 * 0.001 = 0.0144 = 1.44% error rate
      - alert: SLOErrorBudgetFastBurn
        expr: |
          (
            sum(rate(http_requests_total{service="api-gateway",status=~"5.."}[1h]))
            /
            sum(rate(http_requests_total{service="api-gateway"}[1h]))
          )
          > (1 - 0.999) * 14.4
        for: 5m
        labels:
          severity: page
          category: slo
          window: 1h
        annotations:
          summary: "Error budget burning fast - FAST PAGE"
          description: |
            Error budget is being consumed {{ $value | humanize }}x faster than sustainable.
            At this rate, your entire 30-day budget will be exhausted in ~7 hours.
            SLO: 99.9% | Current error rate: {{ $value | humanizePercentage }}
            Action: Page on-call immediately and investigate error spike.

Medium Burn Alert (6-Hour Window)

# 6-hour window: Warning if burning 6x sustainable rate
# 6 * 0.001 = 0.006 = 0.6% error rate
- alert: SLOErrorBudgetMediumBurn
  expr: |
    (
      sum(rate(http_requests_total{service="api-gateway",status=~"5.."}[6h]))
      /
      sum(rate(http_requests_total{service="api-gateway"}[6h]))
    )
    > (1 - 0.999) * 6
  for: 30m
  labels:
    severity: warning
    category: slo
    window: 6h
  annotations:
    summary: "Error budget burning - INVESTIGATE"
    description: |
      Error budget is being consumed {{ $value | humanize }}x faster than sustainable.
      At this rate, 10% of your 30-day budget will be burned in ~6 hours.
      SLO: 99.9% | Current error rate: {{ $value | humanizePercentage }}
      Action: Investigate elevated error patterns during business hours.

Slow Burn Alert (3-Day Window)

# 3-day window: Long-term trend detection
# 3 * 0.001 = 0.003 = 0.3% error rate
- alert: SLOErrorBudgetSlowBurn
  expr: |
    (
      sum(rate(http_requests_total{service="api-gateway",status=~"5.."}[3d]))
      /
      sum(rate(http_requests_total{service="api-gateway"}[3d]))
    )
    > (1 - 0.999) * 3
  for: 3h
  labels:
    severity: warning
    category: slo
    window: 3d
  annotations:
    summary: "Error budget slow leak - REVIEW"
    description: |
      Error budget is being consumed {{ $value | humanize }}x faster than sustainable.
      At this rate, 10% of your 30-day budget will be burned in ~3 days.
      SLO: 99.9% | Current error rate: {{ $value | humanizePercentage }}
      Action: Schedule reliability review; may indicate systemic issues.

Combined Multi-Window Alert

# Combined: Fire if ANY window exceeds threshold
# This catches both fast spikes and slow leaks
- alert: SLOErrorBudgetMultiWindowBurn
  expr: |
    (
      sum(rate(http_requests_total{service="api-gateway",status=~"5.."}[1h]))
      /
      sum(rate(http_requests_total{service="api-gateway"}[1h]))
    )
    > (1 - 0.999) * 14.4
    or
    (
      sum(rate(http_requests_total{service="api-gateway",status=~"5.."}[6h]))
      /
      sum(rate(http_requests_total{service="api-gateway"}[6h]))
    )
    > (1 - 0.999) * 6
    or
    (
      sum(rate(http_requests_total{service="api-gateway",status=~"5.."}[3d]))
      /
      sum(rate(http_requests_total{service="api-gateway"}[3d]))
    )
    > (1 - 0.999) * 3
  for: 5m
  labels:
    severity: page
    category: slo
  annotations:
    summary: "Error budget burning across multiple time windows"
    description: |
      Multi-window burn-rate alert triggered.
      One or more time windows show unsustainable error rates.
      1h burn rate: {{ printf "%.1f" (neilyz (index $alerts "0" | value)) }}x threshold
      6h burn rate: {{ printf "%.1f" (neilyz (index $alerts "1" | value)) }}x threshold
      3d burn rate: {{ printf "%.1f" (neilyz (index $alerts "2" | value)) }}x threshold
      Action: On-call should investigate and consider declaring incident.

Burn-Rate Alerting Template (Parameterizable)

# Template for burn-rate alerting with configurable SLO
# Use $slo_threshold as a template variable or replace manually
- alert: SLOBudgetBurnGeneric
  expr: |
    (
      sum(rate(http_requests_total{service="$service",status=~"5.."}[1h]))
      /
      sum(rate(http_requests_total{service="$service"}[1h]))
    )
    > (1 - $slo_threshold) * 14.4
    or
    (
      sum(rate(http_requests_total{service="$service",status=~"5.."}[6h]))
      /
      sum(rate(http_requests_total{service="$service"}[6h]))
    )
    > (1 - $slo_threshold) * 6
    or
    (
      sum(rate(http_requests_total{service="$service",status=~"5.."}[3d]))
      /
      sum(rate(http_requests_total{service="$service"}[3d]))
    )
    > (1 - $slo_threshold) * 3
  for: 5m
  labels:
    severity: page
    category: slo
  annotations:
    summary: "Error budget burning for {{ $labels.service }} (SLO: {{ $labels.slo_target }})"
    description: |
      Multi-window burn-rate alert for {{ $labels.service }}.
      SLO: {{ $labels.slo_target }}% | Windows: 1h / 6h / 3d
      Refer to incident runbook: {{ $labels.runbook_url }}

Error Budget Dashboard

{
  "dashboard": {
    "title": "SLO Error Budget Dashboard",
    "panels": [
      {
        "title": "Error Budget Remaining",
        "type": "gauge",
        "targets": [
          {
            "expr": "(1 - (sum(rate(http_requests_total{service=\"api-gateway\",status=~\"5..\"}[30d])) / sum(rate(http_requests_total{service=\"api-gateway\"}[30d])))) * 100",
            "legendFormat": "Budget Used %"
          }
        ],
        "fieldConfig": {
          "defaults": {
            "min": 0,
            "max": 100,
            "unit": "percent",
            "thresholds": {
              "mode": "absolute",
              "steps": [
                { "value": 0, "color": "red" },
                { "value": 25, "color": "orange" },
                { "value": 50, "color": "yellow" },
                { "value": 75, "color": "green" }
              ]
            }
          }
        }
      },
      {
        "title": "Burn Rate by Window",
        "type": "graph",
        "targets": [
          {
            "expr": "(sum(rate(http_requests_total{service=\"api-gateway\",status=~\"5..\"}[1h])) / sum(rate(http_requests_total{service=\"api-gateway\"}[1h]))) / (1 - 0.999)",
            "legendFormat": "1h Burn Rate"
          },
          {
            "expr": "(sum(rate(http_requests_total{service=\"api-gateway\",status=~\"5..\"}[6h])) / sum(rate(http_requests_total{service=\"api-gateway\"}[6h]))) / (1 - 0.999)",
            "legendFormat": "6h Burn Rate"
          },
          {
            "expr": "(sum(rate(http_requests_total{service=\"api-gateway\",status=~\"5..\"}[3d])) / sum(rate(http_requests_total{service=\"api-gateway\"}[3d]))) / (1 - 0.999)",
            "legendFormat": "3d Burn Rate"
          }
        ],
        "gridPos": { "x": 0, "y": 8, "w": 12, "h": 8 }
      },
      {
        "title": "Projected Budget Exhaustion (Hours)",
        "type": "stat",
        "targets": [
          {
            "expr": "((1 - (sum(rate(http_requests_total{service=\"api-gateway\",status=~\"5..\"}[30d])) / sum(rate(http_requests_total{service=\"api-gateway\"}[30d])))) * 30 * 24) / ((sum(rate(http_requests_total{service=\"api-gateway\",status=~\"5..\"}[1h])) / sum(rate(http_requests_total{service=\"api-gateway\"}[1h]))) / (1 - 0.999))",
            "legendFormat": "Hours remaining"
          }
        ],
        "gridPos": { "x": 12, "y": 8, "w": 6, "h": 4 }
      },
      {
        "title": "Error Rate vs SLO Target",
        "type": "timeseries",
        "targets": [
          {
            "expr": "sum(rate(http_requests_total{service=\"api-gateway\",status=~\"5..\"}[5m])) / sum(rate(http_requests_total{service=\"api-gateway\"}[5m])) * 100",
            "legendFormat": "Current Error Rate %"
          },
          {
            "expr": "(1 - 0.999) * 100",
            "legendFormat": "SLO Target Error Rate %"
          }
        ],
        "gridPos": { "x": 18, "y": 8, "w": 6, "h": 8 }
      }
    ]
  }
}

Observability Hooks for Metrics Monitoring

This section defines what to log, measure, trace, and alert for metrics monitoring systems themselves.

Log (What to Emit)

Event	Fields	Level
Scrape target added	target, job, endpoint	INFO
Scrape target removed	target, job, reason	INFO
Alert state change	alert_name, old_state, new_state, duration	INFO
Recording rule evaluation error	rule_name, error	ERROR
Alert evaluation error	alert_name, error	ERROR
Remote write failure	remote_url, error, retry_count	WARN
TSDB checkpoint created	checkpoint_size, duration	DEBUG

Measure (Metrics to Collect)

Metric	Type	Description
prometheus_tsdb_head_samples	Gauge	Samples in TSDB head
prometheus_tsdb_head_chunks	Gauge	Chunks in TSDB head
prometheus_tsdb_head_duration_seconds	Gauge	Time head has existed
prometheus_target_scrapes_total	Counter	Total scrape attempts
prometheus_target_scrapes_failed_total	Counter	Failed scrape attempts
prometheus_target_scrapes_exceeded_target_limit_total	Counter	Targets exceeding limit
prometheus_remote_write_requests_total	Counter	Remote write requests
prometheus_remote_write_requests_failed_total	Counter	Failed remote write requests
prometheus_alertmanager_alerts_total	Counter	Alerts sent to Alertmanager
prometheus_alerting_rules_evaluated_total	Counter	Rule evaluations
prometheus_notifications_queue_length	Gauge	Pending notifications
prometheus_http_request_duration_seconds	Histogram	HTTP request latency

Trace (Correlation Points)

Operation	Trace Attribute	Purpose
Scrape cycle	scrape.job, scrape.target	Track scrape performance
Remote write	remote_write.endpoint, remote_write.status	Monitor write health
Alert evaluation	alert.name, alert.severity	Correlate alerts

Alert (When to Page for Monitoring System Itself)

Alert	Condition	Severity	Purpose
Prometheus Down	Prometheus instance unreachable	P1 Critical	Monitoring unavailable
TSDB Head Growing	Head chunks > 2 weeks of data	P2 High	Storage issue
Scrape Failure Rate	Scrape failures > 10% for 10 min	P2 High	Missing metrics
Remote Write Failing	Remote write failures > 5%	P1 Critical	Backup missing
Alert Queue Full	Pending alerts > 100	P3 Medium	Notification delay
High Cardinality	Cardinality > configured limit	P3 Medium	Memory pressure

Metrics Monitoring System Observability Template

# monitoring-system-observability.yaml
groups:
  - name: prometheus-self-monitoring
    rules:
      # Prometheus instance down
      - alert: PrometheusDown
        expr: up{job="prometheus"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Prometheus instance {{ $labels.instance }} is down"
          description: "Prometheus monitoring is unavailable. Investigate immediately."

      # TSDB head growing unbounded
      - alert: PrometheusTSDBHeadOld
        expr: prometheus_tsdb_head_min_time{job="prometheus"} < (time() - 3600 * 24 * 14)
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "Prometheus TSDB head is more than 2 weeks old"
          description: "TSDB head has not compacted in 2 weeks. Check storage and compaction settings."

      # High scrape failure rate
      - alert: PrometheusScrapeFailureRate
        expr: |
          sum(rate(prometheus_target_scrapes_failed_total{job="prometheus"}[10m]))
          /
          sum(rate(prometheus_target_scrapes_total{job="prometheus"}[10m])) > 0.1
        for: 10m
        labels:
          severity: high
        annotations:
          summary: "Prometheus scrape failure rate above 10%"
          description: "{{ $value | humanizePercentage }} of scrapes are failing."

      # Remote write failures
      - alert: PrometheusRemoteWriteFailing
        expr: |
          sum(rate(prometheus_remote_write_requests_failed_total[5m]))
          /
          sum(rate(prometheus_remote_write_requests_total[5m])) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Prometheus remote write failure rate above 5%"
          description: "Remote write to long-term storage is failing. Historical metrics at risk."

      # Notification queue backing up
      - alert: PrometheusNotificationQueueFull
        expr: prometheus_notifications_queue_length{job="prometheus"} > 100
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Prometheus alert notification queue is backing up"
          description: "{{ $value }} alerts pending delivery. Alertmanager may be unreachable."

      # High HTTP request latency
      - alert: PrometheusHighQueryLatency
        expr: |
          histogram_quantile(0.95,
            sum(rate(prometheus_http_request_duration_seconds_bucket{job="prometheus"}[5m])) by (le)
          ) > 2
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Prometheus query latency above 2 seconds (p95)"
          description: "Query performance is degraded. Check TSDB load and query complexity."

      # Burn-rate alerting for SLO itself (meta-monitoring)
      - alert: SLOBurnRateAlertMetaMonitoring
        expr: |
          (
            sum(rate(prometheus_target_scrapes_failed_total{job="prometheus"}[1h]))
            /
            sum(rate(prometheus_target_scrapes_total{job="prometheus"}[1h]))
          ) > 0.01
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Prometheus monitoring quality degraded"
          description: "Scrape failure rate is {{ $value | humanizePercentage }}. Monitoring accuracy reduced."

DORA Metrics for Delivery Performance

DORA (DevOps Research and Assessment) research identified four metrics that predict software delivery performance. You can use your existing monitoring infrastructure to track all of them.

The Four DORA Metrics

DORA (DevOps Research and Assessment) is a research program that ran multi-year studies across thousands of practitioners to identify which operational practices actually predict organizational performance. Their finding: these four metrics together are a reliable proxy for how well your team delivers software. The data shows elite performers deploy hundreds of times more frequently than low performers, with change failure rates up to 60x lower. The causation is not perfectly clear — high-performing teams may naturally adopt better practices, or the practices may enable the performance — but the correlation is strong enough that tracking these metrics tells you something real about your delivery capability.

Metric	Definition	Target
Deployment Frequency	How often code deploys to production	Multiple per day
Lead Time for Changes	Time from commit to production	Under 1 week
Change Failure Rate	Percentage of deployments causing failures	Under 15%
Mean Time to Recovery (MTTR)	Time to restore service after failure	Under 1 hour

A note on targets: these are team-level aggregates, not individual deployment targets. A team that deploys once per week to a single staging environment does not have high deployment frequency by DORA’s definition — the metric tracks production deployments. Similarly, lead time measures the full path from code commit to production, which includes review, testing, and deployment pipeline time. If your lead time is four weeks but two of those weeks are waiting in a manual approval queue, the metric is telling you something uncomfortable about your process.

Measuring DORA Metrics

# Deployment frequency (deployments per day)
sum(increase(deployments_total[1d]))

# Lead time for changes (p95 time from commit to deploy)
histogram_quantile(0.95,
  sum(rate(deploy_build_to_deploy_seconds_bucket[1w])) by (le)
)

# Change failure rate (failed deployments / total deployments)
sum(increase(deployments_total{status="failed"}[1w]))
/
sum(increase(deployments_total[1w]))

# MTTR from incident start to resolution
avg(incident_resolution_seconds) by (service)

Connecting DORA to SLOs

Each DORA metric maps to a corresponding SLO signal. Deployment frequency shows up in how fast your error budget burns across releases. Change failure rate appears as the percentage of deploy-related burn-rate spikes. MTTR is the time between an SLO alert firing and error rates returning to normal. When you track these pairs side by side, patterns appear that neither framework shows alone.

Start by overlaying deployment events on SLO burn-rate timelines. Most monitoring tools support annotations for deploys. If a burn-rate spike appears within 30 minutes of a deploy, that change just consumed error budget. Over a quarter, you can see which teams and services most often trigger burn-rate alerts after their deploys. That is a much more precise reliability signal than change failure rate alone.

Your error budget also works as a deployment gate. When less than 20% of the budget remains in a 30-day window, pause non-critical deploys. When the budget recovers, resume. This gives teams a data-driven policy instead of subjective judgment calls about stability. SRE sets the thresholds; teams plan around them.

Build a dashboard that shows SLO burn rate alongside DORA metrics for each service. The two views cover different time horizons. DORA shows what already happened to your delivery pipeline. SLO burn rate shows what is happening to reliability right now. When burn rate climbs but DORA metrics are stable, you caught a reliability problem before it became a delivery bottleneck. That early signal is the practical reason to connect both frameworks.

Monitoring Cost Optimization

Monitoring costs have a way of creeping up on you. What starts as a lean setup — a single Prometheus server, a handful of dashboards, maybe one remote write destination — can quietly balloon into a budget line item that rivals your cloud compute spend. The three usual suspects — cardinality explosions, excessive retention, and over-scraping — each silently multiply storage, memory, and network costs in ways that are easy to miss until the monthly bill arrives 😬.

Cardinality is the silent killer. Every unique label combination creates a new time series, and these multiply faster than most teams expect. A metric like http_requests_total{endpoint, method, status, user_tier} with 10 endpoints × 4 methods × 5 status codes × 50 user tiers already generates 10,000 series. Add a single unbounded label such as session_id or request_id — values that are never reused — and your Prometheus instance quietly turns into a memory furnace. The fix is to audit your label sets before they hit production, not after your TSDB starts swapping to disk. The queries in the Controlling Cardinality section help you detect which metrics are the worst offenders, and recording rules let you pre-aggregate high-cardinality data before it reaches the storage layer.

Retention is a storage trap 🧠. Keeping raw high-resolution data for months feels like the safe choice — until your disk fills up and queries slow to a crawl. Most alerting and operational decisions only need the last 15-30 days of raw data; anything older should be aggressively downsampled. A proven strategy: 30 days raw → 90 days at 1-hour resolution → 1-2 years at daily resolution for capacity planning and business reporting. The recording rules under Retention and Downsampling show exactly how to set this up without losing the ability to spot long-term trends.

Scrape frequency is a cost you choose every day ⏱️. Scraping every target at 15-second intervals because “it might matter someday” is like keeping every tap in your house running because you might need water later. Match the scrape interval to the decision speed you actually need: 15s for P1-critical paths, 60s for standard services, 5m for slow-changing infrastructure like disk usage and queue depths. The Scrape Interval Tuning table below gives a quick cheat-sheet you can hand to any team onboarding onto your monitoring stack.

The three sections that follow — cardinality control, tiered retention, and interval tuning — are the practical levers you pull to keep monitoring costs under control without sacrificing visibility. Pick one to implement per quarter, and watch your Prometheus heap usage flatten out.

Controlling Cardinality

Every unique combination of label values creates a new time series. High-cardinality labels like user IDs or request IDs quickly exhaust Prometheus memory.

# Check cardinality per metric
topk(10,
  count by (__name__) (
    {__name__=~".+"}
  )
)

# Identify high-cardinality label combinations
count by (job, __name__) (
  {__name__="http_requests_total"}
)

Practical limits per metric:

• Keep unique label combinations under 10,000 per metric
• Avoid labels with unbounded values (user_id, session_id, request_id)
• Use recording rules to pre-aggregate high-cardinality queries

Retention and Downsampling

Long retention periods multiply storage costs. Use downsampling to preserve historical patterns at lower resolution.

# prometheus.yml
rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  - job_name: "federate"
    metrics_path: /federate
    params:
      match[]:
        - '{__name__=~"job:.+"}'
    rule_files:
      - /etc/prometheus/rollups.yml

# rollups.yml - pre-computed aggregations
groups:
  - name: hourly_rollups
    interval: 1h
    rules:
      - record: job:http_requests_total:hourly_rate
        expr: |
          sum by (job, status) (
            rate(http_requests_total[1h])
          )

Retention guidelines:

• Raw data: 15-30 days (sufficient for most alerting)
• 1-hour resolution: 90 days (for capacity planning)
• 1-day resolution: 1-2 years (for business reporting)

Scrape Interval Tuning

Frequent scrapes improve resolution but increase resource usage. Match scrape intervals to alert requirements.

Use Case	Scrape Interval	Notes
Real-time alerting	15s	Use for critical P1 metrics only
Standard metrics	60s	Sufficient for most use cases
Slow-changing metrics	5m	Disk usage, job queue depths
Historical analysis	15m+	Roll up to reduce storage

Multi-Region Monitoring Strategies

Running monitoring across regions means dealing with network latency, aggregation headaches, and the question of whether to set SLOs per region or globally.

Federation Architecture

Federation allows hierarchical collection where regional Prometheus servers scrape local targets, then a global Prometheus federation scrapes the regional instances.

# Regional Prometheus (scapes local targets)
scrape_configs:
  - job_name: "local-services"
    static_configs:
      - targets: ["app-us-east-1:9090", "db-us-east-1:9090"]

# Global Prometheus (federates regional)
scrape_configs:
  - job_name: "federate-us-east-1"
    metrics_path: /federate
    params:
      match[]:
        - '{__name__=~"job:.+"}'
    static_configs:
      - targets: ["prometheus-us-east-1:9090"]

Considerations:

• Network latency affects metric freshness; use longer evaluation intervals for federated data
• Not all metrics need global aggregation; filter with match[] rules
• Cross-region bandwidth costs money; only federate what you need

Regional vs Global SLOs

A user in Frankfurt cares about Frankfurt availability, not your global average. Global SLOs can hide regional outages entirely.

Approach	Pros	Cons
Global SLO only	Simpler reporting	Masks regional issues
Regional SLOs	Reflects user experience	More dashboards to manage
Both	Accurate + simple reporting	Requires good tooling

For global services, set regional SLOs per region plus a composite global SLO. Alert on regional SLOs to catch degradation early.

Synthetic Monitoring from Multiple Locations

Real-user monitoring and synthetic checks from multiple geographic points catch regional issues that internal metrics miss.

import requests
import time

# Run synthetic checks from multiple regions
REGIONS = ["us-east-1", "eu-west-1", "ap-southeast-1"]

def synthetic_check_region(region):
    start = time.time()
    try:
        resp = requests.get(
            f"https://api.example.com/health",
            timeout=5,
            headers={"X-Region": region}
        )
        duration = time.time() - start
        # Report per-region metrics
        metrics.gauge(f"health_check_{region}_duration", duration)
        metrics.gauge(f"health_check_{region}_success", 1 if resp.status == 200 else 0)
    except Exception as e:
        metrics.gauge(f"health_check_{region}_success", 0)

Route alerts based on both regional synthetic checks and internal metrics. A regional synthetic failure plus elevated internal latency in the same region strongly indicates a regional outage.

Service Level Agreements

SLAs are contractual commitments to customers, often backed by financial penalties. They are usually less aggressive than SLOs.

SLA vs SLO

Aspect	SLA	SLO
Audience	Customers	Internal
Enforced by	Contracts, penalties	Team discipline
Target	Usually less strict	Usually more strict
Consequences	Financial	Operational

Don’t set SLAs until you have SLOs you are confident you can meet. Adding contractual SLAs before you understand your system’s behavior is asking for trouble.

Building Dashboards

Dashboards translate metrics into actionable information. Good dashboards answer specific questions. Bad dashboards show everything and answer nothing.

Dashboard Design Principles

Dashboards serve different audiences and purposes, and designing without clarity about who will use it leads to tools that neither on-call engineers nor executives find useful. The design process starts by defining the primary user and the specific question they need answered. An on-call engineer needs to diagnose a live incident in under 60 seconds, which means high-level health indicators, clear error signals, and the ability to drill into specific services or traces. An executive needs trend analysis and business impact, which means longer time ranges, comparisons to prior periods, and correlation with business metrics like revenue or user signups. A developer debugging a regression needs detailed breakdown by endpoint, method, and version. Mixing these audiences on a single dashboard creates a compromise that serves none of them well. The solution is separate dashboards per audience, with links between them for cross-context navigation.

Grouping related metrics into logical sections does more than reduce scrolling. When error rate spikes during an incident, you want to immediately see if latency also spiked and if traffic increased at the same time. Placing these three panels adjacent vertically means you take in the correlation in a single gaze rather than scanning across a dashboard. Group panels by the operational question they answer rather than by metric type. A row titled “Deprecation Risks” might contain panels for library version vulnerabilities, sunsetting API endpoints, and expiring TLS certificates. These are unrelated metric types, but they all answer “what needs my attention soon?” Grouping them together respects the engineer’s mental model. Avoid grouping by service when debugging production issues; group by symptom instead.

Context transforms raw numbers into actionable information. A p95 latency of 500ms means nothing without knowing whether your SLO target is 200ms or 1000ms. The most effective dashboards layer three types of context onto every metric. First, target context: horizontal reference lines showing SLO thresholds or business targets. Second, temporal context: a comparison to the same metric 7 or 30 days ago, showing whether conditions are improving or degrading. Third, causal context: vertical annotations marking deployments, configuration changes, or other known events that explain why a metric changed. Without these layers, viewers spend time interpreting data rather than acting on it. Color-coding panels against thresholds (green/yellow/red) gives immediate situational awareness, but thresholds must be calibrated to actual SLO targets, not arbitrary values set during initial dashboard creation.

Every visual element that does not convey information adds cognitive overhead and slows dashboard loading. Gridlines aid estimation but can be replaced with axis labels at key intervals. Legends are necessary when a single panel shows multiple series, but redundant when each panel shows only one series and the panel title already describes it. Y-axis labels at every tick mark clutter the visualization when a threshold line already shows the value of interest. The test for any dashboard element is simple: if removing it would make the dashboard harder to interpret, keep it. If removing it makes interpretation easier or has no effect, remove it. Audit your dashboards quarterly. Panels accumulate as teams add new views for new hypotheses, but old panels rarely get removed even when the questions they answer no longer matter. A dashboard with 40 panels loads slowly and buries the 5 panels that actually matter. Aim for under 20 panels per dashboard, with the most critical panels in the top two rows where attention naturally lands.

Example Dashboard Panels

# Grafana dashboard JSON (abbreviated)
{
  "dashboard":
    {
      "title": "API Gateway Overview",
      "panels":
        [
          {
            "title": "Request Rate",
            "type": "graph",
            "targets":
              [
                {
                  "expr": "sum(rate(http_requests_total{service='api-gateway'}[5m])) by (service)",
                  "legendFormat": "{{service}}",
                },
              ],
            "gridPos": { "x": 0, "y": 0, "w": 12, "h": 8 },
          },
          {
            "title": "Error Rate",
            "type": "stat",
            "targets":
              [
                {
                  "expr": "sum(rate(http_requests_total{service='api-gateway',status=~'5..'}[5m])) / sum(rate(http_requests_total{service='api-gateway'}[5m])) * 100",
                  "legendFormat": "Error %",
                },
              ],
            "fieldConfig":
              {
                "defaults":
                  {
                    "thresholds":
                      {
                        "mode": "absolute",
                        "steps":
                          [
                            { "value": 0, "color": "green" },
                            { "value": 0.1, "color": "yellow" },
                            { "value": 1, "color": "red" },
                          ],
                      },
                  },
              },
            "gridPos": { "x": 12, "y": 0, "w": 6, "h": 4 },
          },
          {
            "title": "P95 Latency",
            "type": "gauge",
            "targets":
              [
                {
                  "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{service='api-gateway'}[5m])) by (le)) * 1000",
                  "legendFormat": "p95 ms",
                },
              ],
            "gridPos": { "x": 18, "y": 0, "w": 6, "h": 4 },
          },
        ],
    },
}

Essential Dashboard Sections

At a Glance: Key health indicators in a single row with green/yellow/red status for critical metrics. This row is what on-call engineers scan first when they open a dashboard during an incident. Include four to six stat panels maximum: availability percentage, error rate, p95 latency, and current RPS. Each should show the current value, the SLO target, and the trend direction. If any panel is red, the row itself should be visually prominent so the problem surfaces before the engineer reads a single number.

Request Metrics: Rate, error rate, and latency percentiles for your main endpoints. Break these down by endpoint group rather than showing an aggregate across all routes. A 0.1% error rate might hide a 10% error rate on your authentication endpoint. Include a time-series graph for each major endpoint group with the SLO threshold line overlaid. Add a top-10 endpoints table showing highest error rates or slowest p95 latencies to surface problems that aggregates would hide.

Infrastructure Metrics: CPU, memory, disk, and network for your hosts. These answer “why is the application slow?” when application metrics show degradation. Include utilization gauges for CPU and memory with thresholds at 70% (warning) and 85% (critical). Disk I/O graphs should show read/write latency and throughput separately. Elevated read latency often points to memory pressure causing swap, not disk problems. Network graphs should distinguish between inbound and outbound traffic and show TCP retransmit rates, which indicate congestion or connectivity issues.

Application Metrics: Business-specific metrics like queue depths, cache hit rates, or background job counts. These vary by service but are often the most actionable signals for product teams. Queue depth for message consumers tells you whether work is backing up faster than workers can process it. Cache hit ratio tells you whether your application is reading from disk instead of memory. Background job queue depth and processing rate tell you whether batch work is keeping up with arrival rate. Define these for your specific application architecture rather than using generic infrastructure metrics.

Dependency Health: Metrics for databases, caches, and external services your application depends on. For each dependency, show connection pool utilization, error rates, and latency percentiles. A database connection pool at 90% capacity is a leading indicator of imminent latency spikes. Requests start queuing for connections before they fail. For external APIs, show both availability (can we reach it?) and latency (does it respond fast enough?). Include the dependency’s own SLO target where available so you can immediately see if a dependency is breaching its commitment.

Alert Design

Alerts should wake someone up only when they need to act. Too many alerts cause fatigue; too few cause outages.

Alert Severity Levels

Severity levels only work if your whole organization agrees on what they mean. A P1 at 2am means someone is getting a phone call and has to act immediately, with management notified within minutes. A P2 means the on-call engineer handles it within 30 minutes but does not necessarily wake anyone up. If engineers escalate a P2 to page-level, or if managers get paged for every P2, you have effectively eliminated your severity system and created noise that erodes trust in alerts.

Severity	Response Time	Examples
P1 Critical	Minutes, 24/7	Complete outage, data loss, security breach
P2 High	30 minutes	Degraded performance affecting many users
P3 Medium	Business hours	Minor degradation, non-critical failures
P4 Low	Next sprint	Predictable issues, capacity planning

The response times in the table are guidelines, not contracts. A P1 in a product with no active users at 3am still requires a phone call, even if the actual impact is currently zero. Conversely, a P2 during peak traffic hours might warrant skipping the 30-minute wait and pulling in extra hands. Use judgment. The severity level should reflect the real-world urgency at the moment the alert fires, not a static label assigned when the alert rule was written.

One common mistake: assigning severity based on infrastructure metrics rather than user impact. “Disk 90% full” might be P1 if it is the last shard of a critical database, or P4 if it is a test environment. Same metric, different severity. Always tie severity to business or user impact, not to the raw metric value.

Alert Quality Checklist

Before creating an alert, ask:

• Does this indicate a real problem affecting users?
• Is the root cause something we can fix?
• Is the alert actionable? Can the recipient do something about it?
• Is the alert specific? Does it point toward the likely cause?
• Is the threshold calibrated? Are we alerting on symptoms or causes?

Alerting on symptoms leads to noise. Alerting on causes requires understanding your system well enough to know what indicates true problems.

Example Alert Rules

# Prometheus alerting rules
groups:
  - name: api-gateway
    rules:
      # High error rate
      - alert: APIGatewayHighErrorRate
        expr: |
          sum(rate(http_requests_total{service="api-gateway",status=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{service="api-gateway"}[5m])) > 0.01
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "API Gateway error rate above 1%"
          description: "Error rate is {{ $value | humanizePercentage }} over the last 5 minutes"

      # Latency degradation
      - alert: APIGatewayHighLatency
        expr: |
          histogram_quantile(0.95,
            sum(rate(http_request_duration_seconds_bucket{service="api-gateway"}[5m])) by (le)
          ) > 1
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "API Gateway P95 latency above 1 second"
          description: "P95 latency is {{ $value | humanizeDuration }}"

      # Slow queries
      - alert: DatabaseSlowQueries
        expr: |
          rate(django_dbqueries_total{type="slow"}[5m]) > 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "More than 10 slow queries per second"
          description: "Database is experiencing query performance issues"

Alert Routing

Route alerts to the right people based on severity and service:

# Alertmanager configuration
route:
  receiver: default
  routes:
    - match:
        severity: critical
      receiver: pagerduty
      continue: true
    - match:
        service: database
      receiver: database-oncall
    - match:
        severity: warning
      receiver: slack-warnings
    - match:
        severity: info
      receiver: none # Don't alert, just log

receivers:
  - name: pagerduty
    pagerduty_configs:
      - service_key: xxx
        severity: critical
  - name: slack-warnings
    slack_configs:
      - channel: "#alerts-warning"
  - name: database-oncall
    pagerduty_configs:
      - service_key: yyy
        routing_key: database-team

Blackbox Monitoring

Blackbox monitoring tests your service from the outside, independent of application metrics. It catches failures that internal metrics miss.

Prometheus Blackbox Exporter

# blackbox.yml
modules:
  http_2xx:
    prober: http
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2"]
      method: GET
      fail_if_ssl: false

# Prometheus scrape config
scrape_configs:
  - job_name: "blackbox-http"
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://api.example.com/health
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

Synthetic Transactions

Run synthetic transactions regularly to catch degradation before users notice:

import requests
import time

def synthetic_checkout():
    start = time.time()

    # 1. Create cart
    cart_resp = requests.post('https://api.example.com/carts', timeout=5)
    cart_id = cart_resp.json()['id']

    # 2. Add item
    requests.post(f'https://api.example.com/carts/{cart_id}/items',
                  json={'product_id': 'PROD123', 'quantity': 1}, timeout=5)

    # 3. Checkout
    checkout_resp = requests.post(f'https://api.example.com/carts/{cart_id}/checkout',
                                   timeout=10)

    duration = time.time() - start

    # Report metrics
    metrics.histogram('synthetic_checkout_duration', duration)
    metrics.gauge('synthetic_checkout_success', 1 if checkout_resp.status == 200 else 0)

Capacity Planning

Monitoring helps predict when you need more capacity.

Trends and Projections

Capacity planning fails when you react to utilization rather than anticipate it. Once CPU reaches 90%, latency degradation has already started and your scaling window closes faster than expected because the remaining headroom disappears quicker than linear extrapolation suggests. Proactive management means reviewing growth trends weekly and acting on projections monthly, not waiting for threshold alerts to fire.

Measuring growth requires a consistent baseline. The most useful comparison is week-over-week percentage change in your primary demand metric. For request-driven services, this is requests per second. For batch systems, it might be jobs queued or items processed per hour. Calculate the ratio using rate() with a 7-day window compared to the same metric from 7 days prior, expressed as a percentage. Plot this over 90 days to distinguish seasonal patterns from sustained growth. E-commerce services typically show 2-3x traffic spikes in the 4 weeks before major holidays, while B2B APIs see traffic drops of 20-30% during summer and winter holiday periods. Without this seasonal awareness, you either over-provision for holidays that will not materialize or under-provision for holidays that will. Track growth rate per endpoint group rather than aggregate across all endpoints. Your public API might be growing 15% week-over-week while your internal admin endpoints are declining. Aggregating masks the difference and leads to mismatched capacity allocation.

The predict_linear() function fits a simple linear regression to your data and projects when the current trend reaches a threshold. For capacity planning, set the threshold at your critical utilization level minus a safety margin. If your target is to never exceed 80% CPU, set the threshold at 75% to give yourself a 5% buffer for unexpected spikes. The query returns seconds since epoch for the projected exhaustion date, which you convert to a calendar date for human readability. Run this projection for CPU, memory, disk usage, and connection pool utilization separately. Each resource has its own growth rate, and the shortest time-to-exhaustion is your binding constraint. An alert on “CPU will exhaust in 60 days” gives you two months to either provision larger instances, optimize consumption, or shift traffic. An alert on “CPU will exhaust in 14 days” is an emergency.

Storage projections follow the same logic but with additional complexity. Disk usage grows from data volume, not just request count, and data volume depends on retention policies, compression ratios, and replica factors. Estimate your daily disk growth by monitoring index_size or disk.used over a 2-week baseline period, then extrapolate forward accounting for any planned data retention changes. A volume that grows 5% per week reaches 80% capacity in 20 weeks, but the actual exhaustion date depends on whether you are using hot-warm-cold architecture where older data moves to cheaper storage. If you are not, then the exhaustion date is exactly what the linear projection says it is. Set calendar reminders to review all capacity projections monthly, and assign an owner to each projection so someone is accountable for acting on the forecast before it becomes an incident.

Scaling Thresholds

# KEDA scaled object for event-driven scaling
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: api-gateway-scaler
  namespace: production
spec:
  scaleTargetRef:
    name: api-gateway
  minReplicaCount: 3
  maxReplicaCount: 100
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus:9090
        metricName: http_requests_per_second
        threshold: "1000"
        query: sum(rate(http_requests_total{service="api-gateway"}[2m]))

When to Use Metrics-Based Monitoring

When to Use Metrics-Based Monitoring:

• Real-time system health monitoring and alerting
• Capacity planning and trend analysis
• SLO/SLA tracking and error budget management
• Correlation with business metrics (revenue, conversions)
• Long-term historical analysis and reporting
• Infrastructure and application layer monitoring

When Not to Use Metrics-Based Monitoring:

• Debugging specific request failures (use logs)
• Understanding request flow across services (use tracing)
• One-off troubleshooting of transient issues
• Monitoring low-volume events that do not aggregate well
• Situations where you need the full context of a single operation

Trade-off Analysis

Aspect	Metrics Monitoring	Log Analysis	Distributed Tracing
Latency Detection	Aggregate patterns	Per-request detail	Request-level timing
Root Cause	Correlation difficult	Full context available	Causal chain is clear
Storage Cost	Low (aggregated)	High (raw events)	Medium (spans)
Query Flexibility	High (PromQL/SQL)	Medium (text search)	Low (structured)
Historical Analysis	Excellent	Limited by cost	Poor
Real-time Alerting	Excellent	Poor	Good
Debug Single Request	Poor	Excellent	Excellent
Capacity Planning	Excellent	Poor	Poor

Production Failure Scenarios

Failure	Impact	Mitigation
Prometheus scrape failure	Gaps in metrics; missing visibility	Configure remote_write to backup; alert on target down; use Prometheus federation
Alert fatigue	Teams ignore alerts; real issues missed	Regularly review alert quality; remove stale alerts; tune thresholds
Metric cardinality explosion	Prometheus OOM; query performance degraded	Limit label cardinality; use recording rules; segment by service
Alert routing failure	Critical alerts not delivered; extended outages	Test alert routing; use multiple notification channels; on-call rotation
Dashboard data source outage	No visibility for teams; blind decision making	Configure backup data sources; cache dashboard definitions
SLO target chronically missed	Customer trust erosion; potential SLA penalties	Investigate root causes; adjust SLO targets if unrealistic; build error budget alerts

Common Pitfalls / Anti-Patterns

Common Pitfalls

1. Alerting on Symptoms Without Context

# Bad: Alerts on symptoms
- alert: HighCPU
  expr: cpu_usage > 80
  # Fires constantly without context

# Good: Alert with actionable context
- alert: HighCPUOnCriticalService
  expr: cpu_usage{service="api-gateway"} > 80 and rate(http_requests_total{service="api-gateway"}[5m]) > 1000
  # Only fires when it matters

2. Missing Alert Aggregation

Firing 1000 alerts for one failure causes chaos. Use grouping, inhibition, and routing to consolidate alerts:

route:
  group_by: ["alertname", "cluster", "service"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

3. SLOs That Do Not Reflect User Experience

Setting SLOs on backend metrics instead of user-visible metrics creates false confidence. SLOs should measure what users experience, not internal implementation details.

4. Dashboard Overload

More panels do not mean better monitoring. Each dashboard should answer specific questions. If you need to scroll, you have too many panels.

5. No Alert Testing

Alerts that never fire in production may have broken queries. Test alerts periodically by simulating conditions that should trigger them.

6. Ignoring Alert Fatigue

If engineers start ignoring alerts, real problems get missed. Review and tune alerts regularly. Remove alerts that no longer serve a purpose.

Observability Checklist

Core Metrics (Golden Signals)

The four golden signals — latency, traffic, errors, and saturation — form the minimum viable observability surface for any service. Google SRE uses these as the baseline for every service they run, and the logic is straightforward: together they answer “is my service healthy?” without requiring deep instrumentation.

Latency deserves special attention because not all slow responses are failures. A 500ms response might be fine for a complex report endpoint but unacceptable for an authentication service. Track latency distributions (p50, p95, p99) rather than averages — averages hide the tail where your most important users live. Separate successful slow responses from failed fast responses, as they point to different problems.

Traffic measures demand in requests per second. It tells you whether your service is seeing normal load, unexpected spikes, or silent drops. Sudden traffic drops often mean upstream failures (your users cannot reach you), while traffic spikes might indicate attack traffic or a viral feature. Correlate traffic changes with error rate and latency changes to distinguish “lots of users having a good experience” from “lots of users having a bad experience.”

Errors come in two flavors: explicit errors (5xx HTTP codes, exceptions) and implicit errors (slow successful responses that technically succeeded but failed to meet user expectations). Track both. Express error rate as a percentage of total requests, not an absolute count — a service handling 10,000 RPS with 10 errors has a 0.1% error rate; a service handling 100 RPS with 10 errors has a 10% error rate and is in much worse shape.

Saturation tells you how full your system is. A service at 80% CPU might handle traffic fine today but fail catastrophically when traffic spikes 20%. Monitor queue depths, memory utilization, and disk I/O saturation as leading indicators. When utilization crosses 80%, you typically have hours before saturation becomes a user-visible problem — enough time to scale or investigate if you catch it early.

• Latency: p50, p95, p99 response times
• Traffic: requests per second, throughput
• Errors: error rate (4xx, 5xx), success rate
• Saturation: CPU, memory, disk, queue depths

Infrastructure Metrics

Infrastructure metrics answer “why is the application slow?” when application-level signals show degradation. The application layer tells you something is wrong; the infrastructure layer tells you why.

Host-level metrics cover the bare metal or VM your services run on. CPU above 80% for sustained periods means your workload exceeds your compute capacity — scale horizontally or upgrade instance types. Memory pressure shows as elevated swap usage or OOM events in kernel logs; once a system starts swapping, every operation slows by an order of magnitude. Disk I/O saturation causes timeouts for databases and message queues. Network throughput saturation creates latency spikes as packets queue at the NIC.

Container-level metrics matter in Kubernetes, Docker Compose, ECS. A container hitting its memory limit gets killed — your application does not gradually slow down, it suddenly vanishes from load balancer rotation. Track restart counts: a container restarting every 10 minutes is not healthy even if it stays responsive between restarts. CPU throttling kicks in when a container exhausts its CPU quota, causing latency spikes that look like application bugs in distributed traces.

Kubernetes metrics add the orchestration layer perspective. Pod status reveals immediate problems (CrashLoopBackOff, ImagePullBackOff) but node conditions reveal underlying issues (MemoryPressure, DiskPressure, PIDPressure). Namespace quotas prevent runaway workloads from consuming all cluster resources, but you need to monitor quota utilization to know when to increase limits. Review resource request vs. actual usage regularly — over-requested resources waste money, under-requested resources cause throttling.

• Host-level: CPU utilization, memory usage, disk I/O, network throughput
• Container-level: resource limits, restart counts, OOM kills
• Kubernetes: pod status, node conditions, namespace quotas

Application Metrics (RED/USE)

Application metrics sit closest to your users. Infrastructure metrics tell you why the system is struggling; application metrics tell you how users experience that struggle. The RED method (Rate, Errors, Duration) covers request-driven services; the USE method (Utilization, Saturation, Errors) covers resource-driven services. Most real-world systems need both.

Rate captures demand patterns that infrastructure metrics cannot see. Tracking requests per second by endpoint reveals which parts of your API get the most traffic — useful for capacity planning and identifying unexpected load spikes. Breaking it down by HTTP method (GET vs POST vs DELETE) tells you whether traffic patterns make sense for your service’s purpose. Tracking by status code range (2xx vs 4xx vs 5xx) gives you real-time visibility into error rates without computing ratios. Rate anomalies often precede error anomalies — a sudden 10x spike in POST requests might indicate a brute force attack before any 5xx errors appear.

Errors at the application layer include exceptions, failed database queries, upstream timeouts, and malformed responses. Not all errors are HTTP 5xx — a 200 response with empty data might be an application error your monitoring tracks separately. Segment error counts by type (timeout vs validation vs auth) to prioritize fixes: a 0.1% rate of timeout errors affecting premium users costs more than a 5% rate of validation errors affecting all users. Error counts by endpoint tell you which part of your API needs the most reliability investment.

Duration histograms let you answer “how long do typical requests take?” without being misled by outliers. Use p50 (median) to understand what most users experience, p95 to understand the tail experience, and p99 to catch severe outliers before they become widespread complaints. Track duration by endpoint and service — if your overall p95 is 500ms but one specific endpoint has p99 of 5 seconds, that endpoint is your reliability risk regardless of what the average looks like.

Utilization for application metrics means connection pools, thread pools, and memory allocation. A connection pool at 80% capacity means 20% of requests wait for a connection to free up — this shows up as latency spikes but not as CPU spikes in infrastructure metrics. Queue lengths for background job processors tell you whether work is backing up faster than workers can process it. Monitor these as rates of change, not just absolute values — a queue depth of 1000 might be normal for a slow consumer but catastrophic for a fast one.

Saturation at the application layer includes backpressure signals: HTTP 429 responses, circuit breaker state (open/half-open/closed), and throttling events. When your service starts sending 429 responses, users are experiencing degradation even if their requests eventually succeed. Circuit breaker state changes indicate upstream dependency problems propagating through your service graph. Track throttling events to catch auto-scaling lag before it becomes a user-visible incident.

• Rate: requests per second by endpoint, method, status
• Errors: error counts and rates by type, service, endpoint
• Duration: histogram of request latencies with percentiles
• Utilization: resource usage, queue lengths, connection pool stats
• Saturation: backpressure indicators, throttling events

SLO/SLA Metrics

SLO/SLA metrics translate raw measurements into business commitments. An availability SLI of 99.9% sounds simple until you ask “99.9% of what?” — and the answer determines whether you meet your targets or face penalties. Precise definitions matter.

Availability SLIs must answer: availability to whom? For what? Over what window? A service available to 99.9% of users making requests between 8am-8pm UTC is not available to users in Sydney at 2am local time even if their requests technically succeed. Define your SLI window to match your users’ time zones and usage patterns, not just your team’s working hours. The 30-day rolling window smooths out day-of-week and hour-of-day variations that would otherwise cause alert noise.

Latency SLIs require understanding what “slow” means for your users. A p95 latency of 500ms might be acceptable for a batch data export but unacceptable for a login request. Define different SLIs for different request types when usage patterns differ significantly. A single aggregate latency SLI across all endpoints hides problems: if your login endpoint has p95 of 2 seconds while your search endpoint has p95 of 200ms, an aggregate p95 of 300ms masks the critical login degradation.

Error rate SLIs must distinguish between errors that matter to users and errors that do not. A 0.1% error rate for a service where 99.9% of requests are OPTIONS preflight checks (which users never notice) is very different from a 0.1% error rate for POST requests that represent user transactions. Segment your error rate SLI by status code range and by endpoint to catch user-visible problems before they breach your target.

Error budget tracking is where SLIs become actionable. An SLI with no burn-rate alert is just a number you check after incidents occur. Build burn-rate alerts that fire before you breach your target, not when you have already breached it. The multi-window burn-rate approach (14.4x for 1-hour, 6x for 6-hour, 3x for 3-day) catches both sudden spikes and slow leaks before they exhaust your budget.

• Availability SLI with 30-day rolling window
• Latency SLI (p95 under 200ms target)
• Error rate SLI (below 0.1% target)
• Error budget tracking and burn rate alerts

Alerting Rules

• P1: Service down or availability SLA breach
• P2: High error rate (>1% for 5 minutes)
• P2: Latency degradation (p95 >500ms)
• P3: Resource utilization >80%
• P4: Error budget 50% consumed in 1 hour

Security Checklist

• Metrics endpoint (/metrics) not exposed publicly
• Prometheus access authenticated (if external)
• Alert manager notifications do not contain sensitive data
• Dashboard links to internal systems use authenticated proxies
• No API keys or secrets visible in dashboard URLs
• Alert routing logs audited
• Metrics cardinality limits enforced to prevent DoS
• Remote write connections use TLS
• Dashboard snapshots sanitized before sharing

Real-world Failure Scenarios

Scenario 1: Alert Storm from Misconfigured Threshold

What happened: A dashboard team deployed a new dashboard with an incorrectly configured alert rule. The alert threshold was set to trigger on any request above 0ms latency instead of the intended threshold of above 1000ms. Within minutes, the alerting system was flooded with thousands of alerts.

Root cause: A copy-paste error introduced an extra zero when configuring the threshold. No unit tests or dry-run validation existed for alert expressions.

Impact: The on-call engineer received over 3,000 notifications in 5 minutes. The alerting system’s queue became saturated, delaying genuine alerts by over 10 minutes.

Lesson learned: Always dry-run alert queries against production data before activating. Set a maximum alert rate limit per team. Implement alert expression validation in CI/CD.

Scenario 2: Metric Gaps from Prometheus Federation Outage

What happened: A network partition between two data centers caused Prometheus federation to fail. Global dashboards showed no data for 45 minutes while regional dashboards continued to function normally.

Root cause: The federation setup had no redundancy. A single network path connected the global Prometheus to regional instances.

Impact: Executive dashboards were blank during a live incident review. The operations team had to manually aggregate regional dashboard data to report status.

Lesson learned: Implement dual redundant federation paths. Alert on federation scrape health. Use remote_write to a central storage backend (Thanos or Cortex) for global query reliability.

Interview Questions

These questions assess practical knowledge of metrics, monitoring, and alerting systems.

Further Reading

• Distributed Tracing - Request flow across microservices
• Prometheus and Grafana - Hands-on implementation patterns
• Logging Best Practices - Structured logging for observability
• SLO Implementation Guide - Error budgets and burn-rate alerting
• Incident Response Runbooks - On-call procedures and escalation

Conclusion

Key Takeaways:

• Start with the four golden signals: latency, traffic, errors, saturation
• Define SLOs that reflect user experience, not internal metrics
• Use error budgets to prioritize reliability work
• Alert on symptoms with context, not raw numbers
• Every alert should be actionable: someone should know what to do
• Review and tune alerts regularly to avoid fatigue

Copy/Paste Checklist:

# Availability SLO query
1 - (
  sum(rate(http_requests_total{status=~"5.."}[30d]))
  /
  sum(rate(http_requests_total)[30d])
)

# Error budget burning alert
(
  sum(rate(http_requests_total{service="api-gateway",status=~"5.."}[1h]))
  /
  sum(rate(http_requests_total{service="api-gateway"}[1h]))
) > (1 - 0.999) * 0.1

# P95 latency with context
histogram_quantile(0.95,
  sum(rate(http_request_duration_seconds_bucket{service="api-gateway"}[5m])) by (le)
) > 1

# Dashboard variable template
sum(rate(http_requests_total{service=~"$service"}[$interval])) by (service)

# Alert routing with grouping
route:
  receiver: default
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  continue: true
  routes:
    - match:
        severity: critical
      receiver: pagerduty

Effective monitoring combines the right metrics, clear SLOs, thoughtful alerts, and actionable dashboards. Start with the four golden signals: latency, traffic, errors, and saturation. Define SLOs that reflect what users actually care about. Build alerts that fire only when someone needs to act.

For deeper observability into distributed systems, our Distributed Tracing guide covers request flow across services. The Prometheus & Grafana guide provides hands-on examples for implementing these patterns.