ELK Stack Deep Dive: Elasticsearch, Logstash, Kibana, and Beats

The ELK Stack is a popular open-source solution for centralized logging. It lets you collect logs from multiple sources, transform them into structured format, store them efficiently, and query them interactively.

This guide covers each component in depth. If you are new to logging concepts, start with our Logging Best Practices guide first.

Introduction

graph LR
    A[Log Sources] -->|Shippers| B[Beats]
    B --> C[Logstash]
    C --> D[Elasticsearch]
    D --> E[Kibana]
    A -->|Direct| D

The ELK Stack has four main components:

• Beats: Lightweight shippers that collect data from various sources
• Logstash: Transforms and enriches data during transit
• Elasticsearch: Stores and indexes data for fast search
• Kibana: Visualizes and explores data

Core Concepts

Elasticsearch is a distributed search and analytics engine built on Apache Lucene. It stores documents in JSON format and provides powerful query capabilities.

Elasticsearch Key Concepts

Understanding how Elasticsearch stores and retrieves data explains most of the operational decisions you will make. The four foundational concepts are index, document, shard, and replica, and each one directly impacts how you design your logging pipeline and query patterns.

An index is the top-level organizational unit in Elasticsearch, analogous to a database in relational systems. It holds all documents of a similar type and defines the settings that govern how data is stored and searched. In the ELK stack, you typically create one index per day for time-series log data, with naming patterns like logs-2026.03.22 that make it easy to identify the time range of the data inside. Indices have settings for the number of primary shards, replica count, refresh interval, and ILM policy. The refresh interval controls how soon new data becomes searchable; the default 1 second means near-real-time search at the cost of indexing overhead. For logging where you query by time range anyway, increasing the refresh interval to 5-10 seconds reduces overhead without practical downside.

A document is a single JSON record stored within an index, analogous to a row in a relational database. Each document has a unique ID and contains the fields that make up your log entry: timestamp, level, message, service name, trace ID, and any custom fields you add. Documents are immutable once written; to update a document, Elasticsearch writes a new version and marks the old one for deletion during the next segment merge. This write-once architecture is why Elasticsearch excels at time-series data: you write sequentially to the latest index and never modify historical entries. Document structure matters for query performance. Fields used in filters and aggregations should be keywords, while fields used in full-text search should be text with appropriate analyzers. A log entry with {"message": "connection timeout", "service": "api-gateway", "level": "ERROR"} has the message as text for searching, service as a keyword for exact filtering, and level as a keyword for severity-based filtering.

A shard is a subset of an index, stored as an independent Apache Lucene index. Elasticsearch distributes shards across nodes in the cluster, and each shard can be located on any node. The number of primary shards is set at index creation and cannot be changed without reindexing. Sharding enables horizontal scaling: as data volume grows, you add nodes and Elasticsearch automatically distributes shards across them. The shard count also controls parallelism for queries. A query against an index with 5 primary shards can be executed by 5 parallel threads scanning 5 separate data sets, making queries faster on multi-core nodes. However, too many small shards create overhead: each shard has its own memory footprint and file handles, and query coordination across many small shards is slower than across fewer larger shards. The sweet spot is 20-50GB per shard for typical log data. An index with 100GB of daily logs should use 2-5 primary shards rather than letting the default create 5 small shards.

A replica is a copy of a primary shard, kept on a different node for fault tolerance and read scalability. The replica count is configurable per index and can be changed at any time without reindexing. With one replica, each primary shard has one copy; if the node holding the primary fails, Elasticsearch promotes the replica to primary automatically and allocates a new replica to another node. Replicas also serve read queries, distributing query load across copies of the data. For read-heavy workloads, increasing replica count from 1 to 2 effectively doubles read throughput for that index. The trade-off is storage: two replicas means 3 copies of every document (primary plus two replicas). In production, use at least one replica for reliability; for hot-warm architectures where hot nodes have SSDs and warm nodes have spinning disk, place primaries on hot nodes and replicas on warm nodes to balance performance with cost.

Index Lifecycle Management

Define policies to manage index data from creation to deletion:

PUT _ilm/policy/logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "7d",
            "max_primary_shard_size": "50gb"
          },
          "set_priority": 100
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": {
            "number_of_shards": 1
          },
          "forcemerge": {
            "max_num_segments": 1
          },
          "set_priority": 50
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "freeze": {},
          "set_priority": 0
        }
      },
      "delete": {
        "min_age": "365d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Mapping and Index Templates

Index templates define mappings and settings for new indices:

PUT _index_template/logs-template
{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 1,
      "index.lifecycle.name": "logs-policy"
    },
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "level": {
          "type": "keyword"
        },
        "message": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "service": {
          "type": "keyword"
        },
        "trace_id": {
          "type": "keyword"
        },
        "user_id": {
          "type": "keyword"
        },
        "duration_ms": {
          "type": "long"
        },
        "host": {
          "properties": {
            "name": { "type": "keyword" },
            "ip": { "type": "ip" }
          }
        }
      }
    }
  }
}

Querying Elasticsearch

GET logs-2026.03.22/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "service": "api-gateway" } },
        { "range": { "@timestamp": { "gte": "now-1h" } } }
      ],
      "filter": [
        { "term": { "level": "ERROR" } }
      ]
    }
  },
  "sort": [
    { "@timestamp": "desc" }
  ],
  "aggs": {
    "error_by_service": {
      "terms": { "field": "service" },
      "aggs": {
        "error_rate": {
          "avg": { "field": "error_count" }
        }
      }
    }
  }
}

Logstash

Logstash processes and transforms data before it reaches Elasticsearch. It handles complex parsing, enrichment, and filtering.

Logstash Pipeline

graph TB
    A[Input] --> B[Filter]
    B --> C[Output]

A pipeline has three sections: input, filter, and output.

A pipeline is the core processing unit in Logstash. Each pipeline configuration you write defines how data moves from ingestion to storage. You can run multiple pipelines in a single Logstash instance by defining separate configuration files in your pipelines.yml file, which is useful when you handle different log types that need separate processing rules.

Input, filter, and output stages run sequentially for each event, but Logstash processes events in parallel. The filter stage is optional, though in practice most setups use at least one filter to parse timestamps or extract fields. Output is where you send the processed result, usually to Elasticsearch but sometimes to multiple destinations.

A key detail worth remembering: if your filter raises an exception for a given event, that event still makes it to the output. The filter block silently skips processing for that event rather than killing the whole pipeline. This matters when you deal with messy log data where every field might not be present.

Input Plugins

Input plugins are the starting point for every Logstash pipeline. They define how data gets into the system. Each input runs in its own thread, so you can have multiple inputs running in parallel without blocking each other. The most common input in production setups is the beats plugin, which listens for data from Filebeat and other Beats shippers over a configurable port.

The beats input supports SSL out of the box, and you should always enable it in production. Logstash can verify the client certificate from each Beat, which prevents unauthorized shippers from pushing data into your pipeline. The http plugin is a solid alternative when you need to accept data from applications that cannot run a Beat, like a custom script or a third-party service that posts JSON directly.

You can also chain multiple inputs in a single pipeline configuration. Logstash merges events from all active inputs into a shared processing queue, so downstream filters see a unified stream regardless of which input the event came from. Just keep an eye on the port numbers to avoid conflicts.

# Receive logs from Beats
input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/ssl/certs/logstash.crt"
    ssl_key => "/etc/ssl/private/logstash.key"
  }

  # Alternative: direct HTTP
  http {
    port => 8080
    content_type => "application/json"
  }
}

Filter Plugins

Filters transform and enrich data:

filter {
  # Parse JSON logs
  json {
    source => "message"
    target => "parsed"
  }

  # Parse timestamp
  date {
    match => ["parsed.timestamp", "ISO8601"]
    target => "@timestamp"
  }

  # Extract fields from message
  grok {
    match => {
      "parsed.message" => "%{DATA:level}\s*%{DATA:logger}\s*%{GREEDYDATA:log_message}"
    }
    overwrite => ["message"]
  }

  # Add computed fields
  mutate {
    add_field => {
      "environment" => "%{[parsed][env]}"
      "[@metadata][index_prefix]" => "logs-%{[parsed][service]}"
    }
  }

  # Enrich with GeoIP
  geoip {
    source => "[parsed][client_ip]"
    target => "[parsed][geoip]"
    database => "/etc/logstash/GeoLite2-City.mmdb"
  }

  # Parse query string
  kv {
    source => "[parsed][request_params]"
    field_split => "&"
    prefix => "param_"
  }
}

Output Plugins

Output plugins define where your processed log data ends up. The elasticsearch output is the most common destination, but you can send data to multiple outputs at the same time. Logstash sends each event to every output you define, so a single pipeline can write to Elasticsearch for storage while also printing to stdout for debugging.

The elasticsearch output gives you control over the index name, authentication, and SSL settings. The example below uses manage_template => false because the index template is managed separately, which is cleaner in production. Index names use metadata fields so each service logs land in their own daily index. Credentials come from environment variables rather than hardcoded values, which keeps secrets out of your config files.

You can add the stdout output with the rubydebug codec for local testing. It prints each event to the terminal with a readable format that shows all fields and their values. This is useful for debugging pipeline issues since you can see exactly how your filters transform each event before it reaches Elasticsearch.

output {
  elasticsearch {
    hosts => ["https://elasticsearch:9200"]
    manage_template => false
    index => "%{[@metadata][index_prefix]}-%{+YYYY.MM.dd}"
    ssl => true
    cacert => "/etc/ssl/certs/ca.crt"
    user => "${ELASTICSEARCH_USER}"
    password => "${ELASTICSEARCH_PASSWORD}"
  }

  # Also send to stdout for debugging
  stdout {
    codec => rubydebug
  }
}

Complete Pipeline Example

input {
  beats {
    port => 5044
  }
}

filter {
  if [fields][log_type] == "application" {
    json {
      source => "message"
      target => "parsed"
    }

    date {
      match => ["parsed.timestamp", "ISO8601"]
      target => "@timestamp"
    }

    if [parsed][level] {
      mutate {
        add_field => { "level" => "%{parsed[level]}" }
      }
    }

    if [parsed][exception] {
      mutate {
        add_tag => ["error"]
      }
    }
  }

  if [fields][log_type] == "access" {
    grok {
      match => {
        "message" => '%{IPORHOST:client_ip} %{DATA:ident} %{DATA:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:status:int} %{NUMBER:bytes:int} "%{DATA:referrer}" "%{DATA:user_agent}"'
      }
    }

    date {
      match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
      target => "@timestamp"
    }

    geoip {
      source => "client_ip"
      target => "geoip"
    }

    useragent {
      source => "user_agent"
      target => "ua"
    }
  }
}

output {
  if "error" in [tags] {
    elasticsearch {
      hosts => ["https://elasticsearch:9200"]
      index => "logs-error-%{+YYYY.MM.dd}"
    }
  } else {
    elasticsearch {
      hosts => ["https://elasticsearch:9200"]
      index => "logs-%{[fields][log_type]}-%{+YYYY.MM.dd}"
    }
  }
}

Beats

Beats are lightweight data shippers that send data from servers to Logstash or Elasticsearch.

Filebeat

Filebeat tails log files and ships them:

# filebeat.yml
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/containers/*.log
    json:
      keys_under_root: true
      add_error_key: true
      message_key: log
    fields:
      log_type: container
    processors:
      - add_kubernetes_metadata:
          host: ${NODE_NAME}
          matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

  - type: log
    enabled: true
    paths:
      - /var/log/nginx/*.log
    fields:
      log_type: nginx
    processors:
      - add_locale: ~

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~

output.logstash:
  hosts: ["logstash:5044"]
  ssl.enabled: true
  ssl.certificate_authorities: ["/etc/filebeat/ca.crt"]
  ssl.certificate: "/etc/filebeat/filebeat.crt"
  ssl.key: "/etc/filebeat/filebeat.key"

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644

Metricbeat

Metricbeat collects system and service metrics:

metricbeat.modules:
  - module: system
    metricsets:
      - cpu
      - memory
      - network
      - process
      - diskio
    period: 10s
    processes: [".*"]

  - module: docker
    metricsets:
      - container
      - cpu
      - diskio
      - healthcheck
      - info
      - memory
      - network
    hosts: ["unix:///var/run/docker.sock"]
    period: 10s

  - module: nginx
    metricsets:
      - stubstatus
    hosts: ["http://nginx:8080/nginx_status"]
    period: 10s

output.elasticsearch:
  hosts: ["https://elasticsearch:9200"]
  ssl.enabled: true
  ssl.certificate_authorities: ["/etc/metricbeat/ca.crt"]

Heartbeat

Heartbeat monitors service availability with synthetic checks:

heartbeat.monitors:
  - type: http
    name: api-health-check
    schedule: "@every 30s"
    urls:
      - https://api.example.com/health
    check.response:
      status: 200
    fields:
      service: api-gateway

  - type: tcp
    name: redis-connectivity
    schedule: "@every 60s"
    hosts: ["redis:6379"]
    timeout: 5s

  - type: icmp
    name: host-ping
    schedule: "@every 5m"
    hosts: ["elasticsearch"]

output.elasticsearch:
  hosts: ["https://elasticsearch:9200"]

Kibana

Kibana provides the visualization and exploration interface for your Elasticsearch data.

Index Pattern Setup

An index pattern tells Kibana which Elasticsearch indices to query. Without one, Kibana has no way to find your data. The pattern supports wildcards, so logs-* matches logs-2026.03.22, logs-api-gateway, and any index starting with logs. Kibana reads the field mappings from matching indices to populate the field list you see in Discover, Visualize, and Dashboard.

Before exploring data, create an index pattern in Kibana:

1. Navigate to Management > Stack Management > Index Patterns
2. Click “Create index pattern”
3. Enter logs-* as the pattern
4. Select @timestamp as the time field

You can create multiple index patterns for different data sources. A common setup uses logs-* for application logs, metricbeat-* for system metrics, and heartbeat-* for uptime data. Each pattern can have its own time field, though most logs use @timestamp. If your logs use a different timestamp field, change the time field selection in step 4 or your queries use the ingest time instead of the actual log time, which causes confusing gaps in time-based visualizations.

Kibana validates the pattern against your Elasticsearch indices before saving. If no indices match, it shows a warning and lets you adjust the pattern. After creation, you can set the pattern as the default so it loads automatically when you open Discover.

Building Visualizations

Error Rate Over Time

{
  "title": "Error Rate",
  "type": "line",
  "params": {
    "type": "line",
    "grid": { "categoryLines": false },
    "categoryAxes": [
      {
        "id": "CategoryAxis-1",
        "type": "category",
        "position": "bottom"
      }
    ],
    "valueAxes": [
      {
        "id": "ValueAxis-1",
        "name": "LeftAxis-1",
        "type": "value",
        "position": "left",
        "scale": {
          "type": "linear",
          "mode": "normal"
        }
      }
    ]
  },
  "aggs": [
    {
      "id": "1",
      "type": "avg",
      "schema": "metric",
      "params": {
        "field": "error_rate"
      }
    },
    {
      "id": "2",
      "type": "date_histogram",
      "schema": "segment",
      "params": {
        "field": "@timestamp",
        "interval": "auto"
      }
    }
  ]
}

Service Error Distribution

{
  "title": "Errors by Service",
  "type": "pie",
  "aggs": [
    {
      "id": "1",
      "type": "count",
      "schema": "metric"
    },
    {
      "id": "2",
      "type": "terms",
      "schema": "segment",
      "params": {
        "field": "service.keyword",
        "size": 10
      }
    }
  ]
}

Kibana Discover

Discover provides ad-hoc search and exploration:

// Sample Discover query
{
  "query": {
    "bool": {
      "must": [
        { "match": { "level": "ERROR" } },
        { "range": { "@timestamp": { "gte": "now-24h" } } }
      ]
    }
  },
  "sort": [{ "@timestamp": "desc" }],
  "fields": ["@timestamp", "level", "message", "service", "trace_id"],
  "filter": [
    {
      "meta": {
        "index": "logs-*",
        "negate": false,
        "params": {},
        "type": "phrase"
      },
      "query": {
        "match_phrase": {
          "service": "api-gateway"
        }
      }
    }
  ]
}

Kibana Dashboard Example

A practical operations dashboard starts with a top row of summary metrics, then breaks down into focused panels below. Place your most important metric, error rate, in the top left where it catches attention immediately. Put a line chart showing log volume over time beside it to give immediate context on whether the error rate spike coincides with increased traffic.

The second row handles service-level breakdown. A pie chart grouped by service.keyword shows which services are generating the most errors, sorted by count. Below that, a data table lists the most recent errors with columns for @timestamp, level, service, and message. Configure the table to paginate at 50 rows and allow clicking a row to jump into Discover filtered to that specific trace_id.

The third row addresses infrastructure visibility. A heat map with @timestamp on the X-axis and host.name on the Y-axis, using a bucket size of 15 minutes, shows error density by machine. Any host that lights up disproportionately stands out immediately. Add a latency percentile metric (P50, P95, P99) as a gauge visualization to track whether errors correlate with performance degradation.

A complete operations dashboard typically includes these panels:

Panel	Type	Purpose
Error rate	Metric with trend	Current error count and 24h delta
Log volume by level	Line chart	Spot traffic spikes and error surges
Errors by service	Pie / donut chart	Identify which service is misbehaving
Recent errors	Data table	Investigate individual error context
Error heat map	Heat map	Detect host-specific issues
Latency percentiles	Gauge	Correlate errors with performance
Status codes	Bar chart	HTTP 5xx vs 4xx distribution

Deployment Considerations

Hardware Requirements

Component	CPU	RAM	Disk
Elasticsearch (per node)	4+ cores	8GB+	SSD, 500GB+
Logstash	2+ cores	4GB+	Minimal
Kibana	2 cores	2GB+	Minimal
Beats	1 core	512MB+	Minimal

Elasticsearch is I/O intensive. Use SSDs and ensure adequate disk throughput.

The disk choice matters more than almost any other spec for Elasticsearch. Shard healing, replication, and segment merges all hammer disk I/O, and spinning disks bottleneck long before CPU or RAM becomes the constraint. A NVMe SSD with at least 500GB handles most development and small production clusters. For high-volume production with 500GB+ ingested daily, look for disks that sustain 150MB/s+ sequential writes. Indices accumulate quickly, and running disk above 80% capacity triggers watermark throttling that degrades cluster performance.

RAM and heap sizing go hand in hand: set heap to 50% of available RAM, and keep it under 32GB to benefit from compressed object pointers. A node with 64GB RAM gets a 31GB heap and leaves the other 33GB for the OS page cache, which Elasticsearch uses aggressively for frequently accessed segments. On a 128GB node some admins allocate 64GB heap, but 31GB heap with 97GB page cache often performs better in practice because the larger heap triggers slower garbage collection pauses.

CPU matters less for most logging workloads than disk and RAM. Elasticsearch is not heavily CPU-bound unless you run complex aggregations or use script fields. Four cores handles a typical indexing and search mix comfortably. Logstash is the opposite: grok parsing and filter execution are CPU-intensive, so give Logstash servers 4-8 cores if your pipelines do heavy enrichment.

Logstash and Kibana both need far less hardware than Elasticsearch. Logstash servers need enough CPU for parsing and enough heap for the pipeline queue; tune pipeline.workers and pipeline.batch.size to saturate CPU without OOMing. Kibana is stateless and runs fine on a 2-core 2GB instance for teams under 20 users; scale CPU if dashboard rendering feels sluggish.

Filebeat typically uses 40-80MB of RAM in practice, well under the 512MB spec floor. Run Beats on the same servers as your applications to tail logs locally before shipping over the network.

Security

Elasticsearch ships with security features disabled by default. Turning them on is one of the first things you should do for any cluster that is not a local dev environment. Security in Elasticsearch operates at a few levels: encryption for data in transit, authentication to verify identity, and authorization to control what each user can do.

The settings below enable TLS on both transport layer (node-to-node communication) and HTTP layer (client-to-cluster communication). Transport encryption matters because shard replication and cluster coordination happen over this internal channel. If an attacker gains access to your network, unencrypted transport traffic leaks every document you store. API key authentication is a good fit for automated systems and scripts, while the built-in user database or LDAP integration works better for human users.

Role-based access control lets you restrict access at the index level. For example, the development team can have read-only access to logs-* indices while the security team gets full access. You define these roles in a roles.yml file on each node. Without RBAC, anyone with network access to your Elasticsearch HTTP port can read or delete any index, which is a common oversight in early-stage deployments.

# Enable security in elasticsearch.yml
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.enabled: true

# API key authentication
xpack.security.api.key.enabled: true

# Role-based access control
xpack.security.authorization:
  roles_path: /etc/elasticsearch/roles.yml

Scaling

Elasticsearch scales by adding nodes. The cluster handles rebalancing automatically, but you still need to decide how many nodes to add, what their roles should be, and when to scale. Shard rebalancing is where most scaling decisions play out.

Shards are the basic unit of data distribution. When you add a new node, the master node reassigns shards to achieve even disk usage across the cluster. This runs in the background without blocking queries, but it consumes network and disk I/O. If your cluster is already under load when a new node joins, rebalancing causes latency spikes. The cluster.routing.allocation.cluster_concurrent_rebalance setting controls how many shards move at once; the default is higher than most people want in a production environment with heavy indexing. Setting it to 2 or 3 keeps rebalancing from degrading search performance during peak hours.

Replica shards matter for scaling just as much as primary shards. A common early mistake is running with number_of_replicas: 0 to save storage, then wondering why search performance does not improve when nodes are added. Replicas serve read queries, so adding nodes without replicas just gives you more storage but not more search throughput. For read-heavy workloads, aim for at least one replica per shard. The hot-warm-cold architecture extends this further: hot nodes with SSDs handle all writes and recent queries, warm nodes with larger spinning disks serve read-only historical data, and cold nodes on cheap bulk storage handle rarely-queried archival data.

Master nodes are separate from data nodes and have different resource requirements. An overloaded master node causes cluster-wide instability even if data nodes have plenty of capacity. For clusters with more than a few nodes, dedicate master-eligible nodes with plenty of RAM and CPU but minimal data storage. The rule of thumb is to run three dedicated master nodes for any cluster with five or more data nodes. Set discovery.zen.minimum_master_nodes to (n/2) + 1 where n is the number of master-eligible nodes, which prevents split-brain scenarios when network partitions occur.

# Minimum master nodes for cluster stability
discovery.zen.minimum_master_nodes: 2  # for 3-node cluster

# Adjust shard allocation
PUT _cluster/settings
{
  "transient": {
    "cluster.routing.allocation.enable": "all",
    "cluster.routing.allocation.cluster_concurrent_rebalance": 2
  }
}

When to Use the ELK Stack

Use the ELK Stack when:

• You need centralized logging from multiple services and environments
• You need full-text search across log entries and application data
• You need log analysis and pattern detection with Kibana
• You need security analytics and threat detection
• You need compliance audit logging and archival
• You need infrastructure log aggregation (syslog, nginx, apache)

Don’t use the ELK Stack when:

• You have simple applications with minimal logging needs
• You only need metrics and dashboards (use Prometheus + Grafana instead)
• You have high-volume streaming use cases (Kafka is better suited)
• You need real-time alerting on log data (use dedicated alerting tools)
• You need large-scale time-series metrics (Elasticsearch is not optimized for pure metrics)

ELK Stack vs Alternatives

The table below compares ELK with the main alternatives. Numbers are directionally accurate; your mileage depends on data shape and query patterns.

Aspect	ELK Stack	Loki	Splunk
Cost	Open source (self-hosted)	Open source (self-hosted)	Commercial (expensive)
Storage efficiency	Medium (indexed)	High (log-structured)	Medium
Query language	KQL (Kibana)	LogQL (Prometheus-style)	SPL
Scalability	Excellent (horizontal)	Excellent	Excellent
Ease of setup	Moderate	Easy	Easy
Full-text search	Excellent	Limited	Excellent
Metrics integration	Via Metricbeat	Native Prometheus	Native
Best for	Complex log analysis, security analytics	High-volume Kubernetes logs	Enterprise compliance, security

Pick ELK when full-text search across logs matters. If your log messages contain errors, stack traces, or search terms that you need to find without knowing exact field structure, Elasticsearch inverted index gives you that capability. Security analytics teams also favor ELK because the indexed structure makes threat hunting queries fast. The operational overhead is real: you manage the cluster, tune shard counts, and monitor heap pressure.

Pick Loki when you run Kubernetes and cost is the primary constraint. Loki does not index log content; it only indexes labels from Prometheus-style metadata. That makes it cheap to run and easy to scale, but you lose the ability to search arbitrary text within log messages. If your logging is structured (JSON) and most queries filter by known labels like service name, severity, and environment, Loki handles it. Grafana ships with native Loki support, so the visualization stack is simpler than ELK.

Pick Splunk when your organization already has Splunk expertise or you need enterprise compliance features like fixed-volume indexing with cryptographic timestamping. Splunk’s SPL is expressive and the product is mature. The licensing cost is the main deterrent; enterprise contracts easily run into six figures annually at meaningful scale.

Pick CloudWatch Logs when you are all-in on AWS and do not want to manage infrastructure. It integrates naturally with other AWS services and Serverless, and pricing scales with ingested volume. The query language is limited compared to KQL or SPL, and cross-account log aggregation gets messy at scale. If you need to correlate AWS service logs (RDS, Lambda, CloudFront) with application logs, CloudWatch is the path of least resistance.

Capacity Planning

Choosing the right hardware for Elasticsearch prevents performance issues down the line. These are rough guidelines for typical workloads.

Elasticsearch Node Sizing

Tier	RAM	CPU	Disk (SSD)	Use Case
Hot	64GB+	8+ cores	1TB+	Active indexing, recent data
Warm	32GB	4+ cores	2TB+	Read-only, older indices
Cold	16GB	2 cores	4TB+	Archival, rare queries

The heap size should be at most 50% of available RAM. Keep heap under 32GB if possible to benefit from compressed object pointers. Set -Xms and -Xmx to the same value to avoid heap resizing during runtime.

# Check JVM settings
GET _nodes/jvm?filter_path=nodes.*.jvm.memory

Estimating Storage Requirements

Calculate expected index size using this formula:

index_size = source_log_volume × compression_ratio × replica_factor

For JSON logs with ILM enabled, expect 3-5x compression from raw log size. Without ILM, indices can grow 10-20x beyond raw log volume due to normalization and extra fields.

The three variables in the formula work like this. source_log_volume is your raw log output per day, measured before any processing. If your application writes 5GB of JSON logs per day across all services, that is your source volume. compression_ratio accounts for how Elasticsearch stores data internally. Raw JSON with repeated field names compresses well, typically 3-5x on SSDs. Text-heavy logs with high cardinality fields like URLs or user IDs compress less, sometimes only 2x. replica_factor is straightforward: 1.0 means no replicas, 2.0 means one replica copy of every shard.

For example, if you generate 10GB of raw logs per day and run with one replica:

10GB × 4 (compression ratio) × 2 (replica_factor) = 80GB/day
80GB × 30 days = 2.4TB per month

That 2.4TB covers hot tier storage. With ILM moving data through warm and cold tiers, your monthly storage cost depends on where indices sit at any given time. If recent data stays on expensive SSDs and older data migrates to bulk storage, your effective cost per GB drops significantly. A node with 4TB of SSD and 16TB of spinning disk can handle 90 days of hot data at 10GB/day ingestion plus a full year of cold data under the same roof.

Add 20% overhead for segment merges, translog, and headroom. Storage running above 80% causes Elasticsearch to slow down shard operations to protect the cluster, so provision accordingly. Set up disk watermarks at 85% warning and 90% flood stage, and monitor the disk.available metric so you catch saturation before it affects ingestion.

Log Volume Estimation

Getting a rough sense of your log volume early on helps you size your cluster correctly from day one. The numbers below assume a moderately busy service handling about 1,000 requests per minute with a 10KB average log entry per request. Real-world volumes vary a lot depending on how verbose your logging is and whether you log request bodies.

The compression math matters because Elasticsearch stores more than your raw log text. It adds inverted indices, doc values, and field data. With ILM compression and best-effort indexing, the storage overhead stays manageable. But if you skip ILM or use aggressive mapping that generates many extra fields, that 14GB/day can balloon to 100GB or more. The best approach is to measure actual volume for a week during your pilot phase, then adjust these estimates before ordering hardware.

A practical starting point: plan for 3x your raw log volume as index storage with a single replica. If you need higher availability with more replicas, multiply accordingly. Watching your actual index growth in the first month will tell you if your assumptions were right or if you need to tune your logging verbosity.

# Estimate daily log volume per service
# Assume: 1000 requests/min × 10KB avg log size × 60 min × 24h = ~14GB/day

# With 3 replicas and 30% overhead:
# 14GB × 4 (3 replicas + overhead) × 30 days = ~1.7TB/month per service

Scaling Triggers

Watch these metrics to decide when to add nodes:

• Cluster health: Yellow or red status means you need capacity
• Indexing latency: P95 above 500ms indicates saturation
• Search latency: P95 above 1s for interactive queries
• Disk usage: Nodes approaching 80% capacity
• JVM heap pressure: Old generation spending more than 30% time in GC

Cluster health is the simplest starting point. A yellow cluster has unassigned replica shards; searches still work but fault tolerance is compromised. A red cluster has unassigned primaries, which means data is at risk and new writes may fail. When yellow persists for more than an hour, investigate: a node probably dropped out or a disk watermark threshold was breached. Red status needs immediate action.

Disk usage is the most common scaling trigger for logging clusters. Set explicit watermark thresholds rather than relying on defaults. The low watermark at 85% stops Elasticsearch from allocating new shards to a node. The high watermark at 90% forces relocation of existing shards. The flood stage at 95% makes all indices read-only until you free space. Above 80% and you are in the danger zone. Many operators set 80% as warning and 88% as critical to catch saturation before it causes index blocks.

Indexing latency tells you when the write pipeline is saturated. The P95 threshold depends on your SLA, but 500ms is a reasonable starting point. If indexing latency climbs above that, check heap usage first. Heap above 70% causes Java GC pressure that shows up directly in indexing throughput. If heap looks healthy, the bottleneck is disk I/O. CPU queuing shows up as high indexing_pressure metrics.

Search latency P95 above 1 second makes dashboards feel sluggish. The culprit is usually large aggregations on hot data, or cold queries pulling from warm or cold tiers. Add replicas to distribute read load before adding nodes; replicas serve the same data with no additional indexing cost. If you are already at one replica per primary, adding data nodes is the next step.

JVM heap pressure above 70% in old gen is a warning sign. Elasticsearch takes heap off-heap for network buffers, Lucene segment caches, and the page cache, leaving a smaller window for doc values and filter caches. When old gen stays above 70% for sustained periods, you get long GC pauses that look like query timeouts and indexing stalls. Add heap (up to 31GB) or reduce load.

Queue depth in Logstash matters when Beats are buffering. Filebeat and Heartbeat hold events locally when Logstash is unreachable, but prolonged backpressure means events age out before processing. Watch the Beats output pipeline metrics and alert on queue depth growing for more than a few minutes.

Observability Stack Integration

ELK works best as part of a broader observability setup. Here is how it fits with other tools.

Prometheus + Grafana Integration

Beats can export metrics to Prometheus:

# metricbeat.yml - Enable prometheus output
metricbeat.modules:
  - module: elasticsearch
    metricsets:
      - node
      - node_stats
    period: 10s
    hosts: ["https://elasticsearch:9200"]

output.prometheus:
  enabled: true
  host: "0.0.0.0"
  port: 9424

Then scrape those metrics in Prometheus and build Grafana dashboards for cluster health, indexing throughput, and search latency.

Elasticsearch metrics break into a few groups. Node resource metrics show CPU time in each mode (user, system, iowait), memory used versus the page cache, and disk I/O rates for reads and writes. When iowait climbs above 20%, your disks are the bottleneck. Lucene segment metrics show open file handles, segment count per index, and merge queue depth. A segment count that keeps climbing means merges are falling behind and you risk exponential index growth.

Indexing and search metrics are where most teams focus. indexing_index_primary_* shows documents indexed per second on primaries; indexing_index_throttle_time shows when indexing pauses because the merge queue is overloaded. On the search side, search_query_total and search_query_time give you P50 latency, but for alerting you want P95 and P99 which requires histogram data. If the search_latency histogram shows P99 climbing past 2 seconds, users feel slowness even when average latency looks fine.

JVM metrics need their own panel. Heap used versus heap max, old generation collection time, and young generation GC frequency tell you whether heap is sized correctly. Frequent young gen GCs with occasional long old gen pauses means you need either more heap or smaller batch sizes. Alert on heap usage crossing 85%.

Cluster health and shard distribution round out the overview. Track shards in INITIALIZING, RELOCATING, and UNASSIGNED states. A cluster that stays yellow because of unassigned replicas will not recover on its own. Put data node count, primaries per index, and replicas per index on a single panel so you can verify redundancy at a glance.

For alerts, start with: cluster health yellow or red, disk usage above 85% on any node, heap usage above 85%, indexing latency P99 above 5 seconds, and search latency P99 above 2 seconds.

Distributed Tracing with Jaeger

Add trace context to your logs so you can jump from a Kibana error directly into Jaeger:

# Logstash filter to extract Jaeger trace context
filter {
  if [message] =~ /trace_id:/ {
    grok {
      match => { "message" => "trace_id:%{DATA:trace_id}\s+span_id:%{DATA:span_id}" }
    }
    mutate {
      add_field => {
        "trace_url" => "https://jaeger.example.com/trace/%{trace_id}"
      }
    }
  }
}

This lets you correlate log entries with the full request trace in Jaeger.

Trade-off Analysis

Factor	ELK Stack (Self-Hosted)	Loki (Grafana Cloud)	Splunk Enterprise	CloudWatch Logs
Deployment Model	Fully self-managed	SaaS / managed	Self-managed	Fully managed SaaS
Storage Cost	Infrastructure + ops	Pay-per-GB stored	License + infra	Pay-per-GB ingested
Query Performance	Excellent on large data	Good	Excellent	Moderate (throttling)
Operational Burden	High (cluster ops)	Low	High (infra + license)	Very low
Scalability	Manual shard management	Auto-scaling	Manual	Auto-scaling
Learning Curve	Steep (ES DSL)	Low (LogQL)	Steep (SPL)	Low (CloudWatch)
Ecosystem	Beats, Fluentd, Logstash	Promtail, Grafana	Heavy Agents	AWS-native

Production Failure Scenarios

Failure	Impact	Mitigation
Elasticsearch cluster red/yellow	Logs not indexing; search degraded	Monitor cluster health; provision more shards; adjust replica settings
Logstash pipeline errors	Logs stuck in queue; processing backlog	Monitor pipeline errors; implement dead-letter queues; alert on queue depth
Hot tier disk saturation	New indices cannot be created; ingestion fails	Monitor disk usage; implement ILM rollover; add nodes
Kibana performance degradation	Slow searches; dashboards timeout	Optimize queries; use filter context; limit time ranges
Beats shipper failure	Logs not forwarded; blind spots in coverage	Monitor Beats health; implement local buffering; alert on forward failures
Index template mismatch	Fields not indexed correctly; search failures	Version index templates; validate mappings; test before deployment

Common Pitfalls / Anti-Patterns

1. Too Many Indices with Few Documents

Each index has overhead. Too many small indices overwhelms the cluster:

// Bad: Index per day per service creates thousands of indices
PUT logs-service-a-2026.03.22
PUT logs-service-b-2026.03.22
// ... thousands more

// Good: Use rollover with larger time intervals
PUT logs-service-a
{
  "aliases": {
    "logs-service-a": { "is_write_index": true }
  }
}

2. Dynamic Field Mapping Without Controls

Dynamic mapping can create unexpected field types and blow up cardinality:

// Bad: Unrestricted dynamic mapping
{
  "mappings": {
    "dynamic": "true" // Creates any field
  }
}

// Good: Strict dynamic mapping or disabled
{
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "@timestamp": { "type": "date" },
      "level": { "type": "keyword" },
      "message": { "type": "text" }
    }
  }
}

3. Not Using Filter Context for Simple Queries

Filter context is faster because it does not score:

// Bad: Query context for term filter
{
  "query": {
    "match": { "level": "ERROR" } // Scores, slower
  }
}

// Good: Filter context for exact match
{
  "query": {
    "bool": {
      "filter": [
        { "term": { "level": "ERROR" } } // No scoring, faster
      ]
    }
  }
}

4. Ignoring Index Lifecycle Management

Without ILM, indices grow unbounded and performance degrades:

// Good: ILM with hot/warm/cold/delete
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": { "rollover": { "max_age": "7d" } }
      },
      "warm": { "min_age": "7d", "actions": { "shrink": 1, "forcemerge": 1 } },
      "cold": { "min_age": "30d", "actions": { "freeze": {} } },
      "delete": { "min_age": "365d", "actions": { "delete": {} } }
    }
  }
}

5. Loading Too Much Data into Memory

Kibana visualizations on large time ranges cause OOM:

// Bad: Visualize 90 days of minute-level data
{
  "query": { "range": { "@timestamp": { "gte": "now-90d" } } }
}

// Good: Use date histogram with appropriate interval
{
  "aggs": {
    "over_time": {
      "date_histogram": {
        "field": "@timestamp",
        "fixed_interval": "1h" // Or auto with proper configuration
      }
    }
  }
}

Real-world Failure Scenarios

Scenario 1: Elasticsearch Cluster Red/Yellow

What happened: After an unexpected traffic spike, the Elasticsearch cluster’s hot tier disk became saturated. New indices could not be created, causing log ingestion to fail across multiple services simultaneously.

Root cause: Index Lifecycle Management (ILM) was not configured, and disk alerts were set too high to catch gradual saturation before it became critical.

Impact: Approximately 4 hours of log data were lost. Engineers lost visibility into production systems during a post-incident analysis window.

Lesson learned: Configure ILM policies before going to production. Set disk usage alerts at 70% as a warning threshold and 80% as a critical threshold.

Scenario 2: Logstash Pipeline Backlog

What happened: A misconfigured Logstash filter caused an infinite loop in a conditional expression, causing the pipeline to process zero events while the input queue grew to millions of pending messages.

Root cause: A Grok filter pattern with an overly broad regex caused Logstash to CPU-thrash while failing to match any events.

Impact: Logs were delayed by 12 hours before the queue depth alert fired. Correlating logs with real-time events during the incident window was impossible.

Lesson learned: Implement dead-letter queues for failed events. Monitor pipeline-to-queue depth. Test filter patterns on sample data before deploying to production.

Scenario 3: Kibana Dashboard Timeout

What happened: A dashboard aggregating logs across 30-day windows with complex Vega visualizations began timing out. Users reported that the Kibana UI became unresponsive for all users on the cluster.

Root cause: The cluster’s coordination node was memory-constrained. Large aggregations caused heap pressure and triggered long GC pauses, affecting all coordination operations.

Impact: All Kibana users lost access to dashboards for approximately 45 minutes during peak business hours.

Lesson learned: Set query timeout limits in Kibana. Limit the default time range for users. Implement query caching and use rollup indices for historical data.

Interview Questions

Further Reading

• Metrics, Monitoring & Alerting - Complete observability stack integration
• Distributed Tracing - End-to-end request tracing with correlation IDs
• OpenTelemetry Official Docs - Vendor-neutral instrumentation standard
• Elasticsearch ILM Documentation - Index lifecycle management
• Fluent Bit Documentation - High-performance log agent
• SLO Embedded on-call Handbook - SLO-based incident management

Quick Recap

Key Takeaways:

• Beats collect, Logstash transforms, Elasticsearch stores, Kibana visualizes
• Index lifecycle management prevents unbounded growth
• Use filter context for exact matches; query context only when scoring needed
• Monitor cluster health and pipeline metrics proactively
• Implement security early: authentication, TLS, RBAC
• Design index templates carefully to control field mapping

Copy/Paste Checklist:

# Check cluster health
GET _cluster/health?pretty

# Monitor index size and document count
GET _cat/indices?v&s=store.size:desc

# Check Logstash pipeline status
GET _nodes/stats/ingest?filter_path=nodes.*.ingest

# ILM policy check
GET _ilm/policy/logs-policy?pretty

# Dead letter queue inspection
GET _all/_doc/_search?q=tags:_dead_letter_queue

# Index template validation
GET _index_template/logs-template?pretty

# Secure your cluster (Elasticsearch)
PUT _security/user/kibana_admin
{
  "password": "${KIBA...ORD}",
  "roles": ["kibana_admin"]
}

Observability Checklist

Infrastructure Monitoring

• Elasticsearch cluster health (green/yellow/red)
• Primary shard and replica distribution
• Index count and size per index
• Node resource utilization (CPU, heap, disk)
• Search and indexing latency percentiles
• JVM heap usage and GC frequency
• Segment count and merge queue depth

Log Pipeline Monitoring

• Beats shipper metrics (bytes sent, errors, lag)
• Logstash pipeline throughput and latency
• Logstash queue depth and worker utilization
• Dead-letter queue size and age
• Log parsing error rate

Kibana Monitoring

• Search response time (p95, p99)
• Dashboard load time
• Visualization render time
• Active users and session count

Data Management

• Index count within expected bounds
• Document count growth rate
• Disk usage trend and forecasting
• ILM policy execution success/failure
• Archive tier accessibility

Security Checklist

• Elasticsearch security enabled (XPack Security)
• User authentication configured (LDAP, SAML, or built-in)
• Role-based access control for indices and spaces
• TLS encryption for all network traffic
• API keys rotated regularly
• Kibana spaces isolation (dev/staging/prod separation)
• Audit logging enabled for security events
• No sensitive data in index names or field names
• Snapshot repositories secured and access logged
• Cross-cluster search secured if used

Conclusion

The ELK Stack provides a powerful platform for centralized logging and analysis. Beats collect data efficiently, Logstash transforms it into structured format, Elasticsearch stores and indexes it, and Kibana makes it explorable.

Start with Filebeat shipping container logs to Elasticsearch, and build from there. Add Logstash for complex parsing, Kibana for visualizations, and ILM policies for efficient data retention.

For monitoring beyond logs, see our Prometheus & Grafana guide for metrics visualization. For distributed tracing, see the Jaeger and Distributed Tracing guides for correlating logs with request traces.