Elasticsearch: Full-Text Search at Scale

Elasticsearch stores data in inverted indexes — the key data structure that makes full-text search fast across billions of documents. This post covers how it works: inverted indexes, shards, replicas, and the Query DSL for writing searches that actually perform.

Core Concepts

The Inverted Index

Elasticsearch stores data in Lucene indices. The core data structure is the inverted index — a mapping from terms to the documents containing those terms. When you search for a word, the engine looks it up and retrieves matching documents immediately. No full table scan. This is what makes Elasticsearch fast.

{
  "inverted_index": {
    "elasticsearch": [{ "doc_id": 1, "positions": [0, 3] }],
    "tutorial": [{ "doc_id": 1, "positions": [1] }],
    "full-text": [{ "doc_id": 2, "positions": [0] }]
  }
}

When you search for “elasticsearch tutorial,” Elasticsearch looks up both terms in the inverted index and finds matching documents instantly. No full table scan required.

The index also stores metadata about each term: document frequency, term frequency, positions for phrase queries, and norms for field-length normalization. This metadata is what makes relevance scoring, phrase matching, and fuzzy search possible.

Analyzer Pipeline

Before terms enter the inverted index, they pass through an analyzer consisting of three stages:

1. Character filters remove HTML tags, convert characters, or apply language-specific normalizations.
2. Tokenizer splits text into individual terms (tokens).
3. Token filters lowercase terms, remove stop words, apply synonyms, and perform stemming.

{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_analyzer": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "snowball", "asciifolding"]
        }
      }
    }
  }
}

Choosing the right analyzer matters. The standard analyzer works fine for most English text. For domain-specific vocabularies, you might need custom analyzers with synonym filters or language-specific stemmers.

Sharding: Distributing Data Across Nodes

A single Elasticsearch node can store millions of documents, but eventually you will need more storage, CPU, or memory than one machine provides. Elasticsearch solves this with shards: horizontal partitions of your index.

When you create an index, you specify the number of primary shards. Each primary shard is an independent Lucene index that stores a subset of your documents.

{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1
  }
}

With three primary shards, Elasticsearch distributes documents roughly evenly across them. A document with ID doc123 is routed to a specific shard using a hash of the ID: shard = hash(_id) % num_primary_shards.

Shard Routing Explained

graph TD
    Client[Client] -->|"search request"| LB[Load Balancer]
    LB --> Node1[Node 1]
    Node1 -->|"coordination"| Coordinator[Coordinator]
    Coordinator -->|"scatter"| Shard1[Primary Shard 1]
    Coordinator -->|"scatter"| Shard2[Primary Shard 2]
    Coordinator -->|"scatter"| Shard3[Primary Shard 3]
    Shard1 -->|"gather"| Coordinator
    Shard2 -->|"gather"| Coordinator
    Shard3 -->|"gather"| Coordinator
    Coordinator -->|"reduce"| Client

The node receiving the search request becomes the coordinator. It broadcasts the query to all relevant shards, collects results, and merges them into a single response. The scatter-gather pattern here is what lets you search across all shards in parallel.

Replica Shards

Every primary shard can have replicas for fault tolerance and read scalability. Replicas are never allocated on the same node as their primary. If a node fails, Elasticsearch promotes replicas to primaries automatically. Add replicas to linearly scale search throughput for read-heavy workloads.

The Query DSL: Expressing Search Logic

Elasticsearch’s Query DSL is a JSON-based language for expressing complex searches. It separates leaf queries (match, term) from compound queries (bool). Use filter context for non-scoring queries to leverage caching. Use query context when relevance scoring matters.

The Bool Query

The bool query is the workhorse of Elasticsearch search. It supports four clauses:

• must: Documents must match (AND logic)
• should: Documents should match (OR logic)
• filter: Same as must but without scoring (faster)
• must_not: Documents must not match (NOT logic)

{
  "query": {
    "bool": {
      "must": [{ "match": { "title": "elasticsearch" } }],
      "filter": [
        { "range": { "publish_date": { "gte": "2024-01-01" } } },
        { "term": { "status": "published" } }
      ],
      "should": [
        { "match": { "tags": "search" } },
        { "match": { "tags": "database" } }
      ],
      "minimum_should_match": 1
    }
  }
}

The filter context is particularly important. Queries in filter context bypass scoring entirely, and Elasticsearch caches filter bitsets for reuse. If you filter by a static field like status, subsequent queries become faster.

Relevance Scoring

Elasticsearch uses BM25 (Okapi BM25) as its default similarity algorithm. BM25 considers term frequency saturation and field length normalization. A term appearing 10 times in a 100-word field scores roughly the same as appearing 5 times in a 50-word field.

You can debug relevance with the explain parameter:

GET /my-index/_search
{
  "explain": true,
  "query": {
    "match": { "content": "elasticsearch tutorial" }
  }
}

The response shows exactly how each document scored, including term frequencies, inverse document frequencies, and field norms.

Designing Indices for Performance

Index design profoundly impacts search performance. A few principles:

Size your shards wisely. Shards between 10GB and 50GB work well. Too many small shards cause overhead; too few large shards cause memory pressure.

Use aliases for flexibility. Index aliases let you reindex without downtime:

POST /_aliases
{
  "actions": [
    { "remove": { "index": "my-index-v1", "alias": "my-index" } },
    { "add": { "index": "my-index-v2", "alias": "my-index" } }
  ]
}

Denormalize for read performance. Elasticsearch does not support joins like SQL. If you frequently query documents with nested objects, consider flattening the structure or using denormalization.

Capacity Estimation

Sizing an Elasticsearch cluster means estimating heap and document counts.

Heap allocation per node:

heap_per_node ≈
    (shards_per_node × segment_overhead)
  + (indexing_buffer × 10-20% of heap)
  + (query_cache × 10% of heap)
  + (fielddata × 10-20% of heap)
  + OS reserve (1GB)

Elasticsearch heap is shared across all shards on a node. Segment overhead alone is roughly 25MB per shard. With 30 shards on one node, that is 750MB before you even touch indexing buffers or caches.

Docs per shard guidelines:

Shard Size Target	Docs per Shard (approx)
10GB shard	10-30M docs (variable by doc size)
30GB shard	30-100M docs
50GB shard	50-150M docs

Doc count is harder to predict than shard size. A 1KB doc and a 50KB doc land at very different doc counts even at the same shard size. Watch both.

Example: 500GB index with 3 primary shards and 1 replica each:

shards_per_node = 3 primaries + 3 replicas = 6 shards
at 30GB/shard = 180GB per node

heap needed:
  segment overhead: 6 × 25MB = 150MB
  indexing buffer: 512MB (default, scales with heap)
  query cache: 276MB (10% of ~2.7GB heap)
  fielddata: 276MB
  OS reserve: 1GB
  total ≈ 2.5GB minimum, 4GB recommended

The sweet spot is 30GB-32GB heap per node. Above 32GB, Lucene starts using direct memory buffers outside the Java heap, which shifts the memory math. Most production clusters run 31GB heap on 64GB machines — the remaining RAM goes to the OS page cache, which Lucene uses heavily for file system caching.

Under the Hood: Segments, Translog, and Merge Policies

Elasticsearch is built on Apache Lucene, and understanding how Lucene stores your data at the file level demystifies many production behaviors — sudden latency spikes, disk usage patterns, and heap growth all trace back to segment internals.

Lucene Segments

Every shard contains one or more Lucene segments. A segment is a self-contained, immutable inverted index. When you index a document, it goes into a small in-memory buffer. On refresh, that buffer is flushed into a new segment file on disk. Searches hit all segments and merge results.

Segment immutability means writes never modify existing segments — new documents create new segments. This design enables locking-free writes but creates a problem: many small segments degrade search performance.

Segment Lifecycle

    index    flush    merge
doc -> buffer -> segment -> (merged) -> larger segment
         \                  /
          \----------------/
           translog backup

The refresh_interval controls how often in-memory buffers become searchable (default: 1 second). Between flushes, data lives in both the buffer and the translog.

The Translog

The translog is a write-ahead log that durability-safes every indexed document before it is committed to a segment. If a crash occurs, Elasticsearch replays the translog to recover un-flushed documents. This is the key difference between Elasticsearch’s near-real-time promise and true real-time guarantees.

Translog truncation happens automatically after a successful flush. You can force a translog flush:

POST /my-index/_flush

For write-heavy workloads, sizing the translog matters:

{
  "index": {
    "translog": {
      "disable_threshold": "1h",
      "sync_interval": "5s",
      "retention": {
        "size": "512MB",
        "age": "12h"
      }
    }
  }
}

Setting retention.size too low causes data loss on crashes. Setting it too high wastes disk space. The right value depends on your crash-recovery SLA.

Merge Policies

When segments accumulate, Lucene runs a merge policy to coalesce small segments into larger ones. The default policy is TieredMergePolicy (Lucene 9+) or LogByteSizeMergePolicy (older).

TieredMergePolicy targets a balanced segment size ladder:

{
  "index": {
    "merge_policy": {
      "type": "tiered",
      "max_merge_at_once": 10,
      "segments_per_tier": 10,
      "max_merged_segment_size_bytes": "5GB"
    }
  }
}

Key parameters:

• max_merge_at_once — how many segments can be merged in a single merge round (default: 10)
• segments_per_tier — how many small segments are allowed before a merge is triggered (default: 10)
• max_merged_segment_size_bytes — caps the size of merged segments to prevent giant segments

LogByteSizeMergePolicy (simpler, older):

{
  "index": {
    "merge_policy": {
      "type": "log_byte_size",
      "min_merge_size": "2MB",
      "max_merge_size": "10GB"
    }
  }
}

For write-heavy workloads (like log ingestion), set max_merge_at_once lower so merges don’t steal CPU from indexing. For read-heavy workloads, higher values mean fewer but larger segments — faster searches.

Force Merge

Running a force merge reduces segment count dramatically, at the cost of heavy I/O:

POST /my-index/_forcemerge?max_num_segments=1

This collapses all segments into one. It dramatically speeds up searches on read-heavy indices and reduces file handle usage. On large indices, run it during maintenance windows — it is I/O and CPU intensive, and Elasticsearch blocks new searches against indices being force-merged.

When to Use / When Not to Use

When to Use Elasticsearch

• Log and event analysis at scale, especially with the ELK stack
• Full-text search with complex relevance tuning and fuzzy matching
• Real-time analytics on time-series data with aggregations
• Distributed search where horizontal scalability is a requirement
• Autocomplete and type-ahead features via completion suggester

When Not to Use Elasticsearch

• Primary data store requiring ACID transactions (use a proper database)
• Simple key-value lookups where a document store suffices
• Heavy join operations across multiple entity types
• Systems requiring strong consistency (Elasticsearch is eventually consistent by default)
• Small datasets where a single PostgreSQL tsvector or SQLite FTS5 would suffice

Trade-off Analysis

Elasticsearch is not the only game in town. Depending on your use case, an alternative may serve you better.

Elasticsearch vs Apache Solr

Factor	Elasticsearch	Apache Solr
Ecosystem	ELK Stack (Kibana, etc.)	Solr + SolrCloud, Banana (older)
Operational model	Born for distributed	Distributed via SolrCloud
Schema	Dynamic mappings	Strict schema (SolrCell for parsing)
Query DSL	JSON-based, developer-friendly	XML-based (older), JSON supported
Scaling	Horizontal by default	Horizontal with auto-replication
Security	XPack (paid), builtin	Apache Shiro-based, configurable
Learning curve	Lower	Higher (deeper Lucene knowledge needed)
Use case fit	Logs, metrics, search	Enterprise search, faceted catalogs

Solr has a deeper history in enterprise search and handles complex faceted search (like e-commerce product catalogs) beautifully. Elasticsearch wins on operational simplicity, the Kibana ecosystem, and native horizontal scaling.

Elasticsearch vs OpenSearch

Factor	Elasticsearch	OpenSearch
Licensing	SSPL (since 2021)	Apache 2.0
Governance	Elastic N.V.	Linux Foundation (community-driven)
Feature parity	Faster innovation cycles	Elasticsearch features, ported over
Plugins	Commercial XPack	OpenSearch plugins
Security	XPack Security (paid)	OpenSearch Security (free)
Use case fit	Proprietary stack	If you need Apache-licensed search

If licensing concerns (SSPL) are a blocker, OpenSearch is the most mature drop-in replacement with near-complete API compatibility.

Elasticsearch vs Meilisearch

Factor	Elasticsearch	Meilisearch
Complexity	Full cluster management	Single binary, runs anywhere
Relevance tuning	Deep BM25 + scoring	Basic relevance, typo tolerance
Indexing speed	Very fast at scale	Extremely fast, even at small scale
Memory footprint	Several GB (JVM heap)	< 100MB
Distributed	Yes (native sharding)	Limited (single replica only)
Use case fit	Billions of docs, complex queries	Small-medium datasets, typo-tolerant search, fast prototype
Configuration	Expert-level tuning	Works out of the box

Meilisearch is the right choice when you want Elasticsearch-grade relevance without the operational overhead. It excels at typo-tolerant search, autocomplete, and datasets under 100 million documents.

Elasticsearch vs Typesense

Factor	Elasticsearch	Typesense
License	SSPL	GPL 3 (self-hosted), cloud-only (hosted)
Memory	JVM heap-heavy	~100MB-1GB
Query model	Full Query DSL	Simple, typo tolerance built-in
Distributed	Native sharding	Raft-based clustering
Faceting	Rich aggregations	Limited (planned)
Use case fit	Complex enterprise search	Typo-tolerant search, autocomplete, ranking
Dev experience	Steep learning curve	Quick start, friendly API

Typesense is optimized for developer experience and typo-tolerant search. If you need sub-100ms search with minimal DevOps and can live with simpler faceting, Typesense wins on simplicity.

Decision Matrix

Your need	Best choice
Billions of logs + Kibana dashboards	Elasticsearch
Enterprise faceted catalog search	Apache Solr
License-free search engine	OpenSearch
Fast prototype, small team, typo tolerance	Meilisearch
Autocomplete, typo-tolerant, low-ops	Typesense
Deep relevance tuning, complex queries	Elasticsearch
ACID transactions with search	PostgreSQL + tsvector

Production Failure Scenarios

Failure	Impact	Mitigation
Primary shard allocation fails	Index unavailable for writes	Set index.number_of_replicas >= 1 and use automatic retry
Coordinating node OOM	Search queries fail across cluster	Limit search.max_buckets, add circuit breakers, increase heap
Split-brain during network partition	Duplicate data or conflicting primaries	Use minimum_master_nodes ( quorum), prefer single-zone clusters
Bulk indexing queue overflow	Documents rejected, indexing lag	Size queue with thread_pool.write.queue_size, implement backpressure
Hot/Warm node imbalance	Some nodes run out of disk or CPU	Use Index Lifecycle Management (ILM), allocate shards manually
Incorrect analyzer causing data loss	Documents return no results	Test analyzers with _analyze API before applying to production indices

Common Pitfalls / Anti-Patterns

Over-Sharding

Creating too many shards wastes memory and increases cluster metadata overhead. Each shard maintains its own segment files, segment metadata, and caches. A rule of thumb: keep shard size between 10GB and 50GB. If you have 1TB of data, five 200GB shards is better than fifty 20GB shards.

Fix: Plan shard count based on expected data volume. Use index templates with ILM to auto-delete old indices.

Using Query Context for Filters

Queries in must context score every document, which is expensive when you only need filtering. Move static filters to filter context.

Before (slow):

{
  "query": {
    "bool": {
      "must": [
        { "match": { "content": "search term" } },
        { "term": { "status": "published" } }
      ]
    }
  }
}

After (faster):

{
  "query": {
    "bool": {
      "must": [{ "match": { "content": "search term" } }],
      "filter": [{ "term": { "status": "published" } }]
    }
  }
}

Ignoring Refresh Interval

New documents are not searchable until the next refresh (default 1 second). For bulk indexing, this is fine, but for near-real-time requirements, understand the tradeoff. Setting refresh_interval: -1 disables auto-refresh and dramatically speeds up bulk ingestion.

Fix: Adjust refresh_interval based on write vs. read latency requirements.

Not Using Aliases for Zero-Downtime Reindexing

Reindexing directly into an existing index causes downtime and potential data inconsistency.

Fix: Use index aliases with swap operations:

POST /_aliases
{
  "actions": [
    { "remove": { "index": "my-index-v1", "alias": "my-index" } },
    { "add": { "index": "my-index-v2", "alias": "my-index" } }
  ]
}

Quick Recap

Key Bullets

• Elasticsearch stores data in inverted indexes, enabling millisecond full-text queries
• Shard count is fixed at index creation — plan wisely (10GB-50GB per shard target)
• Use filter context for non-scoring queries to leverage caching
• Replicas provide fault tolerance and read scaling; they do not help write throughput
• The Query DSL separates leaf queries (match, term) from compound queries (bool)
• Index aliases enable zero-downtime reindexing and blue-green deployments
• BM25 is the default similarity algorithm; it handles term frequency saturation

Copy/Paste Checklist

# Check cluster health
GET /_cluster/health

# View shard allocation
GET /_cat/shards?v

# Monitor search latency
GET /_nodes/stats/indices/search?filter_path=**.*.query_total&timeout=30s

# Force merge to reduce segments
POST /my-index/_forcemerge?max_num_segments=1

# Update replica count
PUT /my-index/_settings
{
  "number_of_replicas": 2
}

# Check slow logs
GET /_nodes/stats/indices/indexing?filter_path=**.*.indexing.index.failed

# Set ILM policy
PUT /_ilm/policy/my-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": { "max_age": "7d" }
        }
      }
    }
  }
}

Observability Checklist

Metrics to Monitor

{
  "cluster_metrics": {
    "cluster_health": "green/yellow/red status",
    "number_of_pending_tasks": "< 100 typically",
    "task_duration_avg": "< 1s for search, < 5s for bulk"
  },
  "node_metrics": {
    "heap_used_percent": "< 85% sustained",
    "cpu_usage_percent": "< 70% sustained",
    "disk_io_read/write": "baseline + anomaly detection",
    "open_file_handles": "< 80% of ulimit"
  },
  "index_metrics": {
    "indexing_rate": "documents/second",
    "search_latency_p99": "< 500ms for interactive, < 2s for batch",
    "refresh_latency": "< 1s per segment",
    "segments_count": "< 50 per shard (force merge if higher)"
  }
}

Key Logs to Capture

• Error logs: logs/elasticsearch.log — shard failures, OOM events, circuit breaker trips
• Deprecation logs: deprecated API usage, settings scheduled for removal
• Slow search logs: configure index.search.slowlog.threshold to capture queries exceeding latency thresholds
• Indexing slow logs: index.indexing.slowlog.threshold for bulk insert issues

# Enable slow logs via API (persistent across restarts)
PUT /my-index/_settings
{
  "index.search.slowlog.threshold.query.warn": "10s",
  "index.search.slowlog.threshold.fetch.warn": "1s",
  "index.indexing.slowlog.threshold.index.warn": "10s"
}

Alerts to Configure

Alert	Condition	Severity
Cluster health	status == red	Critical
Node heap usage	heap_used_percent > 90% for > 5 min	Warning
Search latency	search_latency_p99 > 2s	Warning
Unassigned shards	unassigned_shards > 0 for > 5 min	Critical
Bulk queue rejection	bulk_queue_rejections > 0	Warning
Disk watermark	disk.watermark.low exceeded	Warning

Security Checklist

• Enable XPack Security (or OpenSearch Security if using OpenSearch distro)
• Use role-based access control (RBAC) — define roles with least-privilege principles
• Encrypt node-to-node communication with TLS certificates
• Restrict JMX/REST endpoints to internal networks; do not expose publicly
• Validate input to prevent injection via query strings and aggregations
• Audit access logs — log all write operations to sensitive indices
• Rotate credentials regularly; use a secrets manager for API keys
• Configure field-level security if certain fields must be hidden from some roles
• Enable audit logging to track unauthorized access attempts

# Example: Create a read-only role for an index
POST /_security/role/read_only_blogs
{
  "indices": [
    {
      "names": ["blogs-*"],
      "privileges": ["read"]
    }
  ]
}

Interview Questions

Further Reading

Elasticsearch: The Definitive Guide — official comprehensive guide

• Lucene Documentation — understanding segments, merge policies, and analyzers
• Elasticsearch Query DSL docs — full leaf and compound query reference
• Index Lifecycle Management — hot-warm-cold tier management
• Search Tuning Guide — official performance tuning
• Understanding BM25 — relevance scoring deep dive
• Monitoring with Metricbeat — pre-built Elasticsearch monitoring dashboards

The inverted index is what makes Elasticsearch fast. Queries that would scan a relational database for minutes return in milliseconds. Sharding and replicas give you horizontal scalability and fault tolerance.