Cache & Buffer Management

Your CPU does not access RAM directly. Between the ALU and main memory sits a ladder of SRAM caches — the fastest storage in your system, measured in single-digit nanoseconds. But these caches introduce problems that do not exist in a simple flat memory model: coherence across multiple cores, replacement policies that determine what gets evicted, and the OS’s role in managing how disk blocks and memory-mapped files flow through the cache hierarchy.

Understanding cache and buffer management is the key to predicting performance bottlenecks, explaining why multi-threaded programs sometimes run slower than single-threaded ones, and making sense of I/O performance anomalies that defy naive analysis.

Introduction

When to Use / When Not to Use

Cache and buffer management is always running — it is built into the hardware and kernel. However, you make high-level choices about write policy and buffering that shape performance.

When understanding this helps:

• Writing multi-threaded code that shares data (cache coherence overhead becomes the bottleneck)
• Optimizing sequential I/O (write-back vs write-through dramatically changes throughput)
• Database tuning (buffer pool size, checkpoint behavior, WAL writing)
• Understanding O_DIRECT vs buffered I/O trade-offs
• Analyzing performance profiles showing cache-miss-dominant hotspots

When this is less relevant:

• Purely sequential, single-threaded workloads with ample working set fit in cache
• Event-driven I/O-bound services (network polling, async I/O) where CPU caches are rarely the bottleneck

Architecture or Flow Diagram

The CPU cache hierarchy and the OS page cache interact through separate but coupled mechanisms. The diagram below shows the flow of a write through both hierarchies.

flowchart TD
    CPU["CPU Core 0<br/>Issues store to addr X"]
    L1D["L1 Data Cache<br/>64-byte lines<br/>Write-back policy"]
    L2["L2 Cache<br/>Inclusive<br/>256 KB - 1 MB"]
    L3["L3 Cache<br/>Shared<br/>8-64 MB per socket"]
    DRAM["DRAM"]
    PGFAULT["Page Cache<br/>4 KB pages<br/>mmap'd or buffered I/O"]
    DISK["Storage Device<br/>NVMe / HDD"]

    CPU --> L1D
    L1D -->|"Write-back<br/>not flushed"| L2
    L2 --> L3
    L3 --> DRAM
    DRAM --> PGFAULT
    PGFAULT -->|"dirty page<br/>periodic flush"| DISK

    subgraph Coherence["Cache Coherence (MESI)"]
        CPU1["CPU Core 1<br/>L1 Cache"]
        L1D -->|"MESI invalidation<br/>on write"| CPU1
    end

    style PGFAULT stroke:#ff00ff,stroke-width:2px
    style L1D stroke:#ff00ff,stroke-width:2px

On a write, the store goes to L1D (write-back). If the line is shared with another core’s L1 (MESI Shared state), the write issues an invalidation to all other caches before updating the line (MESI Modified state). The line is not written to DRAM or disk immediately — it sits in the cache until evicted or explicitly flushed.

Core Concepts

CPU Cache Architecture

Modern CPU caches are organized in a hierarchy that reflects the physics of wire delay and die area constraints. The L1 cache is split into instruction (L1 I) and data (L1 D) caches, each small (~32-48 KB per core) and fast (~4 cycles). The L2 cache is unified and larger (~256-512 KB per core, ~12 cycles). The L3 cache (also called LLC — Last Level Cache) is shared across all cores on the socket (~8-64 MB, ~20-40 cycles).

Cache sets and associativity: A cache is divided into sets (buckets), and each set has ways (slots for cache lines). A cache line maps to a specific set determined by its physical address. If a set has 8 ways (8-way set associative), it can hold 8 different cache lines for that set index before evicting one.

With a 64 KB L1 data cache and 64-byte cache lines:

• Number of sets = 64 KB / 64 bytes / 8 ways = 128 sets
• Set index = (address / 64) % 128

If 129 different addresses map to the same set, the 129th access evicts the oldest line regardless of recency — this is cache set thrashing.

Cache Coherence: MESI and Derivatives

When multiple CPU cores have caches, and a line is cached in more than one core’s L1, writes by one core must be visible to readers in other cores immediately — otherwise the program produces incorrect results. This is the cache coherence problem.

The MESI protocol (Modified, Exclusive, Shared, Invalid) is the canonical write-invalidate coherence protocol:

• Modified (M): This cache has the only valid copy; memory is stale; the line is “owned” and will be written back on eviction
• Exclusive (E): This cache has the only valid copy; memory is up-to-date; the line is clean
• Shared (S): This cache and possibly others have a valid copy; memory is up-to-date; no writing allowed without invalidating others
• Invalid (I): This cache does not have a valid copy of the line

MESI transitions on a write by core 0 to a line currently Shared:

1. Core 0 sends invalidate to all other caches
2. Other caches respond with acknowledge
3. Core 0 upgrades the line to Modified
4. All other caches mark their copies Invalid

This invalidation traffic on every write to a shared cache line is why false sharing — two unrelated variables on the same cache line, modified by different threads — collapses multi-core scaling.

Modern derivatives:

• MOESI (AMD): Adds an Owned state where a core can provide data to another core without writing to memory first
• MESI with Forward (Intel): Similar optimization; the core with Modified state supplies the line to requesters

Cache Replacement Policies

When a cache set is full and a new line needs to be placed, the cache must choose which line to evict. Common policies:

LRU (Least Recently Used): Evicts the least recently accessed line. Ideal but expensive to track — true LRU requires updating ordering on every access. Hardware caches typically use approximations.

Pseudo-LRU (PLRU): A tree-based approximation using binary flags. Faster to update than true LRU, with acceptable hit rate for most workloads. Used in many Intel and AMD L2/L3 caches.

Random: Evicts a random line. No tracking overhead, surprisingly competitive hit rate, used in ARM L2 caches in some configurations.

RRIP (Re-Reference Interval Prediction): Used in Intel Skylake+ L2/L3 caches. Each line has a ReReference Value (RRV). On insertion, lines get a high RRV (not recently used). On hit, they get low RRV. Eviction targets lines with highest RRV. This provides a form of dead block tracking — lines that have not been used recently take priority for eviction.

Buffer Cache and Page Cache

The page cache is the kernel’s main disk cache. When a file is read, the kernel checks if the requested blocks are already in the page cache. If so, the read is served from RAM — microseconds. If not, the kernel reads from disk and caches the pages for future reads.

The page cache is built on top of the buffer cache (which manages individual disk blocks) and the inode/dentry caches (which manage filesystem metadata). This multi-tier caching means that after reading a file once, subsequent reads are served from the page cache at DRAM speed.

Write-back (delayed write): By default, writes to files are stored in the page cache (marked dirty) and written to disk later. This batches multiple small writes into larger, more efficient I/O operations. The sync() system call forces all dirty pages to disk; the fsync() call forces a specific file’s dirty pages to disk.

Write-through: Some filesystems (or mount options like sync) write through the page cache directly to disk. This is slower but safer — a system crash does not lose the write. Databases typically use O_DIRECT to bypass the page cache entirely and write directly to disk, controlling their own durability guarantees.

Write Policies and Their Trade-offs

Write-back: Write to cache, mark line dirty. Only written to main memory on eviction. Fastest for repeated writes to the same location, but risks losing writes on crash until the dirty line is flushed.

Write-through: Write to cache and main memory simultaneously. Every store hits main memory, consuming bandwidth. Safe — memory always reflects the latest write. Used for write-combining memory regions (I/O devices, graphics RAM).

Write-combining: Buffers multiple writes and issues them as a burst. Used by GPU drivers and memory-mapped I/O regions where strict ordering is handled separately. Combines multiple small stores into one larger bus transaction.

Production Failure Scenarios

False Sharing Destroying Multi-Core Scalability

Failure: A thread-safe counter implemented as two volatile 64-bit counters on the same cache line (adjacent in memory), each updated by different threads, shows near-linear scaling degradation beyond 4 cores. The cache coherence protocol invalidates one core’s cache line on every increment of the other thread’s counter. Thread A writes to its counter, invalidating thread B’s cache line copy; thread B writes to its counter, invalidating thread A’s copy. Every cross-thread update generates a round-trip of invalidation traffic.

Mitigation: Align critical data to cache line boundaries (64 bytes). Use __attribute__((aligned(64))) in C or padding fields in Java. Use per-thread accumulators (thread-local counters that are merged periodically) to reduce cross-thread sharing. Profile with perf stat events: bus_trashes and l2_lines_out.all reveal coherence traffic.

Cache Stampede (Thrashing the Last-Level Cache)

Failure: A service that processes items from a working set larger than LLC by repeatedly scanning a large hash table. Each cache line that fits in the LLC is quickly evicted by the next access. Effective cache hit rate approaches zero; the workload becomes DRAM-bandwidth-bound, not compute-bound. Throughput does not improve with more CPU cores.

Mitigation: Use huge pages (2 MB) to reduce TLB pressure in addition to cache pressure. Redesign the data structure to have better spatial locality (one cache line per item instead of scattered across many pages). Consider partitioning the working set across cores and merging results rather than sharing a single structure.

Dirty Page Cache Pressure Causing Write Latency Spikes

Failure: A batch job that generates many small writes overwrites the page cache with dirty pages. When the dirty-write threshold (/proc/sys/vm/dirty_ratio, default 10% of total memory) is reached, the kernel blocks further writes to background writeback (pdflush/flush/kworker). The batch job stalls while the kernel writes accumulated pages to disk.

Mitigation: Call fsync() or use sync() periodically to acknowledge write durability. Use O_DIRECT for the batch writer if it manages its own durability (databases). Tune vm.dirty_ratio, vm.dirty_background_ratio, and vm.dirty_writeback_centisecs to match workload characteristics.

Trade-off Table

Write Policy	Write Latency	Read Hit Advantage	Crash Safety	Bandwidth Efficiency
Write-back cache	Low (hits cache)	Excellent (line in cache)	Poor (dirty line lost on crash)	High (batched eviction)
Write-through cache	High (must write memory)	Good	Partial (memory has latest)	Low (every write to memory)
Write-combining	Very low (buffered)	None (no caching)	Poor	Very high (burst)
Buffered I/O (page cache)	Low (hitting page cache)	Excellent	Requires sync/fsync	High
O_DIRECT	High (bypasses cache)	None	User-controlled (fdatasync)	Low (no read caching)

Implementation Snippets

Detecting False Sharing in Practice (C + pthread)

#define _GNU_SOURCE
#include <pthread.h>
#include <stdatomic.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <sys/time.h>

/* False sharing demonstration.
 * Two atomic counters on the same cache line (adjacent in memory)
 * vs. separated by a cache line (padding approach). */

#define ITERATIONS 100000000

/* Cache line size on x86/x64 is 64 bytes */
#define CACHE_LINE_SIZE 64

/* Method 1: Adjacent variables — same cache line, FALSE SHARING */
struct shared_counter_adjacent {
    atomic_long counter_a;  /* Written by thread 0 */
    atomic_long counter_b;  /* Written by thread 1 */
};

/* Method 2: Separated by cache line — NO false sharing */
struct shared_counter_separated {
    atomic_long counter_a;  /* Written by thread 0 */
} __attribute__((aligned(CACHE_LINE_SIZE)));

struct shared_counter_separated_b {
    atomic_long counter_b;  /* Written by thread 1 */
} __attribute__((aligned(CACHE_LINE_SIZE)));

static void *thread_func_adjacent(void *arg) {
    atomic_long *counter = (atomic_long *)arg;
    for (atomic_long i = 0; i < ITERATIONS; i++) {
        atomic_fetch_add(counter, 1);
    }
    return NULL;
}

static void *thread_func_adjacent_split(void *arg) {
    atomic_long *counter = (atomic_long *)arg;
    for (atomic_long i = 0; i < ITERATIONS; i++) {
        atomic_fetch_add(counter, 1);
    }
    return NULL;
}

double measure_throughput(const char *name, void *data_a, void *data_b,
                          void *(*thread_func)(void *)) {
    pthread_t t0, t1;
    struct timeval start, end;

    gettimeofday(&start, NULL);
    pthread_create(&t0, NULL, thread_func, data_a);
    pthread_create(&t1, NULL, thread_func, data_b);
    pthread_join(t0, NULL);
    pthread_join(t1, NULL);
    gettimeofday(&end, NULL);

    double elapsed = (end.tv_sec - start.tv_sec) +
                    (end.tv_usec - start.tv_usec) / 1e6;
    double ops_per_sec = (2.0 * ITERATIONS) / elapsed;
    printf("%s: %.2f million ops/sec (%.3f sec)\n", name, ops_per_sec / 1e6, elapsed);
    return ops_per_sec;
}

int main(void) {
    printf("=== False sharing detection ===\n");
    printf("Iterations per thread: %d\n", ITERATIONS);
    printf("Cache line size assumed: %d bytes\n\n", CACHE_LINE_SIZE);

    /* Method 1: Same cache line (adjacent) */
    struct shared_counter_adjacent adjacent = {0};
    measure_throughput("Adjacent (FALSE SHARING)", &adjacent.counter_a,
                       &adjacent.counter_b, thread_func_adjacent);

    /* Method 2: Cache-line-separated */
    struct shared_counter_separated counter_a_obj = {0};
    struct shared_counter_separated_b counter_b_obj = {0};
    measure_throughput("Separated (no false sharing)", &counter_a_obj.counter_a,
                       &counter_b_obj.counter_b, thread_func_adjacent_split);

    return 0;
}

Examining Page Cache Statistics (bash)

#!/bin/bash
# Page cache and buffer management monitoring

echo "=== Page Cache Statistics ==="
cat /proc/meminfo | grep -E "^(Buffers|Cached|Active|Inactive|Dirty|Writeback)"


echo ""
echo "=== Buffer Cache (block device) Statistics ==="
cat /proc/meminfo | grep -E "^( Buffers|Cached)"

echo ""
echo "=== Memory and Cache Details ==="
cat /proc/meminfo | grep -E "^(MemFree|MemAvailable|MemTotal|Shmem)"

echo ""
echo "=== Writeback Daemon Activity ==="
cat /proc/vmstat | grep -E "^(nr_dirty|nr_writeback|nr_writeback_temp|nr_pages_mapped)"

echo ""
echo "=== Per-CPU Cache Statistics (L1/L2/L3 hits) ==="
# Requires perf or likwid-tools
if command -v likwid &> /dev/null; then
    echo "Using likwid..."
    # likwid-topology -c 0 -M  # Uncomment if likwid is available
elif command -v perf &> /dev/null; then
    echo "Using perf stat (run as root for cache metrics)..."
    echo "perf stat -e cache-misses,cache-references, L1-dcache-load-misses ..."
fi

Observability Checklist

• CPU cache hit/miss rates: perf stat -e cache-misses,cache-references,L1-dcache-load-misses,L1-dcache-loads ./program
• Cache coherence traffic: perf stat -e bus_transactions,bus_trashes,l2_lines_out.all
• Page cache hit rate: cat /proc/vmstat | grep -E "^(pgpgin|pgpgout|pgactivate|deactivate)" — high pgactivate without corresponding pgdeactivate suggests working set not fitting
• Dirty page accumulation: cat /proc/meminfo | grep Dirty and cat /proc/vmstat | grep nr_dirty
• Writeback daemon activity: cat /proc/vmstat | grep -E "^(nr_writeback|writeback)" — persistent writeback activity indicates write-heavy workload
• LLC (last-level cache) occupancy: perf stat -e LLC-references,LLC-misses ./program or likwid -M
• TLB miss rate: perf stat -e dTLB-load-misses,dTLB-store-misses,iTLB-load-misses
• Per-socket memory bandwidth: likwid -M -C 0 or intel_pcm_bandwidth
• Cache line alignment check: perf c2c (cache-to-cache transfer analysis) to detect false sharing

Common Pitfalls / Anti-Patterns

Cache timing side-channel attacks: Spectre, Meltdown, and related variants exploit the timing difference between a cache hit and a cache miss. By measuring access time to a cache line, an attacker can infer whether another process accessed a particular line — leaking cryptographic keys, passwords, or any data that influences memory access patterns. Mitigations include:

• IBRS/STIBP: Microcode patches that serialize branch prediction and address translation, closing the timing oracle
• L1D flush on context switch: l1d_flush=on kernel parameter forces L1 cache flush on each context switch, reducing cross-process cache residue
• SpecCtrl/IBRS: Kernel and hypervisor settings that isolate speculative execution windows

Cache coherence traffic as information leakage: Even without malicious timing attacks, cache coherence invalidation patterns can reveal which memory addresses another process is accessing — a form of reverse-engineering. In high-security environments, cache activity should be monitored for anomalies.

Huge pages and secure enclaves (Intel SGX): SGX enclaves have restricted cache access — enclave pages cannot be cached in the LLC (last-level cache) to prevent cross-enclave cache snooping. This means enclave workloads have higher cache miss rates than equivalent non-enclave workloads, which is a performance consideration for SGX-based applications.

Common Pitfalls / Anti-patterns

Pitfall: Assuming cache line size is 64 bytes on all architectures. Cache line sizes vary: ARM Cortex-A72 uses 64 bytes; some IBM POWER chips use 128 bytes; older embedded ARM may use 32 bytes. Code that relies on CACHE_LINE_SIZE must be defined per-architecture. On x86 and ARM64, 64 bytes is correct. If you use __attribute__((aligned(64))) on an ARM with 128-byte lines, you still get false sharing — the compiler can only align to what you specify.

Pitfall: Over-padding in multi-threaded data structures. Adding a full cache line of padding between every field in a shared structure wastes memory bandwidth and cache capacity. Padding should be added only between fields that are written by different threads. Two fields that are always accessed by the same thread can share a cache line without false sharing — the coherence protocol does not invalidate on reads.

Pitfall: Ignoring the write-combining buffer in I/O-heavy code. Memory-mapped I/O regions (device registers) use write-combining, not cache. A store to an MMIO address does not update a cache line — it goes to a write-combining buffer. If you write multiple values to the same MMIO register in quick succession, each write may be dropped if the buffer is not flushed. Use wmb() (write memory barrier) or mmio_write() functions that handle the barrier.

Anti-pattern: Disabling cache entirely with uncached memory accesses. Setting the MTRR (Memory Type Range Register) to uncached (UC) for a memory region forces all accesses to bypass the cache. This is occasionally necessary for hardware registers but devastating for performance on any frequently-accessed data. Use write-combining (WC) or write-back (WB) instead.

Quick Recap Checklist

• CPU cache hierarchy (L1/L2/L3) provides 10-100x faster access than DRAM; cache lines are 64 bytes on x86/ARM
• MESI protocol maintains cache coherence across cores: Modified, Exclusive, Shared, Invalid states govern when writes must invalidate other copies
• False sharing occurs when two unrelated variables on the same cache line are written by different threads, triggering constant invalidation traffic
• Cache replacement policies (LRU approximations, RRIP) determine which line to evict when a set is full
• The page cache buffers disk I/O; write-back delays writes to disk for batching, while write-through writes immediately
• O_DIRECT bypasses the page cache for user-controlled durability; databases use this to avoid double-buffering
• perf stat -e cache-misses,cache-references reveals how well your workload utilizes the cache hierarchy
• Huge pages (2 MB) improve cache efficiency by reducing TLB pressure and page table overhead
• Dirty page thresholds (vm.dirty_ratio) control when the kernel blocks writers to flush dirty pages to disk
• Cache timing side channels (Spectre/Meltdown) exploit the timing difference between cache hits and misses to leak data
• Tools: perf stat (cache events), perf c2c (false sharing detection), slabtop (kernel slab cache stats)

Interview Questions

Further Reading

MESI Protocol Variants

MOESI (AMD) — adds the Owned state:

• Owned: this cache has the only valid copy AND memory is stale; this core can supply data to requesters without writing to memory first
• Reduces memory bandwidth by eliminating unnecessary writes-back when sharing data

MESI with Forward (Intel) — similar to MOESI but the line is written to memory before being supplied to other cores

Dragon Protocol (research) — used in some ARM implementations:

• Uses update (write-broadcast) instead of invalidate
• Better for workloads that read more than write to shared lines
• Higher bandwidth cost for writes but fewer invalidation storms

Cache Replacement Policy Deep Dive

LRU approximations:

• True LRU: O(n) per access for n lines in a set — used only in small, software-managed structures
• PLRU (Pseudo-LRU): Tree-based, ~3 comparisons per access, used in many Intel L2/L3 caches
• RRIP (Re-Reference Interval Prediction): Intel Skylake+ L2/L3 — each line has an N-bit counter; recently used lines have low values, idle lines have high values; eviction targets highest counter

Thrashing-resistant policies:

• SIP (Set-dueling Predictive): Some sets use LRU, others use RRIP; the policy that performs better wins for the whole cache
• MLP-aware replacement: Lines accessed in a memory-level-parallel manner get higher priority for retention

Page Cache and Buffer Cache Relationship

The page cache is the primary disk cache in modern Linux. The buffer cache (legacy, pre-2.6) managed individual disk blocks and was eventually subsumed by the page cache.

Layer	Unit	Purpose
Page cache	4 KB pages	Caches file content (page-mapped files)
Buffer cache	512 B - 4 KB blocks	Caches metadata and raw block device I/O
dentry cache	VFS structures	Caches directory entry lookups
inode cache	Filesystem metadata	Caches inode structures

free command shows -/+ buffers/cache line which combines page cache + buffer cache + slab reclaimable.

Key Takeaways

• CPU caches (L1/L2/L3) are hardware-managed transparently; page cache is kernel-managed for disk I/O
• MESI (Modified, Exclusive, Shared, Invalid) maintains coherence across cores — writes trigger invalidations
• False sharing (unrelated variables on the same cache line, written by different threads) collapses multi-core scaling
• Cache replacement policies (PLRU, RRIP) approximate LRU without per-access overhead
• Page cache write-back batches writes to disk; fsync() forces durability for specific files
• O_DIRECT bypasses the page cache for user-controlled durability and to avoid double-buffering
• perf c2c detects false sharing; perf stat reveals cache hit/miss rates

Conclusion

Cache and buffer management represents the invisible infrastructure that makes modern computing fast. The CPU cache hierarchy — L1/L2/L3 — operates automatically on every memory access, applying the MESI coherence protocol to maintain consistency across cores. What appears to your code as a simple memory read actually involves cache line lookups, potential invalidation traffic, and sophisticated replacement policies like RRIP.

The page cache extends this thinking to disk I/O, batching writes and serving reads from RAM. Write-back delivers maximum throughput but requires explicit fsync calls for durability. O_DIRECT bypasses the page cache entirely, giving databases control over their own buffering — essential for applications that cannot afford the double-buffering penalty of both the OS page cache and their own in-memory structures.

For deeper exploration, study how memory-mapped I/O interacts with write-combining buffers, why SGX enclaves have higher cache miss rates, and how the L1D flush on context switch mitigates cross-process cache residue. The performance implications of cache behavior become most visible in NUMA systems, where cross-socket memory access latencies dwarf local cache hit times.