Shared Memory

If you need to move large amounts of data between processes, every mechanism we’ve looked at so far — pipes, message queues, sockets — requires the kernel to copy data at least once. Each read() from a pipe copies data from the kernel’s pipe buffer into user space. Each msgsnd() copies a message into the kernel’s queue. This is fine for small messages but becomes a bottleneck when you are sharing megabytes or gigabytes of data between processes at high frequency.

Shared memory fixes this by letting multiple processes map the same physical memory pages into their address spaces. No kernel copy is involved — one process writes to an address and another process reads from a different address that maps to the same physical memory. This is the fastest form of IPC available, and understanding it is essential for building high-performance systems like databases, video processing pipelines, and real-time data distribution engines.

Introduction

Shared memory is a region of memory that multiple processes can access. Unlike pipes where data flows in one direction and is consumed, shared memory is a bidirectional, persistent data space that all participants share. The operating system coordinates access through page table entries and optional synchronization primitives (typically semaphores or mutexes).

There are two primary mechanisms for shared memory on Unix/Linux systems:

Memory-mapped files (mmap()) — Maps a file (or a file-backed region) into memory. Changes are eventually written back to the file. The file provides persistence and a way to back the shared region.

System V shared memory (shmget()/shmat()/shmdt()/shmctl()) — A pure shared memory mechanism backed by kernel memory (not a file). Data does not persist across system reboots. Offers more control over memory attributes.

POSIX shared memory (shm_open()/mmap()) — A hybrid approach using filesystem-like paths but backed by kernel memory rather than a file. More modern API than System V.

All three approaches work by mapping the same physical memory pages into multiple processes’ page tables. The CPU’s memory management unit (MMU) handles the translation from virtual addresses (different per process) to physical addresses (shared).

When to Use / When Not to Use

Use shared memory when:

• You need maximum throughput between processes (zero-copy data sharing)
• You are sharing large data structures (buffers, matrices, databases)
• You are implementing a producer-consumer pattern where the data volume is high
• Multiple processes need simultaneous read/write access to the same data
• You want to avoid the kernel overhead of repeated copy operations

Do not use shared memory when:

• You need data persistence across system reboots (use memory-mapped files)
• Your data sizes are small and simple (pipes or message queues are simpler)
• You cannot tolerate the complexity of synchronization
• You are on a platform where shared memory is not well-supported
• You need security isolation between processes (use separate address spaces)

Architecture or Flow Diagram

graph TD
    subgraph Process A
        A_VA[Virtual Address Space<br/>Process A] --> A_PT[A's Page Table]
        A_PT --> A_MMU[MMU Translation]
    end

    subgraph Process B
        B_VA[Virtual Address Space<br/>Process B] --> B_PT[B's Page Table]
        B_PT --> B_MMU[MMU Translation]
    end

    A_MMU --> PHYS[Physical Memory Page<br/>Shared Physical Page]
    B_MMU --> PHYS

    subgraph Memory Mapping Flow
        C[shmget / mmap] --> D[Kernel allocates<br/>physical pages]
        D --> E[Kernel maps same physical pages<br/>into both processes' page tables]
        E --> F[Process A writes to address 0x7f...<br/>Process B reads from 0x556...]
        F --> G[Both virtual addresses<br/>→ same physical address]
    end

    subgraph Cache Coherence Issue
        H[CPU 1 writes to shared cache line] --> I[Cache line invalidated<br/>in CPU 2's cache]
        I --> J[MES protocol or MESI protocol<br/>handles coherence]
    end

Core Concepts

Memory-Mapped Files (mmap)

The mmap() function creates a mapping between a file (or anonymous region) and a process’s virtual address space:

#include <sys/mman.h>
#include <fcntl.h>
#include <unistd.h>

// Open or create a file
int fd = open("/tmp/shared_data.bin", O_RDWR | O_CREAT, 0666);
if (fd == -1) {
    perror("open");
    exit(1);
}

// Extend file to needed size
ftruncate(fd, 1024 * 1024);  // 1MB

// Map the file into memory
void *addr = mmap(NULL, 1024 * 1024, PROT_READ | PROT_WRITE,
                  MAP_SHARED, fd, 0);
if (addr == MAP_FAILED) {
    perror("mmap");
    exit(1);
}

// Now read/write like regular memory
memcpy(addr, "Hello shared world!", 18);

// Cleanup
munmap(addr, 1024 * 1024);
close(fd);

Key points:

• PROT_READ | PROT_WRITE defines the access mode
• MAP_SHARED means changes are visible to other processes and written to the underlying file
• MAP_PRIVATE creates a copy-on-write mapping (changes go to a private copy)
• MAP_ANONYMOUS creates a mapping backed by swap space, not a file (no fd needed)

System V Shared Memory

#include <sys/ipc.h>
#include <sys/shm.h>

// Create a shared memory segment (1MB)
int shmid = shmget(IPC_PRIVATE, 1024 * 1024, IPC_CREAT | 0666);
if (shmid == -1) {
    perror("shmget");
    exit(1);
}

// Attach the segment to our address space
void *addr = shmat(shmid, NULL, 0);
if (addr == (void *)-1) {
    perror("shmat");
    exit(1);
}

// Use the shared memory
strcpy((char *)addr, "Hello from System V shared memory!");

// Detach when done
shmdt(addr);

// Clean up (only when no one needs it anymore)
shmctl(shmid, IPC_RMID, NULL);

POSIX Shared Memory

#include <sys/mman.h>
#include <sys/stat.h>
#include <fcntl.h>

// Create a POSIX shared memory object
int fd = shm_open("/my_shm", O_CREAT | O_RDWR, 0666);
if (fd == -1) {
    perror("shm_open");
    exit(1);
}

// Set the size
ftruncate(fd, 1024 * 1024);

// Map it into address space
void *addr = mmap(NULL, 1024 * 1024, PROT_READ | PROT_WRITE,
                  MAP_SHARED, fd, 0);

// Use it...

munmap(addr, 1024 * 1024);
close(fd);
shm_unlink("/my_shm");  // Remove from filesystem

Cache Coherence

Shared memory works at the page level, but modern CPUs operate at the cache line level (typically 64 bytes). When two processes running on different CPU cores access the same memory location, their respective CPU caches may hold the same cache line. This creates cache coherence challenges.

Modern CPUs implement MESI (Modified, Exclusive, Shared, Invalid) or MOESI cache coherence protocols. When one CPU writes to a cache line that another CPU holds in a shared state, the write causes an invalidation on the other CPU’s cache line, forcing the other CPU to re-fetch the data from memory on its next access.

In a high-contention shared memory scenario with many writers on different cores, cache line bouncing can severely degrade performance. This is why shared memory with high write contention often benefits from careful data layout — separating frequently written fields to different cache lines to avoid false sharing.

Production Failure Scenarios

Race Conditions Without Synchronization

The most common shared memory failure is two processes writing to the same data simultaneously. Without synchronization primitives, one write overwrites the other, or partial writes create garbage data. A process reads a half-updated data structure and crashes.

Mitigation: Always use synchronization primitives with shared memory. Options include System V semaphores (semget/semop), POSIX mutexes (with pthread_mutexattr_setpshared), or memory-mapped files with msync() and atomic operations. Design your data layout to minimize contention and false sharing.

Data Corruption from Partial Writes

If a writer is interrupted mid-write (preempted by the scheduler) and another process reads the data structure, it sees a partially updated state. This is especially dangerous with multi-field data structures that need atomic updates.

Mitigation: Use atomic operations for simple values, use reader-writer locks for complex data, or structure updates so they can be done in a single write (e.g., write to a versioning field last, so readers can check version before trusting data).

Lost Updates from Non-Atomic Operations

Even simple operations like counter++ are not atomic on most architectures — they involve a read, an increment, and a write. If two processes do this simultaneously on a shared counter, one update is lost.

Mitigation: Use atomic operations (__sync_fetch_and_add() GCC builtin, or C11 <stdatomic.h>), or use mutex-protected critical sections.

Resource Leaks from Improper Cleanup

If a process exits without detaching from shared memory (shmdt() or munmap()), the kernel marks the mapping as detached when the process’s page table entries are destroyed. However, the shared memory segment itself may persist if IPC_RMID or shm_unlink() has not been called.

Mitigation: Implement robust cleanup in signal handlers (SIGTERM), use wrapper frameworks that track shared memory lifecycle, and monitor for orphaned shared memory segments with ipcs -m.

Page Fault Overhead on First Access

When a shared memory segment is first attached, the pages may not be in physical memory. A page fault occurs for each page, which adds latency on first access.

Mitigation: Prefault the pages by touching each page immediately after attachment to force them into memory. Use mlock() to lock pages in RAM and prevent paging to disk.

Trade-off Table

Feature	mmap (file-backed)	System V shm	POSIX shm_open	Pipe/Message Queue
Data persistence	Yes (file)	No (kernel RAM)	No (kernel RAM)	No
Zero-copy semantics	Yes (after initial map)	Yes (direct access)	Yes (direct access)	No (kernel copy)
Access model	Random access, file-like	Direct memory access	Direct memory access	Sequential/random
Typical use case	Memory-mapped I/O, file sharing	High-speed IPC	High-speed IPC	Task distribution
Synchronization	Optional (file locks or external)	Semaphores (sysv)	POSIX mutex/sem	Built-in (send/recv)
Maximum size	Limited by disk space	Limited by kernel limits	Limited by kernel limits	Limited by buffer size
Cleanup model	munmap + file remains	shmctl(IPC_RMID)	shm_unlink()	Auto when all close
Portable	Very portable	Portable	POSIX (widely available)	Very portable

Implementation Snippet(s)

C: Shared Memory with POSIX Mutex Synchronization

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <pthread.h>
#include <unistd.h>

#define SHM_NAME "/prod_cons_shm"
#define BUF_SIZE 1024

typedef struct {
    pthread_mutex_t mutex;
    pthread_cond_t cond;
    int data_ready;
    char data[BUF_SIZE];
} shared_data_t;

int main() {
    int fd = shm_open(SHM_NAME, O_CREAT | O_RDWR, 0666);
    ftruncate(fd, sizeof(shared_data_t));

    shared_data_t *shm = mmap(NULL, sizeof(shared_data_t),
                              PROT_READ | PROT_WRITE,
                              MAP_SHARED, fd, 0);
    close(fd);

    // Initialize synchronization primitives (once)
    pthread_mutexattr_t mattr;
    pthread_mutexattr_init(&mattr);
    pthread_mutexattr_setpshared(&mattr, PTHREAD_PROCESS_SHARED);
    pthread_mutex_init(&shm->mutex, &mattr);
    pthread_condattr_init(&cattr);
    pthread_condattr_setpshared(&cattr, PTHREAD_PROCESS_SHARED);
    pthread_cond_init(&shm->cond, &cattr);
    shm->data_ready = 0;

    pid_t pid = fork();
    if (pid == 0) {
        // Child: producer
        while (1) {
            pthread_mutex_lock(&shm->mutex);
            snprintf(shm->data, BUF_SIZE, "Message at %ld", (long)time(NULL));
            shm->data_ready = 1;
            pthread_cond_signal(&shm->cond);
            pthread_mutex_unlock(&shm->mutex);
            sleep(1);
        }
    } else {
        // Parent: consumer
        while (1) {
            pthread_mutex_lock(&shm->mutex);
            while (!shm->data_ready) {
                pthread_cond_wait(&shm->cond, &shm->mutex);
            }
            printf("Received: %s\n", shm->data);
            shm->data_ready = 0;
            pthread_mutex_unlock(&shm->mutex);
        }
    }

    return 0;
}

Python: Using mmap for Shared Memory

import mmap
import os

# Memory-mapped file as shared memory
FILE_PATH = "/tmp/shared_mmap.bin"
SIZE = 1024 * 1024  # 1MB

# Create the file and map it
fd = os.open(FILE_PATH, os.O_RDWR | os.O_CREAT)
os.ftruncate(fd, SIZE)

mmap_obj = mmap.mmap(fd, SIZE, mmap.MAP_SHARED, mmap.PROT_READ | mmap.PROT_WRITE)

# Write data
mmap_obj.seek(0)
mmap_obj.write(b"Shared data via mmap!")

# Read data
mmap_obj.seek(0)
print(mmap_obj.readline())

mmap_obj.close()
os.close(fd)
os.unlink(FILE_PATH)

Bash: Using shared memory monitoring tools

# Check existing System V shared memory segments
ipcs -m

# Show detailed info about a specific segment
ipcs -m -i <shmid>

# Remove a stuck segment
ipcrm -m <shmid>

# Check POSIX shared memory
ls /dev/shm/

# Remove POSIX shared memory object
# shm_unlink("/name") from C code

# Watch shared memory consumption
watch -n 1 'ipcs -m'

Observability Checklist

• Shared memory segments: ipcs -m shows all System V shared memory segments with IDs, sizes, attached processes
• POSIX shared memory: ls -la /dev/shm/ lists POSIX shared memory objects
• Attachment count: ipcs -m shows number of attached processes per segment — 0 means orphaned if not marked for destruction
• Page fault analysis: Use perf stat -e page-faults -e minor-faults to measure page fault overhead when first accessing shared segments
• Cache coherence metrics: Use perf stat with hardware counters (if available) to detect cache line bouncing between cores
• Memory usage: Monitor RSS and shared memory size via /proc/<pid>/status and /proc/<pid>/smaps
• strace: Trace mmap, munmap, shmget, shmat, shmdt, shmctl system calls to debug mapping issues

Common Pitfalls / Anti-Patterns

Shared memory provides no access control beyond Unix permissions — any process with read/write permission to the underlying file or segment can read and modify all data. This is a significant security concern.

Mitigations:

• Use appropriate file permissions on backing files for mmap() — chmod 0600 for exclusive access
• For System V and POSIX shared memory, set restrictive permissions (0660 or 0664) and use dedicated groups
• Consider encrypting sensitive data before placing it in shared memory (application-layer encryption)
• Use mlock() to prevent sensitive data from being paged to disk (where it could be recovered from swap)
• Implement process authentication at the application level if untrusted processes may access shared segments

Compliance: Shared memory segments may contain sensitive data in memory dumps (core files). For PCI-DSS, HIPAA, or other compliance regimes, ensure that sensitive data in shared memory is properly protected and that core dumps are handled appropriately.

Common Pitfalls / Anti-patterns

1. Forgetting synchronization — the most dangerous anti-pattern. Without mutexes, semaphores, or atomic operations protecting shared data, you get race conditions, data corruption, and crashes. Always pair shared memory with synchronization.

2. False sharing — when two frequently-modified variables happen to share the same cache line, each modification invalidates the other’s cached copy, causing massive performance degradation. Pad your data structures to align frequently-written fields to separate cache lines (64-byte boundaries).

3. Assuming memory visibility is immediate — when one process writes to shared memory, the other process might not see that write immediately due to CPU caching, compiler optimizations, or store buffers. Use pthread_mutex_lock() which provides memory barriers, or use __sync_synchronize() for explicit barriers.

4. Not handling EINVAL on shmat — shmat() returns (void *)-1 on error, not NULL. Check for this correctly: if (addr == (void *)-1) not if (!addr).

5. Leaving synchronization primitives in inconsistent state — if a process crashes while holding a mutex, other processes deadlock forever. Use robust mutex initialization and consider using robust mutexes (PTHREAD_MUTEX_ROBUST) that detect deadlocks.

6. Not accounting for different page sizes — on systems with huge pages (2MB or 1GB pages), shared memory alignment requirements may differ. Using huge pages can significantly improve performance for large shared memory regions.

7. Mismatched MAP_SHARED vs MAP_PRIVATE — two processes mapping the same file with MAP_PRIVATE each get a private copy of the data. Changes made by one process are invisible to the other. Use MAP_SHARED for true sharing.

Quick Recap Checklist

• Shared memory provides zero-copy IPC by mapping the same physical memory pages into multiple processes’ address spaces
• Three mechanisms: mmap() with file backing, System V shmget()/shmat(), and POSIX shm_open()/mmap()
• Synchronization is mandatory — shared memory alone does not prevent race conditions. Use mutexes, semaphores, or atomic operations.
• Cache coherence between CPU cores is managed by MESI/MOESI protocols; false sharing causes severe performance degradation
• mmap() with MAP_ANONYMOUS creates anonymous shared memory backed by swap, not a file
• System V shared memory persists until explicitly removed with IPC_RMID; POSIX objects persist until shm_unlink()
• Memory-mapped files provide persistence and are useful for memory-mapped I/O where writes are eventually flushed to disk
• For high-performance inter-process data sharing, shared memory is the fastest option but requires careful synchronization design

Interview Questions

Further Reading

• shm_overview(7) — Linux man page — POSIX shared memory overview
• shmget(2) — Linux man page — System V shared memory API
• mmap(2) — Linux man page — Memory mapping API including shared memory
• shm_open(3) — Linux man page — POSIX shared memory objects

Conclusion

Shared memory delivers the highest throughput of any IPC mechanism by eliminating kernel-mediated data copies. Multiple processes access the same physical memory pages through their own virtual address spaces, with the MMU handling address translation and CPU cache coherence protocols maintaining consistency. This raw speed comes at the cost of programmer-managed synchronization — there are no locks or queues to serialize access, so you must provide your own.

The evolution from shared memory toward even faster mechanisms leads to memory-mapped I/O, NUMA-aware data placement, and persistent memory (PMEM) architectures. Database systems like PostgreSQL and Oracle use shared memory extensively for buffer pools, while real-time trading systems use it for low-latency data distribution between processes.

For continued learning, explore how the Linux kernel implements shared memory under the hood (the shmem filesystem), study NUMA-aware memory allocation strategies for multi-socket systems, and investigate persistent memory programming models where shared regions survive system reboots.