Message Queues

Pipes work well for streaming data between processes, but they have limits. What happens when one process produces data faster than the consumer can process it? What if the producer and consumer run at completely different times? Pipes fail here because both ends need to be open simultaneously and they lose data when no reader is available. Message queues fix this by providing persistent, asynchronous, message-oriented communication between processes.

Introduction

Message queues are a form of inter-process communication where messages sit in the kernel until a receiving process retrieves them. Each message has a type field and optional payload, and the kernel maintains message ordering and priority. This makes message queues a good fit for producer-consumer patterns, task distribution, and situations where processes need to operate independently in time.

Unix systems have two main message queue implementations:

POSIX Message Queues (mq_* functions) — The more modern API, introduced in POSIX.1b. They use filesystem-like paths (e.g., /my_queue) and can notify processes via signals or callbacks.

System V Message Queues (msgget, msgsnd, msgrcv) — The older API that predates POSIX. They use numeric keys instead of paths, identified by integer queue identifiers.

Both are still widely used. POSIX queues have a cleaner API and better Linux integration; System V queues are more portable across Unix variants and offer some advanced features like message undo operations.

When to Use / When Not to Use

Use message queues when:

• You need asynchronous communication — sender and receiver do not need to run simultaneously
• You want message boundaries preserved (discrete, typed messages)
• You need message priorities (urgent messages jump ahead of normal ones)
• You need a queue that persists messages until consumed
• You are implementing a producer-consumer or work queue pattern
• You want to decouple processing stages in a pipeline

Do not use message queues when:

• You need bidirectional communication (use sockets instead)
• You need high-throughput streaming (pipes or shared memory are faster)
• You need total ordering of messages with no priority (use a pipe with proper framing)
• You are communicating between machines (use network sockets)
• You need transactional message delivery with acknowledgments (use a message broker like Kafka or RabbitMQ)

Architecture or Flow Diagram

graph TD
    subgraph POSIX Message Queue
        A[Process A calls mq_open] --> B[Kernel creates/opens queue at path]
        B --> C[Process A calls mq_send<br/>Kernel copies message to queue buffer]
        C --> D[Message stored in kernel queue<br/>with priority and timestamp]
        E[Process B calls mq_receive<br/>Kernel copies message to Process B's buffer]
        D --> E
        E --> F[Kernel removes message<br/>from queue after successful receive]
    end

    subgraph System V Message Queue
        G[Process A calls msgget with key] --> H[Kernel creates/retrieves<br/>System V queue by key]
        H --> I[Process A calls msgsnd<br/>Kernel enqueues message with type]
        I --> J[Kernel maintains message list<br/>ordered by message type]
        K[Process B calls msgrcv<br/>Kernel dequeues by type matching]
        J --> K
        K --> L[Kernel copies message<br/>and removes from queue]
    end

    M((Message Queue Kernel Buffer)) -.-> C
    M -.-> D
    N((System V Kernel Queue)) -.-> I
    N -.-> J

Core Concepts

POSIX Message Queues

POSIX message queues use a filesystem-like API. Open a queue with mq_open(), send messages with mq_send(), and receive them with mq_receive():

#include <mqueue.h>

// Open or create a message queue
mqd_t mq = mq_open("/my_queue", O_CREAT | O_RDWR, 0666, NULL);
if (mq == (mqd_t)-1) {
    perror("mq_open");
}

// Send a message
const char *msg = "Hello, queue!";
mq_send(mq, msg, strlen(msg), 0);  // priority = 0

// Receive a message
char buf[1024];
unsigned int prio;
ssize_t len = mq_receive(mq, buf, sizeof(buf), &prio);
if (len > 0) {
    buf[len] = '\0';
    printf("Received (prio %u): %s\n", prio, buf);
}

mq_close(mq);
mq_unlink("/my_queue");  // Clean up when done

Key attributes of POSIX message queues:

• Path-based: Queues are identified by filesystem paths (must start with /)
• Priority: Messages have a priority (0-lower to higher), higher priority messages are delivered first
• Notification: Queues can notify processes when they become non-empty via signals or thread notification
• Size limits: Each queue has a max message size and max number of messages (queryable via mq_getattr())

System V Message Queues

System V message queues use integer keys and queue IDs:

#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/msg.h>

// Create or get a queue by key
int msgid = msgget(IPC_PRIVATE, IPC_CREAT | 0666);
if (msgid == -1) {
    perror("msgget");
}

// Message structure (must start with long mtype)
struct msgbuf {
    long mtype;       // Message type (must be > 0)
    char mtext[256]; // Message payload
};

struct msgbuf msg;
msg.mtype = 1;
strcpy(msg.mtext, "Hello from System V");

// Send message
if (msgsnd(msgid, &msg, sizeof(msg.mtext), 0) == -1) {
    perror("msgsnd");
}

// Receive message (mtype filter - 0 means any)
if (msgrcv(msgid, &msg, sizeof(msg.mtext), 0, 0) == -1) {
    perror("msgrcv");
}

printf("Got: %s\n", msg.mtext);

// Clean up
msgctl(msgid, IPC_RMID, NULL);

Key attributes of System V message queues:

• Key-based: Queues are identified by integer keys (can be IPC_PRIVATE for anonymous)
• Message typing: Each message has a type field used for selective receive
• Operations: Full set of control operations via msgctl() including removal, statistics
• Legacy: More portable across different Unix variants, but API feels dated

Kernel Implementation

Both POSIX and System V message queues live in the kernel as linked lists of messages. The kernel maintains:

1. Queue descriptor: Holds queue permissions, maximum message size, maximum number of messages, current message count
2. Message list: Doubly-linked list of messages, each with a header (type, size, timestamp) and payload
3. Wait queues: Processes blocked on send (when queue is full) or receive (when queue is empty) are stored here

The kernel copies messages from user space to kernel space on send, and from kernel space to user space on receive. This double-copy is the main performance overhead compared to shared memory solutions.

The kernel-side data structures make this concrete. Each message queue is represented internally by a struct msg_queue (System V) or struct mqueue (POSIX). When a process calls msgsnd() or mq_send(), the kernel allocates a message structure, copies the payload with copy_from_user(), inserts it into the priority-sorted linked list, and wakes any processes waiting in the receive wait queue. On msgrcv() or mq_receive(), the kernel unlinks the highest-priority message (or oldest at equal priority for System V), copies the payload via copy_to_user(), frees the message struct, and wakes senders blocked on a full queue.

Blocked senders and receivers are stored in wait queue entries inside the queue’s kernel struct. When the queue transitions from full to non-full (a receive frees space) or from empty to non-empty (a send adds a message), the kernel calls wake_up() on the corresponding wait queue. The processes get rescheduled but consume no CPU while blocked. With many processes blocked simultaneously, the wait queue entries themselves occupy measurable kernel memory.

The double-copy cost is the primary performance bottleneck. Each message of size N requires two memory copies of N bytes plus a context switch to enter kernel space. For high-throughput scenarios handling thousands of small messages per second, this adds up fast. Shared memory sidesteps the copy entirely by letting processes access the same physical pages directly, at the cost of more involved synchronization logic.

Production Failure Scenarios

Queue Full — Senders Block

When a message queue reaches its maximum capacity (either max bytes or max messages), a sender blocks until a receiver removes messages. In a high-throughput system with slow consumers, producers can accumulate in the kernel wait queue indefinitely.

The blocking behavior can cause cascading latency in producer processes. For a mq_send() call on a full queue, the kernel places the calling process in the send wait queue and deschedules it. The process sleeps until the receive side drains enough messages to make room. Under sustained overload, producers can block for seconds or longer.

Mitigation: Use non-blocking send (MQ_DONTWAIT flag or IPC_NOWAIT for System V), implement timeouts, monitor queue fill levels via mq_getattr() or msgctl(MSG_STAT), and implement backpressure at the application level. Set send timeouts using mq_timedsend() (absolute clock time) or poll the queue fill level before sending. In multi-threaded producers, consider a bounded thread pool where senders drop work rather than block when the queue is full — this prevents head-of-line blocking for urgent messages.

The kernel enforces queue capacity through two independent limits that interact. POSIX queues set mq_maxmsg (maximum number of messages) and mq_msgsize (maximum size of each message) at creation time. System V queues set MSGMNB (maximum bytes in a single queue) and MSGMNI (maximum number of queues system-wide). When either limit is hit, senders block or get an error — it depends on the flags used.

What surprises developers is that a queue with many small messages can become full even when the total byte count is low, and vice versa. A queue with mq_maxmsg = 100 and mq_msgsize = 1024 holds at most 100 messages even if the total byte count is far below the limit. Conversely, a queue with MSGMNB = 16384 can fill up with just a handful of oversized messages. Always check both limits when debugging queue-full behavior.

For POSIX queues, call mq_getattr() to read mq_curmsgs and compare it to mq_maxmsg. For System V, msgctl(msgid, IPC_STAT, &buf) fills a msqid_ds structure with msg_qbytes (queue byte limit) and msg_cbytes (current byte count). Polling these values before each send and logging a warning when the queue passes 80% capacity gives you an early warning before blocking occurs. For high-throughput producers, put this check on a timer rather than inline with the send path so it does not add latency to the send operation itself.

Queue Overflow — Messages Lost

System V message queues can be configured with MSGMAX (max message size) and MSGMNB (max queue bytes) limits. When a queue reaches capacity, the behavior depends on the flags used at open time. With blocking sends (the default), the sender blocks until space becomes available — no message is ever lost silently. With IPC_NOWAIT set, msgsnd() returns -1 and sets errno=EAGAIN immediately when the queue is full; the message is not queued and the caller must handle the failure.

The “messages lost” scenario takes a different shape here: if the queue has received messages and the consumer reads them, those messages are gone after consumption — they are not replicated or persisted. In a crash-recovery scenario where no one consumed the messages, the kernel persists them (for System V) until the queue is removed or the system reboots. But when a non-blocking sender fails with EAGAIN and the application does not retry or log the failure, the message disappears from the application’s perspective.

Mitigation: Monitor queue fill levels, implement retry logic with exponential backoff, use larger queue sizes, or switch to a persistent message broker. Also implement application-level acknowledgment: have the consumer send a confirmation back on a response queue after processing, so the producer knows the message was handled. This lets the producer detect and recover from silent losses.

The distinction between blocking and non-blocking behavior matters for design. Blocking sends are safe — you never lose a message — but they make your producer vulnerable to latency spikes when the consumer is slow. Non-blocking sends are safe from stalling, but they require explicit handling of the EAGAIN case. The common mistake is using non-blocking sends without that handling, which silently drops messages. If you use IPC_NOWAIT, treat EAGAIN as a first-class error: log it, retry with backoff, and consider whether the message should be written to a fallback store.

For applications that need guaranteed delivery even under load, pair a message queue with a disk-backed retry log. When msgsnd() returns EAGAIN, write the message to a local file or a backup queue. When the primary queue becomes available, drain the retry log first before accepting new messages. This gives you non-blocking sends in the normal case and durability when the queue backs up.

Permission Issues on Queue Access

POSIX message queues are accessed via filesystem paths under /dev/mqueue/. The kernel creates virtual files there for each queue, and standard Unix file permissions govern access. Several error codes can surface from permission mismatches: EACCES when the calling process lacks read or write permission on the queue; ENOENT when the queue path does not exist (queue not created yet or already unlinked); EMFILE when the process has exhausted its file descriptor limit; ENFILE when the system has hit its total file descriptor limit.

The EACCES case is especially tricky in multi-process scenarios. Process A creates the queue with 0666 permissions, but if the directory /dev/mqueue/ itself has restrictive permissions (common on hardened systems), Process B cannot traverse the directory to reach the queue file. Process B then gets EACCES even though the queue permissions are wide open.

System V queues work differently. Queue IDs are visible in /proc/sysvipc/msg (readable by root) and via ipcs -q. Anyone with access to these interfaces can enumerate active queues and attempt msgctl() operations. The queue creator’s effective UID must match the queue’s creator UID for privileged operations, but regular send and receive operations use the permission bits set at queue creation.

Mitigation: Create queues in directories with proper permissions (e.g., /var/run/myapp/), use consistent permission modes (0666 or 0660 with appropriate group), ensure the directory itself is traversable by all processes that need access, and handle EACCES, ENOENT, EMFILE, and ENFILE explicitly in error handling. For System V, restrict access to /proc/sysvipc/ in containerized environments and use msgctl() to set appropriate permission bits immediately after msgget().

The directory traversal problem is the most frequently missed part. POSIX queue permissions have two layers: the queue file itself and the directory containing it. A queue created with mode 0666 in a directory with mode 0700 is inaccessible to anyone other than the directory owner. The queue file permissions are irrelevant because no process can open the directory to reach the queue. This catches people who harden /dev/mqueue/ thinking they are improving security while breaking their own queue access. Always check both the queue path permissions and the directory permissions when debugging EACCES on POSIX queues.

For multi-tenant or containerized environments, create queues in a dedicated directory with controlled access. A directory like /var/run/myapp/ with mode 0755 and queues created with 0660 group-owned by a specific group gives you selective access without world-readability. The directory itself must be traversable by all processes that need to reach the queues, while the queues themselves control the read/write access.

Message Queue Key Conflicts (System V)

System V queues are identified by integer keys of type key_t. If two unrelated applications use the same key for different queues, they may accidentally share a queue or encounter EEXIST on creation. IPC_PRIVATE generates a unique key but then requires some external mechanism to share the queue ID with other processes.

The standard key generation function is ftok():

#include <sys/types.h>
#include <sys/ipc.h>

// Generate a key from a path and project ID character
key_t key = ftok("/var/run/myapp/queue_dir", 'M');
if (key == (key_t)-1) {
    perror("ftok failed — path does not exist");
    exit(1);
}

int msgid = msgget(key, IPC_CREAT | IPC_EXCL | 0666);
if (msgid == -1 && errno == EEXIST) {
    // Queue already exists — try to open it instead
    msgid = msgget(key, 0666);
}

The ftok() function combines the device number and inode number of the given path with the project ID character. If the filesystem is remastered, backed up and restored to a different inode, or the file is deleted and recreated, ftok() returns a different key — breaking existing queue references. This is a notoriously footgun in production systems.

Collisions also occur when different applications choose overlapping project IDs on shared systems. The project ID is a single byte, so only 256 distinct values exist. Pick a path specific to your application (a directory that persists) and a unique character from the application name to keep collisions unlikely.

Mitigation: Use a consistent key generation scheme (e.g., ftok() with a known project ID and path), use IPC_CREAT | IPC_EXCL to detect collisions, or use POSIX queues which use path-based discovery. For production systems where queue sharing across processes is needed, store the queue ID in a state file rather than calling ftok() every time the application restarts.

Orphaned Queues After Process Crash

If a process that created a queue terminates without calling mq_unlink() (POSIX) or msgctl(IPC_RMID) (System V), the queue persists in the kernel until the system is rebooted or an administrator removes it manually. For POSIX queues, the queue survives even after all processes close their file descriptors — it remains in /dev/mqueue/ until explicitly unlinked. For System V queues, the queue is marked for deletion when IPC_RMID is called, but lingers until the last process detaches.

The orphaning behavior is a double-edged sword. If a producer crashes mid-execution, queued messages survive and are available to the consumer when it restarts — useful for recovery. But if the crash scrambles the application’s sense of which queues it owns (it forgets, or creates new ones instead of reopening existing ones), the old queues pile up and consume kernel memory.

Orphaned queues are visible via ipcs -q for System V and ls /dev/mqueue/ for POSIX queues. Each orphaned queue consumes a small amount of kernel memory for its queue descriptor and any remaining messages. Under heavy churn — many crashed applications each leaving queues behind — this can add up.

System V queues created with IPC_PRIVATE are especially troublesome. The creating process is supposed to pass the queue ID to other processes through some IPC channel, but if it crashes before doing that, the queue becomes unreachable (no key, no path) and only an administrator with knowledge of the queue ID can clean it up with ipcrm.

Mitigation: Use mq_getattr() to check queue state at application startup, implement a cleanup mechanism that scans for and removes queues created by previous instances, use wrapper scripts or daemons to clean up stale queues, and monitor for orphaned queues in production. For System V, add a startup step that runs ipcrm -q <id> on any stale queues tied to your application, filtering ipcs -q output by creation time or owner. For POSIX queues, track queue names in a state file so a restart handler can unlink old queues before creating new ones.

Trade-off Table

Feature	POSIX mq_*	System V msg	Anonymous Pipe	Named Pipe (FIFO)
Identification	Filesystem path (/queue)	Integer key (msgget)	File descriptor (inheritance)	Filesystem path
Message Boundaries	Yes (preserved)	Yes (preserved)	No (byte stream)	No (byte stream)
Message Priority	Yes (0-max prio)	Yes (by mtype)	No	No
Async (no receiver needed)	Yes (messages persist)	Yes (messages persist)	No (reader required)	No (reader required)
Non-blocking ops	MQ_DONTWAIT flag	IPC_NOWAIT flag	O_NONBLOCK	O_NONBLOCK
Notification mechanism	Signal or thread callback	None (poll manually)	None	None
Kernel buffer location	Fixed-size kernel buffers	Variable-size kernel buffers	In-memory circular buffer	Same as pipe
Performance	Moderate (kernel copy)	Moderate (kernel copy)	Fast (kernel copy)	Fast (kernel copy)
Cleanup	mq_unlink()	msgctl(IPC_RMID)	Auto (last fd close)	unlink()

Implementation Snippet(s)

C: POSIX Message Queue with Notification

#include <mqueue.h>
#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
#include <errno.h>

static mqd_t g_mq;

void handle_mq_notification(int sig) {
    char buf[1024];
    unsigned int prio;
    ssize_t len;

    // Read all messages
    while ((len = mq_receive(g_mq, buf, sizeof(buf), &prio)) >= 0) {
        buf[len] = '\0';
        printf("Notification received (prio %u): %s\n", prio, buf);
    }

    // Re-arm notification
    struct sigevent sev;
    sev.sigev_notify = SIGEV_SIGNAL;
    sev.sigev_signo = SIGRTMIN;
    mq_notify(g_mq, &sev);
}

int main() {
    struct mq_attr attr = {
        .mq_flags = 0,
        .mq_maxmsg = 10,
        .mq_msgsize = 1024,
        .mq_curmsgs = 0,
    };

    g_mq = mq_open("/demo_queue", O_CREAT | O_RDWR, 0666, &attr);
    if (g_mq == (mqd_t)-1) {
        perror("mq_open");
        exit(1);
    }

    // Register for notification
    struct sigevent sev;
    sev.sigev_notify = SIGEV_SIGNAL;
    sev.sigev_signo = SIGRTMIN;
    signal(SIGRTMIN, handle_mq_notification);
    mq_notify(g_mq, &sev);

    // Block forever, handling notifications
    while (1) {
        pause();
    }

    mq_close(g_mq);
    mq_unlink("/demo_queue");
    return 0;
}

Python: POSIX Message Queue (via POSIX module)

import os
import mmap
import struct

# Note: Python's standard library doesn't have native POSIX mq support
# For production use, consider the posix_ipc module or use System V msg queues

# Example using a workaround with pipes for simple cases:
import subprocess
import signal

# Simpler approach: use a pipe with proper framing
pipe_path = "/tmp/demo_pipe"
os.mkfifo(pipe_path)

# For actual message queues in Python, use the 'posix_ipc' module:
# import posix_ipc
# mq = posix_ipc.MessageQueue("/demo_queue", posix_ipc.O_CREAT)
# mq.send("Hello", priority=5)
# msg, prio = mq.receive()

Bash: System V Message Queue Basics

# Create a System V message queue
# Using ipcs to inspect current queues
ipcs -q

# C program needed for actual usage:
# The key is generated with ftok()
# msgget(ftok("/some/path", 'A'), IPC_CREAT | 0666)

# Cleanup orphaned queues
ipcrm -q <msgid>

# Monitor queue statistics
ipcs -q -i <msgid>

Observability Checklist

• Queue existence and attributes: Use ipcs -q to list all System V message queues and their parameters
• Queue fill level: Check current number of messages and total bytes with ipcs -q or mq_getattr()
• Blocking processes: Check which processes are blocked on send/receive with ps aux | grep msgsnd/msgrcv or strace
• Queue limits: Check system-wide limits in /proc/sys/kernel/msg* parameters (System V) or mq_open() will fail with EMFILE if process limit reached
• Notification delivery: Verify signal handlers are properly installed for POSIX queue notifications
• Resource leaks: Monitor that queues are being properly closed/unlinked — orphaned queues accumulate over time
• strace/dtrace: strace -e trace=msgsnd,msgrcv,msgget,msgctl -p <pid> for System V queue operations

Common Pitfalls / Anti-Patterns

Queue Permissions & Security: Message queues respect standard Unix permissions — any user can read/write queues they have access to. Use 0660 or 0664 when creating queues to restrict access. In shared hosting environments, other users can potentially discover queue identifiers if they have read access to /proc/sysvipc/. Messages stored in kernel queues are readable by any process with appropriate permissions, so do not store sensitive data (passwords, tokens, PII) without encryption.

Queue Limits as DoS Vector: An attacker could fill all message queues (system-wide limits on total bytes and number of queues) by sending messages to public queues. Monitor /proc/sys/kernel/msgmnb, /proc/sys/kernel/msgmni, and implement proper input validation.

Audit Requirements: Message queue operations (creation, send, receive) may not generate standard filesystem audit events. Consider application-level logging for compliance requirements.

1. Not handling EAGAIN on non-blocking send — when the queue is full and IPC_NOWAIT/MQ_DONTWAIT is set, msgsnd/mq_send returns -1 with errno=EAGAIN. Ignoring this causes message loss.

2. Confusing message type with priority — System V mtype is a filter for selective receive (receive only messages of type N), not a priority. POSIX message queues have explicit priority levels.

3. Forgetting to unlink queues — POSIX queues persist until explicitly unlinked, even after all processes close them. This causes resource leaks. Always call mq_unlink() during cleanup.

4. Reading with wrong buffer size — mq_receive() returns the message length. If your buffer is smaller than mq_msgsize, the message is truncated and the call fails with EMSGSIZE. Always allocate buffers large enough for mq_msgsize.

5. Not accounting for message ordering under load — with priority queues, a steady stream of high-priority messages can starve low-priority ones. Design your priority scheme carefully.

6. Mixing POSIX and System V for the same queue — they are completely separate implementations and do not interoperate. Pick one and stick with it.

7. Assuming queue persistence across reboots — System V message queues are kernel memory and do not survive system reboots. For persistence, use a disk-backed message broker.

Quick Recap Checklist

• Message queues preserve message boundaries and support priority-based delivery
• POSIX message queues use path-based API (mq_open/send/receive/close/unlink); System V uses key-based API (msgget/msgsnd/msgrcv/msgctl)
• Messages persist in the kernel until explicitly received — sender does not need to wait for receiver
• Both implementations have kernel-enforced limits on queue size and message count
• Use non-blocking mode (IPC_NOWAIT/MQ_DONTWAIT) to avoid blocking when queue is full
• Always clean up with mq_unlink() or msgctl(IPC_RMID) to prevent orphaned queues
• Message queue notifications can be delivered via signals (POSIX) or polled manually
• For production message passing with persistence, durability, and clustering, consider dedicated message brokers (Kafka, RabbitMQ, NATS)

Interview Questions

Further Reading

• mq_overview(7) — Linux man page — POSIX message queues overview
• mq_open(3) — Linux man page — POSIX message queue API reference
• msgop(2) — Linux man page — System V message queue operations
• mq_notify(3) — Linux man page — POSIX queue notification mechanism
• Shared Memory — zero-copy IPC mechanism — Memory-mapped IPC with direct physical memory access
• Pipes & Named Pipes (FIFO) — unidirectional IPC — Anonymous and named pipe communication

Conclusion

Message queues fill the gap between pipes (streaming, connection-oriented) and shared memory (high-speed, complex to synchronize). Their persistence and asynchronous delivery make them ideal for work distribution patterns where producers and consumers operate independently in time. Both POSIX and System V implementations remain relevant today — POSIX for its cleaner API and notification mechanisms, System V for its portability and selective receive capabilities.

At scale, message queues evolve into full message brokers (Kafka, RabbitMQ, NATS) that provide persistence, durability, routing, and clustering. Understanding the kernel-level fundamentals of POSIX and System V message queues makes these higher-level systems comprehensible because you grasp what the middleware is abstracting away.

For continued learning, explore how ZeroMQ or Nanomsg provide socket-like APIs over various transport mechanisms including in-process, inter-process, and network, and study the design of durable message brokers that guarantee delivery across system restarts.