Pipes & Named Pipes (FIFO)

If you have ever typed ls | grep "error" in a terminal, you have used a pipe without even knowing it. Pipes are one of the oldest inter-process communication mechanisms in Unix-like operating systems. They embody the Unix philosophy of building complex behavior from simple, composable parts. Yet beneath that deceptively simple | operator lies a rich set of kernel-level details that every systems programmer should understand.

Introduction

A pipe is a unidirectional data channel that connects two processes. Data written to the write end of a pipe can be read from the read end. The kernel implements pipes as an in-memory circular buffer, which means the data never touches disk unless the system is under extreme memory pressure.

Anonymous pipes are the simpler variant. They are created using the pipe() system call and exist only as kernel objects — there is no filesystem representation. Because they rely on inheritance through fork(), anonymous pipes only work between processes with a common ancestor (typically a parent-child or sibling relationship).

Named pipes, also known as FIFOs (First In, First Out), are filesystem objects. You create them with mkfifo() and they persist in the filesystem just like a regular file. Any process that knows the path to a named pipe can open it for reading or writing, regardless of their ancestry. This makes FIFOs the tool of choice for communication between completely unrelated processes.

The key distinction is provenance. Anonymous pipes are for “processes I already know” — ones I fork or inherit from. Named pipes are for “processes I will eventually meet” — daemons, co-processes, and scripts that need to find each other through filesystem paths.

When to Use / When Not to Use

Use anonymous pipes when:

• You have a parent-child or sibling process relationship
• You want to stream data between a filter command and its input (shell pipelines)
• You need a simple, low-overhead one-directional channel
• The communicating processes share a common ancestor

Use named pipes when:

• You need to communicate between unrelated processes
• You want the communication channel to persist across process lifetimes
• You need multiple writers or multiple readers for the same channel
• You need to coordinate between processes that are started independently
• You want to debug communication between two long-running processes

Do not use pipes when:

• You need bidirectional communication (use a socket pair instead)
• You need to persist data across system reboots (use a regular file)
• You need to seek within the data (pipes are stream-oriented, not random access)
• You want to communicate between machines (use TCP/UDP sockets)
• You need message boundaries preserved with multiple discrete messages (use message queues)

Architecture or Flow Diagram

Anonymous Pipe Creation: pipe() System Call

The pipe() system call creates an anonymous pipe. When a process calls pipe(pipefd), the kernel allocates an in-memory circular buffer and gives you two file descriptors — pipefd[0] for reading, pipefd[1] for writing. No filesystem entry is created; the pipe lives only as a kernel object. The call returns 0 on success, -1 on failure. Calling it twice simply creates two independent pipes, so there is no danger in calling it again.

The diagram above traces the call sequence. The parent process calls pipe(), the kernel creates the buffer and descriptors, and you end up with two file descriptors with opposite semantics. Now no data has flowed — it is just two open handles. By default they are in blocking mode; use fcntl(fd, F_SETFL, O_NONBLOCK) after creation to change that.

graph LR
    A[Parent process calls pipe] --> B[Kernel creates pipe buffer<br>and two file descriptors]
    B --> C[fd0 = read end, fd1 = write end]

Fork Pattern

Anonymous pipes become useful when you combine them with fork(). The sequence matters: first the parent calls pipe() to create the buffer and two file descriptors, then it calls fork() to create a child. When fork() returns, both parent and child have a copy of the file descriptor table — meaning both now hold references to both ends of the pipe. This is where things go wrong for most developers. Having both ends open in both processes creates a bidirectional channel, which is almost never what you want.

The standard pattern for unidirectional flow is for each side to close the end it does not use. The parent keeps the write end (pipefd[1]) and closes the read end (pipefd[0]). The child does the opposite — keeps the read end, closes the write end. Once the unneeded ends are closed, the pipe becomes a clean one-way channel. The diagram above shows this: the parent writes to fd1, the child reads from fd0, and data flows in one direction.

One detail that often surprises: fork() duplicates the file descriptor table entry, not the underlying file description. Both descriptors in parent and child still point to the same kernel pipe buffer. The buffer reference count stays up until both processes close their ends. If either side leaves its unused end open, the reader will block forever waiting for EOF that never comes.

graph LR
    C[fd0 = read end, fd1 = write end] --> D[Parent forks child process]
    D --> E[Child inherits file descriptors]
    E --> F[Parent closes read end<br>Child closes write end]
    F --> G[Data flows: Parent writes to fd1<br>Child reads from fd0]

Named Pipe Creation

Named pipes (FIFOs) get around the ancestry requirement that makes anonymous pipes limiting. An anonymous pipe lives only in kernel memory and can only be reached through inherited file descriptors. A FIFO is a filesystem entry that any process can open by path. mkfifo() takes a path and a mode and returns 0 on success. Where pipe() creates something referenced by descriptors, mkfifo() creates something ls can show you.

The mode argument sets access. mkfifo("/tmp/my_fifo", 0660) makes a FIFO readable by owner and group but not the world. It respects the process umask, so passing 0666 does not give world-read/write — umask masks bits. If the FIFO already exists, mkfifo() fails with EEXIST; check for that and treat it as OK. Once created, the FIFO stays in the filesystem until you remove it with unlink().

The diagram above shows the creation flow. A process calls mkfifo(), the kernel creates the FIFO file entry, and any process that knows the path can open it. Unlike a regular file, opening a FIFO blocks until both ends are connected — this blocking-on-open behavior is the reason most FIFO bugs happen, which brings us to the next section.

graph LR
    H[Process calls mkfifo] --> I[Kernel creates FIFO file<br>in filesystem]
    I --> J[Any process with path<br>can open FIFO for R/W]

FIFO Reader/Writer Connection

The blocking behavior of open() on a FIFO catches most people off guard. When a process opens a FIFO for reading (O_RDONLY), the call blocks until some other process opens the same FIFO for writing. Opening for writing blocks until a reader appears. This chicken-and-egg problem trips up most developers the first time they use a FIFO — if the writer starts first, it blocks waiting for a reader; if the reader starts first, it blocks waiting for a writer. Neither side makes progress, and you have a deadlock.

Once you see why it happens, the fix is obvious. The most common approach is non-blocking mode with O_NONBLOCK. The read side opens and gets ENXIO until a writer appears — handle that with select() or poll(). Another approach is to coordinate startup order: always start the reader first, or create the FIFO before either process runs. Linux also lets you open with O_RDWR, which does not block — the fd works as both read and write ends, though this is Linux-specific.

The diagram above shows the blocking handshake. Both the reader and writer call open() on the same FIFO path, and both block until the other side connects. After that initial handshake, the FIFO behaves like a normal pipe — reads block when the buffer is empty, writes block when it is full, and both sides can use select() or poll() to multiplex.

graph LR
    K[Reader process] --> L[Blocks on open FIFO until writer connects]
    M[Writer process] --> L
    L --> N[Both sides now communicate]

Core Concepts

Anonymous Pipe Creation via pipe()

The pipe() system call creates an anonymous pipe and returns two file descriptors:

#include <unistd.h>

int pipefd[2];
if (pipe(pipefd) == -1) {
    perror("pipe");
    exit(1);
}
// pipefd[0] = read end
// pipefd[1] = write end

When you call pipe(), the kernel allocates a kernel buffer (typically 64KB on modern Linux systems) and creates two file objects pointing to it. The read file descriptor references the buffer’s head, and the write file descriptor references its tail.

The Fork Pattern

Anonymous pipes are almost always used with fork(). When a process forks, its file descriptor table is duplicated, so the child inherits the pipe’s file descriptors. This creates a situation where both parent and child have both ends of the pipe open — which is usually wrong. The standard pattern is for one side to close the end it does not need:

int pipefd[2];
pipe(pipefd);

pid_t pid = fork();
if (pid == 0) {
    // Child process — reads from pipe
    close(pipefd[1]);  // Close write end
    char buf[128];
    read(pipefd[0], buf, sizeof(buf));
    close(pipefd[0]);
} else {
    // Parent process — writes to pipe
    close(pipefd[0]);  // Close read end
    write(pipefd[1], "hello child", 10);
    close(pipefd[1]);
}

Pipe Capacity and Blocking

Every pipe has a finite capacity. When the buffer is full, a write() blocks until a reader drains data. On Linux, you can query and modify pipe capacity with fcntl(F_GETPIPE_SZ) and fcntl(F_SETPIPE_SZ). The default is typically 65536 bytes (one page), but it can be increased up to /proc/sys/fs/pipe-max-size (usually 1MB).

If all file descriptors referring to the write end are closed and a reader attempts to read(), it returns 0 (end-of-file). This is how the reading side detects that the writer has finished.

Partial writes trip up most developers the first time they encounter them. A write() call may succeed with fewer bytes than requested if the buffer fills up mid-write — check the return value to see what actually went through. For non-blocking I/O with O_NONBLOCK, a write that would block returns -1 with errno set to EAGAIN.

Two separate limits matter here. The pipe capacity (buffer size) controls how much data can be buffered between producer and consumer. The PIPE_BUF limit (4096 bytes on Linux) governs atomicity: writes of up to PIPE_BUF bytes are atomic and will not be interleaved with other writers’ data. Writes larger than PIPE_BUF may be split and interleaved if multiple processes write simultaneously. If your protocol requires atomic message delivery, keep individual writes below PIPE_BUF and use length-prefixed framing on the receiving end.

Named Pipes (FIFO)

A FIFO is created with mkfifo():

#include <sys/stat.h>

if (mkfifo("/tmp/my_fifo", 0666) == -1) {
    perror("mkfifo");
    // Might fail if already exists — handle appropriately
}

Once created, a FIFO appears in the filesystem and can be opened with open() just like a regular file. However, the open() call has special semantics for FIFOs:

• open() for reading blocks until another process opens the same FIFO for writing
• open() for writing blocks until another process opens the same FIFO for reading
• You can pass O_RDWR to open both ends without blocking (but this is rarely the right approach)

This blocking-on-open behavior is a common source of confusion and bugs. Both sides of a FIFO need to be opened carefully to avoid deadlocks.

Pipe Buffer Internals

On Linux, pipe buffers are managed as a circular buffer of struct pipe_buffer entries. Each buffer entry can hold up to one page (4KB) of data. When you write large amounts of data, the kernel may allocate multiple buffer slots. The pipe inode tracks these entries and their consumer/producer positions.

The pipe_buffer structure holds a pointer to the page frame containing data plus some metadata flags. The kernel allocates pipe pages on first write rather than at creation time — a brand-new pipe consumes almost no physical memory. Pages are allocated as data flows through and released once the buffer drains.

The splice() and vmsplice() system calls exploit this structure for zero-copy data transfer. splice() moves data directly between a pipe and another descriptor (like a socket or file) by swapping pipe_buffer pointers instead of copying through userspace. vmsplice() maps userspace memory pages straight into the pipe buffer, letting an application send data it already has in memory without an extra copy. Both show up in production networking stacks where per-byte copying would otherwise be a bottleneck.

You can inspect pipe buffer state via /proc/<pid>/fdinfo/<fd> — it shows current capacity and allocated pages. If you see pages allocated but throughput is low, that usually means the consumer is not draining the pipe fast enough or the pipe is undersized for your workload.

Production Failure Scenarios

Broken Pipe Signal (SIGPIPE)

When you write to a pipe whose read end has been closed, the process receives a SIGPIPE signal. By default, this terminates the process. This happens frequently in shell pipelines like yes | head -n 1 — yes writes to a pipe whose reader (head) exits after producing one line, and yes receives SIGPIPE.

Mitigation: Set SIGPIPE to SIG_IGN in your signal handlers, or use the MSG_NOSIGNAL flag on send():

signal(SIGPIPE, SIG_IGN);
// or
send(fd, data, len, MSG_NOSIGNAL);

FIFO Deadlock Due to Blocking Open

If both a reader and writer process try to open the same FIFO in blocking mode, and the reader opens first and waits for a writer (and the writer does the same), you have a classic deadlock. The open() call blocks indefinitely because each side is waiting for the other.

Mitigation: Open FIFOs in non-blocking mode, or ensure the reader runs first and creates the FIFO if it does not exist:

int fd = open("/tmp/my_fifo", O_RDONLY | O_NONBLOCK);
// Must also open write end to prevent blocking even in non-blocking mode
int wfd = open("/tmp/my_fifo", O_WRONLY | O_NONBLOCK);

Forgetting to Close Unused Pipe Ends

In the fork pattern, if you forget to close the unused end of the pipe, the reading side may never see end-of-file because there is still a writer descriptor open (even if the original writer process does not actually write). This leads to the reader blocking forever on read().

Mitigation: Be disciplined about closing both ends immediately after fork() before any data operations begin. Use a wrapper function or clearly comment which side closes which end.

The mechanism is reference counting. When fork() duplicates the file descriptor table, both parent and child hold a reference to the same pipe buffer. The kernel keeps the pipe buffer alive as long as at least one file descriptor pointing to either end remains open. When all references to both ends are closed, the kernel releases the buffer and any reader subsequently calling read() gets immediate EOF (return 0). But if the child inherits and fails to close the write end, the buffer reference count never drops to zero, even after the parent closes its write end. The child process that is supposed to be reading then blocks indefinitely because the kernel still sees an active writer.

This mistake is insidious because it produces no error at compile time, runtime, or even during initial testing. A process that forks, does not close the unused end, and then immediately writes to and reads from the pipe may work fine. Problems surface only under load or with specific timing, when the reader has been running longer than expected and the buffer fills up. Run lsof -p <pid> to check: if the reader process has both ends of the pipe open, that is your problem. The reader is in state S (interruptible sleep) on a read() that never returns.

The fix is mechanical. Immediately after fork(), before any data operations, each process should close the end it does not use. The parent closes pipefd[0] (read end) and the child closes pipefd[1] (write end). Automating this in a wrapper function prevents the error from creeping back in during maintenance:

void spawn_reader(int pipefd[2]) {
    pid_t pid = fork();
    if (pid == 0) {
        close(pipefd[1]);  // Always close write end first in child
        // Now use pipefd[0] for reading
    }
}

A common variant is closing the wrong end — parent closes write end, child closes read end — which swaps the direction. Data flows the wrong way and neither side sees what it expects. Comment each close explicitly and test with a simple pipe program before integrating into a larger pipeline.

Pipe Capacity Exhaustion

If a fast producer writes to a pipe faster than a slow consumer reads from it, the buffer fills up and the producer blocks. In a pipeline, this can cause the entire pipeline to stall — a slow consumer causes the producer to block, which causes the pipeline to back up.

Mitigation: Use pipe() with larger capacity via fcntl(F_SETPIPE_SZ), implement backpressure signals in your protocol, or use non-blocking I/O with an event loop (e.g., epoll or select).

The underlying cause is the finite buffer. A pipe is not an infinite queue — on Linux the default capacity is 65536 bytes (one page), though it can be increased to the value in /proc/sys/fs/pipe-max-size (typically 1048576 bytes). When the write end tries to write and the buffer has no free space, the kernel puts the writer process to sleep. The writer wakes only when the reader drains data from the buffer, freeing space.

What makes this tricky in practice is that the bottleneck may not be obvious. A pipeline like producer | processor | consumer stalls when the middle processor blocks on writing to its output pipe because the consumer is reading slowly. producer then blocks on writing to processor, and so on. The chain backs up from the slowest consumer. top or ps shows all pipeline stages in state S (sleeping), which can look like a deadlock but is actually a throughput mismatch.

You can detect this before it stalls by monitoring pipe buffer usage. The /proc/<pid>/fdinfo/<fd> file shows current buffer state. When these values stay near capacity between polling intervals, the pipe is a bottleneck. Raising capacity with fcntl(F_SETPIPE_SZ) helps, but only up to the system maximum. If you need more buffering, a message queue or disk-backed pipe may be more appropriate.

For real-time data pipelines, build backpressure awareness into the design. Use non-blocking I/O on all stages and an event loop that monitors both the input side and the output side. When the output pipe is full, the event loop stops reading from the input and waits. This naturally throttles the pipeline without any stage blocking indefinitely.

FIFO Permissions and Security

FIFOs created with mkfifo() respect the umask of the creating process. If you create a FIFO that is world-readable/writable, any user on the system can read or write to it. This is a security concern in multi-user environments.

Mitigation: Set appropriate permissions (e.g., mkfifo(path, 0660) for a group-owned FIFO), use filesystem ACLs, or place FIFOs in directories with restricted access.

Trade-off Table

Feature	Anonymous Pipe	Named Pipe (FIFO)	Unix Domain Socket
Creation	pipe() system call	mkfifo() system call	socketpair() or socket() + bind()
Filesystem Presence	None (kernel-only)	Yes (persistent filesystem entry)	None (kernel-only) or filesystem path for named sockets
Process Relationship	Must share ancestry (fork inheritance)	Any process (no relationship required)	Any process (no relationship required)
Bidirectional	No (unidirectional only)	No (unidirectional only)	Yes (full-duplex)
Capacity	Configurable (default ~64KB)	Same as anonymous pipe (kernel buffer)	Configurable, typically larger
Message Boundaries	None (byte stream)	None (byte stream)	None (byte stream for SOCK_STREAM) / Yes (for SOCK_DGRAM)
Persistence	Destroyed when all descriptors close	Persists until unlinked	Destroyed when all descriptors close (unnamed) or until unlinked (named)
Multiple Readers/Writers	Supported but complex	Supported but complex	Supported natively
Typical Use Case	Shell pipelines, parent-child streaming	Daemon communication, cross-process coordination	Network IPC, full-duplex local communication

Implementation Snippet(s)

C: Fork-based Anonymous Pipe

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/wait.h>

int main() {
    int pipefd[2];
    if (pipe(pipefd) == -1) {
        perror("pipe");
        exit(1);
    }

    pid_t pid = fork();
    if (pid == -1) {
        perror("fork");
        exit(1);
    }

    if (pid == 0) {
        // Child: read from pipe
        close(pipefd[1]);  // Close write end
        char buf[256];
        ssize_t n = read(pipefd[0], buf, sizeof(buf) - 1);
        if (n > 0) {
            buf[n] = '\0';
            printf("Child received: %s\n", buf);
        }
        close(pipefd[0]);
        _exit(0);
    } else {
        // Parent: write to pipe
        close(pipefd[0]);  // Close read end
        const char *msg = "Hello from parent!";
        write(pipefd[1], msg, strlen(msg));
        close(pipefd[1]);
        wait(NULL);  // Wait for child to finish
    }

    return 0;
}

Python: Using Named FIFO

import os
import time
import errno

FIFO_PATH = "/tmp/my_python_fifo"

# Create FIFO if it doesn't exist
try:
    os.mkfifo(FIFO_PATH)
except FileExistsError:
    pass

# Writer process
def writer():
    with open(FIFO_PATH, 'w') as fifo:
        fifo.write("Message via named pipe\n")
        fifo.flush()

# Reader process (in practice, run this as a separate process)
def reader():
    with open(FIFO_PATH, 'r') as fifo:
        line = fifo.readline()
        print(f"Reader received: {line.strip()}")

# To test: open two terminal sessions
# Terminal 1: python3 reader.py
# Terminal 2: python3 writer.py

Bash: Anonymous Pipe in Shell

#!/bin/bash
# Classic shell pipeline
cat /var/log/syslog | grep "error" | head -n 20

# Programmatically create and use a pipe
mkfifo /tmp/my_pipe

# In one terminal
# cat /tmp/my_pipe

# In another terminal
# echo "Hello from bash FIFO" > /tmp/my_pipe

# Cleanup
rm /tmp/my_pipe

Observability Checklist

• Process state: Check if processes are blocked on pipe read/write using ps aux or lsof -p <pid>
• File descriptors: Monitor open file descriptors for pipes with ls -la /proc/<pid>/fd/
• Pipe capacity: Query current pipe size with fcntl(fd, F_GETPIPE_SZ) in C or check via /proc/<pid>/fdinfo/
• SIGPIPE handling: Verify signal handler is set (cat /proc/<pid>/status | grep SigPnd)
• FIFO existence: Check if FIFOs exist in expected locations with ls -la /tmp/*.fifo or similar
• strace: Use strace -e trace=pipe,mkfifo,read,write -p <pid> to trace pipe operations
• dtrace/SystemTap: Trace pipe create/open/read/write events across the system

Common Pitfalls / Anti-Patterns

Permissions & Security: FIFOs respect the standard Unix permission model. Ensure the FIFO is created with appropriate access controls — world-readable FIFOs allow any local user to participate in the communication channel. Use mode 0660 or 0664 with a specific group when possible. Unlike sockets, FIFOs are filesystem objects and their content can be inspected with ls. If you are communicating sensitive data, ensure the FIFO resides in a protected directory (e.g., /var/run/myapp/) with restricted permissions.

Denial of Service: A malicious process could open a FIFO for reading or writing and never close it, causing the legitimate counterpart to block on open(). Use non-blocking open (O_NONBLOCK) and implement timeouts for all I/O operations.

No Authentication: Pipes and FIFOs provide no built-in authentication. Any process that can reach the pipe (by file descriptor inheritance or by path) can read or write data. Implement your own application-layer authentication if needed.

Audit Trail: Pipe operations do not generate filesystem audit events by default (since FIFOs are not regular files). Monitor process-level audit logs for access patterns.

1. Forgetting to close unused pipe ends — causes readers to hang forever waiting for EOF that never comes because a write end is still open somewhere.

2. Assuming read returns the same write unit — write() of 100 bytes does not guarantee read() of 100 bytes. It may return 50, then another 50, or 1 byte at a time. Always handle partial reads.

3. Ignoring SIGPIPE — writing to a closed pipe kills your process by default. Set signal(SIGPIPE, SIG_IGN) or use MSG_NOSIGNAL.

4. Blocking on FIFO open without a counterpart — opening a FIFO in blocking mode hangs if no other process has the other end open. Use non-blocking mode or coordinate startup order.

5. Using pipes for message-oriented data — pipes are byte streams with no message boundaries. If you send discrete messages, you need to frame them yourself (length prefix, delimiter, etc.).

6. Not checking for errors — pipe(), mkfifo(), read(), and write() can all fail. Always check return values.

7. Leaving FIFOs in filesystem after crash — if a process creating a FIFO terminates abruptly, the FIFO may remain. Implement cleanup on startup or use a process ID in the filename.

Quick Recap Checklist

• Pipes are unidirectional byte streams implemented as in-memory circular buffers in the kernel
• Anonymous pipes (pipe()) require a common ancestor (fork inheritance); named pipes (mkfifo()) work between any processes via filesystem path
• Always close the unused end of a pipe in both parent and child after fork() to avoid hangs
• Set SIGPIPE to SIG_IGN or use MSG_NOSIGNAL to prevent unexpected process termination
• FIFOs block on open() until both ends are connected — use O_NONBLOCK or coordinate startup order
• Pipes are for streaming data; they have no message boundaries — implement your own framing for discrete messages
• Pipe capacity is finite (default ~64KB on Linux) — large writes will block when buffer is full
• FIFOs persist in the filesystem and require cleanup (unlink) when no longer needed
• Named pipes enable unrelated processes to communicate but lack built-in authentication

Interview Questions

Further Reading

• Unix Pipes and FIFOs — Linux man pages — Official documentation for named pipes
• pipe(2) — Linux man page — System call reference for pipe creation
• select(2) — Linux man page — Multiplexing I/O across file descriptors
• epoll(7) — Linux man page — Linux-specific scalable I/O event notification
• Signals — IPC mechanism for async notifications — Signal handling and async process communication
• Sockets & Network IPC — bidirectional communication endpoints — Unix domain and TCP/UDP sockets for IPC

Conclusion

Pipes and FIFOs are foundational IPC primitives in Unix systems — simple in concept but packed with kernel-level details. Anonymous pipes excel at connecting related processes in parent-child or pipeline arrangements, while FIFOs extend that capability to unrelated processes through the filesystem. Their byte-stream nature makes them ideal for streaming data, though the lack of message boundaries means you need careful protocol design when discrete messages are involved.

As systems become more distributed, pipes give way to socket-based IPC for cross-machine communication. But on a single host, pipes remain among the fastest mechanisms available — data moves through kernel-managed buffers without per-byte system call overhead. Understanding pipe semantics — blocking behavior, capacity limits, SIGPIPE handling — sets you up for all forms of stream-oriented communication.

For continued learning, explore Unix domain sockets (which offer bidirectional communication while keeping pipe-like efficiency), and study how select()/poll()/epoll() APIs let you multiplex multiple pipes within a single thread.