Sockets & Network IPC

If pipes and message queues are the local delivery trucks of the IPC world, sockets are the postal service — they can deliver data not just between processes on the same machine, but across the network to any reachable host. Sockets are the most versatile and widely-used form of IPC on Unix systems, and understanding them is essential for every systems programmer. Whether you are building a web server, a database client, a microservice communication layer, or a local daemon, sockets are the foundational building block.

Introduction

A socket is a bidirectional communication endpoint. Unlike pipes which are unidirectional and unnamed, sockets provide bidirectional, connection-oriented (TCP) or connectionless (UDP) communication that can be local (Unix domain) or network-based (TCP/IP, UDP/IP).

There are two main families of sockets:

Unix domain sockets (AF_UNIX / AF_LOCAL) — Use filesystem paths as addresses. Data never leaves the kernel. Used for local inter-process communication with TCP-like or UDP-like semantics. Comparable in speed to shared memory for many workloads.

Internet domain sockets (AF_INET / AF_INET6) — Use IP addresses and port numbers. Data flows through the full network stack. Used for network communication between processes on different hosts.

Within each family, there are two main protocols:

SOCK_STREAM (TCP) — Connection-oriented, reliable, byte-stream, no message boundaries. Similar to a pipe but bidirectional.

SOCK_DGRAM (UDP) — Connectionless, unreliable, message-oriented with preserved boundaries. Each send delivers a discrete message.

The socket API was originally developed for BSD Unix and standardized in POSIX. It consists of socket(), bind(), listen(), accept(), connect(), send(), recv(), close(), and related functions.

When to Use / When Not to Use

Use Unix domain sockets when:

• You need IPC between processes on the same machine with TCP-like semantics
• You need bidirectional communication
• You want a simpler alternative to shared memory (with built-in synchronization at the kernel level)
• You need to use select()/poll()/epoll for multiplexing multiple connections
• You need a connection-oriented channel with backpressure (TCP flow control)

Use TCP sockets when:

• You need network communication between different machines
• You need reliable, ordered, connection-oriented delivery
• You need to handle many concurrent connections efficiently

Use UDP sockets when:

• You need low-latency communication and can tolerate some packet loss
• You are building systems that handle brief disconnections gracefully
• You are doing broadcast or multicast communication

Do not use sockets when:

• You need maximum throughput for local communication (shared memory may be faster)
• You need simple unidirectional streaming (pipes are simpler)
• You need message queue semantics with priorities (message queues fit better)
• You are communicating between threads in the same process (use condition variables or channels)

Architecture or Flow Diagram

Server Socket Setup

Server side socket setup starts by creating a file descriptor, binding it to a filesystem path or port, then marking it ready to accept connections. The diagram below shows each kernel call and what it returns.

sequenceDiagram
    participant S as Server
    participant K as Kernel (Socket Layer)

    S->>K: socket(AF_UNIX, SOCK_STREAM, 0)
    K-->>S: fd = socketfd

    S->>K: bind(socketfd, "/tmp/mysock")
    S->>K: listen(socketfd, backlog=5)

Client Connection & Accept

Client side is simpler: create a socket, then call connect to reach the server’s address. The kernel completes the three-way handshake and drops the finished connection into the server’s accept queue.

sequenceDiagram
    participant S as Server
    participant K as Kernel (Socket Layer)
    participant C as Client

    Note over S,C: TCP Server-Client Flow (Unix Domain or TCP)

    C->>K: socket(AF_UNIX, SOCK_STREAM, 0)
    C->>K: connect(socketfd, "/tmp/mysock")

    K-->>S: Notification: new connection
    S->>K: accept(socketfd)
    K-->>S: connfd = new connection socket

    Note right of K: Kernel maintains accept queue

Data Exchange

Once connected, both sides exchange data through send and recv. The kernel buffers data on each end, so writes go into the send buffer and reads pull from the receive buffer. Neither side has to wait for the other.

sequenceDiagram
    participant S as Server
    participant K as Kernel (Socket Layer)
    participant C as Client

    C->>K: send(data)
    K-->>C: 15 bytes accepted

    K-->>S: data available on connfd
    S->>K: recv(connfd, buf, 1024)
    K-->>S: "Hello from client"

    S->>K: send(connfd, "Hi back", 8)
    K-->>C: "Hi back"

    Note over S,C: close both sockets when done

Core Concepts

Unix Domain Socket Creation and Binding

#include <sys/socket.h>
#include <sys/un.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>

int main() {
    int server_fd = socket(AF_UNIX, SOCK_STREAM, 0);
    if (server_fd == -1) {
        perror("socket");
        exit(1);
    }

    struct sockaddr_un addr;
    memset(&addr, 0, sizeof(addr));
    addr.sun_family = AF_UNIX;
    strncpy(addr.sun_path, "/tmp/my_socket", sizeof(addr.sun_path) - 1);

    // Remove existing socket file (avoid EADDRINUSE)
    unlink("/tmp/my_socket");

    if (bind(server_fd, (struct sockaddr *)&addr, sizeof(addr)) == -1) {
        perror("bind");
        exit(1);
    }

    if (listen(server_fd, 5) == -1) {
        perror("listen");
        exit(1);
    }

    printf("Server listening on %s\n", addr.sun_path);

    // Accept a connection
    int client_fd = accept(server_fd, NULL, NULL);
    if (client_fd == -1) {
        perror("accept");
        exit(1);
    }

    char buf[256];
    ssize_t n = recv(client_fd, buf, sizeof(buf) - 1, 0);
    if (n > 0) {
        buf[n] = '\0';
        printf("Received: %s\n", buf);
    }

    send(client_fd, "Hello from server", 17, 0);

    close(client_fd);
    close(server_fd);
    unlink("/tmp/my_socket");

    return 0;
}

Socket Pairs (Anonymous Connected Sockets)

A socket pair is a pair of connected sockets where data written to one can be read from the other. Created with socketpair(), they are useful for creating bidirectional communication channels between related processes:

#include <sys/socket.h>
#include <unistd.h>
#include <stdio.h>
#include <string.h>

int main() {
    int sv[2];  // Two connected sockets

    if (socketpair(AF_UNIX, SOCK_STREAM, 0, sv) == -1) {
        perror("socketpair");
        exit(1);
    }

    pid_t pid = fork();
    if (pid == 0) {
        // Child: close write end, read from read end
        close(sv[1]);
        char buf[128];
        ssize_t n = recv(sv[0], buf, sizeof(buf), 0);
        if (n > 0) {
            buf[n] = '\0';
            printf("Child received: %s\n", buf);
        }
        close(sv[0]);
        _exit(0);
    } else {
        // Parent: close read end, write to write end
        close(sv[0]);
        send(sv[1], "Hello from parent!", 18, 0);
        close(sv[1]);
        wait(NULL);
    }

    return 0;
}

TCP Server with select() Multiplexing

#include <sys/socket.h>
#include <netinet/in.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

#define PORT 8080
#define MAX_CLIENTS 10

int main() {
    int server_fd = socket(AF_INET, SOCK_STREAM, 0);
    int opt = 1;
    setsockopt(server_fd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));

    struct sockaddr_in addr;
    memset(&addr, 0, sizeof(addr));
    addr.sin_family = AF_INET;
    addr.sin_addr.s_addr = INADDR_ANY;
    addr.sin_port = htons(PORT);

    bind(server_fd, (struct sockaddr *)&addr, sizeof(addr));
    listen(server_fd, 5);

    fd_set readfds;
    int client_fds[MAX_CLIENTS] = {0};

    while (1) {
        FD_ZERO(&readfds);
        FD_SET(server_fd, &readfds);
        int maxfd = server_fd;

        for (int i = 0; i < MAX_CLIENTS; i++) {
            if (client_fds[i] > 0) {
                FD_SET(client_fds[i], &readfds);
                if (client_fds[i] > maxfd) maxfd = client_fds[i];
            }
        }

        int activity = select(maxfd + 1, &readfds, NULL, NULL, NULL);
        if (activity < 0) perror("select");

        // New connection?
        if (FD_ISSET(server_fd, &readfds)) {
            int client_fd = accept(server_fd, NULL, NULL);
            for (int i = 0; i < MAX_CLIENTS; i++) {
                if (client_fds[i] == 0) {
                    client_fds[i] = client_fd;
                    break;
                }
            }
        }

        // Client data?
        for (int i = 0; i < MAX_CLIENTS; i++) {
            if (client_fds[i] > 0 && FD_ISSET(client_fds[i], &readfds)) {
                char buf[1024];
                ssize_t n = recv(client_fds[i], buf, sizeof(buf), 0);
                if (n <= 0) {
                    close(client_fds[i]);
                    client_fds[i] = 0;
                } else {
                    // Echo back
                    send(client_fds[i], buf, n, 0);
                }
            }
        }
    }

    return 0;
}

Production Failure Scenarios

EADDRINUSE — Socket Already Bound

EADDRINUSE means bind() was called on an address that is already claimed. For TCP/UDP ports, something is already listening. For Unix domain sockets, the socket file path already exists on the filesystem — even if no process currently has it open. The usual culprit is a server that crashed or got killed, leaving the kernel still holding the address while the socket file lingers.

Set SO_REUSEADDR before calling bind() to allow rebinding to an address in TIME_WAIT:

int opt = 1;
setsockopt(server_fd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));
bind(server_fd, (struct sockaddr *)&addr, sizeof(addr));
// bind() now succeeds even if the previous socket on this port is in TIME_WAIT

The behavior differs by socket type:

For Unix domain sockets, SO_REUSEADDR is not enough on its own. You also need to unlink() the path before rebinding if the file still exists. The kernel releases the address on close(), but the filesystem entry sticks around until you remove it:

unlink("/tmp/my_socket");  // Remove stale socket file before rebinding
bind(server_fd, (struct sockaddr *)&addr, sizeof(addr));

Skip the unlink() and you get EADDRINUSE regardless of SO_REUSEADDR. This trips up a lot of people restarting Unix domain socket servers during development.

For TCP servers, SO_REUSEADDR does not bypass TIME_WAIT safety — it only permits binding to an address in that state. Incoming packets for the old connection still route correctly. The kernel holds the port for 60 seconds (2 * MSL on Linux) to drain delayed packets; SO_REUSEADDR is the standard workaround for restarts within that window.

Connection Refused Under Load

When a server is overwhelmed with connections, the listen queue (backlog) fills up. New connections are refused with ECONNREFUSED or silently dropped (depending on OS). Set an appropriate backlog and monitor queue depth.

The kernel maintains two queues per listening socket: the SYN queue (incomplete connections that have received SYN but not yet a matching SYN-ACK) and the accept queue (completed connections that have finished the three-way handshake and are waiting for accept() to be called). When listen(fd, backlog) is called, the backlog argument sets the maximum size of the accept queue. The actual cap is the lesser of your backlog value and somaxconn (Linux default: 4096). When the accept queue is full, the kernel silently drops the final ACK from the client, causing clients to retransmit until they timeout or give up.

Under heavy load, your server appears to refuse connections even though it is running. Clients see ECONNREFUSED or timeouts. This is distinct from EMFILE (file descriptor exhaustion) and from port exhaustion in the TIME_WAIT state.

Mitigation: Increase the listen backlog with listen(fd, backlog). Set it to a value that handles burst traffic without masking a real overload condition. For high-throughput servers, values between 256 and 1024 are common. You can view and set somaxconn system-wide:

# View current cap
cat /proc/sys/net/core/somaxconn

# Set it (requires root)
echo 4096 > /proc/sys/net/core/somaxconn

Setting a large backlog does not help if your application cannot call accept() fast enough. The queue just fills with completed connections waiting there. If ss -ltn shows a consistently full queue under load, your application-level processing is the bottleneck, not the queue size. Implement connection limiting at the application level and use load balancing to distribute connection attempts across multiple server processes.

Socket Leak — File Descriptors Not Closed

Every socket is represented as a file descriptor, and file descriptors are a finite system resource. On Linux, the default limit per process is 1024 (viewable with ulimit -n or getrlimit(RLIMIT_NOFILE, ...)). Root processes can raise this, but it remains bounded. If sockets are not closed in all code paths — especially error branches — file descriptors accumulate until the process hits its limit. At that point, socket() and open() fail with EMFILE: Too many open files.

Socket leaks commonly occur in two scenarios. First, in error handling paths: after socket() succeeds but before connect() or bind() succeeds, the fd must be closed. Forgetting a close on any error return leaves a leak. Second, in long-running servers: each accepted connection produces a new socket fd. If the server does not close the client fd after handling (e.g., in a non-bugging fork pattern), leaked fds accumulate with every request.

Mitigation: Always close sockets in all code paths. The robust pattern is a cleanup goto block:

int fd = -1;
int server_fd = socket(AF_INET, SOCK_STREAM, 0);
if (server_fd == -1) {
    perror("socket");
    exit(1);
}

if (bind(server_fd, ...) == -1) {
    perror("bind");
    goto cleanup_server;
}

if (listen(server_fd, 5) == -1) {
    perror("listen");
    goto cleanup_server;
}

// ... accept and handle connections ...

cleanup_server:
    close(server_fd);
// Each error site does: goto cleanup_server;
// No duplicate close calls needed

Monitor file descriptor usage in real time:

# List open fds for a process
ls /proc/<pid>/fd/ | wc -l

# Show which files/sockets are open
lsof -p <pid>

# Check the system-wide and per-process limits
cat /proc/sys/fs/file-max # system-wide limit
ulimit -n                          # soft limit for shell
cat /proc/<pid>/limits # hard/soft limits for a process

Watch for a growing fd count over time — a stable server’s fd count should be constant under steady load. If it climbs, you have a leak somewhere in your connection lifecycle.

Partial Send / recv

send() may transmit fewer bytes than requested if the kernel’s send buffer is full (especially on non-blocking sockets). recv() may return fewer bytes than requested. Always check return values and handle partial operations.

On blocking sockets, send() blocks until all data is in the kernel buffer or an error occurs, but the return value may still be less than requested if the buffer fills mid-call. On non-blocking sockets, send() returns immediately with either the number of bytes accepted or EAGAIN/EWOULDBLOCK if the buffer is full. recv() similarly returns what’s available at the moment, which may be fewer bytes than requested even on blocking sockets if data arrives in fragments.

The consequence of not handling partial operations is protocol corruption. If you send a 100-byte message and only 50 bytes arrive, the receiver must know whether to wait for the remaining 50 or treat the partial as complete. Without a framing protocol, both sides go out of sync.

Mitigation: Loop until all data is sent. Use a sendall() wrapper that handles partial sends and EINTR (interrupted system call):

ssize_t sendall(int sockfd, const void *buf, size_t len) {
    size_t total = 0;
    while (total < len) {
        ssize_t n = send(sockfd, (const char *)buf + total, len - total, 0);
        if (n == -1) {
            if (errno == EINTR) continue;  // Retry on signal
            return -1;
        }
        total += n;
    }
    return total;
}

For recv(), accumulate bytes until a complete message is received. Choose a framing strategy:

• Fixed-length: Every message is exactly N bytes. Simple but wastes bandwidth for variable-size data.
• Length prefix: Prepend a fixed-size length field (e.g., 4-byte uint32_t in network byte order) before each message. The receiver reads the length first, then reads exactly that many bytes.
• Delimiter-based: A special byte sequence (e.g., \n for line-oriented protocols) marks message boundaries. Beware of delimiter injection attacks.

Which framing you pick depends on your protocol. Length prefix is the most common for binary protocols. Delimiter-based is common for text protocols like HTTP, though you need to guard against delimiter injection attacks.

UDP Packet Loss and Reordering

UDP has no built-in mechanism for detecting lost packets, reordering out-of-order delivery, or deduplicating. When you send a datagram, the kernel hands it to the IP layer and walks away — no confirmation, no retry, no reordering. For DNS queries, video streams, or game state updates, this is usually fine. For other applications, it is a problem you have to solve yourself.

Packet loss happens when a router’s output buffer fills and it drops the datagram. The sender never finds out unless the application layer handles it. Reordering happens when packets take different paths — a datagram sent after another one can arrive first. Duplication is less common but real, usually from network retransmissions firing duplicates into the mix.

Building reliability on top of UDP means adding four things at the application layer:

1. Sequence numbers — Assign a monotonically increasing number to each datagram. The receiver tracks the last number seen and can spot gaps (missing packets) or duplicates.
2. Acknowledgments — The receiver sends an ACK when it gets a valid packet. If the sender does not see an ACK within a timeout, it retransmits.
3. Retransmission with backoff — After a timeout, resend. After repeated failures, back off exponentially so you do not make congestion worse.
4. Reordering buffer — The receiver holds onto out-of-order packets briefly, waiting for gaps to fill before passing data up to the application.

Reliability patterns:

For most applications, just use TCP. The complexity of building a reliable UDP protocol is hard to justify when TCP already does the job. But when you need custom congestion control tuned to your workload — real-time audio that tolerates a dropped packet but not a latency spike — or when you need multicast delivery to multiple peers, application-layer reliability on UDP is the only path. QUIC (the transport behind HTTP/3) and WebRTC’s data channel both sit on UDP for exactly this reason.

One thing to watch: a sender that retransmits aggressively on a congested link will make packet loss worse, not better. Track round-trip time using timestamps or sequence number timing, and pace your transmissions to stay within estimated available bandwidth. For non-critical logging or metrics over UDP, none of this matters — fire and forget.

Trade-off Table

Implementation Snippet(s)

Python: Unix Domain Socket Client

Python’s built-in socket module talks to the BSD socket API so you don’t manage file descriptors directly. This client connects to a Unix domain socket, sends a message, and prints whatever the server sends back.

import socket
import os

SOCKET_PATH = "/tmp/my_socket"

# Create Unix domain socket
client = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)

try:
    client.connect(SOCKET_PATH)
    client.sendall(b"Hello from Python client")
    data = client.recv(1024)
    print(f"Server said: {data.decode()}")
except ConnectionRefusedError:
    print("Server not running")
except Exception as e:
    print(f"Error: {e}")
finally:
    client.close()

Bash: Using netcat for Socket Testing

netcat (nc) is a command-line tool for probing socket connections during debugging. The examples below show Unix domain sockets, TCP servers, and some common diagnostic commands.

# Connect to a Unix domain socket (Linux only)
# nc -U /tmp/my_socket

# Listen on a Unix domain socket
# nc -l -U /tmp/my_socket

# Test TCP server
# nc localhost 8080

# Send HTTP request to test server
# echo -e "GET / HTTP/1.0\r\n\r\n" | nc localhost 80

# Check what is listening on TCP ports
ss -tlnp | grep :8080
netstat -tlnp | grep :8080

Observability Checklist

• Open sockets: lsof -p <pid> shows all file descriptors including sockets
• Listening ports: ss -tlnp (preferred over netstat on modern Linux) shows listening sockets with process info
• Established connections: ss -tnp shows all established TCP connections
• Socket buffers: Check with cat /proc/sys/net/core/rmem_default and /proc/sys/net/core/wmem_default
• Connection state: ss -ti shows TCP connection state, retransmissions, congestion window
• strace: strace -e trace=bind,listen,accept,connect,send,recv,close -p <pid> to trace socket operations
• perf: perf stat -e syscalls:sys_enter_bind,syscalls:sys_enter_connect to measure socket call frequency

Common Pitfalls / Anti-Patterns

Socket Permissions & Network Exposure: Unix domain sockets respect filesystem permissions on the socket file path — use appropriate permissions on the containing directory (0700 for sensitive IPC). TCP/UDP sockets bound to 0.0.0.0 or :: are network-accessible; bind to localhost (127.0.0.1 or ::1) for local-only access and use firewall rules for additional protection. Local processes can potentially sniff Unix domain socket traffic if they have access to the socket path — use filesystem permissions and separate namespaces for sensitive IPC.

DoS via Connection Flood & Audit: An attacker can exhaust server resources by opening many connections (SYN flood for TCP, connection flood for SOCK_STREAM). Use connection limits, rate limiting, proper timeout configuration, and consider a load balancer or SYN cookies. Socket operations (bind, listen, connect) generate standard audit events on most Linux distributions — for compliance, monitor for unexpected socket creation or binding to unusual ports.

1. Not handling EINTR on socket calls — same as pipes and other blocking calls, socket operations can return EINTR. Handle it or use SA_RESTART.

2. Ignoring partial send/recv — send() and recv() may process fewer bytes than requested. Loop until all data is transferred.

3. Forgetting SO_REUSEADDR — not setting this causes “Address already in use” errors after server restart, especially during development.

4. Buffer overflow in recv — always bounds-check the buffer size. Malicious clients can send more data than expected.

5. Using UDP for reliable data — UDP makes no guarantee of delivery, order, or uniqueness. If you need reliability on top of UDP, implement sequence numbers, ACKs, and retransmission.

6. Not setting socket timeouts — default socket operations may block forever. Set SO_RCVTIMEO and SO_SNDTIMEO for production code.

7. Leaving sockets in TIME_WAIT too long — after closing a connection, the kernel holds the port in TIME_WAIT state. Use SO_REUSEADDR to allow rebinding, or design your protocol to use longer-lived connections.

8. Mixing select/poll with edge-triggered epoll — if using epoll() in edge-triggered mode and not draining all pending data, you may miss events. Use level-triggered mode or drain completely.

Quick Recap Checklist

• Sockets provide bidirectional IPC for both local (AF_UNIX) and network (AF_INET) communication
• SOCK_STREAM is connection-oriented, reliable, byte-stream (like a bidirectional pipe)
• SOCK_DGRAM is connectionless, unreliable, message-oriented (preserves boundaries)
• Unix domain sockets (AF_UNIX) are the fastest local IPC mechanism with full socket API features
• Always set SO_REUSEADDR before bind() to handle server restarts gracefully
• Handle EINTR on all socket calls and partial send/recv by looping
• Use select()/poll()/epoll() for multiplexing many connections in a single thread
• Monitor socket file descriptor usage to prevent leaks; always close in all code paths
• UDP requires application-level reliability if your use case needs it
• Socket buffer sizes affect performance — tune with SO_RCVBUF and SO_SNDBUF

Interview Questions

int opt = 1;
setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));
bind(sockfd, ...);

ssize_t sendall(int sockfd, const void *buf, size_t len) {
    size_t total = 0;
    while (total < len) {
        ssize_t n = send(sockfd, (const char *)buf + total, len - total, 0);
        if (n == -1) {
            if (errno == EINTR) continue;  // Retry
            return -1;
        }
        total += n;
    }
    return total;
}

// Read exactly len bytes
ssize_t recvall(int sockfd, void *buf, size_t len) {
    size_t total = 0;
    while (total < len) {
        ssize_t n = recv(sockfd, (char *)buf + total, len - total, 0);
        if (n == 0) return total;  // EOF
        if (n == -1) {
            if (errno == EINTR) continue;
            return -1;
        }
        total += n;
    }
    return total;
}

Further Reading

• socket(7) — Linux man page — Socket API overview and options
• unix(7) — Linux man page — Unix domain socket API
• tcp(7) — Linux man page — TCP socket options and behavior
• epoll(7) — Linux man page — Scalable I/O event notification for sockets
• OS Networking Stack — network protocol layers — TCP/IP protocol stack and network layer internals
• Shared Memory — high-speed zero-copy IPC — Memory-mapped IPC for maximum throughput

Conclusion

Sockets represent the pinnacle of IPC flexibility—bidirectional, connection-oriented or connectionless, local or network-spanning. Unix domain sockets provide highest-performance local communication, while TCP/UDP sockets extend that capability across network boundaries. The layered API (socket, bind, listen, accept, connect, send, recv) has remained remarkably stable since BSD introduced it decades ago.

Mastering socket programming means mastering edge cases: partial sends and receives requiring loops, EINTR handling for interrupted system calls, TIME_WAIT state and SO_REUSEADDR for server restarts, and appropriate use of select/poll/epoll for scalable multiplexing. These fundamentals apply whether you’re building a local daemon, a network service, or a distributed system.

Looking forward, the evolution continues with zero-copy socket APIs, io_uring integration for asynchronous I/O, and increasingly sophisticated offloading to network hardware. The socket API adapts, but the underlying principles of bidirectional communication endpoints and kernel-mediated data transfer remain constant.