Distributed Transactions: ACID vs BASE Trade-offs

Introduction

In a single database, transactions are straightforward. The database handles atomicity, isolation, consistency, and durability (ACID). When you commit, it is committed. When you read data, it is the latest.

In distributed systems, this simplicity breaks. Data lives across multiple databases, services, and nodes. Keeping everything consistent requires coordination—and coordination has costs: latency, availability, and complexity.

This post explores trade-offs between consistency models, from strict ACID to relaxed eventual consistency, and how to choose the right model for your system.

Topic-Specific Deep Dives

Consistency Models

ACID vs BASE

These are two philosophies for handling distributed data.

ACID (Atomicity, Consistency, Isolation, Durability) prioritizes consistency above all else. Every transaction sees a consistent state. The database will wait (or fail) to maintain consistency.

BASE (Basically Available, Soft state, Eventually consistent) prioritizes availability. The system may return stale data temporarily, but given enough time, all replicas converge to the same value.

flowchart TD
    A[ACID] -->|Strong consistency| B[Reads always see latest writes]
    A -->|Isolation| C[Concurrent tx look serial]
    A -->|Durability| D[Committed data survives crashes]

    E[BASE] -->|Availability| F[System stays available]
    E -->|Eventual| G[Data converges over time]
    E -->|Soft state| H[State may change without input]

ACID is what you want for financial transactions. BASE is what you get with many NoSQL databases and distributed caches.

The CAP Theorem

Eric Brewer’s CAP theorem says you can have at most two of three: Consistency, Availability, and Partition tolerance. Since network partitions happen in real systems, you must choose between consistency and availability.

flowchart LR
    C[Consistency] -->|CP| DB[(Database)]
    C -->|CA| DB
    A[Availability] -->|AP| DB
    P[Partition Tolerance] --> C
    P --> A

A CP system (like ZooKeeper, etcd) will refuse to serve requests if it cannot confirm consistency. An AP system (like Cassandra, DynamoDB) will serve requests even during partitions but may return stale data.

Most real systems choose AP and use compensating transactions (like saga) to handle inconsistency.

Distributed Protocols

Consensus Algorithms

When 2PC’s coordinator fails, participants are left in limbo. Consensus algorithms solve this by using a quorum-based approach where any node can take over if the leader fails.

Paxos, developed by Leslie Lamport, uses two phases:

1. Prepare phase: A node proposing a value sends a prepare request to a quorum of nodes. Nodes promise not to accept requests with a lower proposal number.
2. Accept phase: If a majority responds, the proposer sends an accept request. Nodes accept if they haven’t seen a higher-numbered proposal.

Paxos guarantees safety (no two proposals can be chosen for the same value) but is notoriously difficult to implement correctly. Multi-Paxos optimizes for the common case where one node is the stable leader.

Raft was designed as a more understandable alternative to Paxos. It separates concerns into three sub-problems:

• Leader election: Nodes vote for a leader using randomized election timeouts. The first node to timeout wins and becomes leader.
• Log replication: The leader accepts entries and replicates them to followers via a heartbeat pipeline.
• Safety: If a leader fails, only nodes with enough log entries can become the new leader.

flowchart TD
    A[Node A<br/>Candidate] -->|RequestVote| B[Node B<br/>Follower]
    A -->|RequestVote| C[Node C<br/>Follower]
    B -->|Vote granted| A
    C -->|Vote granted| A
    A -->|Elected Leader| D[Node A<br/>Leader]
    D -->|Heartbeat| B
    D -->|Heartbeat| C

Comparison: ZooKeeper and etcd use Raft. Google Spanner uses Paxos (specifically Multi-Paxos with TrueTime). CockroachDB uses a Raft variant called Raft + HLC.

Two-Phase Commit

Two-phase commit (2PC) coordinates a distributed transaction across multiple participants. A coordinator asks all participants to prepare. If all say yes, it tells them to commit. If any says no, it tells them all to rollback.

For a detailed explanation including failure handling, see Two-Phase Commit.

The problem with 2PC is that it is blocking. If the coordinator crashes after participants have prepared but before sending commit, participants must wait indefinitely. For this reason, many systems avoid 2PC in favor of saga or other patterns.

Distributed Locking

When you need exclusive access to a distributed resource, you need distributed locking. Unlike local locks, distributed locks must survive node failures and network partitions.

Chubby, Google’s lock service, uses Paxos to achieve consensus on lock ownership. Applications like BigTable and MapReduce use Chubby for leader election and distributed locking.

Redis Distributed Locks (Redlock): The Redlock algorithm uses multiple Redis instances to acquire a lock. A client writes a unique value to multiple Redis nodes. If a majority of nodes respond within a timeout, the lock is acquired. This approach is simpler than Paxos but has known safety issues under certain network partition scenarios.

ZooKeeper ephemeral nodes: A node can create an ephemeral file. If the client disconnects, ZooKeeper automatically deletes the file. Applications use this for leader election (the first node to create the file becomes leader) and distributed locks (lock files created in a specific order).

sequenceDiagram
    Client1 -->|Create lock| ZooKeeper
    ZooKeeper -->|Created| Client1
    Client2 -->|Create lock| ZooKeeper
    ZooKeeper -->|Node exists| Client2
    Client2 -->|Watch lock| ZooKeeper
    Client1 -->|Release lock| ZooKeeper
    ZooKeeper -->|Node deleted| Client2
    Client2 -->|Create lock| ZooKeeper
    ZooKeeper -->|Created| Client2

Key considerations for distributed locks:

• Lock lifetime: How long should a lock be held? Too short and work doesn’t complete. Too long and other tasks wait unnecessarily.
• Clock skew: If nodes have different clock speeds, lock timeouts can behave unexpectedly. Use logical clocks (Lamport timestamps or hybrid logical clocks) when possible.
• Fencing tokens: Always include a monotonically increasing token with locked operations. Storage systems can reject stale writes using the token.

Consistency Patterns

Transaction Isolation Levels in Distributed Systems

Local database isolation levels (READ UNCOMMITTED, READ COMMITTED, REPEATABLE READ, SERIALIZABLE) don’t map directly to distributed systems. The guarantees differ.

Isolation Level	Local Meaning	Distributed Equivalent
Read Uncommitted	Dirty reads allowed	Not typically used
Read Committed	Only committed reads	Eventual consistency with read repair
Repeatable Read	Same row, same value	Session consistency (read your writes)
Serializable	Full serial order	Linearizable consistency (Paxos/Raft)

Snapshot isolation is common in distributed databases. Each transaction reads from a consistent snapshot (a point-in-time view). Writes are validated at commit time to ensure no conflicts. PostgreSQL uses snapshot isolation. CockroachDB implements it via its MVCC (Multi-Version Concurrency Control) layer.

Serialized Snapshot Isolation (SSI) extends snapshot isolation by detecting write-write conflicts at commit time rather than at write time. This eliminates the “write skew” anomaly but has higher overhead. CockroachDB uses SSI by default for serializable transactions.

The Saga Pattern

Saga breaks a distributed transaction into a sequence of local transactions, each with a compensating transaction. If step 3 fails, steps 1 and 2 are compensated (undone).

See Saga Pattern for full details.

Saga sacrifices isolation and atomicity for availability. Unlike 2PC, saga does not block. Unlike ACID, saga allows intermediate states to be visible.

sequenceDiagram
    participant Saga
    participant Inv as Inventory
    participant Pay as Payment
    Saga-->|Reserve| Inv
    Inv-->|OK| Saga
    Saga-->|Charge| Pay
    Pay-->|OK| Saga
    Note over Saga,Pay: If payment fails, compensate inventory

Eventual Consistency

Eventual consistency guarantees that, if no new updates are made, all replicas will eventually return the same value. The system is available during the “eventually” window.

Amazon DynamoDB, Cassandra, and many distributed databases use eventual consistency by default or offer it as an option.

The “eventually” window can be milliseconds or seconds. Under high load or network partitions, it stretches.

Read Repair

One way to achieve eventual consistency: when you read data and detect inconsistency, you repair the stale replica on the fly.

def read(key):
    values = [replica.read(key) for replica in replicas]
    if not all_equal(values):
        # Repair stale replicas
        latest = max(values, key=lambda v: v.version)
        for replica in replicas:
            if replica.value != latest.value:
                replica.write(key, latest)
    return values[0]

This makes reads slightly more expensive but ensures convergence over time.

Anti-Entropy

Anti-entropy is background repair. Nodes periodically compare their data and fix inconsistencies. This runs continuously in the background, separate from read operations.

Consistency Levels

Many databases let you tune consistency per operation.

Strong consistency: All reads see the result of the latest write. No staleness.

Sequential consistency: All nodes see operations in the same order, but that order may lag behind wall clock time.

Causal consistency: If operation A causally precedes operation B, all nodes see A before B.

Eventual consistency: Given no new updates, all replicas eventually converge.

Read-your-writes consistency: Your own writes are always visible to subsequent reads from the same client. A special case of causal consistency.

# Strong (block until confirmed by quorum)
result = db.read('key', consistency='strong')

# Eventual (return immediately from nearest replica)
result = db.read('key', consistency='eventual')

# Session consistency (read from same replica as write)
result = db.read('key', consistency='session', session_id=client_session)

# Quorum read (R out of N replicas must respond)
result = db.read('key', consistency='quorum', read_quorum=3)

# Bounded staleness (read from replica within time bound)
result = db.read('key', consistency='bounded', max_staleness='5s')

Latency and Throughput by Consistency Level

The consistency level you choose directly affects both latency and throughput. Here’s what to expect in practice:

Consistency Level	Read Latency	Write Latency	Throughput	Notes
Strong (linearizable)	20-100ms	30-150ms	Lowest	Must confirm with quorum or leader
Sequential	10-50ms	20-80ms	Low	Total order broadcast required
Session	5-15ms	10-30ms	Medium	Reads from preferred replica
Bounded staleness	2-10ms	10-30ms	High	Return stale data within bound
Eventual	1-5ms	5-20ms	Highest	Nearest replica, no confirmation

These numbers assume 3-5 replicas in a single region. Cross-region and global deployments push these higher. Strong consistency across regions can hit 200-500ms round-trip.

Why the gap matters: A user-facing API with a 200ms timeout can barely afford one strong-consistency read cross-region. It can handle dozens of eventual reads in the same window. For high-frequency operations (analytics, caching, social feeds), eventual consistency is the only practical choice.

Throughput scaling: With eventual consistency, you can scale read throughput horizontally by adding replicas. Every replica can serve reads. With strong consistency, every read must contact a quorum, so adding replicas does not help read throughput (though it does help with read availability).

Database-Specific Consistency Implementations

Different databases handle consistency very differently under the hood. Understanding what your database actually does helps you make better trade-offs.

Google Spanner uses TrueTime (GPS + atomic clocks) to bound clock uncertainty to ±7ms. This lets Spanner provide strong consistency without a single leader — any replica can serve reads at any timestamp, as long as it waits out the uncertainty window. The cost is read latency: strong reads must wait for time to advance past the uncertainty bound.

CockroachDB uses Hybrid Logical Clocks (HLC) instead of physical clocks. HLC combines physical time with a logical counter, giving you causality tracking without TrueTime’s hardware requirements. CockroachDB’s default consistency level is snapshot (like eventual), but you can request SYNC reads that force consensus.

Amazon DynamoDB uses last-write-wins by default for eventual consistency. Its conditional writes provide optimistic concurrency control, but it doesn’t support true multi-item transactions without DynamoDB Transactions (which uses 2PC internally). DynamoDB Streams provides ordered change capture per partition.

Apache Cassandra uses eventual consistency with tunable consistency per query. You can request ONE, QUORUM, or ALL reads/writes. Cassandra’s lightweight transactions (LWT) use Paxos for linearizable consistency on a per-partition basis — but LWT is expensive (4 round-trips) and should be used sparingly.

Database	Consistency Model	Transaction Support	Max Safe Latency
Spanner	Strong (TrueTime)	2PC + Paxos	~50ms (single region)
CockroachDB	Strong (HLC + Raft)	Raft-based distributed SQL	~20ms (single region)
DynamoDB	Eventual (LWW)	Per-item with Transactions	~10ms (provisioned)
Cassandra	Eventual + LWT	Paxos (LWT) per partition	~5ms (LOCAL_* modes)
etcd	Strong (Raft)	Linearizable reads by default	~5ms

Application-Level Consistency

Handling Inconsistency at the Application Level

When you choose eventual consistency, your application must handle partial states. This is where things get tricky.

Scenario: User orders a product. Payment succeeds. Inventory reservation succeeds. Shipment creation fails. The saga compensates and refunds the payment.

During the failure window, the user sees “order placed” but shipment does not exist. The saga eventually undoes everything, but the user may have seen inconsistent data briefly.

Mitigation options:

• Show pending states clearly to the user
• Use idempotency keys to safely retry
• Implement compensating transactions that notify the user
• Design sagas to minimize the inconsistency window

When to Use / When Not to Use Strong Consistency

Criteria	Strong Consistency (ACID)	Eventual Consistency (BASE)
Latency	Higher (waits for confirmations)	Lower (returns immediately)
Availability	Lower (may refuse requests during partitions)	Higher (always serves requests)
Isolation	Full serializable isolation	No isolation (dirty reads possible)
Complexity	Simple transactions	Complex conflict resolution
Rollback	Automatic and free	Requires compensation logic
Use Case	Financial, inventory, compliance	Analytics, social features, caching
Data Returned	Always latest	May be stale

When to Use Strong Consistency

Use strong consistency when:

• Financial transactions where double-charging is unacceptable
• Inventory systems where overselling has direct business impact
• Operations with regulatory compliance requirements
• Any case where stale reads cause business damage
• Systems where users expect immediate visibility of their actions

When to Use Eventual Consistency

Avoid strong consistency when:

• High-volume, low-value operations where blocking is too expensive
• Systems that must stay available during network partitions
• User-generated content (likes, comments) where occasional duplication is acceptable
• Analytics and reporting where slightly stale data is fine
• Operations where eventual consistency is explicitly acceptable to the business

Decision Tree: Which Consistency Model Should I Choose?

flowchart TD
    Start{What are you building?}
    Start --> Financial{Financial transaction?}
    Financial -->|Yes| Strong1[Use Strong Consistency / ACID transactions or 2PC / Accept higher latency]
    Financial -->|No| Inventory{Inventory management?}
    Inventory -->|Yes| Strong2[Use Strong Consistency or Saga with careful design / Prevent overselling]
    Inventory -->|No| UserContent{User-generated content?}
    UserContent -->|Yes| Eventual1[Use Eventual Consistency / Likes, comments, posts / Accept duplicates]
    UserContent -->|No| Analytics{Analytics or reporting?}
    Analytics -->|Yes| Eventual2[Use Eventual Consistency / Bounded staleness OK / Focus on throughput]
    Analytics -->|No| Partition{Need availability during partitions?}
    Partition -->|Yes| Eventual3[Use Eventual Consistency / BASE model / Saga for transactions]
    Partition -->|No| Latency{Critical latency requirements?}
    Latency -->|Yes| Eventual4[Use Eventual Consistency with quorum reads / Tune R/W values]
    Latency -->|No| Regulatory{Regulatory compliance requirements?}
    Regulatory -->|Yes| Strong3[Use Strong Consistency / Audit trails, compliance / Document decisions]
    Regulatory -->|No| Default[Default to Eventual / Add Strong only where needed / Profile before optimizing]

Quick Reference by Use Case:

Use Case	Recommended Model	Reasoning
Banking/Financial	Strong (ACID/2PC)	Double-charging unacceptable
E-commerce inventory	Strong or Saga	Overselling has direct cost
Social media likes	Eventual	Duplicates tolerable, speed matters
Product catalog	Eventual	Stale reads acceptable
Order processing	Saga	Cross-service, compensating transactions
Analytics/Dashboard	Eventual	Stale data fine, throughput critical
User profiles	Session/Read-your-writes	User expects to see own changes
Stock trading	Strong	Regulatory, exact values required

Consistency and Microservices

In microservices, each service owns its data. Cross-service consistency is the hardest problem. You cannot wrap a transaction around service A’s database and service B’s database.

Options:

1. Accept eventual consistency: Use saga or choreography. Handle partial failures explicitly.
2. Shared database: Both services use the same database. This creates coupling.
3. 2PC: Atomic transaction across databases. Blocked during commit. Rarely used.
4. Event sourcing: Store events, not state. Reconstruct state by replaying.

Most microservice architectures choose option 1: eventual consistency with saga or orchestration.

For related patterns, see Saga Pattern, Event-Driven Architecture, and Message Queue Types.

Use Cases: Strong Consistency

Strong consistency matters for financial transactions where double-charging is unacceptable. Inventory systems where overselling is unacceptable. Operations with regulatory compliance requirements. Any case where stale reads cause business damage.

For these cases, use 2PC or synchronous replication. Accept the latency and availability trade-offs.

Use Cases: Eventual Consistency

Eventual consistency makes sense for user-generated content (likes, comments) where occasional duplication is acceptable. Analytics and reporting where slightly stale data is fine. High-volume, low-value operations where blocking is too expensive. Systems that must stay available during network partitions.

For more on read and write patterns, see Database Replication.

Trade-off Analysis

Understanding the trade-offs between consistency models helps you make informed decisions for your specific use case.

Consistency vs Availability Trade-offs

The CAP theorem is not a menu where you pick three items. It is a forced choice: when the network breaks, you either serve stale data or you stop serving entirely. That is the constraint everything else bends around.

The table below maps concrete scenarios to outcomes under each model. A 150ms quorum round-trip sounds fine in a blog post. It stops sounding fine at 3am when a mobile client times out at 100ms and a user files a bug.

CP systems choose consistency over availability during partitions. When a network partition occurs, CP databases like ZooKeeper and etcd stop accepting writes rather than risk serving divergent data. The moment quorum is lost, the cluster becomes unavailable for writes, even if reads still work. This is not a failure mode to discover in production. It is the intended behavior, and it is why CAP documentation for each service should be explicit before you deploy.

AP systems make the opposite trade. Cassandra and DynamoDB will continue accepting writes on both sides of a partition, converging when the partition heals. The risk is that each side has been accepting independent writes, and conflict resolution now falls to your application or the database’s last-write-wins mechanism. For social media likes this is fine. For a payment ledger it is catastrophic. The CAP label (CP or AP) tells you what happens during a partition. It does not tell you how painful the recovery will be.

A concrete example that brings this home: a 150ms quorum round-trip sounds manageable until you are debugging a production incident at 3am. A mobile client times out at 100ms. The user sees an error. They retry. The retry succeeds against a stale replica. Now you have duplicate charges or double inventory reservation. The database made no mistake, but your consistency level was calibrated wrong for the client timeout, and that mismatch cost you a support ticket and a compensation payment. This is why the CAP trade-off is not an academic checkbox. It is a client experience decision that lives in your error handling and retry logic as much as in your database configuration.

The 150ms threshold matters for another reason: it is the approximate round-trip time for a quorum write across three data centers in the same region. Add cross-region replication and you are at 200-500ms easily. Most mobile clients give you 100-200ms before they timeout and retry. This means strong consistency across regions is functionally incompatible with mobile user experience unless you explicitly educate clients to tolerate longer round-trips. If you are building a mobile-first application, eventual consistency is not a concession to scale. It is the only choice that fits the latency budget.

Scenario	Strong Consistency (ACID)	Eventual Consistency (BASE)
Network partition during write	Request blocked or rejected	Write accepted, synced later
Replica unavailable	Write fails (quorum not met)	Write succeeds to available replicas
Read during partition	May return error	Returns local replica (possibly stale)
Recovery after partition	Automatic reconciliation	Requires conflict resolution
Latency per operation	20-150ms (quorum round-trip)	1-5ms (local write)
Throughput ceiling	Limited by slowest replica	Horizontally scalable with replicas

2PC vs Saga vs Event Sourcing Trade-offs

Three patterns dominate cross-service transaction design, and none of them wins outright. The choice depends on how long your transactions run, how tightly coupled your services are, and how much you can tolerate partial state.

2PC gives you atomicity: all-or-nothing semantics across participants, which matters for financial moves. The cost is blocking. If the coordinator dies after participants have voted yes, those participants wait until someone recovers the coordinator. This happens in production at companies that did not read the failure modes closely enough.

Saga drops the blocking but keeps the complexity. Each step runs as a local transaction, and if a later step fails, you undo the earlier ones in reverse order. The catch is that intermediate states leak. Nothing prevents a user from reading between step 2 and step 3 of a saga. Saga is not ACID compliant, and that matters only if your business cannot absorb brief inconsistencies.

Event sourcing takes a different approach. Instead of storing current state, you store every state change as an immutable event. The current state is reconstructed by replaying the log. This gives you a full audit trail without extra instrumentation, and it pairs naturally with CQRS for read-heavy systems. The downside is operational complexity: event schema changes, event replay, and projection maintenance are all real failure modes that are hard to debug.

The 2PC blocking problem trips up teams that skipped the failure modes. In phase 1, the coordinator sends a prepare message to all participants. Each participant votes yes and holds a lock. At this point the participant is committed to the outcome but does not yet know what that outcome is. If the coordinator crashes here, every participant holds its lock and waits. There is no timeout on the participant side that can safely resolve this state. Someone must either recover the coordinator from its log or manually resolve the in-doubt transaction. This is why production 2PC deployments often pair with a coordinator that is itself backed by a WAL and a recovery process.

Saga compensation complexity is often underestimated until you hit a non-idempotent step. Consider a transfer saga: reserve inventory, charge payment, create shipment. If payment succeeds but shipment creation fails, you must credit the payment back. That credit operation must itself be idempotent, and it must succeed exactly once. Now add retry storms from transient failures, and you end up with compensation logic that is more complex than the original transaction. Event sourcing has its own version of this: when an event schema changes, old events must be upcast to the new schema before replay. Miss one edge case in the upcaster and your projection produces wrong state for events that happened before the schema change. Schema evolution in event-sourced systems is a genuinely hard problem that does not get easier over time. Every schema change is permanent because the old events cannot be rewritten. You end up with upcasters that must coexist, sometimes for years, and every new engineer on the team has to learn the full history of every schema version before they can safely change anything.

The compensation complexity problem in saga is most visible when you hit a non-idempotent step. Consider a transfer saga: reserve inventory, charge payment, create shipment. If payment succeeds but shipment creation fails, you must credit the payment back. That credit operation must itself be idempotent, and it must succeed exactly once. If the credit call times out on the response, you do not know whether the credit went through or not. Do you retry? You might double-credit. Do you skip it? You might leave the customer undercharged. This is not a hypothetical edge case. It is the first non-trivial saga most teams implement, and it is why the choreography model where each service listens to events from others tends to produce compensation logic that is harder to reason about than the original transaction flow.

Criteria	Two-Phase Commit (2PC)	Saga Pattern	Event Sourcing
Atomicity	Full atomicity across participants	No atomicity (partial states visible)	Full atomicity via event log
Blocking	Yes (coordinator failure blocks participants)	No (compensations are async)	No (events appended immutably)
Latency	High (multiple round-trips)	Medium (sequential local transactions)	Low write, higher read complexity
Complexity	Moderate (protocol handling)	High (compensation logic per step)	High (event reconstruction)
Rollback	Automatic (all or nothing)	Manual (compensation transactions)	Via compensating events
Audit Trail	No (only final outcome)	Partial (via compensation logs)	Full (immutable event history)
Use Case	Financial, tight coupling	Cross-service business transactions	Event-driven systems, CQRS

Consensus Algorithm Trade-offs

Consensus is the problem of getting a group of nodes to agree on one value, even when some nodes crash or the network drops messages. Simple in description, brutal in implementation. The theoretical foundations took years to establish, and production implementations still regularly surprise people.

Paxos has a historical problem worth knowing: Leslie Lamport submitted “The Part-Time Parliament” to ACM Transactions on Computer Systems in 1990. The reviewers could not follow the proofs. The paper was rejected. Lamport resubmitted with a more narrative style, and it was eventually published in 1998. By then the damage was done: several production systems had implemented Paxos incorrectly based on vague summaries rather than the actual paper. This is why Multi-Paxos, which simplifies the common case with a stable leader, became the practical variant, and why Raft’s designers explicitly targeted understandability as a primary goal, publishing their paper with an English algorithm that engineers could actually read.

Raft’s understandability goal was not an academic exercise. Diego Ongaro’s dissertation at Stanford documents that during Raft’s development, they user-tested both Paxos and Raft implementations with students. Raft implementations were debugged faster and more completely. The decomposition into leader election, log replication, and safety is deliberate: it means you can reason about each piece independently. In practice, this means etcd’s Raft implementation has fewer subtle bugs than a typical Paxos port, and when bugs do appear, they are easier to isolate.

Paxos has the longest pedigree. Leslie Lamport proved it correct in a paper that was initially rejected because reviewers could not follow the proofs. In practice, Paxos works, but the gap between Paxos in theory and Paxos in production code is real. Multi-Paxos improves things by assuming a stable leader, which matches most production workloads.

Raft was designed specifically to be more understandable than Paxos. It decomposes the problem into three relatively independent pieces: leader election, log replication, and safety. The result is that engineers at etcd and CockroachDB can reason about their implementation and debug it when things go wrong. Raft has become the default choice for most new distributed databases partly because engineers can actually follow the logic.

2PC is not a consensus algorithm, but it ends up in these comparisons because it solves a similar coordination problem. The difference is that 2PC requires all participants to be available, has a single point of failure in the coordinator, and blocks when the coordinator dies. You will see it on architecture diagrams. You should not use it in new microservices work.

Paxos has a specific historical problem worth knowing: Leslie Lamport submitted “The Part-Time Parliament” to ACM Transactions on Computer Systems in 1990. The reviewers could not follow the proofs. The paper was rejected. Lamport resubmitted it with a more narrative style, and it was eventually published in 1998. By then the damage was done: several production systems had implemented Paxos incorrectly based on vague summaries rather than the actual paper. This is why Multi-Paxos (which simplifies the common case with a stable leader) became the practical variant, and why Raft’s designers explicitly targeted understandability as a primary goal, publishing their paper with an appendix containing an English algorithm that engineers could actually read.

Raft’s understandability goal was not an academic exercise. Diego Ongaro’s dissertation at Stanford documents that during Raft’s development, they user-tested both Paxos and Raft implementations with students. Raft implementations were debugged faster and more completely. The decomposition into leader election, log replication, and safety is deliberate: it means you can reason about each piece independently. In practice, this means etcd’s Raft implementation has fewer subtle bugs than a typical Paxos port, and when bugs do appear, they are easier to isolate.

Algorithm	Implementation Complexity	Latency	Fault Tolerance	Use Cases
Paxos	Very high	Medium	N/2 failures	Google Spanner, some distributed DBs
Raft	High	Medium	N/2 failures	etcd, ZooKeeper, CockroachDB
2PC	Moderate	High	Coordinator SPOF	Rarely used in microservices

Lock-based vs Lock-free Trade-offs

Distributed locks feel reassuring until you actually need one. Then the questions start: what happens if the lock holder gets network-partitioned? What if your lock expires but the work keeps running? What if two processes both think they hold the lock?

Pessimistic locking acquires the lock before doing any work. Chubby, ZooKeeper ephemeral nodes, and Redis-based Redlock all follow this model. If you hold the lock, you are the only writer. That simplicity has a cost: under high load, processes spend more time waiting for locks than doing actual work. Lock lifetime management is also tricky. Hold a lock too long and you block everyone. Hold it too short and work does not complete before the lock expires.

Optimistic locking flips this. You skip the lock entirely and check at commit time whether a conflict occurred. MVCC databases use this by assigning version numbers to writes: if your version is behind, someone else changed the data and you must retry. Under low contention this is fast. Under high contention you get retry storms, and a system that looked fast in testing can become unusable in production if your access patterns create hot spots.

Fencing tokens address the partition problem. When you acquire a lock, the lock service gives you a monotonically increasing token. Every write under that lock carries the token. Storage systems reject writes with stale tokens, which prevents a partition-induced lock expiry from resulting in duplicate writes. This is the piece most tutorials skip, and it is the reason Redlock has known safety issues in certain network partition scenarios: without strict token validation on the storage side, you cannot guarantee that a failed lock holder has not already written data that will now be overwritten by a new lock holder.

Martin Kleppmann’s analysis of Redlock is worth reading directly. He identifies that Redlock assumes clocks are monotonic and non-skewed across Redis nodes, but in practice clock adjustments (NTP corrections, virtualization pauses, OS scheduler pauses) can cause a lock to appear valid when it has already expired. A client that holds an expired lock can still write, and a new client can acquire the same lock simultaneously. This is not a theoretical concern. It is a real race condition that occurs under load with virtualized infrastructure. The fix is either a consensus-based lock service (ZooKeeper, etcd) or a lease model where the storage layer enforces token validity regardless of what the lock service says.

Optimistic locking with MVCC fails visibly under high contention. The version check detects a conflict at commit time, and the transaction rolls back. Your application retries. Under moderate contention this works. Under high contention with hot keys, you get retry storms: dozens of transactions all reading the same version, all trying to commit, all rolling back and retrying. This is the failure mode that looks fast in single-threaded testing and falls over at 10,000 TPS on a hot key. The mitigation is careful partition design (spread hot keys across partitions), lease-based pessimistic locking for the hottest keys, and monitoring of abort rates as a leading indicator.

The Redlock safety issue Martin Kleppmann identified is concrete and worth understanding in detail. Redlock assumes clocks are monotonic and non-skewed across Redis nodes. In practice, NTP corrections, virtualization pauses, and OS scheduler pauses can cause a lock to appear valid when it has already expired. A client holding an expired lock can still write, and a new client can acquire the same lock simultaneously. This is not a theoretical concern. Under load with virtualized infrastructure, it occurs. The fix is not to tune timeouts better. The fix is either a consensus-based lock service like ZooKeeper or etcd, where the consensus protocol itself enforces lease validity regardless of what the clock says, or a lease model where the storage layer validates token monotonicity independently of the lock service. If you are using Redlock today, audit whether your storage layer enforces fencing tokens strictly. If it does not, you have a silent correctness violation that only manifests as data corruption under specific failure conditions.

Criteria	Distributed Locks (Chubby, Redlock, ZK)	Lock-free (Optimistic, MVCC)
Contention handling	Pessimistic (lock before operation)	Optimistic (detect conflicts at commit)
Latency under low contention	Higher (lock acquisition overhead)	Lower (no lock needed)
Latency under high contention	Predictable (queued)	High (retry storms)
Starvation risk	Yes (if lock held too long)	Yes (if retries fail)
Implementation	Complex (fencing, renewal, expiry)	Moderate (version checking)

Failure Scenarios Overview

Real-world Failure Scenarios

Failure	Impact	Mitigation
Coordinator crashes in 2PC prepare phase	Participants hold locks indefinitely	Use Paxos-based consensus for coordinator; implement timeout-based lock release
Network partition during commit	Some participants commit, others rollback; inconsistent state	Design for partition tolerance; use saga for compensation
Replica lag in synchronous replication	Read latency increases; writes may timeout	Use semi-synchronous replication; monitor replica lag; adjust consistency level
Read repair produces wrong result	Stale data returned under high contention	Use version vectors; implement conflict resolution; use quorum reads
Eventual consistency window too long	Users see stale data for unacceptable period	Monitor convergence time; adjust replication factor; use read repair
Deadlock in distributed transaction	Transactions timeout; partial state possible	Implement deadlock detection; set appropriate timeout values
Fencing token reuse after lock expiration	Duplicate writes accepted by storage	Always include monotonically increasing tokens; storage must validate strictly
Clock skew causing read inconsistencies	Nodes disagree on latest value	Use logical clocks (HLC) instead of wall-clock time; bound uncertainty
Saga compensation failure	Partial transaction state committed; inconsistent data	Design idempotent compensations; implement retry with exponential backoff
Quorum split-brain during partition	Both partitions accept writes	Require majority quorum; reject writes on minority partition

Consistency Failure Flow

flowchart TD
    Start[Read Request] --> Quorum{Quorum Reached?}
    Quorum -->|Yes| Latest{All Replicas Current?}
    Latest -->|Yes| ReturnLatest[Return Latest Value]
    Latest -->|No| Repair[Trigger Read Repair]
    Repair --> ReturnLatest
    Quorum -->|No| StaleOK{Can Return Stale?}
    StaleOK -->|Yes| ReturnStale[Return Stale Value / Mark for Repair]
    StaleOK -->|No| Error[Return Error]

2PC Failure Flow

flowchart TD
    Start[Start 2PC] --> Prepare[Coordinator sends Prepare to all participants]
    Prepare --> Wait{All Participants Prepare OK?}
    Wait -->|Yes| Commit[Coordinator sends Commit to all]
    Wait -->|No| Rollback[Coordinator sends Rollback to all]
    Commit --> CoordCrash{Coordinator Crashes?}
    CoordCrash -->|Yes| ParticipantsWait[Participants wait indefinitely]
    ParticipantsWait --> Recovery[Recovery protocol: Timeout + Query participants]
    Recovery --> Outcome{Determine outcome}
    Outcome -->|Commit| DoneC[Distributed transaction committed]
    Outcome -->|Rollback| DoneR[Distributed transaction rolled back]

Common Pitfalls / Anti-Patterns

Defaulting to strong consistency everywhere: Strong consistency is expensive. Not every operation needs it. Profile your access patterns and choose consistency levels appropriately.

Ignoring the CAP tradeoff: You cannot have both strong consistency and availability during partitions. Decide what your system needs and design accordingly—don’t pretend this constraint doesn’t exist.

Using 2PC without understanding blocking: The coordinator blocking problem is real. If you use 2PC, understand the failure modes and have mitigation plans ready before production.

Treating eventual consistency as failure: Eventual consistency is not a bug—it is a feature for scalability. Design your application to handle it gracefully from day one.

Not monitoring replication lag: If you use replica reads for scalability but do not monitor lag, you may serve stale data without knowing it until users complain.

Ignoring conflict resolution: With eventual consistency and concurrent writes, conflicts happen. Have a clear strategy: last-write-wins, application-level merge, or quorum-based resolution.

Interview Questions

Further Reading

• Two-Phase Commit - Deep dive into 2PC, including failure handling and alternatives
• Saga Pattern - Detailed coverage of choreography vs orchestration approaches
• Database Replication - Replication strategies and their consistency implications
• Event-Driven Architecture - How eventual consistency fits into event-driven systems
• Message Queue Types - Understanding messaging patterns that underpin distributed transactions
• Designing Data-Intensive Applications - Martin Kleppmann’s comprehensive guide to distributed systems
• The Part-Time Parliament - Lamport’s original Paxos paper

Quick Recap Checklist

Before shipping distributed transaction logic to production, run through this checklist:

• CAP tradeoff documented — For each service, have you explicitly decided between CP and AP?
• Consistency level chosen per operation — Not defaulting to strongest; choosing based on business requirements
• Saga pattern implemented — For cross-service business transactions needing atomicity without 2PC blocking
• 2PC avoided for microservices — Using saga or event sourcing instead where appropriate
• Replication lag monitored — Alerting on lag exceeding defined thresholds
• Transaction latency SLAs defined — P50/P95/P99 tracked and alerted
• Conflict resolution strategy documented — Last-write-wins, application-level merge, or quorum-based
• Idempotency implemented — All distributed operations are idempotent or use idempotency keys
• Fencing tokens in place — Monotonically increasing tokens with all locked operations
• Circuit breakers configured — Preventing cascade failures during partial outages
• Correlation IDs in transaction logs — Every distributed operation traceable across services
• Failure scenarios tested — Chaos testing or game days covering coordinator crash, partition, replica failure
• Compensation logic verified — Saga compensations tested with injected failures at each step
• Read repair and anti-entropy understood — Know which consistency mechanism your database uses
• Lock lifetime tuned — Lock expiry long enough to complete work, short enough to unblock

Conclusion

Distributed transactions do not have a one-size-fits-all answer. ACID gives you consistency but sacrifices availability and latency. BASE gives you availability but introduces inconsistency windows.

In practice, most systems use a mix. Core business operations get strong consistency. Peripheral operations get eventual consistency. Saga patterns handle cross-service business transactions without 2PC’s blocking problems.

The right choice depends on your domain, your tolerance for inconsistency, and what your users expect. Do not default to strong consistency everywhere. It is expensive. Do not default to eventual consistency everywhere. Your users will notice bugs.

flowchart TD
    A[Consistency Models] --> B[Strong ACID]
    A --> C[Eventual BASE]
    B --> D[2PC, Synchronous Replication]
    C --> E[Saga, Async Replication]
    D --> F[High Latency, Lower Availability]
    E --> G[Low Latency, Higher Availability]

Key Points

Quick recap of the core trade-offs covered in this post.

The CAP theorem is the starting constraint, not the whole story. CAP tells you that during a partition you must choose between consistency and availability. It does not tell you that the choice plays out differently depending on whether you are serving reads or writes, whether your clients are browsers or mobile apps with different timeout budgets, or whether your recovery process after the partition is automated or manual. The practical CAP decision for your service is a configuration choice with real client-facing consequences, not a binary label.

Saga and event sourcing are the two patterns that handle what 2PC cannot. Saga handles long-running business transactions across independent services by trading atomicity for availability. Event sourcing handles audit and replay requirements by trading query simplicity for temporal queries and complete history. Neither replaces 2PC; they exist to solve problems 2PC was never designed for.

Consensus algorithms exist because distributed systems need a tiebreaker when nodes disagree. Paxos, Raft, and their variants all solve the same problem with different implementation complexity. Raft won in practice not because it is more correct but because engineers can actually reason about it and debug it when it misbehaves. That understandability advantage is real and documented.

• ACID prioritizes consistency; BASE prioritizes availability
• CAP theorem: you cannot have consistency, availability, and partition tolerance simultaneously
• Choose consistency level per operation based on business requirements
• Saga pattern handles distributed transactions without 2PC’s blocking problems
• Eventual consistency requires application-level handling for conflicts

Production Checklist

These items exist because distributed transaction bugs are rarely found in unit tests. They surface under load, during network partitions, and at 2am when a coordinator crashes. Each item below prevents a specific class of production failure.

The most commonly skipped item is idempotency. Distributed operations fail partially: a network timeout between the commit response and the retry means you cannot tell whether the operation succeeded. If your operation is not idempotent, the retry double-executes. Payment charges twice, inventory decrements twice, emails send twice. Idempotency keys or idempotent operation design are not optional for financial operations. They are the baseline.

Fencing tokens are skipped more often than they should be. When a lock holder gets network-partitioned and the lock expires while work is still in-flight, a new lock holder can start work on the same resource. Without a monotonically increasing fencing token validated by the storage layer, both holders write, and one overwrite is silent. Chubby and Spanner validate fencing tokens. Most Redis-based locks do not. This is the gap that causes duplicate writes in distributed lock implementations that look correct until a specific failure sequence triggers them.

Circuit breakers belong in every distributed transaction path even when they are not your idea of a correctness mechanism. They do not prevent inconsistency, but they prevent cascade failures from turning one participant outage into a full-system degradation. If your payment service is slow, you want the inventory reservation to fail fast rather than queue up behind retries that exhaust the payment service completely.

# Distributed Transactions Production Readiness

- [ ] Consistency level chosen per operation based on requirements
- [ ] Saga pattern implemented for cross-service business transactions
- [ ] Replication lag monitored and alerted
- [ ] Transaction latency SLAs defined and monitored
- [ ] Conflict resolution strategy documented and implemented
- [ ] Idempotency implemented for all distributed operations
- [ ] CAP tradeoff understood and documented for each service
- [ ] Circuit breakers implemented to prevent cascade failures
- [ ] Correlation IDs in all distributed transaction logs