The Saga Pattern: Managing Distributed Transactions

Introduction

An order can be only partly complete when a service fails. Inventory may commit a reservation, then the payment service times out before the order is confirmed. A saga records that reservation as a completed local transaction and, if payment cannot proceed, releases the inventory with an idempotent compensation. The customer may see a pending state while recovery runs because the workflow is not an atomic database transaction.

This guide explains how to model those steps, choose choreography or orchestration, and recover safely from retries, ambiguous outcomes, and failed compensation.

Topic-Specific Deep Dives

Core Concepts

A saga represents a distributed transaction as a series of steps. Each step is a local transaction on one service. After each step, the saga either continues to the next step or, if a step fails, runs compensating transactions to undo the previous steps.

Step 1: Reserve Inventory (compensate: release inventory)
Step 2: Charge Payment (compensate: refund payment)
Step 3: Create Shipment (compensate: cancel shipment)

If step 3 fails, you run the compensation for step 2 (refund) and then step 1 (release inventory). The saga undoes what it did, leaving the system consistent.

sequenceDiagram
    participant Saga
    participant Inv as Inventory
    participant Pay as Payment
    participant Ship as Shipping
    Saga->>+Inv: Reserve
    Inv->>-Saga: OK
    Saga->>+Pay: Charge
    Pay->>-Saga: OK
    Saga->>+Ship: Ship
    Ship->>-Saga: OK
    Note over Saga: Success - all steps complete

Saga does not provide isolation.

Example: order processing saga

A complete order processing saga with three services:

def create_order_saga(order):
    saga = OrderSaga()

    # Step 1: Reserve inventory
    reservation = saga.reserve_inventory(order.items)
    if not reservation.success:
        return OrderResult(rejected=True, reason="Insufficient inventory")

    # Step 2: Authorize payment
    authorization = saga.authorize_payment(order.payment, order.total)
    if not authorization.success:
        saga.compensate_inventory(reservation.id)
        return OrderResult(rejected=True, reason="Payment declined")

    # Step 3: Create shipment
    shipment = saga.create_shipment(order.address, order.items)
    if not shipment.success:
        saga.reverse_payment(authorization.id)
        saga.compensate_inventory(reservation.id)
        return OrderResult(rejected=True, reason="Shipping unavailable")

    # All steps succeeded
    return OrderResult(confirmed=True, order_id=order.id, shipment=shipment)

Each compensate_* method is a compensating transaction. They run in reverse order on failure.

Saga vs two-phase commit

Two-phase commit (2PC) locks resources until all participants confirm. It provides atomicity but at the cost of availability. If the coordinator fails during the commit phase, participants may be left waiting indefinitely.

Saga takes a different approach. It sacrifices isolation and atomicity for availability and scalability. Steps execute one at a time. Failures trigger compensation, not rollback.

For a deep dive on 2PC and why it is often avoided, see Two-Phase Commit.

Architecture Deep Dives

The choice between choreography and orchestration shapes how teams debug a saga and how it behaves under load. Neither approach is better across the board. Each makes different trade-offs in coupling, visibility, and operational complexity.

Choreography distributes saga logic across services. Each service knows its own step and reacts to events emitted by the previous one. There is no central coordinator, so teams stay loosely coupled and can evolve independently. The trade-off is visibility: no single place shows the overall workflow. Debugging a failed choreographed saga means following an event chain across multiple services, which requires distributed tracing and domain knowledge of each service’s event handlers.

Orchestration centralises the workflow in a dedicated orchestrator. It holds the full step sequence, decides what runs next, and triggers compensations on failure. This gives you a single source of truth for saga state. Debugging is more straightforward: you check the orchestrator logs. The cost is that the orchestrator becomes a coupling point and a potential bottleneck. Teams that own steps must coordinate with the orchestrator team when adding or modifying workflows.

After comparing the two patterns, this section also covers implementation concerns that apply regardless of which style you choose: designing deterministic compensations, ensuring idempotency across unreliable networks, and decomposing large workflows into nested sagas.

Choreographed saga

Each service knows its own step and its own compensation. When a step completes, the service emits an event. The next service reacts to that event. If something fails, services emit failure events that trigger compensations.

graph LR
    Order[Order Service] -->|OrderCreated| Inv[Inventory]
    Inv -->|InventoryReserved| Pay[Payment]
    Pay -->|PaymentCharged| Ship[Shipping]
    Ship -->|ShipmentCreated| Order

In this flow, each service reacts to the previous step’s event. The behavior is distributed. Each service knows only its own piece.

When Payment fails after Inventory is reserved:

graph LR
    Pay -->|PaymentFailed| Inv
    Inv -->|InventoryReleased| Order

The compensation logic lives in each service. Inventory responds to the failure by releasing the reservation.

Orchestrated saga

A central orchestrator manages the sequence. It decides what step runs next, handles failures, and triggers compensations. The orchestrator knows the entire workflow.

graph LR
    Orch[Order Orchestrator] -->|Reserve| Inv
    Inv -->|OK| Orch
    Orch -->|Charge| Pay
    Pay -->|OK| Orch
    Orch -->|Ship| Ship
    Ship -->|OK| Orch

The orchestrator keeps state about what has completed. If Payment fails, the orchestrator tells Inventory to release and returns an error to the client.

For a full comparison, see Service Orchestration.

Implementing saga compensation

Compensation logic must be deterministic. If step 3 fails after step 2 succeeded, you must undo step 2. Running compensation twice or in the wrong order causes problems.

class OrderSaga:
    def execute(self, order):
        try:
            # Step 1: Reserve inventory
            reservation = self.inventory.reserve(order.items)
            self.steps.append(('reserve', reservation))

            # Step 2: Process payment
            charge = self.payment.charge(order.payment_info, order.total)
            self.steps.append(('charge', charge))

            # Step 3: Create shipment
            shipment = self.shipping.create(order.address)
            self.steps.append(('ship', shipment))

            return SagaResult(success=True, shipment=shipment)

        except PaymentDeclined:
            # Compensate step 1
            self.inventory.release(self.steps[0][1].reservation_id)
            return SagaResult(success=False, reason="Payment declined")

        except ShippingUnavailable:
            # Compensate step 2
            self.payment.refund(self.steps[1][1].charge_id)
            # Compensate step 1
            self.inventory.release(self.steps[0][1].reservation_id)
            return SagaResult(success=False, reason="Shipping unavailable")

The saga tracks completed steps in order. On failure, it runs compensations in reverse order. This is the compensating transaction pattern.

A compensation is a business recovery action, not a database rollback. It may only partially offset the original action: refunding a captured payment cannot erase the time the customer was charged, and a parcel already handed to a carrier may need a return workflow instead of a cancellation. If an effect cannot be undone, decide before starting the saga whether to delay it until other risky steps finish, continue forward with a corrective action, or route the case to an operator. Record that outcome explicitly so the saga does not report a clean rollback when the business state still needs attention.

Idempotency in sagas

Sagas execute across unreliable networks. Messages may be delivered twice. Services may crash mid-operation. Your saga must handle idempotency.

Reserve the same inventory twice should not double-reserve. Charge the same payment twice should not double-charge.

def reserve_inventory(self, reservation_id, items):
    # Idempotency check
    if self.inventory.is_reserved(reservation_id):
        return self.inventory.get_reservation(reservation_id)

    # Actually reserve
    return self.inventory.create_reservation(reservation_id, items)

def charge_payment(self, charge_id, payment_info, amount):
    # Idempotency check using charge_id
    existing = self.payment.get_charge(charge_id)
    if existing:
        return existing

    # Process charge
    return self.payment.create_charge(charge_id, payment_info, amount)

Include an idempotency key (the reservation ID from step 1) in the compensation call. Check before acting.

The check and the operation must be protected from concurrent retries. Store the idempotency key and result atomically with the side effect, usually with a unique constraint or a transaction in the service that owns that effect; a separate check followed by a write can still let two requests perform it. Scope keys to the saga, step, and operation (including whether it is a compensation), and keep them for at least the full retry and recovery window. If a timeout leaves the result unknown, query by that key before deciding to retry or advance.

Nested sagas

Large workflows sometimes need sub-sagas. Rather than one monolithic saga with dozens of steps, you can decompose into nested sagas where a step’s “execute” is itself a saga.

For example, an order fulfillment saga might have a step called ProcessPayment that internally runs authorize → capture as a nested saga. If the nested saga fails, the parent saga treats it as a single failed step and compensates accordingly.

Why nest sagas?

• Reusability: The ProcessPayment nested saga can be reused across multiple parent sagas (order, subscription, refund)
• Readability: Top-level saga reads like a business workflow, not a technical protocol
• Scoped failures: If payment fails, you know exactly which sub-step failed without scrolling through 20 parent steps

Concurrency control with nested sagas:

Nested sagas introduce concurrency at the parent level. While the payment nested saga is running, other parent sagas may also be running and trying to access shared resources. Use optimistic or pessimistic locking at the parent level.

class ParentSaga:
    def execute(self):
        # Step 1: Reserve inventory (parent-level lock on inventory record)
        with self.lock('inventory', self.order.inventory_id):
            self.reserve_inventory()

        # Step 2: Run payment as nested saga (has its own compensation)
        payment_result = PaymentNestedSaga().execute(self.payment_context)

        if not payment_result.success:
            # Parent-level compensation for inventory
            self.release_inventory()  # Uses same lock
            return Failed

        # Step 3: Create shipment
        self.create_shipment()

Optimistic locking: Read the resource version before modifying. On update, check the version hasn’t changed. If it has, abort and retry.

Pessimistic locking: Acquire a lock before accessing the resource. Blocks other sagas from accessing it until the lock releases. Simpler but reduces concurrency.

Use optimistic locking for most cases (higher throughput). Use pessimistic locking only when the cost of a concurrent modification is very high (financial transactions, inventory with hard limits).

class SagaState:
    """Persisted saga state for crash recovery."""

    def __init__(self, saga_id: str):
        self.saga_id = saga_id
        self.steps: list[tuple[str, str, dict]] = []  # [(step_name, status, data)]
        self.status = "pending"
        self.created_at = datetime.utcnow()

    def mark_step_started(self, step_name: str, step_data: dict):
        self.steps.append((step_name, "started", step_data))
        self._persist()

    def mark_step_completed(self, step_name: str, result_data: dict):
        # Update step status
        for i, (name, status, data) in enumerate(self.steps):
            if name == step_name and status == "started":
                self.steps[i] = (name, "completed", {**data, **result_data})
                break
        self._persist()

    def mark_step_compensated(self, step_name: str):
        for i, (name, status, data) in enumerate(self.steps):
            if name == step_name and status == "completed":
                self.steps[i] = (name, "compensated", data)
                break
        self._persist()

    def get_pending_steps(self) -> list[str]:
        return [name for name, status, _ in self.steps if status == "started"]

    def get_completed_steps(self) -> list[str]:
        return [name for name, status, _ in self.steps if status == "completed"]

    def _persist(self):
        # Save to durable storage (database, etc.)
        db.sagas.upsert(self.saga_id, self.to_dict())

Failure Handling

Saga failures fall into three categories, and each requires a different response.

Failure Type	Examples	Default Behaviour	Fallback
Transient	Network timeout, connection pool exhaustion, 503 Service Unavailable	Retry with exponential backoff (3-5 attempts)	Treat as permanent after max retries
Permanent	Insufficient inventory, card declined, invalid input	Trigger compensation immediately	N/A, will never succeed
Unknown State	Service crashes mid-step, response lost in transit	Check idempotency key / query step status	Recover state or treat as permanent

Transient failures are temporary glitches that resolve on their own. A network blip, a connection pool draining under load, or a service restarting mid-deployment. The right response is retry with exponential backoff. Start small (100-500ms), double each attempt, cap at 10-15s, and stop after 3-5 tries. Add jitter so concurrent sagas don’t pile up on the same retry window.

def execute_with_retry(step_fn, context, max_retries=3):
    delay = 0.5
    for attempt in range(max_retries):
        try:
            return step_fn(context)
        except TransientError as e:
            if attempt < max_retries - 1:
                sleep(delay + random.uniform(0, delay * 0.5))
                delay *= 2
                continue
            raise PermanentError(f"Failed after {max_retries} retries") from e

Permanent failures will never succeed on retry. Retrying a declined card or zero-inventory item is wasted time that keeps the system inconsistent longer. Detect them early and trigger compensation immediately with no retry delay.

def handle_permanent_failure(saga, failed_step_index):
    for index in range(failed_step_index - 1, -1, -1):
        compensate_step(saga, saga.steps[index])
    saga.status = "failed"
    saga.persist()

Unknown state is the hardest category. You sent “reserve inventory” and the service crashed or the response was lost. Did it execute? The answer is the idempotency check-then-act pattern: check whether a step already completed using its idempotency key before running it.

def execute_step_with_idempotency(step_id, step_fn):
    existing = get_step_result(step_id)
    if existing is not None:
        return existing
    result = step_fn()
    store_step_result(step_id, result)
    return result

When state cannot be determined, mark the step as ambiguous and alert an operator. Workflow engines like Temporal eliminate this problem through durable execution: step progress is persisted atomically so a crash cannot leave state indeterminate.

Compensation failure is the meta-risk. If the compensation itself fails (payment service is down when you try to refund), you are stuck with an inconsistent system. Retry compensation with exponential backoff, just like a forward step. If it keeps failing, move the saga to a dead letter queue for manual review and alert aggressively.

Design compensations to be idempotent as well. If a compensation runs twice, the second run should have no effect.

Performance considerations

Saga trades a synchronous two-round-trip 2PC for a sequential multi-step approach. The math is worth understanding.

2PC latency profile: Two network round-trips (prepare + commit), but all participants vote and commit in parallel. Typical: 10-50ms per participant phase.

Saga latency profile: Each step adds its own latency. If step 1 takes 20ms, step 2 takes 30ms, step 3 takes 15ms, your total is 65ms plus orchestration overhead. Steps run sequentially, not in parallel.

For a 3-step saga vs 2PC across the same 3 services:

Metric	2PC	Saga
Happy path latency	~20-40ms (parallel phases)	~50-80ms (sequential steps)
Failure recovery	Blocks until coordinator recovers	Compensation runs immediately
Availability	Lower (blocking on coordinator)	Higher (no coordinator SPOF)
Lock duration	All locks held during both phases	Each lock released after its step

Saga’s latency overhead is real but often acceptable. If each step is 10-30ms (typical for service calls), a 5-step saga runs in 50-150ms total. Compare that to a user-facing API timeout (usually 1-5 seconds) and the overhead is negligible.

Where saga latency hurts is high-throughput, low-latency paths (trading systems, real-time pricing). In those cases, consider whether you can pipeline steps (some steps don’t depend on previous step results and can overlap).

Trade-off Analysis

The saga pattern involves several fundamental trade-offs compared to other distributed transaction approaches.

Testing Strategies

Test compensations for idempotency, inject failures at each step to verify reverse-order recovery, and exercise crash and network behavior in staging. The examples below cover those checks; use the release gate at the end of this section before deployment.

Saga vs Alternative Patterns

Aspect	Saga	Two-Phase Commit	TCC (Try-Confirm-Cancel)
Atomicity	Eventual (via compensation)	True atomic (all-or-nothing)	Eventual (via Confirm/Cancel)
Isolation	None (partial states visible)	Full isolation	Partial (Try reserves)
Availability	High (no participant locks held; orchestration still needs recovery)	Low (coordinator is SPOF)	Medium (Try phase reserves resources)
Complexity	Medium (compensation logic)	High (coordinator protocol)	High (3-phase interface per service)
Rollback Mechanism	Compensation transactions	Automatic rollback	Cancel / Confirm semantics
Latency	Per-step cumulative	Two round-trips (parallel phases)	Three round-trips per operation
Service Coupling	Each service owns compensation	All services must support 2PC	All services must implement TCC
Failure Handling	Application-defined compensation	Coordinator-driven	Try phase failures auto-cancel

Choreography vs Orchestration Trade-offs

Factor	Choreography	Orchestration
Central Point of Failure	None (fully distributed)	Orchestrator is a potential failure point
Logic Location	Distributed across services	Centralized in orchestrator
Adding New Steps	All participating services must change	Only orchestrator changes
Understanding Workflow	Harder to see overall flow	Easier to see complete workflow
Debugging	Requires distributed tracing	Easier (centralized logs)
Scalability	Scales with number of services	Limited by orchestrator capacity
Team Independence	High (services evolve independently)	Low (teams depend on orchestrator)
Suitable For	2-4 services, simple event flows	5+ services, complex workflows

Design Decision Matrix

Your Constraint	Recommended Approach
Maximum availability required	Saga with durable state and recovery
True ACID isolation required	Single database or 2PC (with caveats)
Cross-service transactions	Saga
Can’t modify participant code	Saga (only compensation needed)
High-frequency, low-latency	Pipelined saga or avoid distributed transaction
Long-running (hours to days)	Orchestrated saga with durable execution (Temporal)
Simple 2-3 step workflow	Choreographed saga
Complex multi-team workflow	Orchestrated saga
Already using workflow engine	Use the engine’s saga support (Temporal, Step Functions)

Cost-Benefit Summary

Saga Benefits:

• Choreography has no central coordinator; an orchestrated saga depends on a recoverable orchestrator
• Steps execute sequentially — each lock is released after its step completes
• Works across service boundaries with separate databases
• Compensation is application-defined and domain-appropriate
• Easier to retrofit onto existing services

Saga Costs:

• No isolation — application must handle partial states
• Compensation can fail, leaving the system in an inconsistent state
• Longer cumulative latency (sum of step latencies vs parallel phases)
• Application must implement idempotency throughout
• Testing is more complex (failure scenarios, compensation ordering)

Unit testing compensations

Test each compensation in isolation first. The compensation is the most critical piece — if it fails, your saga is stuck.

# Test that releasing inventory twice has no effect (idempotency)
def test_release_inventory_idempotent():
    inventory = InMemoryInventory()
    inventory.reserve("order-123", ["item-a"])

    # First release — succeeds
    result1 = inventory.release("order-123")
    assert result1.success
    assert not inventory.is_reserved("order-123")

    # Second release — should be idempotent (no error)
    result2 = inventory.release("order-123")
    assert result2.success  # Still succeeds, even though already released

    # Third release — still idempotent
    result3 = inventory.release("order-123")
    assert result3.success

Integration testing failure scenarios

The real test is whether your saga handles failures correctly. Set up test infrastructure that simulates failures at each step.

# Test: step 3 fails, step 2 compensation runs correctly
def test_saga_step3_failure_triggers_step2_compensation():
    inventory = MockInventory()  # Always succeeds
    payment = MockPayment()      # Always succeeds
    shipping = MockShipping(fail_on="create")  # Fails on create

    saga = OrderSaga(inventory, payment, shipping)
    result = saga.execute(order_with_3_items)

    assert result.failed
    assert result.failed_step == "create_shipment"
    assert payment.refund_was_called()        # Step 2 compensated
    assert inventory.release_was_called()      # Step 1 compensated
    assert not shipping.shipment_created      # Step 3 never ran

Testing compensations run in correct order

The most common saga bug is compensation running in the wrong order. Write a test that explicitly verifies reverse order.

def test_compensation_runs_in_reverse_order():
    call_order = []

    class TrackingService:
        def do_step(self):
            call_order.append(f"do-{self.name}")
            return Success()

        def compensate(self):
            call_order.append(f"compensate-{self.name}")
            return Success()

    class OrderedSaga(Saga):
        def __init__(self):
            self.s1 = TrackingService(name="step1")
            self.s2 = TrackingService(name="step2")
            self.s3 = TrackingService(name="step3")

        def execute(self):
            self.do_step(self.s1)
            self.do_step(self.s2)
            self.do_step(self.s3)  # Fails here
            return Success()

        def compensate(self):
            # Should run in reverse: s3, s2, s1
            self.compensate(self.s3)
            self.compensate(self.s2)
            self.compensate(self.s1)

    saga = OrderedSaga()
    saga.execute()  # Fails on step 3
    saga.compensate()

    assert call_order == [
        "do-step1", "do-step2", "do-step3",
        "compensate-step3", "compensate-step2", "compensate-step1"
    ]

Chaos testing before release

In staging, inject failures to verify recovery: terminate a service mid-saga, add network latency, exhaust a connection pool, and interrupt communication between participants.

Release gate

Before deployment, verify that each compensation is idempotent, step failures trigger compensations in reverse order, and recovery resumes correctly after a crash or timeout.

Observability & Tracing

A shared trace context connects each forward step to its compensation, while structured logs make saga state searchable. The trace hierarchy and propagation example below show how to preserve that context across service calls.

Trace structure for sagas

A saga trace has a parent span for the overall saga, with child spans for each step and compensation.

Trace: order-123-saga (trace-id: abc123)
├── Span: saga_created (service: order-service)
├── Span: step.reserve_inventory (service: inventory-service)
│   └── Span: compensate.reserve_inventory (service: inventory-service)
├── Span: step.charge_payment (service: payment-service)
│   └── Span: compensate.charge_payment (service: payment-service)
├── Span: step.create_shipment (service: shipping-service)
└── Span: saga_completed / saga_failed

Implementing trace context propagation

Pass trace context through saga steps using baggage or span links.

from opentelemetry import trace
from opentelemetry.propagate import inject, extract

tracer = trace.get_tracer(__name__)

class OrderSaga:
    def execute(self, order, context=None):
        # Extract incoming trace context (if triggered by HTTP request)
        if context:
            ctx = extract(context)
            span = tracer.start_span("order_saga", context=ctx)
        else:
            span = tracer.start_span("order_saga")

        with span:
            span.set_attribute("saga.id", order.saga_id)
            span.set_attribute("saga.type", "order_fulfillment")

            # Inject trace context into step calls
            headers = {}
            inject(headers)  # Injects current trace into headers

            try:
                # Step 1: Reserve inventory (pass headers for trace propagation)
                inventory_ctx = self.inventory.reserve(order.items, headers)

                # Step 2: Charge payment
                payment_ctx = self.payment.charge(order.payment, headers)

                # Step 3: Create shipment
                shipment = self.shipping.create(order.address, headers)

                span.set_status(trace.Status.OK)
                return Success(shipment)

            except Exception as e:
                span.record_exception(e)
                span.set_status(trace.Status.ERROR, str(e))
                raise

When a step fails and compensation runs, link the compensation span to the original step span so you can see the pairing in the trace.

def compensate_inventory(self, reservation_id, original_span):
    with tracer.start_as_current_span(
        "compensate.reserve_inventory",
        links=[Link(original_span.get_span_context())]
    ) as span:
        span.set_attribute("compensation.for", original_span.name)
        span.set_attribute("reservation.id", reservation_id)
        self.inventory.release(reservation_id)

This way, in your trace viewer, you see the compensate span linked back to its original do span — paired visually rather than guessing from logs.

Framework & Decision Guides

Building saga from scratch is educational but not practical for production. You end up writing state persistence, retry logic, idempotency checks, and distributed tracing integration yourself. All of these take time to get right and are easy to get subtly wrong. A framework that handles these concerns out of the box gets you to production faster and with fewer surprises.

The main options fall into three categories. Durable workflow engines like Temporal give you crash-resilient execution with automatic state recovery. Managed cloud services like AWS Step Functions handle persistence and retries without any infrastructure overhead. Self-hosted orchestrators like Netflix Conductor offer control over the runtime without depending on a cloud vendor.

The comparison below covers each framework’s strengths, ideal use cases, and trade-offs. Following that, a decision matrix maps your constraints (availability needs, team size, latency requirements) to the right approach.

Temporal

The strongest choice for saga orchestration. Temporal provides durable workflow execution — if your service crashes mid-saga, Temporal persists the workflow state and resumes it from where it left off. No need to build your own saga state machine.

• Strengths: Durable execution (survives worker crashes), built-in retries with backoff, activity heartbeats, sandboxed workflow code, strong OpenTelemetry integration
• Good for: Complex multi-step business workflows, long-running sagas (hours to days)
• Trade-offs: Self-hosting is operationally heavy; Temporal Cloud pricing can be significant at scale

# Temporal workflow example
@workflow.defn
class OrderWorkflow:
    @workflow.run
    async def run(self, order: Order) -> OrderResult:
        # Activities with automatic retry
        reservation = await workflow.execute_activity(
            reserve_inventory,
            order.items,
            start_to_close_timeout=timedelta(seconds=30),
            retry_policy=ActivityRetryPolicy(maximum_attempts=3),
        )

        try:
            charge = await workflow.execute_activity(
                charge_payment,
                order.payment,
                order.total,
                start_to_close_timeout=timedelta(seconds=30),
            )
        except PaymentDeclined:
            await workflow.execute_activity(
                release_inventory,
                reservation.id,
            )
            return OrderResult(rejected=True, reason="Payment declined")

        shipment = await workflow.execute_activity(
            create_shipment,
            order.address,
        )

        return OrderResult(confirmed=True, shipment=shipment)

AWS Step Functions

Managed saga orchestration on AWS. Integrates tightly with AWS services (Lambda, ECS, SQS, DynamoDB). Good if you’re already all-in on AWS.

• Strengths: Fully managed, pay-per-state-transition, tight Lambda integration, visual workflow designer
• Good for: AWS-centric architectures, medium-complexity workflows
• Trade-offs: Vendor lock-in, expensive at high step counts, debugging can be opaque

Conductor (Netflix)

Conductor is an open-source saga orchestrator from Netflix. Good for microservices that need workflow orchestration without heavy operational overhead.

• Strengths: Open source (self-hostable), JSON-based workflow definitions, HTTP-based workers
• Good for: Teams wanting open-source without Temporal complexity
• Trade-offs: Not as battle-tested as Temporal at extreme scale, less mature ecosystem

Comparison

Framework	Durability	Open Source	Complexity	Best For
Temporal	Excellent (durable execution)	Yes (server) + cloud	Medium	Complex long-running workflows
AWS Step Functions	Good (managed)	No	Low	AWS-centric, simple workflows
Conductor	Good	Yes (fully)	Medium	Open-source preference

For most production scenarios, Temporal is the right call. The durable execution guarantee alone eliminates a whole class of saga state loss bugs.

Saga Applicability and Coordination Trade-offs

First ask whether the operation can stay inside one service and one database. If it can, use a local transaction: it is simpler and provides ACID guarantees without distributed retries, partial states, or compensation. Choose a saga only when a business operation must coordinate independent services or databases and eventual consistency is acceptable. Within a saga, prefer choreography for a short, stable flow with few participants; use orchestration when the workflow needs centralized state, branching, long waits, or coordinated recovery.

Criteria	Saga (Choreography)	Saga (Orchestration)	Two-Phase Commit
Atomicity	Eventual	Eventual	True atomicity
Isolation	None	None	Full isolation
Availability	High (depends on messaging and participants)	High (also depends on orchestrator recovery)	Low (blocks on coordinator failure)
Complexity	Distributed logic	Centralized logic	Distributed but synchronous
Compensation	Each service knows its own	Orchestrator directs	Automatic rollback
Latency	Per-step latency	Per-step + orchestration overhead	Two round-trips
Debugging	Harder (distributed)	Easier (centralized)	Moderate
Rollback Cost	Compensation required	Compensation required	Free (automatic)

Use saga when:

• Operations span multiple services with separate databases
• You cannot use 2PC (which is usually the right call)
• Business transactions map naturally to a sequence of steps
• You can define compensating transactions for each step
• Eventual consistency is acceptable for your domain
• You need availability over strong isolation

Avoid saga when:

• Steps have tight interdependencies that require strict isolation
• Rollback must be immediate and guaranteed
• Compensation is expensive or impossible (sending an email, charging a card with long refund times)
• Your domain requires all-or-nothing atomicity that saga cannot provide
• The inconsistency window is unacceptable for your use case

Production Considerations

Staging environments rarely expose the failure modes that surface in production. A saga that passes every integration test can still break when real network partitions occur, connection pools exhaust under concurrent load, or Kubernetes evicts a pod mid-transaction. Production introduces latencies, failures, and load patterns that are difficult to simulate reliably in pre-production environments.

The failure scenarios below walk through the most common production failures and their specific mitigations. After that come two diagram types that help teams reason about saga behaviour at the system level: failure flow diagrams and state machine diagrams. These are not decorative. They are the shared vocabulary teams use when debugging a step failure at 3 AM.

Key areas where production reality diverges from staging:

• Network partitions: Staging uses controlled network conditions. Production sees partial partitions, asymmetric latency, and DNS failures that staging cannot replicate.
• Concurrent load: A saga tested in isolation behaves differently under real concurrent load. Two sagas racing for the same inventory reveal race conditions that single-threaded tests miss.
• Resource exhaustion: Connection pools, thread pools, and memory limits behave differently under production load. A misconfigured pool limit that passes staging tests can cause cascading failures under real traffic.
• Partial deployments: A rolling deployment can leave some services on version N and others on N+1 mid-saga. The saga must handle step results from a service running different code than expected.

Saga Failure Scenarios

Failure	Impact	Mitigation
Step fails and compensation also fails	System left in inconsistent state	Design idempotent compensations; implement retry with exponential backoff; alert on repeated compensation failures
Service crashes mid-step	Step may or may not have completed; unknown state	Use idempotency keys; implement saga state tracking; use durable workflow engines
Concurrent sagas interfere	One saga’s uncommitted data affects another saga’s read	Implement optimistic concurrency control; use application-level locks for critical resources
Compensation runs on already-succeeded step	Double compensation causes incorrect state	Track completed steps explicitly; prevent compensation on committed steps
Saga state lost (orchestrator crash)	Cannot determine what steps completed	Persist saga state to durable storage; use workflow engines that handle this
Infinite retry loop	System stuck in repeated failed attempts	Implement max retry count; move to dead letter state after threshold; alert

Diagrams & State Machines

Diagrams serve two purposes when designing sagas. A failure flow diagram maps the decision tree that the saga follows when each step succeeds or fails, making it clear which compensations fire and in what order. A state machine diagram defines the legal state transitions a saga instance can go through, from creation to terminal completion or failure.

The two diagrams below show the standard patterns. They are not specific to any framework or implementation. You can map them directly onto Temporal workflows, Step Functions state machines, or a custom orchestrator.

Failure Flow Diagram

graph TD
    Start[Start Saga] --> Step1[Execute Step 1]
    Step1 --> Step1OK{Step 1 OK?}
    Step1OK -->|No| Fail1[Return Error]
    Step1OK -->|Yes| Step2[Execute Step 2]
    Step2 --> Step2OK{Step 2 OK?}
    Step2OK -->|No| Comp1[Compensate Step 1]
    Comp1 --> Fail2[Return Error]
    Step2OK -->|Yes| Step3[Execute Step 3]
    Step3 --> Step3OK{Step 3 OK?}
    Step3OK -->|No| Comp2A[Compensate Step 2]
    Comp2A --> Comp2B[Compensate Step 1]
    Comp2B --> Fail3[Return Error]
    Step3OK -->|Yes| Success[Saga Complete]

Saga Execution State Machine

stateDiagram-v2
    [*] --> Pending: Saga created
    Pending --> Executing: First step started
    Executing --> Executing: Step N completed
    Executing --> Compensating: Step failed
    Compensating --> Compensating: Compensation in progress
    Compensating --> Compensated: All compensations done
    Executing --> Completed: Final step succeeded
    Compensating --> Failed: Compensation failed after retries
    Pending --> Failed: Immediate failure (validation error)
    Completed --> [*]
    Compensated --> [*]
    Failed --> [*]

State Definitions:

State	Description
Pending	Saga created but not yet started. Initial state before first step execution.
Executing	One or more steps have completed successfully. Saga is processing subsequent steps.
Compensating	A step failed and compensation is running in reverse order to undo completed steps.
Completed	Every forward step succeeded. Terminal business success state.
Compensated	Completed steps were compensated after a failure. The workflow ended without business success.
Failed	Saga reached an unrecoverable state (compensation failed or max retries exceeded).

State Transition Rules:

1. Once in Executing, cannot return to Pending
2. Compensating can only be entered from Executing
3. Completed, Compensated, and Failed are terminal states
4. From Compensating, successful cleanup leads to Compensated, persistent failure leads to Failed
5. Only Pending and Failed can be safely retried at the saga level

Choreography vs Orchestration: Trade-off Analysis

Factor	Choreography	Orchestration	Recommendation
Coupling	Low — services own their own logic	High — orchestrator knows all steps	Choreography for independent teams
Complexity	Distributed — each service knows its trigger	Centralised — orchestrator is complex	Orchestration for complex flows
Visibility	Poor — no central view of saga state	High — orchestrator tracks all state	Orchestration for debugging
Scalability	High — no central bottleneck	Limited by orchestrator capacity	Choreography at scale
Failure Handling	Each service handles its own failures	Centralised retry and compensation	Orchestration for guaranteed ordering

Production Failure Scenarios

The following scenarios are illustrative examples, not sourced incident reports. They show failure modes and recovery lessons teams should validate in their own systems.

Scenario 1: Payment Service Unavailable During Saga

What happened: A saga was executing an order placement across three services: inventory, payment, and shipping. The payment service became unavailable for 45 seconds mid-saga after the inventory reservation succeeded.

Root cause: The payment service’s database connection pool was exhausted due to a misconfigured max-connections setting. Requests queued up behind the pool limit and timed out.

Impact: 127 sagas entered the compensating state simultaneously. The compensating transaction to release inventory ran successfully, but the refund for charges already processed was delayed. 23 customers reported duplicate charges before idempotency checks caught them.

Lesson learned: Always implement idempotency keys at every saga step. The payment step used an idempotency key derived from the saga ID and step number, which prevented double-charging on retry — but the compensation path did not use an idempotency key, causing the delayed refund to appear as a duplicate in monitoring.

Scenario 2: Circular Dependency in Choreography

What happened: Two sagas in an e-commerce platform developed a circular dependency. Saga A needed to update product pricing and Saga B needed to update product availability. Both sagas published events that triggered each other’s next steps, creating an infinite loop that saturated the message broker.

Root cause: The choreography design did not account for the relationship between pricing updates and availability updates. A pricing change can affect which warehouse fulfils an order, which changes availability, which can affect pricing through promotional rules.

Impact: Message broker queue depth grew to 2.3 million messages in 3 minutes before the circuit breaker triggered. The system recovered after restarting both sagas and replaying from the last stable checkpoint.

Lesson learned: Model all cross-saga dependencies explicitly before deploying choreography. Implement circuit breakers per event type, not per saga. Always set message TTL and dead-letter queues for choreography events.

Scenario 3: Orchestrator State Loss

What happened: A Kubernetes pod running the saga orchestrator was terminated during a 12-step order fulfilment saga. The orchestrator had been persisting state every 10 steps, and the saga was at step 7. The replacement pod could not determine whether step 6 had completed.

Root cause: The orchestrator persisted saga state asynchronously to reduce latency, using a write-ahead log with batch commits. The batch had not been committed when the pod was terminated, losing visibility into steps 1–6.

Impact: The replacement pod retried from step 1, causing double reservation of inventory and double payment capture. The idempotency layer caught the double payment but not the double inventory reservation, leading to inventory inconsistency that took 4 hours to reconcile manually.

Lesson learned: Persist saga state synchronously at every step boundary, not in batches. Use the outbox pattern to guarantee that step completion and state persistence are atomic. Consider checkpointing after every step, not every N steps.

Scenario 4: Compensating Transaction Loop

What happened: A saga that books travel (flight, hotel, car) experienced a network partition between the hotel and flight services. The saga compensated the flight booking, then the network partition resolved, and the hotel service’s compensation event arrived late. The orchestrator interpreted the late event as a new failure and began compensating again.

Root cause: Events arrived out of order because the hotel service had a slower consumer group than the flight service. The compensation for the hotel ran after the saga had already reached the Completed state.

Impact: Three customers had their flight rebooked unnecessarily when the second compensation attempt ran after the original compensation had already released the inventory. One customer was moved from their booked flight to a later flight with a 6-hour delay.

Lesson learned: Use version numbers or vector clocks on saga events to detect and discard late events. Implement idempotency in compensation handlers using the original saga ID. Never compensate a saga that has already reached a terminal state.

Common Pitfalls / Anti-Patterns

Treating saga as ACID transaction: Saga does not provide isolation. Concurrent sagas can see each other’s partial results. If saga A reserves inventory and saga B reads inventory before A completes, B may make decisions on uncommitted data. Handle this at the application level.

Non-idempotent steps or compensations: If a step or compensation runs twice due to retries, the effect should be the same as running once. Reserve inventory twice should not double-reserve. Always check before acting.

Compensation order errors: Compensations must run in reverse order of execution. If step 2’s compensation runs before step 1’s, you may leave the system in a worse state. Explicitly track execution order.

Ignoring the inconsistency window: During a saga execution, the system is in an inconsistent state. Other operations may read partial results. Design your application to handle this (show pending states, use optimistic UI).

Long-running compensations: Compensations can take time (refunds, cancellations). While compensating, the system is still inconsistent. Minimize compensation time and alert if compensations are taking too long.

Not planning for compensation failure: What happens if compensation fails? The saga is stuck. Implement retry with backoff, then move to a dead letter state that requires manual intervention.

Quick Recap Checklist

The saga pattern manages distributed transactions without 2PC. Break the transaction into steps, and define a compensation for each step. If a step fails, run the compensations in reverse order. Saga provides eventual consistency, not ACID isolation. Two implementations exist: choreography distributes behavior across services, while orchestration centralizes coordination in an orchestrator. Both work; the choice depends on workflow complexity and team structure. Idempotency is essential for safe retries and preventing duplicate operations.

Reference Checklists

The checklists below consolidate the key requirements from this guide into actionable items. Use the production checklist as a pre-deployment gate: every item should be verified before a saga goes live. The observability checklist covers metrics, logs, and alerts to monitor in production. The security checklist covers authentication, validation, and audit concerns that are easy to overlook when focusing on compensation logic.

Production Checklist

• Saga breaks distributed transactions into steps with compensating transactions
• If a step fails, previous steps are compensated in reverse order
• Saga provides eventual consistency, not ACID isolation
• Two implementations: choreography (distributed) and orchestration (centralized)
• Idempotency is essential for safe retries

Saga Production Checklist

• Each saga step has a defined compensation transaction
• Compensation transactions are tested with injected failures
• Idempotency implemented for all saga steps
• Saga state persisted for crash recovery
• Timeout values tuned for each step (long enough to complete, short enough to recover)
• Maximum retry count configured per step
• Concurrent saga execution handled correctly
• Monitoring and alerting for saga failures configured
• User-facing pending state handling designed
• Dead letter handling for unrecoverable saga states
• Correlation IDs included in saga logs

Observability Checklist

Track progress as well as terminal success and failure. A saga can be neither completed nor failed while a step or compensation is stuck, so alerts should identify stalled instances and the step that needs attention.

Metrics

• Saga completion rate (success vs failure vs compensating)
• Saga execution duration by type
• Compensation execution count and success rate
• Step failure rate by step type
• Concurrent saga count
• Average number of steps per saga

Logs

• Log saga start, each step start and completion, compensation, and outcome with correlation IDs
• Include step index and relevant saga, step, and compensation IDs
• Exclude sensitive input parameters and payload fields

Alerts

• Alert when saga duration exceeds its expected threshold
• Alert when compensation repeatedly fails or a retry limit is reached
• Alert when saga failure rate exceeds its normal baseline
• Alert on sagas with no progress for an extended period

Security and Compliance Notes

Saga state and messages can outlive the request that created them and may be replayed during recovery. Treat the orchestrator, persisted state, event payloads, and replay path as security boundaries.

• Authenticate and authorize saga trigger endpoints; restrict each service identity to the steps and resources it needs
• Validate and minimize saga inputs; avoid copying credentials or unnecessary personal data into persisted state or events
• Encrypt saga state at rest and protect service-to-service traffic in transit
• Redact payment data, secrets, and personal data from logs and traces
• Audit state transitions and compensation decisions with a correlation ID
• Authorize replay and reprocessing operations, and make replayed steps and compensations idempotent

When to Use / When NOT to Use

Use a saga when one business operation commits changes in multiple services, those changes cannot share a local database transaction, and a failure may arrive after earlier steps have committed. The workflow should be allowed to remain pending while services recover, and each completed step needs a defined recovery action. Retries also need stable operation IDs so a timeout does not repeat a charge, reservation, or other side effect.

Do not add a saga when one service and database can enforce the invariant with a local transaction. For a short request that has no independently committed side effects or need to resume after a process restart, a normal request handler is simpler than persisted saga state.

An orchestrator is unnecessary for a small, linear workflow if one service can publish the next event and each participant can act independently. Add orchestration when the workflow needs a durable central state machine, branching decisions, or a clear owner for retries and compensation.

If a completed action cannot be compensated, decide how the business will recover before choosing a saga. For example, a shipment already delivered may require a replacement or a manual case rather than a rollback. If no acceptable compensation or forward-recovery path exists, a saga cannot promise that the workflow will return to its prior state.

Interview Questions

Further Reading

For distributed transaction fundamentals, see Distributed Transactions. For two-phase commit, see Two-Phase Commit.

For workflow patterns, see Service Orchestration and Service Choreography. AWS Prescriptive Guidance: Saga patterns compares choreography and orchestration, while the Temporal workflow documentation explains durable workflow execution.

Conclusion

A saga makes partial commits recoverable by pairing each step with an idempotent compensation and durable state. Choose choreography for simple, independently owned event flows or orchestration when centralized recovery and visibility matter, and design the application to handle the intermediate states that come with eventual consistency.

sequenceDiagram
    participant Saga
    participant Inv as Inventory
    participant Pay as Payment
    Saga->>+Inv: Step 1: Reserve
    Inv->>-Saga: OK
    Saga->>+Pay: Step 2: Charge
    Pay->>-Saga: Fail
    Saga->>+Inv: Compensate