The Saga Pattern in Node.js: How to Roll Back Distributed Transactions Across Microservices

<p>Building reliable workflows across multiple microservices is challenging. In a monolith, a database transaction can ensure that multiple operations either succeed or fail together. But once data is spread across different services and databases, that guarantee disappears.</p> <p>This is where the Saga Pattern comes in. Instead of using distributed transactions, a saga coordinates a sequence of local transactions and runs compensation actions when something goes wrong.</p> <p>In this article, we'll build an orchestrated Saga Pattern using NestJS, gRPC, PostgreSQL, and Sequelize. You'll learn how to coordinate work across services, implement compensation-based rollbacks, handle idempotency, and track workflow progress in a production-style microservice architecture.</p> <h2 id="heading-table-of-contents">Table of Contents</h2> <ul> <li><p><a href="#heading-prerequisites">Prerequisites</a></p> </li> <li><p><a href="#heading-1-introduction">1. Introduction</a></p> </li> <li><p><a href="#heading-2-the-problem-in-one-picture">2. The Problem in One Picture</a></p> </li> <li><p><a href="#heading-3-why-you-need-a-saga">3. Why You Need a Saga</a></p> </li> <li><p><a href="#heading-4-choreography-vs-orchestration">4. Choreography vs Orchestration</a></p> <ul> <li><p><a href="#heading-choreography">Choreography</a></p> </li> <li><p><a href="#heading-orchestration">Orchestration</a></p> </li> </ul> </li> <li><p><a href="#heading-5-the-example-project">5. The Example Project</a></p> </li> <li><p><a href="#heading-6-architecture">6. Architecture</a></p> </li> <li><p><a href="#heading-7-the-saga-flow-step-by-step">7. The Saga Flow, Step by Step</a></p> </li> <li><p><a href="#heading-8-the-state-machine">8. The State Machine</a></p> </li> <li><p><a href="#heading-9-implementing-the-orchestrator">9. Implementing the Orchestrator</a></p> <ul> <li><p><a href="#heading-creating-the-saga-record">Creating the Saga Record</a></p> </li> <li><p><a href="#heading-the-main-loop">The Main Loop</a></p> </li> <li><p><a href="#heading-a-single-step-in-detail">A Single Step in Detail</a></p> </li> <li><p><a href="#heading-habits-worth-copying">Habits Worth Copying</a></p> </li> </ul> </li> <li><p><a href="#heading-10-implementing-the-participant">10. Implementing the Participant</a></p> </li> <li><p><a href="#heading-11-rollback-compensation">11. Rollback (Compensation)</a></p> <ul> <li><p><a href="#heading-on-the-orchestrator-side">On the Orchestrator Side</a></p> </li> <li><p><a href="#heading-on-the-participant-side">On the Participant Side</a></p> </li> <li><p><a href="#heading-rules-of-a-good-compensation">Rules of a Good Compensation</a></p> </li> <li><p><a href="#heading-what-happens-if-the-compensation-itself-fails">What Happens if the Compensation Itself Fails?</a></p> </li> </ul> </li> <li><p><a href="#heading-12-tracking-idempotency-and-observability">12. Tracking, Idempotency and Observability</a></p> <ul> <li><p><a href="#heading-orchestrator-side-agency_onboarding_sagas">Orchestrator Side — agency_onboarding_sagas</a></p> </li> <li><p><a href="#heading-participant-side-agency_provision_records">Participant Side — agency_provision_records</a></p> </li> <li><p><a href="#heading-observability-for-free">Observability for Free</a></p> </li> </ul> </li> <li><p><a href="#heading-13-testing-a-saga">13. Testing a Saga</a></p> </li> <li><p><a href="#heading-14-when-not-to-use-a-saga">14. When NOT to Use a Saga</a></p> </li> <li><p><a href="#heading-15-trade-offs-and-lessons-learned">15. Trade-offs and Lessons Learned</a></p> </li> <li><p><a href="#heading-16-conclusion">16. Conclusion</a></p> </li> </ul> <h2 id="heading-prerequisites">Prerequisites</h2> <p>This article assumes you're already familiar with some backend development concepts. You don't need prior experience with the Saga Pattern, but you should be comfortable with:</p> <ul> <li><p>JavaScript, TypeScript, Node.js</p> </li> <li><p>NestJS fundamentals (controllers, services, dependency injection)</p> </li> <li><p>Basic PostgreSQL concepts</p> </li> <li><p>Database transactions</p> </li> <li><p>Docker (recommended for local development)</p> </li> <li><p>Microservice architecture basics</p> </li> <li><p>gRPC fundamentals (helpful but not required)</p> </li> </ul> <p>If you've already built a few backend services with NestJS and PostgreSQL, you'll have everything you need to follow this guide.</p> <h2 id="heading-1-introduction">1. Introduction</h2> <p>A <strong>saga</strong> is a sequence of local transactions across multiple services. Each step commits its own database transaction. If a later step fails, the saga runs <strong>compensating transactions</strong> to semantically undo the work already committed.</p> <p>The pattern was first described by Hector Garcia-Molina and Kenneth Salem in 1987 for long-lived database transactions. It was rediscovered a decade ago when companies started splitting monoliths into microservices and realised that the database transaction — the single most powerful tool in a backend developer's belt — stops working at the service boundary.</p> <p>This article walks through an orchestrated saga in Node.js (NestJS + gRPC) for onboarding an agency, where two services must agree on a single business outcome:</p> <ul> <li><p><code>agency-service</code> — owns the agency record.</p> </li> <li><p><code>auth-service</code> — owns the organization, user and role.</p> </li> </ul> <p>If either side fails, the system must end up as if nothing ever happened. No half-created users, orphan organizations, or 3am Slack threads.</p> <h2 id="heading-2-the-problem-in-one-picture">2. The Problem in One Picture</h2> <p>Here's the bug a saga is built to prevent:</p> <pre><code class="language-plaintext">Step 1: auth-service ✅ creates Organization #42 Step 2: auth-service ✅ creates User #99 Step 3: agency-service ❌ fails (DB down, validation, network blip…) Result without a saga: Organization #42 and User #99 still exist. There is no Agency row. The user can log in but has nothing to manage. Support gets a ticket. Engineer writes a one-off SQL cleanup. Repeat every week. </code></pre> <p>The saga's job is to detect that step 3 failed and <strong>explicitly delete Organization #42 and User #99</strong>, so the system is consistent again — even though those rows live in a different service's database.</p> <h2 id="heading-3-why-you-need-a-saga">3. Why You Need a Saga</h2> <p>In a monolith, you wrap everything in one DB transaction and let the database handle atomicity:</p> <pre><code class="language-ts">await sequelize.transaction(async (tx) =&gt; { await Organization.create({...}, { transaction: tx }); await User.create({...}, { transaction: tx }); await Agency.create({...}, { transaction: tx }); }); </code></pre> <p>In microservices, each service has its own database. You can't wrap two services in one ACID transaction. The classic alternatives all have problems:</p> <table> <thead> <tr> <th>Option</th> <th>Problem</th> </tr> </thead> <tbody><tr> <td><strong>Two-Phase Commit (2PC)</strong></td> <td>Locks rows across services, coordinator is a single point of failure, and doesn't scale. Most modern databases don't support it well across HTTP/gRPC.</td> </tr> <tr> <td><strong>"Just hope it works"</strong></td> <td>Leaves orphan users / billing rows when half the flow fails. Real data corruption — and the longer the system runs, the more orphans accumulate.</td> </tr> <tr> <td><strong>Manual cleanup scripts</strong></td> <td>Works for a week. Bugs hide for months. New engineers don't know they exist.</td> </tr> <tr> <td><strong>Eventual consistency without compensation</strong></td> <td>Fine for some domains (analytics) but completely wrong for billing, identity, or anything with money.</td> </tr> <tr> <td><strong>Saga pattern</strong></td> <td>Each service commits locally. The orchestrator owns the workflow and runs explicit compensation on failure. It's auditable, restartable, and reasonable.</td> </tr> </tbody></table> <p>The saga gives you eventual consistency with a clear, auditable rollback path — without distributed locks.</p> <h2 id="heading-4-choreography-vs-orchestration">4. Choreography vs Orchestration</h2> <p>There are two ways to implement a saga:</p> <h3 id="heading-choreography">Choreography</h3> <p>With Choreography, services emit events and other services subscribe and react.</p> <pre><code class="language-plaintext">auth-service → emits "UserCreated" agency-service → listens, creates agency, emits "AgencyCreated" billing-service → listens, creates subscription… </code></pre> <p>It's simple at first, but brittle later. The workflow is scattered across N codebases. Nobody owns it. Debugging means tracing events across logs. Adding a step means changing several services.</p> <h3 id="heading-orchestration">Orchestration</h3> <p>With Orchestration, one service is the conductor. It calls the others in order.</p> <pre><code class="language-plaintext">orchestrator: 1. authClient.provisionAccount(...) 2. agencyRepo.create(...) 3. authClient.sendWelcomeEmail(...) </code></pre> <p>There's slightly more coupling here (the orchestrator imports clients), but the entire workflow lives in one file. Onboarding new engineers becomes a one-hour task. Adding a step is a single PR.</p> <p><strong>Pick orchestration unless you have a strong reason not to.</strong> This article — and the reference implementation — uses orchestration.</p> <h2 id="heading-5-the-example-project">5. The Example Project</h2> <p>Our goal here is to create an Agency in the system. This is the moment a new B2B customer signs up.</p> <p>It requires two services to agree on a single outcome:</p> <p><code>auth-service</code> <strong>must create:</strong></p> <ul> <li><p>an <code>Organization</code> row (the tenant)</p> </li> <li><p>a <code>User</code> row (the agency admin who will log in)</p> </li> <li><p>a <code>UserRole</code> row linking the user to the <code>AGENCY_ADMIN</code> role</p> </li> </ul> <p><code>agency-service</code> <strong>must create:</strong></p> <ul> <li>an <code>Agency</code> row containing business details (size, registration number, website, branches…), linked to the user/organization above</li> </ul> <p>These rows have foreign-key relationships <em>within</em> a service, but <em>not</em> across services — Postgres can't enforce that the user in auth's DB matches the <code>authUserId</code> in agency's DB. The application has to do it.</p> <pre><code class="language-plaintext">auth-service DB agency-service DB ───────────────── ───────────────── organizations ◄────────┐ │ │ │ (1:1) │ foreign reference (no FK) ▼ │ agencies users ──────► user_roles ─ authUserId └ authOrganizationId </code></pre> <p>If step 2 fails <em>after</em> step 1 succeeded, we end up with a user who can authenticate but has no agency — the exact bug from 2. That's what the saga prevents.</p> <h2 id="heading-6-architecture">6. Architecture</h2> <pre><code class="language-plaintext"> ┌───────────────────────────────┐ │ API Gateway │ └──────────────┬────────────────┘ │ HTTP ▼ ┌──────────────────────────────────────────────────┐ │ agency-service │ │ ┌─────────────────────────────────────────┐ │ │ │ AgencyOnboardingOrchestrator (SAGA) │ │ │ └───────────────┬─────────────────────────┘ │ │ │ writes state │ │ ▼ │ │ agency_onboarding_sagas (Postgres) │ └───────────────┬─────────────────┬────────────────┘ │ gRPC │ gRPC provisionAgencyAccount compensateAgencyAccount │ │ ▼ ▼ ┌──────────────────────────────────────────────────┐ │ auth-service │ │ AgencyProvisioningService (Participant) │ │ │ │ organizations · users · user_roles │ │ agency_provision_records ← idempotency log │ └──────────────────────────────────────────────────┘ </code></pre> <p>Three components do all the work:</p> <ol> <li><p><code>AgencyOnboardingOrchestrator</code> in <code>agency-service</code> — drives the workflow.</p> </li> <li><p><code>agency_onboarding_sagas</code> table in <code>agency-service</code> — the durable log of the saga's progress.</p> </li> <li><p><code>AgencyProvisioningService</code> in <code>auth-service</code> — exposes a <code>do</code> operation (<code>provisionAgencyAccount</code>) and an <code>undo</code> operation (<code>compensateAgencyAccount</code>). It's backed by its own <code>agency_provision_records</code> idempotency table.</p> </li> </ol> <p>The orchestrator never reaches into the auth database directly. The boundary is enforced by gRPC.</p> <h2 id="heading-7-the-saga-flow-step-by-step">7. The Saga Flow, Step by Step</h2> <p>This sequence diagram shows the complete lifecycle of the onboarding saga. The workflow begins when a client sends a request to create a new agency. The orchestrator first creates a saga record in its database and marks it as <code>STARTED</code>, giving it a durable record of the workflow before any business action takes place.</p> <p>At a high level, the orchestrator begins by creating a saga record and then asks <code>auth-service</code> to provision the organization, user, and role. Once that succeeds, the orchestrator creates the agency record in its own database.</p> <p>If every step succeeds, the saga reaches the <code>COMPLETED</code> state. If the agency creation fails after the auth resources have already been created, the orchestrator triggers a compensation step that instructs <code>auth-service</code> to remove everything it previously provisioned.</p> <p>The key idea is that each service commits its own local transaction, while the saga coordinates the overall business workflow and ensures the system can return to a consistent state when failures occur.</p> <pre><code class="language-mermaid">sequenceDiagram autonumber participant C as Client participant AS as agency-service&lt;br/&gt;Orchestrator participant DB1 as saga store participant AU as auth-service participant DB2 as auth DB C-&gt;&gt;AS: POST /agencies AS-&gt;&gt;DB1: INSERT saga (STARTED, payload) AS-&gt;&gt;AU: provisionAgencyAccount(sagaId, …) AU-&gt;&gt;DB2: BEGIN TX AU-&gt;&gt;DB2: create org + user + role + provision_record AU-&gt;&gt;DB2: COMMIT AU--&gt;&gt;AS: { userId, organizationId, roleId } AS-&gt;&gt;DB1: UPDATE saga (AUTH_PROVISIONED) AS-&gt;&gt;AS: create Agency row alt Agency row OK AS-&gt;&gt;DB1: UPDATE saga (AGENCY_CREATED → COMPLETED) AS-&gt;&gt;AU: sendAgencyWelcomeEmail (non-critical) AS--&gt;&gt;C: 200 OK + sagaId else Agency row fails AS-&gt;&gt;DB1: UPDATE saga (COMPENSATING) AS-&gt;&gt;AU: compensateAgencyAccount(sagaId) AU-&gt;&gt;DB2: BEGIN TX AU-&gt;&gt;DB2: delete role + token + user + org + record AU-&gt;&gt;DB2: COMMIT AS-&gt;&gt;DB1: UPDATE saga (COMPENSATED → FAILED) AS--&gt;&gt;C: 5xx + error code end </code></pre> <p>Read this once top to bottom and you'll understand the entire onboarding workflow. That's the value of orchestration — the sequence diagram <em>is</em> the architecture.</p> <h2 id="heading-8-the-state-machine">8. The State Machine</h2> <p>Every transition is written to <code>agency_onboarding_sagas</code> <strong>before</strong> the next step runs. That is what makes the saga observable and recoverable.</p> <pre><code class="language-ts">export enum AgencyOnboardingSagaStatus { STARTED = 'STARTED', // Row exists, no side effects yet AUTH_PROVISIONED = 'AUTH_PROVISIONED', // Auth side committed AGENCY_CREATED = 'AGENCY_CREATED', // Agency row committed COMPLETED = 'COMPLETED', // Happy-path terminal state COMPENSATING = 'COMPENSATING', // Rollback in progress COMPENSATED = 'COMPENSATED', // Rollback finished FAILED = 'FAILED', // Terminal failure (with or without compensation) } </code></pre> <p>Why so many states? Because <em>"what went wrong here?"</em> is a question someone will ask at 2am. A saga that only stores <code>success | failure</code> is useless for forensics.</p> <pre><code class="language-plaintext"> ┌── auth fails ──────────► FAILED (nothing to compensate) │ STARTED ──► AUTH_PROVISIONED ──► AGENCY_CREATED ──► COMPLETED (happy path) │ agency fails ───┘ ▼ COMPENSATING │ ▼ COMPENSATED ──► FAILED (consistent again) </code></pre> <p>The “point of no return” is <code>AUTH_PROVISIONED</code>. Before it, we can fail fast — there's nothing to undo. After it, every failure path <em>must</em> go through compensation.</p> <h2 id="heading-9-implementing-the-orchestrator">9. Implementing the Orchestrator</h2> <p>The orchestrator is the <em>only</em> place that knows the workflow. Each step is a private method, and each step persists its result before returning.</p> <h3 id="heading-creating-the-saga-record">Creating the Saga Record</h3> <pre><code class="language-ts">// agency-onboarding.saga.repository.ts async createSaga(payload: CreateAgencyOrchestrationInput) { return this.sagaModel.create({ sagaId: randomUUID(), // correlation id for everything status: AgencyOnboardingSagaStatus.STARTED, currentStep: 'STARTED', payload, // full input snapshot for replay }); } </code></pre> <p>The <code>sagaId</code> is a UUID generated once and <strong>propagated to every downstream call</strong>. It's the single identifier that ties the saga log on the orchestrator side to the provision record on the participant side.</p> <h3 id="heading-the-main-loop">The Main Loop</h3> <pre><code class="language-ts">// agency-onboarding.orchestrator.ts (trimmed for the article) async execute(input: CreateAgencyOrchestrationInput) { const saga = await this.sagaRepository.createSaga(input); // STARTED try { // Step 1 — auth-service work const authStep = await this.provisionAuth(saga, input); if (!authStep.ok) { await this.markFailed(saga, authStep.failure); // nothing to compensate return authStep.failure; } // Step 2 — agency-service work let activeSaga = authStep.saga; // status: AUTH_PROVISIONED try { activeSaga = await this.createAgencyRow(activeSaga, input, authStep.authIds); } catch (err) { // The expensive case: undo what auth-service did await this.compensateAuth(activeSaga, 'SAGA_FAILED'); const failure = mapSagaFailure(err.message, 'SAGA_FAILED', 'CREATE_AGENCY'); await this.markFailed(activeSaga, failure); return failure; } // Step 3 — mark done and run non-critical side effects activeSaga = await this.sagaRepository.updateSaga(activeSaga, { status: AgencyOnboardingSagaStatus.COMPLETED, }); await this.sendWelcomeEmail(input, activeSaga); // best-effort return mapSagaSuccess(activeSaga, await this.agencyModel.findByPk(activeSaga.agencyId!)); } catch (error) { // Defensive catch-all (lost DB connection, unexpected throw) await this.compensateAuth(saga, 'SAGA_FAILED'); const failure = mapSagaFailure(error.message, 'SAGA_FAILED', 'SAGA'); await this.markFailed(saga, failure); return failure; } } </code></pre> <h3 id="heading-a-single-step-in-detail">A Single Step in Detail</h3> <pre><code class="language-ts">private async provisionAuth(saga: AgencyOnboardingSaga, input: ...) { this.logger.log(`[${saga.sagaId}] PROVISION_AUTH`); const auth = await firstValueFrom( this.authClient.provisionAgencyAccount({ sagaId: saga.sagaId, // &lt;-- correlation organizationName: input.agencyName.trim(), email: input.email.trim().toLowerCase(), // … }), ); if (!auth.status || !auth.data) { return { ok: false, failure: mapAuthProvisionFailure(auth) }; } // Persist the IDs we will need if we have to compensate later const updated = await this.sagaRepository.updateSaga(saga, { authOrganizationId: Number(auth.data.organizationId), authUserId: Number(auth.data.userId), authUserRoleId: Number(auth.data.userRoleId), status: AgencyOnboardingSagaStatus.AUTH_PROVISIONED, }); return { ok: true, saga: updated, authIds: auth.data }; } </code></pre> <p>The line that does most of the work is the <code>updateSaga</code> call. It stores the foreign IDs returned by <code>auth-service</code> on the saga row, so even if the orchestrator process crashes and restarts, a recovery job can read that row and still know what to compensate.</p> <h3 id="heading-habits-worth-copying">Habits Worth Copying</h3> <ul> <li><p><strong>Persist after every successful step</strong>, including the IDs you'll need to undo it.</p> </li> <li><p><strong>Distinguish critical vs non-critical steps.</strong> Welcome emails, audit logs and analytics events are <em>not</em> worth rolling a saga back for. They're best-effort.</p> </li> <li><p><strong>One log line per transition</strong>, prefixed with <code>[${sagaId}]</code>. Grep is your debugger.</p> </li> </ul> <h2 id="heading-10-implementing-the-participant">10. Implementing the Participant</h2> <p>The participant (<code>auth-service</code>) wraps all of its own work in a local DB transaction. Inside that boundary it's still ACID — the saga only handles the cross-service problem.</p> <pre><code class="language-ts">// agency-provisioning.service.ts (trimmed) async provisionAgencyAccount(req: ProvisionAgencyAccountInput) { // 1. Idempotency — return the previous result if this sagaId already provisioned. const existing = await this.provisionRecordModel.findOne({ where: { sagaId: req.sagaId }, }); if (existing) { return serviceSuccess('Agency admin already onboarded', { userId: Number(existing.userId), organizationId: Number(existing.organizationId), userRoleId: Number(existing.roleId), }); } // 2. Domain validation BEFORE the transaction (fail fast). if (await this.emailExists(req.email)) { return serviceFailure('Email already exists', { code: 'EMAIL_EXISTS' }); } if (await this.organizationExists(req.organizationName)) { return serviceFailure('Organization already exists', { code: 'ORGANIZATION_EXISTS' }); } // 3. The actual work — atomic at the auth-service boundary. return withSequelizeTransaction(this.sequelize, async (tx) =&gt; { const org = await this.organizationModel.create({ ... }, { transaction: tx }); const user = await this.userModel.create({ ..., organizationId: org.id }, { transaction: tx }); await this.userRoleModel.create({ userId: user.id, roleId: agencyAdminRole.id }, { transaction: tx }); // The audit record that makes compensation possible later. await this.provisionRecordModel.create( { sagaId: req.sagaId, organizationId: org.id, userId: user.id, roleId: agencyAdminRole.id }, { transaction: tx }, ); return serviceSuccess('Provisioned', { userId: user.id, organizationId: org.id, userRoleId: agencyAdminRole.id, }); }); } </code></pre> <p>Three things make this method "saga-safe":</p> <ol> <li><p><strong>Idempotency check first:</strong> If the orchestrator retries (network blip, gRPC timeout), the second call is a no-op that returns the same IDs. No duplicate users.</p> </li> <li><p><strong>Validation outside the transaction:</strong> Cheap reads first, expensive writes second.</p> </li> <li><p><strong>One transaction wraps every write:</strong> If any insert fails, the whole thing rolls back automatically. The orchestrator sees a clean failure response and knows nothing was persisted.</p> </li> </ol> <p>The <code>agency_provision_records</code> table is the single most important piece of the participant. It's <strong>both</strong> the idempotency key <em>and</em> the compensation lookup — keyed by the same <code>sagaId</code> the orchestrator uses.</p> <h2 id="heading-11-rollback-compensation">11. Rollback (Compensation)</h2> <p>Compensation is just another gRPC call. The orchestrator sends the <code>sagaId</code> and the IDs it remembers. The participant deletes everything it created, <strong>in reverse dependency order</strong>, inside its own DB transaction.</p> <h3 id="heading-on-the-orchestrator-side">On the Orchestrator Side</h3> <pre><code class="language-ts">private async compensateAuth(saga: AgencyOnboardingSaga, errorCode?: string) { if (!saga.authUserId &amp;&amp; !saga.authOrganizationId) { // Nothing was provisioned — nothing to compensate. return; } // Mark the saga as compensating BEFORE the call, so the row is consistent // even if the compensating RPC times out. await this.sagaRepository.updateSaga(saga, { status: AgencyOnboardingSagaStatus.COMPENSATING, currentStep: 'COMPENSATING', errorCode, }); try { const rollback = await firstValueFrom(this.authClient.compensateAgencyAccount({ sagaId: saga.sagaId, organizationId: saga.authOrganizationId, userId: saga.authUserId, })); if (!rollback.status) { this.logger.error(`[\({saga.sagaId}] Auth compensation returned failure: \){rollback.message}`); } } catch (err) { this.logger.error(`[\({saga.sagaId}] Auth compensation RPC failed: \){err.message}`); } await this.sagaRepository.updateSaga(saga, { status: AgencyOnboardingSagaStatus.COMPENSATED, currentStep: 'COMPENSATED', }); } </code></pre> <h3 id="heading-on-the-participant-side">On the Participant Side</h3> <pre><code class="language-ts">private async rollbackProvisionedAuth(req, sagaId: string, tx: Transaction) { // Use the saga log as the source of truth — even if the caller forgot IDs. const record = await this.provisionRecordModel.findOne({ where: { sagaId }, transaction: tx, }); const userId = req.userId ?? record?.userId; const organizationId = req.organizationId ?? record?.organizationId; if (userId) { const user = await this.userModel.findByPk(userId, { transaction: tx, attributes: ['email'] }); await this.userRoleModel.destroy({ where: { userId }, transaction: tx }); if (user?.email) { await this.passwordResetTokenModel.destroy({ where: { email: user.email }, transaction: tx }); } await this.userModel.destroy({ where: { id: userId }, transaction: tx }); } if (organizationId) { await this.organizationModel.destroy({ where: { id: organizationId }, transaction: tx }); } if (record) { await record.destroy({ transaction: tx }); } } </code></pre> <h3 id="heading-rules-of-a-good-compensation">Rules of a Good Compensation</h3> <ol> <li><p><strong>Reverse the order of creation:</strong> Children first (user_roles, tokens), then parents (users, organizations). The same rule you follow for <code>DROP TABLE</code> statements.</p> </li> <li><p><strong>Be idempotent:</strong> Receiving the same <code>sagaId</code> twice must be safe — every <code>destroy</code> is a no-op if the row is already gone.</p> </li> <li><p><strong>Use the saga log, not just the request:</strong> If the caller forgets an ID or sends a partial payload, look it up by <code>sagaId</code>. Defence in depth.</p> </li> <li><p><strong>Wrap it in a local transaction:</strong> The rollback must itself be atomic — half-undone is worse than not-undone.</p> </li> <li><p><strong>Always close the loop on the orchestrator side:</strong> Mark <code>COMPENSATED</code> even if the RPC failed. The failure should also be surfaced (log, metric, alert). A stuck <code>COMPENSATING</code> row is an operational landmine.</p> </li> </ol> <h3 id="heading-what-happens-if-the-compensation-itself-fails">What Happens if the Compensation Itself Fails?</h3> <p>This is the worst case in any saga design. There are three reasonable strategies:</p> <p>First, you can retry with exponential backoff. This works for transient failures (network, deadlocks).</p> <p>Second, you can dead-letter the saga — write it to a "needs human attention" queue and alert.</p> <p>Third, you can expose a manual rollback endpoint. This reference implementation does that via <code>RollbackAgencyOnboarding</code> gRPC, so an operator can replay compensation with the same <code>sagaId</code>.</p> <p>A production system should combine all three. The pattern doesn't decide for you. <em>You</em> decide based on your business risk.</p> <h2 id="heading-12-tracking-idempotency-and-observability">12. Tracking, Idempotency and Observability</h2> <p>Two tables, both keyed by the same UUID <code>sagaId</code>, give you full traceability across services.</p> <h3 id="heading-orchestrator-side-agencyonboardingsagas">Orchestrator Side — <code>agency_onboarding_sagas</code></h3> <table> <thead> <tr> <th>column</th> <th>purpose</th> </tr> </thead> <tbody><tr> <td><code>sagaId</code> (UUID, unique)</td> <td>Propagated to every RPC. The join key across services.</td> </tr> <tr> <td><code>status</code></td> <td>Current state in the state machine.</td> </tr> <tr> <td><code>currentStep</code></td> <td>Human-readable label for dashboards (<code>PROVISION_AUTH</code>, <code>CREATE_AGENCY</code>…).</td> </tr> <tr> <td><code>payload</code> (JSONB)</td> <td>Snapshot of the input — used for replay, debug, support.</td> </tr> <tr> <td><code>authOrganizationId</code>, <code>authUserId</code>, <code>authUserRoleId</code></td> <td>Foreign IDs needed for compensation.</td> </tr> <tr> <td><code>agencyId</code></td> <td>Set once the agency row exists.</td> </tr> <tr> <td><code>errorCode</code>, <code>errorMessage</code></td> <td>Filled on failure.</td> </tr> <tr> <td><code>createdAt</code>, <code>updatedAt</code></td> <td>Timeline for the saga.</td> </tr> </tbody></table> <p>A real row in <code>COMPLETED</code> state looks roughly like this:</p> <pre><code class="language-json">{ "sagaId": "0a4f3e2c-7b11-4f8d-9a2c-90b6f5f5b8a1", "status": "COMPLETED", "currentStep": "COMPLETED", "agencyId": 17, "authOrganizationId": 42, "authUserId": 99, "authUserRoleId": 3, "errorCode": null, "errorMessage": null, "payload": { "agencyName": "Acme Education", "email": "admin@acme.com", "...": "..." }, "createdAt": "2026-05-22T10:14:32.118Z", "updatedAt": "2026-05-22T10:14:33.412Z" } </code></pre> <h3 id="heading-participant-side-agencyprovisionrecords">Participant Side — <code>agency_provision_records</code></h3> <table> <thead> <tr> <th>column</th> <th>purpose</th> </tr> </thead> <tbody><tr> <td><code>sagaId</code> (unique)</td> <td>Idempotency key. The same <code>sagaId</code> from the orchestrator.</td> </tr> <tr> <td><code>userId</code>, <code>organizationId</code>, <code>roleId</code></td> <td>What to delete on compensation.</td> </tr> <tr> <td><code>createdAt</code>, <code>updatedAt</code></td> <td>Audit timestamps.</td> </tr> </tbody></table> <h3 id="heading-observability-for-free">Observability for Free</h3> <p>Because every log line is prefixed with <code>[${sagaId}]</code>, a single grep across both services gives the full timeline:</p> <pre><code class="language-plaintext">[0a4f3e2c…] PROVISION_AUTH agency-service [0a4f3e2c…] provisionAgencyAccount: ok auth-service [0a4f3e2c…] CREATE_AGENCY agency-service [0a4f3e2c…] Agency step failed: ... agency-service [0a4f3e2c…] Auth compensation completed auth-service </code></pre> <p>In a structured-logging setup (Loki, Elasticsearch, Datadog) this becomes a one-click filter. <strong>The</strong> <code>sagaId</code> <strong>is your distributed trace.</strong></p> <h2 id="heading-13-testing-a-saga">13. Testing a Saga</h2> <p>A saga is just a state machine, so the test matrix is finite and small. Cover at least these cases:</p> <table> <thead> <tr> <th>#</th> <th>Scenario</th> <th>Expected end state</th> </tr> </thead> <tbody><tr> <td>1</td> <td>Happy path</td> <td><code>COMPLETED</code>, agency exists, user exists</td> </tr> <tr> <td>2</td> <td>Auth step fails (e.g. email exists)</td> <td><code>FAILED</code>, no rows on either side</td> </tr> <tr> <td>3</td> <td>Agency step fails</td> <td><code>COMPENSATED</code>, auth rows gone, no agency</td> </tr> <tr> <td>4</td> <td>Compensation RPC times out</td> <td><code>COMPENSATING</code> → operator-driven recovery</td> </tr> <tr> <td>5</td> <td>Caller retries with the same <code>sagaId</code></td> <td>Second call returns the first call's result; no duplicate rows</td> </tr> <tr> <td>6</td> <td>Welcome email fails</td> <td><code>COMPLETED</code> still — non-critical step did not cascade</td> </tr> </tbody></table> <p>Two practical tips for testing:</p> <p>First, mock the gRPC client at the orchestrator level, not the network. You want to assert that <code>compensateAgencyAccount</code> <em>was called with the right</em> <code>sagaId</code>, not that bytes hit a socket.</p> <p>Second, spin up a real Postgres in integration tests (Testcontainers, or a Docker Compose <code>postgres</code> service). The saga state machine is too easy to "test" against a mock and too easy to break against a real DB.</p> <h2 id="heading-14-when-not-to-use-a-saga">14. When NOT to Use a Saga</h2> <p>Sagas are not free. Skip them when:</p> <ul> <li><p><strong>One service does all the writes.</strong> Use a regular DB transaction. Don't reinvent the wheel.</p> </li> <li><p><strong>The workflow is read-only or analytical.</strong> No rollback semantics exist for a SELECT.</p> </li> <li><p><strong>The "rollback" is impossible.</strong> You sent a real email. You charged a credit card and the gateway doesn't support refunds. In those cases, design forward: send an apology email, queue a manual refund. Sagas can't unsend physical actions.</p> </li> <li><p><strong>You don't actually have multiple services yet.</strong> A saga in a monolith is over-engineering. Wait until the service boundary is real.</p> </li> </ul> <p>A saga adds a state table, a compensation method per step, and an operational habit of grepping by <code>sagaId</code>. That cost is worth paying when the alternative is orphaned data — and not before.</p> <h2 id="heading-15-trade-offs-and-lessons-learned">15. Trade-offs and Lessons Learned</h2> <p>Things that worked well in this design:</p> <ul> <li><p>Synchronous orchestration is easier to debug than choreography. A new engineer reads one file and understands the whole flow.</p> </li> <li><p>Idempotency at the participant is non-negotiable. Retries from the orchestrator must be safe. Build it in from day one — retro-fitting is painful.</p> </li> <li><p>The saga table replaces tribal knowledge. Ops can answer <em>"what happened to this signup?"</em> with a single SQL query. The payload JSONB is gold during incidents.</p> </li> <li><p><code>sagaId</code> as the trace key plays nicely with OpenTelemetry / Datadog / Loki — no extra infra to set up.</p> </li> </ul> <p>Things to know before copying this pattern:</p> <ul> <li><p>A failing compensation is the worst case. If <code>compensateAgencyAccount</code> itself errors, you have inconsistent state. Plan for retries + dead-letter + a manual rollback endpoint from the start.</p> </li> <li><p>Non-critical steps must be marked explicitly. Here, the welcome email is allowed to fail without rolling back the agency. Don't accidentally compensate over a flaky SMTP provider.</p> </li> <li><p>Sagas aren't a replacement for local transactions. Inside each service, still use a real DB transaction. The saga only handles the cross-service seam.</p> </li> <li><p>Synchronous gRPC is simple but couples availability. If <code>auth-service</code> is down, agency creation fails. Swap the gRPC calls for a durable message bus (RabbitMQ / Kafka) and treat each step as a command + reply when you need higher resilience.</p> </li> <li><p>The orchestrator becomes a critical service. Treat its uptime accordingly — monitor saga durations, alert on stuck <code>COMPENSATING</code> rows, and run more than one replica.</p> </li> </ul> <h2 id="heading-16-conclusion">16. Conclusion</h2> <p>The saga pattern isn't magic. It's a disciplined version of what experienced engineers already do by hand: <em>commit locally, record what you did, and know how to undo it.</em></p> <p>In Node.js with NestJS, you only need three ingredients:</p> <ol> <li><p><strong>A state table</strong> to track the saga.</p> </li> <li><p><strong>An orchestrator</strong> that drives the workflow and writes that state.</p> </li> <li><p><strong>A participant</strong> that exposes a <code>do</code> and an <code>undo</code> operation, both idempotent and keyed by <code>sagaId</code>.</p> </li> </ol> <p>Get those three right and your microservices can offer the same "all-or-nothing" feel as a monolithic transaction — without the operational pain of distributed locks.</p> <p>Start simple, use orchestration, make every step idempotent, persist before you call, and always know how to undo. That's the whole pattern.</p>