Idempotency Keys for Bookings That Fan Out Upstream
This is about idempotency keys for booking systems that fan out one request into three separate reservation calls, and why the "just add a UUID" advice you read in most API docs falls apart the moment you have more than one upstream system involved. The real challenge lies in deciding what happens when leg two fails after leg one already succeeded, and your retry shows up asking to do the whole thing again.
The Setup Nobody Warns You About
Say you're building a trip booking flow. One user action, "book my trip," turns into three separate calls: a flight reservation, a hotel hold, and a rental car confirmation. Each of those lives in a different upstream system, with its own retry behavior, its own timeout window, and its own opinion about what an idempotency key even means.
Single-system idempotency is the easy version of this problem. You've seen the pattern: client generates a key, usually a UUID, sends it in a header, server stores the result keyed off that value, and any retry with the same key just returns the cached response instead of double-charging someone's card. Stripe popularized this approach and it works great when there's one system doing one thing.
The moment you fan out to three systems, that clean story falls apart, because now "the request" isn't one thing. It's three things wearing one coat.
Why One Key Isn't Enough (And Why Three Keys Isn't Either)
Here's the trap engineers fall into first: generate one idempotency key for the whole booking, pass it to all three upstream systems, done. Feels tidy. Feels like you followed the tutorial.
Except now you've got a correlation problem disguised as a solved problem. If the flight system and the hotel system both received the same key, and the flight booking succeeds while the hotel call times out, your retry logic doesn't know which leg actually needs retrying. It just knows "the key" hasn't fully succeeded yet, so it might resend to all three, including the flight system that already did its job and is sitting there, confused, wondering why you're asking it to book the same seat twice with a key it already has on file.
Some systems handle that gracefully and just return the cached response. Others don't, especially third-party reservation APIs you don't control, and you'll find out which kind you're dealing with at the worst possible time (usually during a partial outage, usually on a Friday).
So you swing the other direction: generate three separate keys, one per upstream system. Now each leg is independently idempotent, which solves the resend problem. But you've traded it for a bookkeeping problem, since you need something that ties those three keys back to the single user-facing booking request, or you lose the ability to answer a simple question: did this booking succeed?
The Pattern That Actually Works
The fix is a two-tier key structure. Think of it like a shipping label with a tracking number and three sub-tracking numbers for three separate packages inside one order. One booking-level identifier, generated once when the user hits "confirm," and derived from it, three leg-level idempotency keys, one per upstream system.
A common approach:
- Booking ID: generated client-side or at the API gateway, the moment the request enters your system. This never changes across retries.
- Leg keys: deterministic derivations of the booking ID, something like {bookingid}:flight, {bookingid}:hotel, {booking_id}:car. Hash them if your upstream systems are picky about key format, but keep the derivation deterministic, not random, so a retry produces the exact same leg keys every time.
Retries in distributed systems aren't polite. They don't wait for a clean signal that says "hey, only redo the hotel part." They just fire the whole orchestration again, and your system needs to be able to look at that second attempt and go, "oh, I've seen you before, flight's already done, hotel's still pending, car never got called, let me pick up where I left off."
The real job of the idempotency key here is tracking state per leg, so a retry becomes a resume operation instead of a blind resend.
Where This Gets Genuinely Ugly
Partial failure is the whole game. Full success is easy, and full failure is easy too. It's the middle ground, two legs booked and one timing out, that turns a straightforward feature into a distributed systems problem with a customer's vacation riding on it.
You've got three real options once a leg fails mid-flow:
- Saga pattern with compensating transactions. Flight and hotel succeeded, car failed after retries exhausted? Cancel the flight and hotel bookings, unwind the whole thing, tell the user it didn't work. Clean, but it means every upstream integration needs a working cancel operation, and "working" is doing a lot of heavy lifting in that sentence. Some reservation systems make cancellation slower and less reliable than the original booking call, which is its own special kind of irony.
- Partial success with manual reconciliation. Accept that the user now has a flight and hotel but no car, surface that clearly, and let them rebook the car leg separately. Less elegant, but it doesn't depend on cancel APIs behaving themselves under pressure.
- Hold-then-confirm two-phase pattern. Reserve all three legs first (soft holds, not confirmed bookings), then confirm all three only once every hold succeeds. Airlines and hotels both support this natively in a lot of cases, since it's basically how their own internal systems already work. It's the cleanest option when your upstreams support it, and the most useless option when they don't.
Most production systems I've seen end up running some blend of one and three, with two as the fallback nobody wants to admit is the fallback.
The Part That Actually Ships
Store leg state in a table that survives process restarts. Not in memory, not in a queue message that could get lost, but a real row per leg, with a status column, and the leg-level idempotency key sitting right next to it as a foreign reference back to the booking ID.
When a retry comes in, your orchestrator checks that table before calling anything. Flight already confirmed? Skip it, pull the cached confirmation number. Hotel still pending? Call it again with the same leg key, so if the hotel system already processed the first attempt and just failed to respond in time, it can hand back the same result instead of creating a duplicate reservation.
This is the unglamorous part of the design that determines whether the whole thing actually works. The idempotency key generation is five minutes of code, but the state table that makes retries safe to run is the part that takes an afternoon, a whiteboard, and at least one argument about column naming.
The One-Liner Version of All This
Idempotency in a single system is a lock. Idempotency across three systems is a ledger. You're keeping score of three separate games at once, and the retry logic just needs to know which games are already finished before it decides to play any of them again.
Get the leg-level keys right, get the state table right, and the booking-level ID settles into the role it should've had from the start: a receipt, not a rulebook.
