Webhook and API reconciliation during a marketplace cutover

Last substantive review: August 2026.

A marketplace cutover is a business-continuity event disguised as a deployment. The storefront may load and the new API may return healthy responses while a paid order is missing, a refund is applied twice, or a seller payout is calculated from stale state. Those are the failures that matter. A sound plan accounts for every material change, applies each business effect without duplication, and leaves an audit trail the team can reconcile, replay, or compensate.

This guide is for CTOs, platform leaders, and engineering teams moving a live marketplace between platforms or replacing a core service inside one. It covers the control plane around the migration: event inventory, idempotency, ordering, reconciliation, backfills, shadow traffic, dual-read and dual-write choices, observability, cutover, rollback, and the period after launch when late events still arrive. If you are planning the wider data move as well, pair it with Horizon's marketplace migration checklist.

Define what reconciled means before writing the cutover plan

Reconciliation is not a claim that two databases have the same row count. The old and new systems may model orders, users, inventory, disputes, and payouts differently. The cutover team needs a set of business invariants that both models can satisfy.

Write the invariants in language that operations, finance, support, and engineering can all review. For example:

• Every captured payment maps to one marketplace order and one customer-facing receipt.

• An order can have several fulfillment events, but it cannot move from refunded back to fulfilled without an explicit corrective action.

• The sum of seller proceeds, platform fees, taxes, refunds, and payment adjustments equals the amount represented in the payment provider for the same transaction, subject to documented timing differences.

• Every active listing has one authoritative availability state, even if search and cache layers update later.

• Every migrated user and seller identity has a stable cross-system mapping that support can inspect.

For each invariant, define the allowed delay and the action taken when it fails. A five-minute search-index lag may be acceptable. An unexplained payout mismatch may block the cutover. That distinction belongs in the runbook, not in an improvised call after traffic has moved.

The team should also agree on the cutover boundary. Is the source platform authoritative until DNS changes, until a feature flag flips, or until a ledger watermark is reached? Which system accepts writes during rollback? Can buyers still place orders while the team investigates? Clear authority rules prevent two engineers from making opposite fixes to the same record.

Build an event inventory around business effects

Start with an inventory of events and API operations, not a list of webhook endpoint URLs. A provider topic named order.updated may represent several business effects. A single refund may create a payment event, an order-state change, a seller-balance adjustment, and a notification. The inventory needs to show how those effects connect.

For every event family, record:

• the provider, account or tenant, event type, and API version;

• the provider event ID and the stable ID of the affected business object;

• the source of truth for the object and each important field;

• whether delivery can be duplicated, delayed, or unordered;

• the retry window, API backfill method, pagination behavior, and rate limits;

• the consumer that changes marketplace state and any downstream side effects;

• the reconciliation query, expected tolerance, and accountable owner;

• the data sensitivity, retention rule, and redaction requirements for stored payloads.

Include low-volume paths. Chargebacks, partial refunds, manual seller adjustments, disputes, cancellations after fulfillment, account merges, and delayed tax updates are easy to miss in a happy-path rehearsal. They are also the records most likely to require finance or support intervention later.

Create a crosswalk between old IDs, new IDs, and external provider IDs before the rehearsal. Do not infer identity from an email address, display name, or SKU if those values can change or collide. Store the mapping as cutover data with its own validation, ownership, and recovery procedure.

Make event processing idempotent at two levels

Webhook providers retry. Networks time out after a provider has sent a request but before it knows whether the receiver committed the work. A cutover can also replay the same historical window more than once. The consumer must be safe under repetition.

Transport-level deduplication starts with the provider event ID, scoped by provider and account. Claim the ID atomically and store a processing status. A duplicate can be acknowledged without reapplying the business effect only when the first attempt is complete; pending or failed entries must be resumed or quarantined under the retry policy. Both Stripe's webhook guidance and Shopify's webhook guidance tell integrators to expect duplicates; Shopify exposes X-Shopify-Webhook-Id, while Stripe recommends logging processed event IDs.

Business-level idempotency protects against a subtler case: two distinct provider events that request the same effect. A refund operation needs a stable key such as payment ID plus refund ID, not only the webhook delivery ID. A seller-credit job needs a unique ledger entry tied to the economic event. Enforce those keys with database constraints where possible. A check followed by a separate insert can still race under concurrent delivery.

Keep signature verification outside this distinction. Verification establishes that the provider sent the payload; it does not prove that the payload is new. Verify the signature against the unmodified request body, enforce the provider's replay protections, enqueue the accepted event, and return the expected success response promptly. The worker can then perform slower state transitions. Preserve enough metadata to investigate a failure without copying secrets or unnecessary personal data into logs.

Where one transaction changes application state and records an event for downstream delivery, commit both facts together. A transactional outbox or change-data-capture design avoids the gap where the database commits but the message does not, or the message escapes while the database rolls back. AWS Prescriptive Guidance on the transactional outbox pattern also notes that consumers still need idempotency and that event order must be handled deliberately.

Assume events will arrive out of order

Do not let arrival order define marketplace truth. Stripe and Shopify both document that webhook ordering is not guaranteed. During cutover, backfills, retries, and two active delivery paths make reordering even more likely.

Provider semantics are not interchangeable. Stripe documents duplicate deliveries, no event-order guarantee, and automatic live-delivery retries for up to three days. Shopify likewise warns that ordering and delivery are not guaranteed. By contrast, Sharetribe's Integration API event reference gives events a strict total order through sequenceId, returns them in ascending order, retains live-marketplace event history for 90 days, and warns that an event can become queryable after a delay. Use the provider's actual contract: for Sharetribe, persist the last completed sequence ID and poll forward without assuming immediate availability or unlimited history.

Use a version, sequence, or source update time that belongs to the business object. Store the last applied source version with the destination record. Reject a stale state replacement, or route it to review if the source cannot supply a monotonic version. Timestamp comparison alone is risky when clocks differ or two updates share a timestamp, so add a deterministic tie-breaker when the provider supports one.

Model state transitions explicitly. An event that says an invoice was paid may arrive before the event that created the local invoice. The consumer can fetch the current source object, hold the event until its prerequisite arrives, or create a narrowly defined placeholder. What it should not do is silently invent missing state. Record which path was taken.

For money, prefer immutable entries and corrective entries over overwriting balances. For order and listing state, record the prior state, requested transition, source version, and result. That history turns an unexplained mismatch into an inspectable sequence.

Use a reconciliation ledger, not a folder of logs

A reconciliation ledger is the operating record for the cutover. It does not need to be the financial ledger, but it should link every material source object and event to its destination effect.

A useful record includes the provider and account, provider event ID, entity type, source ID, destination ID, event time, received time, source version, payload hash, processing result, attempt count, last error, and correlation ID. Add the cutover run ID so results from rehearsals cannot be confused with production. Store raw payloads only when retention, encryption, and access controls permit it.

The ledger should answer practical questions without an engineer searching several systems:

• Which source orders have no destination mapping?

• Which events were accepted but have no completed business effect?

• Which destination records have no source object?

• Which records disagree on amount, currency, status, owner, or version?

• Which failures are retryable, quarantined, or waiting for a human decision?

Separate a transport success from a business success. A webhook can receive a 2xx response and still fail in the worker. An API request can return successfully while creating a semantically invalid mapping. Track both stages.

Reconcile money and state with different rules

Money should be reconciled by stable transaction identifiers, currency, signed amount, and economic purpose. Compare totals and individual entries. Define how pending transactions, processor fees, reserves, refunds, disputes, and timing differences appear. Do not force a false zero-difference result by excluding records that are hard to map; classify the difference and assign it.

Marketplace state needs transition-aware comparisons. A destination order marked fulfilled may be valid even if the legacy model calls it completed. Build a mapping table that explains equivalent, terminal, and incompatible states. Reconciliation should detect an impossible transition and a missing record, not merely different labels.

Search indexes, analytics stores, and caches are derived state. They still need validation, but they should be rebuilt or compared against the authoritative record rather than promoted to a source of truth.

Design backfill and replay before the outage

A backfill retrieves source objects or events for a bounded period and applies any missing effects. A replay reprocesses stored events. They solve related problems, but they are not interchangeable. The API may expose current object state without a complete event history; a stored event may reproduce an old effect that is no longer valid.

Use a high-water mark per source and entity type. Each job should record its start cursor, end cursor, page count, object count, failures, and completion state. Query with an overlap around the last successful watermark when provider filters are not perfectly precise, then let idempotency remove repeats. Never advance the watermark past a failed page.

Make backfill jobs restartable and rate-limit aware. Test them against production-shaped volume. A script that handles ten thousand orders in staging may not finish the cutover window when the source contains years of order lines, refund records, media, and user metadata.

Replay needs a side-effect policy. Email, SMS, shipments, refunds, and seller transfers generally should not fire again just because an event is replayed. Put external effects behind their own idempotency keys and allow a dry-run mode that records the intended action without executing it. Quarantine events whose payload version can no longer be parsed; do not discard them.

Choose shadow, dual-read, and dual-write modes separately

Parallel operation is not one switch. Reads and writes have different risks, and a team can combine modes.

Shadow traffic

Shadowing sends a copy of production reads or commands to the new path while users still receive the legacy result. Suppress external side effects in the shadow path. Compare normalized results, not raw JSON, so harmless field ordering or generated timestamps do not bury real differences. Sample enough ordinary and edge-case traffic to expose tenant, currency, locale, and permission boundaries.

Dual reads

A compare-only dual read returns the legacy result and records the new result for analysis. A fallback read tries the new system and consults the old one when the record is absent. Fallback can hide migration gaps, so measure every fallback and require the count to fall before launch. If the two systems disagree, define which response wins rather than merging fields ad hoc.

Dual writes

Directly writing the same command to two systems creates a partial-success problem. If both systems must receive a change, prefer one authoritative write followed by an outbox, change-data-capture stream, or another durable propagation mechanism. If a business transaction spans independent services, document compensation and retry behavior. Do not promise atomicity that the underlying providers cannot deliver.

For some cutovers, a short write freeze plus final backfill is safer than weeks of dual writes. For others, the marketplace cannot pause. Make the choice from order volume, tolerance for downtime, external side effects, provider limits, and rollback needs.

Make observability answer the cutover decision

A general application dashboard is not enough. Build a cutover view that shows event intake rate, processing success, duplicate suppression, queue depth, oldest unprocessed event, reconciliation lag, unmatched objects, invalid transitions, API throttling, and errors by provider and tenant. For payments and payouts, show mismatch count and value by currency without exposing sensitive record details.

Attach release markers and cutover run IDs to the graphs. Alert on age as well as volume; a small queue containing a two-hour-old payout event may matter more than a large queue of fresh product updates. Test the alerts during a rehearsal by creating a controlled failure.

Put business gates next to technical metrics. Examples include no unexplained money mismatch, no unmapped active seller, no invalid terminal-state transition, queue age under the agreed threshold, and support access to the cross-system ID map. Assign one person the authority to stop the cutover when a gate fails.

Run the cutover as a sequence of reversible decisions

A production runbook should name the operator, verifier, decision owner, communication channel, command or dashboard, expected result, timeout, and rollback action for every step. The following sequence is a starting point, not a universal schedule:

1. Complete at least one production-scale rehearsal and retain its reconciliation report.

2. Freeze schema, webhook-subscription, mapping, and API-version changes early enough to investigate drift.

3. Confirm credentials, rate limits, queue capacity, backups, restore procedures, feature flags, and provider contacts.

4. Record source counts, ledger watermarks, open financial items, and current queue age.

5. Start durable tail capture before any write freeze or traffic change.

6. Apply the final incremental backfill and reconcile blocking invariants.

7. Move a small traffic cohort when the architecture permits it, then compare business results.

8. Move the remaining traffic only when the named decision owner accepts every gate.

9. Keep the legacy path recoverable and start the post-cutover tail plan.

Write the customer and operator communication before the event. Support should know which symptoms are expected, which require escalation, and which system to inspect. Finance should know when payout or refund operations are paused. A precise internal status message saves time when the team is already handling an anomaly.

Rollback must cover data and external side effects

Rollback is rarely a simple deployment reversal. Code can move back while new orders, payment intents, messages, or seller actions have already happened in the new system. The plan needs a data authority decision for each point in the runbook.

Define the last point at which rollback is automatic. Beyond that point, the safer response may be to stop new writes, reconcile the affected window, and roll forward. Record irreversible actions such as captured payments, refunds, transfers, notifications, or fulfillment requests. Reversing those effects may require a compensating transaction or a manual process; retrying the original command can make the problem worse.

A rollback rehearsal should prove that routing changes work, consumers can be paused without losing queued events, old subscriptions still receive what they need, and the final source of truth is unambiguous. Preserve the failed cutover run as evidence. Do not clear queues or edit ledger rows to make the dashboard green.

Keep a tail period until late work is exhausted

After traffic moves, old events can still arrive, long-running orders can change, refunds can be opened, and provider retries can continue. Keep the legacy system read-only where possible, maintain event capture, and run frequent reconciliation for high-risk entities.

The tail should cover the longest relevant retry or backfill window and the business cycles that create delayed changes. A marketplace with long fulfillment and dispute periods may need targeted legacy access well beyond the technical retry window. End the tail when documented exit criteria are met: no unexplained mismatches, no stale mappings, queues drained, required records retained, rollback authority closed, and operational owners signed off.

Archive the ID maps, runbooks, reconciliation results, risk decisions, and exception log. Schedule a later check for financial and lifecycle events that could not occur during the launch window.

Bring in senior help where the risk concentrates

Horizon worked with the Rarewaters Sharetribe marketplace from the first day of marketplace operations through its acquisition. That is evidence of sustained marketplace operating context, not a claim that Horizon caused the acquisition or that Rarewaters used the cutover method in this guide.

If your cutover crosses payments, payouts, identity, inventory, or several vendor APIs, Horizon can provide senior marketplace and integration engineers at $150–$200 per hour to audit the event model, build the reconciliation controls, pressure-test the runbook, or work through a risky backlog alongside your team. The output is concrete: code, queries, dashboards, decision gates, and a handoff your operators can use. Discuss the cutover with Horizon Labs.

Frequently asked questions

Why aren't webhooks enough to keep two marketplace systems consistent?

Webhooks are a fast notification channel, but deliveries can be duplicated, delayed, processed out of order, or missed during downtime. Reconciliation jobs use the provider API or another authoritative source to find gaps and verify the resulting business state.

How should a team choose the source of truth during cutover?

Name one authoritative system for each entity and field at every phase of the runbook. The choice should reflect where writes are accepted, which external provider owns the record, and whether a rollback has occurred. Do not merge conflicting values without an explicit rule.

Does a zero-downtime marketplace cutover require dual writes?

No. Some teams use one authoritative write plus durable propagation, shadow reads, or a short write freeze with a final backfill. Direct dual writes add partial-success risk and should be used only with defined idempotency, reconciliation, and compensation behavior.

What must be verified before moving production traffic?

Verify cross-system ID mappings, blocking business invariants, queue age, reconciliation results, observability, operator access, backup and restore paths, rollback steps, and ownership for every unresolved exception. The exact thresholds should be approved before the cutover starts.

How long should the post-cutover tail period last?

Keep it open through the longest relevant delivery or backfill window and any business cycle that can create delayed changes, such as fulfillment, refunds, disputes, or payouts. Close it only after the written exit criteria are met and accountable owners sign off.

Primary technical sources

• Shopify developer documentation: About webhooks

• Shopify developer documentation: Troubleshoot webhooks

• Stripe documentation: Receive events in a webhook endpoint

• Stripe documentation: Handle webhook versioning

• AWS Prescriptive Guidance: Transactional outbox pattern

• Sharetribe API reference: Integration API events