A webhook can be delivered successfully and still leave the receiving system with an older value. The problem appears when events arrive out of order: a later update is processed first, then an earlier update overwrites it.
This is easy to miss because every request may return a normal success response. The destination has a record, the queue is empty, and no obvious error appears. The data is simply stale.
What out-of-order delivery looks like
Imagine a customer changes a subscription from “trial” to “active,” then quickly cancels it. The source system creates two events:
- Event A: subscription becomes active.
- Event B: subscription becomes canceled.
If Event B arrives first, the destination correctly writes “canceled.” If Event A arrives afterward, a naive consumer writes “active” because it processes messages in arrival order rather than state order.
The same pattern can affect a CRM contact, an order, a donation, an inventory item, a marketing-consent field, or a support ticket. The most dangerous cases are not hard failures. They are valid-looking records that no longer reflect the source.
Why the source and destination see different timelines
Webhook delivery is asynchronous. A provider may retry a failed request, distribute events across workers, or route different event types through different paths. Network latency and consumer load can change the order in which messages reach your endpoint.
Some providers document this explicitly. Stripe, for example, says webhook events are not guaranteed to arrive in the order they are generated and recommends that consumers avoid depending on a fixed sequence. Amazon S3 also documents duplicate and out-of-order event delivery for event notifications.
That does not mean every webhook system behaves the same way. It means your integration should treat arrival order as an input to evaluate, not as proof of business order.
Give each event enough information to be compared
A consumer can only reject stale updates if it has a reliable way to compare them. Depending on the source, that may be:
- A monotonically increasing version number for the record.
- A source-side sequence number for the entity or stream.
- An updated-at timestamp generated by the source.
- An event creation timestamp, when it represents the source change rather than delivery time.
- A state transition identifier with documented ordering rules.
Do not confuse delivery time with change time. The time your endpoint receives a request tells you when the message arrived, not when the source record changed. Store both values when they are available:
- Occurred at: when the source says the change happened.
- Received at: when your system accepted the event.
- Processed at: when your worker applied or rejected the event.
If the provider offers no trustworthy ordering field, consider retrieving the current source object before applying an update. That adds a request and may have rate-limit or latency costs, but it can be safer than trusting a partial event payload.
Choose a stale-update policy before writing the handler
There is no universal rule for every integration. Define the policy per entity and field group.
Last source version wins
Store the latest accepted source version with the destination record. Apply an incoming event only when its version is newer than the stored version. This is usually the clearest option when the source provides a reliable version.
Source timestamp wins
Compare the source’s change timestamp and reject an event that is older than the last accepted change. Define how ties work, and document timestamp precision and timezone assumptions. Clock-based ordering is weaker than a source-generated sequence because timestamps can have limited precision or arrive with unexpected corrections.
Fetch current state
Use the event as a signal, then request the source record and write the current state. This is useful when events are incomplete or when the source exposes a dependable current-state endpoint. It can also turn a burst of events into repeated reads, so add coalescing or a short delay when appropriate.
Queue for reconciliation
When the event cannot be safely compared, hold it for review or a reconciliation job. A delayed, explainable update is better than silently writing a value that may be wrong.
Make stale decisions visible
Rejecting an older event is not the same as losing it. Keep an audit record containing the event ID, entity ID, source version or timestamp, received time, current destination version, decision, and reason.
Do not place full payloads or sensitive fields into logs by default. A compact record with identifiers, status, and decision data is usually enough to investigate the workflow. Protect personal, payment, health, and authentication data according to the system’s access and retention rules.
Useful metrics include:
- Events received by type and source.
- Events accepted, rejected as duplicates, or rejected as stale.
- Events that required a source re-fetch.
- Time between source change and destination processing.
- Records whose destination state differs from a later reconciliation read.
A rising stale-event count may signal a slow consumer, a change in provider behavior, or a new concurrency pattern. Without the metric, the destination may look healthy while its state becomes less reliable.
Test the workflow with deliberate disorder
A test that sends one event and checks one record does not exercise ordering. Build test cases that model the conditions the production workflow must survive.
- Create two or more changes to the same entity.
- Deliver the events in the intended order and confirm the final state.
- Deliver them in reverse order and confirm that the older change cannot overwrite the newer one.
- Deliver the same event twice and confirm that the second receipt is harmless.
- Delay one event, then deliver it after a newer source state has been accepted.
- Send an event with a missing, malformed, or unknown ordering value.
- Force a worker failure after the event is received but before the destination write completes.
- Run a reconciliation check against the source of truth.
Also test concurrency. Two workers can read the same destination version before either writes, then both decide that their events are acceptable. Use a transaction, conditional update, row version, or equivalent compare-and-set operation so the ordering rule is enforced at write time, not only in application memory.
Keep recovery separate from normal delivery
When an event is rejected as stale, do not automatically retry it forever. A retry can be useful for a temporary dependency failure, but it will not make an older event newer.
Use a separate recovery path for events that need investigation or a source re-fetch. That path might re-read the current object, replay a bounded range of events, or run an entity-level reconciliation. Record the result so operators can tell the difference between “received,” “applied,” “rejected as stale,” and “repaired from source.”
This is related to idempotency keys, but the problems are different. Idempotency prevents the same operation from creating a second side effect. Ordering protection prevents an older operation from undoing a newer state.
What to review before trusting a webhook sync
Use this short review with each connected system:
- What identifies an event uniquely?
- What identifies the business entity being changed?
- Which value defines source order?
- Can events be duplicated, delayed, or delivered out of order?
- What happens when the ordering value is missing or tied?
- Where is the last accepted version stored?
- Is the comparison enforced during the write?
- Can the current source record be retrieved for recovery?
- Which events are logged as stale, and who reviews them?
- How often does reconciliation run?
A webhook endpoint is only one part of the system. Reliable synchronization also depends on the queue, worker, database write, retry policy, monitoring, and recovery process around it.
Keep arrival order from becoming business truth
Out-of-order events are a normal property of many asynchronous workflows, not an edge case to hide behind a successful HTTP response. The durable pattern is to compare incoming changes against source state, enforce the decision at write time, keep rejected events observable, and provide a deliberate recovery path.
If your integration can create duplicates, stale records, or hard-to-explain status changes, DigitalWerks can review the event model, identifiers, queue behavior, write conditions, logs, and reconciliation checks. Talk with DigitalWerks about validating the workflow before stale data becomes an operational problem.
Sources: Stripe webhook documentation; Amazon S3 event notification documentation.