Webhook security and idempotency: duplicates, ordering and queues

Who is this for?

Backend developers consuming payment, store, messaging or integration events.

Before you start

Provider signature documentation, a durable event table and a work queue.

1. Verify the exact signature contract

If the provider signs raw bytes, parsing and serializing JSON can change the signed material. Follow the documented algorithm and timestamp checks. Keep secrets and complete payloads out of logs. Do not invent a replacement signature format for a provider integration.

2. Persist before acknowledgment

Store the verified identity, provider and state before acknowledging durable acceptance. Otherwise a crash may lose an already acknowledged event. Process long operations outside the handler. Read what the provider considers success; queued and completed are distinct states.

3. Enforce durable uniqueness

An in-memory seen-event set disappears on restart. Use provider/event uniqueness in storage. Coordinate the business effect and processed state through a transaction or outbox. External effects require additional idempotency because a local SQL transaction cannot cover an external service.

Webhook → Verify → UNIQUE(provider, event_id)
        → Durable outbox → Worker → Effect → processed
Duplicate → previous state / safe reprocessing

4. Handle out-of-order updates

An older subscription event must not overwrite newer state. Use source versions when available and retrieve authoritative state for important transitions. Record source and received times separately. Do not assume all providers offer the same ordering guarantees.

5. Bound retries and support inspection

Separate transient network errors from invalid business data. Use bounded backoff and send exhausted work for review. Audit operator replay. Replay from trusted persisted events rather than bypassing signature verification to patch a delivery.

6. Inject failures

Repeat an event, interrupt the worker after the effect, alter a signature, submit old timestamps and reverse delivery order. Track oldest work age, duplicate rate and completion latency. Do not claim exactly-once behavior without defining and testing the business boundary.

• Reject invalid signatures.
• Apply duplicate effects once.
• Recover after worker restart.
• Audit manual replay.