Webhooks at Scale: Idempotency, Retries, Out-of-Order
Every FDE inherits a customer ERP that sends nightly batch files and webhooks that arrive twice, late, or scrambled. Production hardening here is unglamorous and decides whether finance trusts the system.
The problem in prod
- Duplicate deliveries double-post payments, shipments, or compliance events.
- Out-of-order events apply cancellations before bookings.
- Nightly batch files change schema without notice and break morning syncs.
Loop engineering
Idempotency keys on every inbound event, stored before processing. Replays return the original result instead of re-executing.
Sequence numbers per source entity with a reorder buffer. Late events reconcile against current state instead of overwriting it.
Validate batch files against a contract on arrival. Quarantine unknown columns to a review queue and alert, never silently drop.
Production hardening checklist
- Exponential backoff with jitter on outbound calls, dead-letter queue after max retries.
- Poison-message isolation so one bad record never blocks the whole batch.
- Daily reconciliation report against source totals, signed off by ops.
- Contract tests against customer sandbox on every release, not just go-live week.
Deploying this in MY
GLCs and banks run core systems that only expose batch files plus DuitNow and FPX callbacks with strict timeouts. Design for batch-first with webhook fast-path, and keep all personal data on Malaysia-resident infra per PDPA cross-border rules.
Malaysia FDE guide →