We have seen missed deposits at other providers when webhooks fail. How do you avoid this?
We don’t rely on a single notification path. Think of webhooks as the doorbell and the blockchain as the security camera. Even if the doorbell doesn’t ring, we still watch the camera.- Two signals, one source of truth: We continuously read and reconcile directly from the blockchain with our own indexers, while treating external notifications (API/webhooks) as convenience signals. If a notification is delayed or dropped, our indexers still capture the deposit. For extra resilience, we also cross-check with secondary RPC providers (e.g, Alchemy) to validate data and backfill if any node lags.
- No double-counting: All inputs are sequenced and all outputs are hash-chained. If the same event arrives twice (or late), the sequence/chain checks make duplicates harmless and detect any gap, so the state stays correct.
- Finality/Confirmations & reorg safety: Blockchains can occasionally re-organize the latest blocks. We mark deposits as pending until a chain-specific confirmation threshold is met (i.e., enough blocks are built on top). If a rare short-term reorg happens before that point, the system re-processes those blocks and corrects the event sequence automatically, so you don’t see a false “final” credit. Think of it like card payments: “pending” first, then “settled” after the network is sure. (This is exactly why we reconcile against the chain rather than trusting any single push event.)
-
What we monitor (broken down):
- Block progress (“height drift”): Are our indexers keeping up with the network’s latest blocks? If not, we alert and catch up.
- Ingest health (“lag”): How long between an on-chain event and our system recording it? If that exceeds our threshold, we alert.
- Service health: We track the components that fetch chain data, evaluate policy, and publish events, with runbooks for quick rollback/redeploy when needed. These controls sit alongside our audit trail, which makes selective drops or edits detectable.
- Why this works: The combination of (a) reading from the blockchain itself, (b) sequencing/chain-linking all messages, and (c) immutable audit logs means a missed or duplicate notification can’t cause a wrong balance or a silent drop without being caught.