Measuring historical completeness, and what happens when a gap appears
Methodology · 7 min read · representative content
Completeness is a claim you have to keep proving
Every data vendor says their history is complete. The honest version of that sentence is: complete as of the last time we checked, by a method we can describe. Our method is continuous reconciliation — comparing what the archive contains against what the chain says should exist.
For Solana datasets, the reconciler walks slot ranges and compares per-slot instruction counts and content hashes against independently captured references. Instruction counts are audited per slot, so a gap of even one decoded instruction in one slot is detectable.
How gaps happen
Capture nodes fall behind. RPC providers return partial responses without erroring. A decoder rejects an instruction it should have understood. Gaps are not a sign of a broken pipeline; assuming you have none is.
Detected gaps enter a repair queue with the affected dataset, partition, and slot or height range. Repairs re-run capture and decoding for exactly the affected range, then rebuild only the touched partitions.
Repair means republish, never rewrite
A repaired partition never silently replaces the old one. The dataset gets a new version, the manifest records what changed and why, and previously delivered files remain exactly as their checksums describe. If you bought the affected window, you receive the corrected version at no cost.
This is the property we care most about: at no point can the files you hold diverge from what the manifest says they are.