Platform
Boring, verifiable data infrastructure
datastore.sh operates one pipeline from raw chain capture to versioned Parquet publication. This page explains each stage and the discipline that keeps the output trustworthy.
The pipeline
Eight stages from bytes to datasets
01
Raw capture
Blocks, transactions, logs, instructions, and traces captured directly from chain infrastructure.
02
Protocol decoding
Program- and venue-specific decoders turn opaque calldata into typed protocol events.
03
Validation & dedup
Records are checked against chain state, reorgs are resolved, and duplicates are removed.
04
Normalization
Events map onto consistent, documented schemas that hold across protocols and chains.
05
Entity resolution
Tokens, pools, programs, and wallets are resolved to stable identifiers with metadata.
06
Enrichment
Records gain context — pool state, token metadata, pricing references, related events.
07
Versioned publication
Datasets ship with schema versions, lineage records, and change logs. Nothing mutates silently.
08
Delivery — files or firehose
The pipeline forks: datasets publish as partitioned, versioned Parquet for download or cloud drops — and the same decoded tables stream live into your own Kafka, ClickHouse, NATS, or S3.
Methodology
The four disciplines behind every dataset
01
Capture from primary sources
Solana data is captured at instruction level — including inner instructions and account deltas — rather than reconstructed from program logs. Hyperliquid data is captured from L1 blocks and venue records with their native identifiers preserved. Raw capture is retained permanently, so any question about decoding can be settled by replaying the bytes.
02
Decoders pinned to program versions
Protocol upgrades change instruction layouts without warning. Decoders are pinned to specific program deployments; upgrades quarantine the affected stream until the new layout is verified. Every published row records the decoder version that produced it.
03
Reconciliation as a permanent process
Archives are compared continuously against chain state: per-slot instruction counts and content hashes on Solana, monotonic venue identifiers on Hyperliquid. Completeness is a claim we keep re-proving, not a launch-day assertion.
04
Repair means republish
When a gap or defect is found, the affected range is recaptured, rebuilt, and published as a new version with a change log. Previously delivered files remain exactly as their checksums describe — nothing is rewritten under you.
The engineering write-ups go deeper: Solana indexing architecture and measuring historical completeness.
Quality & provenance
Properties you can hold us to
- Documented provenance
- Every dataset records where its raw inputs came from and every transformation applied to them.
- Versioned schemas
- Schemas evolve through explicit versions with change logs. Breaking changes never land silently.
- Dataset lineage
- Derived datasets reference their upstream sources, so you can trace any record back to raw chain data.
- Completeness reconciliation
- Archives are continuously reconciled against chain state to detect and repair gaps.
- Deterministic processing
- Transformations are reproducible: the same raw inputs always produce the same published records.
- Deduplication & validation
- Records are validated against protocol rules and deduplicated across capture paths before publication.
- Freshness indicators
- Each dataset exposes when it was last updated and how far its coverage extends, so pipelines can act on staleness.
- Immutable versions
- Published dataset versions never change under you; corrections ship as new versions with change logs.