Skip to content
datastore.sh

Documentation

Everything you need to buy, verify, query — and stream

There is no SDK to install and no API to integrate. You buy datasets and receive Parquet you verify and query with your own engine — or subscribe to a stream and the same tables arrive in your own sink. This page covers both lifecycles.

01

Quickstart

  1. 1. Pick a dataset in the catalog and download its free sample to check the shape.
  2. 2. Choose a coverage window ($200–$2,000) and pay on the dataset page. Every window includes all of the dataset's tables.
  3. 3. Signed download links and the manifest arrive at your checkout email.
  4. 4. Fetch, verify checksums, query with the engine you already run.

02

Data model

A dataset is one Solana program (62 in the catalog). A table is one instruction or event type inside it (1,455 total). Physically, every dataset ships in the same 15-column shape — the ix_name column selects a table, and the decoded payload lives in the json column.

03

Physical schema

Verified identical across all 387 archive files by footer audit. Failed transactions are included and flagged via success.

ColumnTypeDescription
slotu64Solana slot number
tx_indexu32transaction index in block
ix_indexu32top-level instruction index
inner_indexu32inner-instruction index
stack_heightu32CPI stack depth
is_innerbooltrue for inner (CPI) instructions
is_voteboolvote transaction flag
successbooltransaction succeeded (failed txs included)
programstringdecoded program name
program_idstringprogram address
signaturestringtransaction signature
feeu64transaction fee, lamports
compute_unitsu64compute units consumed
ix_namestringdecoded instruction / event name
jsonstringdecoded payload: accounts + args (or event fields)

04

Delivery layout

what a delivery looks like

solana/<dataset>/
  schema=v1.0/                 # 15-column unified shape
    epoch=.../part-0000.parquet
    epoch=.../part-0001.parquet
    ...
  _manifests/
    <dataset>.v1.0.manifest.json   # schema, lineage, sha256 per file

05

Verifying deliveries

fetch + verify

$ aws s3 sync <signed-prefix> ./data/<dataset>/
$ sha256sum --check manifest.sha256
data/<dataset>/epoch=.../part-0000.parquet: OK

A file that fails its checksum is a defect on our side — we reissue the delivery as a new version at no cost.

06

Querying

duckdb — slice one table

SELECT *
FROM read_parquet(
  'data/pumpfun/epoch=*/part-*.parquet')
WHERE ix_name = 'Buy'
  AND success;

duckdb — read the payload

SELECT slot,
  json_extract(json, '$.data.data') AS args
FROM read_parquet(
  'data/pumpfun/epoch=*/part-*.parquet')
WHERE ix_name = 'Buy'
LIMIT 100;

More engine recipes on the delivery page.

07

Samples

Every table page has a free sample download — a synthetic 50-row Parquet file in the exact delivered shape, generated from the table's documented schema. Use it to wire up your pipeline before spending anything. Samples are clearly synthetic; they demonstrate shape, not market data.

08

Stream destinations

A stream subscription delivers a dataset into a sink you control. After checkout you receive a secure link to submit credentials; we verify them with a live test write before the pipeline starts. Grant the minimum: every guide below is scoped to a single prefix, database, or subject tree.

Kafka

kafka setup

# What we need from you (collected after checkout)
bootstrap_servers: broker-1.example.com:9092
security_protocol: SASL_SSL
sasl_mechanism:    SCRAM-SHA-512
sasl_username:     datastore-writer        # write-only principal
topic_prefix:      acme.solana.pumpfun.    # we create nothing outside it

# Recommended ACL — scoped to the prefix, nothing more
kafka-acls --add --allow-principal User:datastore-writer \
  --producer --topic acme.solana.pumpfun. --resource-pattern-type prefixed

One topic per table under your prefix. Messages are protobuf with schema-registry-compatible framing; keys are the deterministic dedup identity.

ClickHouse

clickhouse setup

-- We generate table DDL from the dataset's proto schemas.
-- ReplacingMergeTree + the dedup key = exactly-once in your warehouse.
CREATE TABLE solana_pumpfun.buy (
  slot        UInt64,
  block_time  DateTime64(3),
  signature   String,
  dedup_key   String,          -- block_hash:tx_index:ix_index
  ...
) ENGINE = ReplacingMergeTree
ORDER BY (slot, dedup_key);

-- Grant only what delivery needs
GRANT INSERT ON solana_pumpfun.* TO datastore_writer;

Direct inserts over the native protocol, batched per block range. The generated DDL ships with your connection link — review it before granting.

NATS JetStream

nats setup

# What we need
url:   nats://nats.example.com:4222
auth:  token or .creds file (publish-only account)

# What we publish
subject:      acme.solana.pumpfun.buy      # one per table
Nats-Msg-Id:  <block_hash>:<tx_index>:<ix_index>

# Your stream config enforces dedup on our IDs — and MUST be bounded:
# an unbounded JetStream stream wedges the server once its store fills.
nats stream add PUMPFUN --subjects "acme.solana.pumpfun.>" \
  --dupe-window 10m --max-bytes 10GiB --max-age 72h

Deterministic Nats-Msg-Id headers mean JetStream's duplicate window turns at-least-once delivery into exactly-once persistence. Always set max-bytes/max-age — size them to your consumers' lag tolerance.

S3 · Parquet drops

s3 setup

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["s3:PutObject"],
    "Resource": "arn:aws:s3:::your-bucket/chain-data/pumpfun/*"
  }]
}

# Partitions land on a fixed cadence, same layout as archives:
# chain-data/pumpfun/schema=v1.0/date=.../part-*.parquet

PutObject on one prefix is the entire required surface. Manifests with checksums accompany every drop, exactly like archive deliveries.

09

Stream semantics

Start point
Chain head, a block you choose, or — on the Pro tier — the exact block your purchased archive ends (“Backfill + Tail”). The archive's final block is recorded at fulfillment and becomes the stream's cursor start; the seam is a number both deliveries share.
Delivery guarantee
At-least-once with deterministic dedup keys (block hash + transaction index + instruction index). The pipeline commits its cursor as it delivers, so restarts resume without replaying committed ranges — and any window where duplicates were possible after a failover is recorded, not hidden.
Gaps
Missing blocks and skipped ranges are detected and recorded per pipeline, with recovery status. Your delivery-health page lists them; repaired ranges are re-delivered under the same dedup keys.
Schema changes
Streams are pinned to a schema version. Version bumps are announced ahead of time and never land silently — the same policy as archives.
Monitoring
Every pipeline has a status page: current block, head lag, throughput, and the complete delivery-health record. No login required — the URL is your secret.