Delivery & data spec

Versioned Parquet, specified precisely

This page is the technical contract for every dataset we sell: how files are laid out, what the manifest guarantees, how to verify downloads, and how versions and corrections behave.

01 · File layout

A deterministic file tree

Every dataset is partitioned by date (UTC), pinned to a schema version in its path, and sorted deterministically inside each file. Two buyers of the same version receive byte-identical files.

Partitioning: one directory per UTC date; large partitions split into multiple parts.
Schema pinning: the schema version is in the path — files never change meaning under you.
Compression: ZSTD Parquet with column statistics for predicate pushdown.

dataset layout

s3://datastore-delivery/<org>/solana/pump_fun/
  buy/
    schema=v1.0/
      date=.../part-0000.parquet
      date=.../part-0001.parquet
      ...
  sell/
    schema=v1.0/...
  _manifests/
    buy.v1.0.manifest.json

02 · The manifest

Every claim, machine-readable

The manifest is the authority on what you bought: the exact schema, the lineage of transformations that produced the data, the quality checks applied, and a checksum for every file.

Manifests are what make our other promises enforceable — if a file doesn't match its checksum, or a schema doesn't match its documentation, that is a defect we correct with a new version.

manifest.json (abridged)

{
  "dataset": "solana/pump_fun",
  "table": "buy",
  "schema_version": "1.0",
  "kind": "instruction",
  "partitioning": ["date"],
  "ordered_by": ["slot", "tx_index", "ix_index"],
  "lineage": [
    { "source": "solana.raw_instructions" },
    { "transform": "idl_decoder pump_fun@pinned" },
    { "transform": "dedup_reorg_resolver" }
  ],
  "files": [
    {
      "path": "date=.../part-0000.parquet",
      "rows": "...",
      "sha256": "..."
    }
  ],
  "checks": {
    "reconciliation": "continuous",
    "deduplication": "enforced",
    "determinism": "reproducible"
  }
}

03 · Download & verify

Signed links, verifiable bytes

After purchase, signed download links arrive at your checkout email. Fetch with any S3-compatible client, then verify every file against the manifest before it enters your pipeline.

Enterprise · drops to your cloud

Instead of downloading, receive new partitions directly in your own S3 or GCS bucket on the dataset's cadence. Enterprise details →

download + verify

# 1. Fetch the manifest, then the files
$ aws s3 sync s3://.../pump_fun/buy/schema=v1.0/ ./data/buy/

# 2. Verify every file against the manifest
$ sha256sum --check manifest.sha256
data/buy/date=.../part-0000.parquet: OK
data/buy/date=.../part-0001.parquet: OK

04 · Versioning & corrections

Published versions never change. Ever.

Immutable publication

Once a version is published, its files and checksums are frozen. Nothing is rewritten in place, for any reason.

Corrections are new versions

A repaired gap or decoder fix produces a new version with a change log describing exactly what changed and why.

Free corrections

If a correction affects a window you purchased, you receive the corrected version at no additional cost.

Schema versions in the path

Queries written against v3.2 files keep working against v3.2 files forever — upgrades are your decision.

Change logs per dataset

Every dataset page carries its version history; every manifest records the lineage that produced it.

Deterministic rebuilds

The same raw capture and decoder version always reproduce the same files, byte for byte.

05 · Engine cookbook

Your engine already speaks Parquet

There is no client library to adopt and no integration project. These are complete, working patterns against the files you download.

duckdb

SELECT date_trunc('day', block_time) AS day,
       count(*)          AS buys,
       count(DISTINCT mint) AS tokens
FROM read_parquet('data/pump_fun/buy/date=*/part-*.parquet')
GROUP BY 1 ORDER BY 1;

pyspark

df = spark.read.parquet("s3://your-bucket/pump_fun/buy/")
df.groupBy("mint") \
  .count().orderBy("count", ascending=False)

clickhouse

SELECT mint, count() AS trades
FROM file('pump_fun/buy/date=*/part-*.parquet', Parquet)
GROUP BY mint
ORDER BY trades DESC
LIMIT 20;

pandas / polars

import polars as pl

buys = pl.scan_parquet(
    "data/pump_fun/buy/date=*/part-*.parquet")
daily = (buys
    .group_by(pl.col("block_time").dt.date())
    .agg(pl.len()))

Ready to pick a dataset? Browse the catalog →