paradigm_absorb 0.3.0

python interface for interacting with flashbots mempool dumpster

absorb 🧽🫧🫧

the sovereign dataset manager

absorb makes it easy to 1) collect, 2) query, 3) manage, and 4) customize datasets from nearly any data source

Features

• limitless dataset library: access to millions of datasets across 20+ diverse data sources
• intuitive cli+python interfaces: collect or query any dataset in a single line of code
• maximal modularity: built on open standards for frictionless integration with other tools
• easy extensibility: add new datasets or data sources with just a few lines of code

Contents

1. Installation
2. Example Usage
   1. Command Line
   2. Python
3. Supported Data Sources
4. Output Format
5. Configuration

Installation

basic installation

uv tool install paradigm_absorb

install with all extras

uv tool install paradigm_absorb[test,datasources,interactive]

install from source

git clone [e-mail]:paradigmxyz/absorb.git
uv tool install --editable .[test,datasources,interactive]

Example Usage

Example Command Line Usage

# collect dataset and save as local files
absorb collect kalshi

# list datasets that are collected or available
absorb ls

# show schemas of dataset
absorb schema kalshi

# create new custom dataset
absorb new custom_dataset

# upload custom dataset
absorb upload custom_dataset

Example Python Usage

import absorb

# collect dataset and save as local files
absorb.collect('kalshi')

# list datasets that are collected or available
datasets = absorb.list()

# get schemas of dataset
schema = absorb.schema('kalshi')

# load dataset as polars DataFrame
df = absorb.load('kalshi')

# scan dataset as polars LazyFrame
lf = absorb.scan('kalshi')

# create new custom dataset
absorb.new('custom_dataset')

# upload custom dataset
absorb.upload('custom_dataset')

Supported Data Sources

absorb collects data from each of these sources:

• 4byte function and event signatures
• allium crypto data platform
• bigquery crypto ETL datasets
• binance trades and OHLC candles on the Binance CEX
• blocknative Ethereum mempool archive
• chain_ids chain id's
• coingecko token prices
• cryo EVM datasets
• defillama DeFi data
• dune tables and queries
• fred federal macroeonomic data
• git commits, authors, and file diffs of a repo
• growthepie L2 metrics
• kalshi prediction market metrics
• l2beat L2 metrics
• mempool dumpster Ethereum mempool archive
• snowflake generalized data platform
• sourcify verified contracts
• tic usa treasury department data
• tix price feeds
• vera verified contract archives
• xatu many Ethereum datasets

To list all available datasets and data sources, type absorb ls on the command line.

Output Format

To display information about the schema and other metadata of a dataset, type absorb help <DATASET> on the command line.

absorb stores each dataset as a collection of parquet files.

Datasets can be stored in any location on your disks, and absorb will use symlinks to organize those files in the ABSORB_ROOT tree.

the ABSORB_ROOT filesystem directory is organized as:

{ABSORB_ROOT}/
    datasets/
        <source>/
            tables/
                <datatype>/
                    {filename}.parquet
                table_metadata.json
            repos/
                {repo_name}/
    absorb_config.json

Configuration

absorb uses a config file to specify which datasets to track.

Schema of absorb_config.json:

{
    'tracked_tables': list[TableDict]
}

schema of dataset_config.json:

{
    "name": str,
    "definition": str,
    "parameters": dict[str, Any],
    "repos": [str]
}