Skip to main content

Bundlebase

A versioned dataset that carries its own context, so an AI agent can query it correctly and every number can be traced back to its source.

Hand an agent a CSV and it writes SQL that looks right. A schema says a column is called amt and holds a number; it doesn't say the value is net of refunds, or what your company means by "revenue". A bundle carries that context with the data: what each column means, how your metrics are defined, where every row came from, and every transformation applied. Serve it over MCP and an agent reads all of it in one call.

The same bundle works for people — Python, the CLI, or any BI tool over Arrow Flight SQL.

Documentation | PyPI | Issues

Features

  • Provenance: Every commit records who changed what, when, and how. Every attached file keeps its source and checksum, so any number can be traced back
  • Semantic model: Column descriptions and named metrics travel in the bundle, so everyone (and every agent) computes a number the same way
  • Agent-native: MCP server with describe, analyze, data_quality, and metric tools. bundlebase setup-agent writes the config
  • Version history: Commit, undo, and reset. History is append-only and never rewritten
  • Zero infrastructure: A bundle is a directory of YAML manifests and Parquet files, on local disk, S3, GCS, or Azure. No server, no catalog
  • Works with your tools: Python (pandas, polars, pyarrow), a CLI, and BI tools over Arrow Flight SQL
  • Many sources: Parquet, CSV, and JSONL from local files, S3, HTTP, SFTP, FTP, Kaggle, or PostgreSQL
  • Streaming: Rust core on Apache Arrow and DataFusion. Datasets larger than RAM, constant memory

Installation

pip install bundlebase

Quick Start

Async API

import bundlebase

# Create a new bundle and chain operations
c = await (bundlebase.create()
    .attach("data.parquet")
    .filter("SELECT * FROM bundle WHERE age >= 18")
    .drop_column("ssn")
    .rename_column("fname", "first_name"))

# Convert to pandas
df = await c.to_pandas()

# Commit changes
await c.commit("Cleaned customer data")

Sync API

import bundlebase.sync as bb

# Same operations, no await needed
c = (bb.create()
    .attach("data.parquet")
    .filter("SELECT * FROM bundle WHERE age >= 18")
    .drop_column("ssn")
    .rename_column("fname", "first_name"))

df = c.to_pandas()
c.commit("Cleaned customer data")

Streaming Large Datasets

Process data larger than RAM efficiently:

import bundlebase

# Stream batches instead of loading everything
c = await bundlebase.create().attach("huge_dataset.parquet")

total_rows = 0
async for batch in bundlebase.stream_batches(c):
    # Each batch is a slice of the data, not the entire dataset
    total_rows += batch.num_rows
    # Memory is freed after each iteration

print(f"Processed {total_rows} rows")

Core Operations

Data Loading

c = await bundlebase.create()
c = await c.attach("data.parquet")      # Parquet files
c = await c.attach("data.csv")          # CSV files
c = await c.attach("data.jsonl")        # JSON Lines files

Data Transformation

c = await c.filter("SELECT * FROM bundle WHERE active = true")  # Filter rows
c = await c.drop_column("temp_field")                           # Remove columns
c = await c.rename_column("old", "new")                         # Rename columns

rs = await c.query("SELECT * FROM bundle WHERE id > 10")        # SQL queries
df = await rs.to_pandas()

Data Export

df = await c.to_pandas()    # → pandas DataFrame
df = await c.to_polars()    # → polars DataFrame
arr = await c.to_numpy()    # → dict of NumPy arrays
data = await c.to_dict()    # → Python dict

Indexing

c = await c.create_index("email", "btree")   # Create index for fast lookups
c = await c.rebuild_index("email")           # Rebuild existing index

Joining

c = await bundlebase.create()
c = await c.attach("customers.parquet")
c = await c.join(
    "orders",
    "bundle.customer_id = orders.id",
    location="orders.parquet",
    join_type="inner",
)

Development

Prerequisites

  • Rust (latest stable)
  • Python 3.9+
  • Poetry

Setup

# Install Python dependencies
poetry install

# Build Rust extension
./scripts/maturin-dev.sh

# Run tests
cargo test --profile test       # Rust tests
poetry run pytest python/tests/ # Python tests

Contributing

Contributions are welcome!

License

Distributed under the Apache 2.0 license.

Metadata

Release files for bundlebase 0.13.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for bundlebase 0.13.0
File Size Uploaded Provenance Actions
bundlebase-0.13.0.tar.gz 956.4 kB Without provenance Details for bundlebase-0.13.0.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for bundlebase 0.13.0
File Actions
bundlebase-0.13.0-cp314-cp314-win_amd64.whl CPython 3.14 CPython 3.14 Windows x86-64 Details for bundlebase-0.13.0-cp314-cp314-win_amd64.whl
bundlebase-0.13.0-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl CPython 3.14 CPython 3.14 Linux glibc 2.17+ x86-64 Details for bundlebase-0.13.0-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
bundlebase-0.13.0-cp314-cp314-macosx_11_0_arm64.whl CPython 3.14 CPython 3.14 macOS 11.0+ ARM64 Details for bundlebase-0.13.0-cp314-cp314-macosx_11_0_arm64.whl
bundlebase-0.13.0-cp313-cp313-win_amd64.whl CPython 3.13 CPython 3.13 Windows x86-64 Details for bundlebase-0.13.0-cp313-cp313-win_amd64.whl
bundlebase-0.13.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl CPython 3.13 CPython 3.13 Linux glibc 2.17+ x86-64 Details for bundlebase-0.13.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
bundlebase-0.13.0-cp313-cp313-macosx_11_0_arm64.whl CPython 3.13 CPython 3.13 macOS 11.0+ ARM64 Details for bundlebase-0.13.0-cp313-cp313-macosx_11_0_arm64.whl

Total release size: 395.5 MB

Release files / bundlebase-0.13.0.tar.gz

Download URL bundlebase-0.13.0.tar.gz
Size 956.4 kB
Tags Source
SHA-256 checksum
How to use checksums
8b0eef539b33bc6ee0603a5d72c61d7f53da25af6d5557de29a9f0f325ba691e
BLAKE2b-256 checksum
How to use checksums
caaf95bd725591570b5a994b8d1bfd858d7c090297e3a7db19e52c263b5191cd
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.14

Release files / bundlebase-0.13.0-cp314-cp314-win_amd64.whl

Download URL bundlebase-0.13.0-cp314-cp314-win_amd64.whl
Size 70.6 MB
Tags CPython 3.14 Windows x86-64
SHA-256 checksum
How to use checksums
46f3a21ad77a0de1275607acc678e12555f23fb04bfecb53df7f39416e748d89
BLAKE2b-256 checksum
How to use checksums
1ac03c56e90b5c521ac03a5f2e1110f94c159f716e3f07147890e0110c41aadf
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.14

Release files / bundlebase-0.13.0-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Download URL bundlebase-0.13.0-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Size 66.2 MB
Tags CPython 3.14 Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums
209f39253d99937c0e710bf7ed625cd5724a157056e122313e07ee80cbb99ab9
BLAKE2b-256 checksum
How to use checksums
3c696800b9a035b28305166d81a0acba2707357cd056f2c3fced9a799ae8b331
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.14

Release files / bundlebase-0.13.0-cp314-cp314-macosx_11_0_arm64.whl

Download URL bundlebase-0.13.0-cp314-cp314-macosx_11_0_arm64.whl
Size 60.4 MB
Tags CPython 3.14 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
2012241edc37c7e7a32b62d9f12095a3ee72fc79da58f451288119e2f2930e07
BLAKE2b-256 checksum
How to use checksums
f218944c15d6b866672620c247b942adb0414350a48fb243f9315d8398ce5f27
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.14

Release files / bundlebase-0.13.0-cp313-cp313-win_amd64.whl

Download URL bundlebase-0.13.0-cp313-cp313-win_amd64.whl
Size 70.6 MB
Tags CPython 3.13 Windows x86-64
SHA-256 checksum
How to use checksums
0bc74555825e925266df7bf0f6bb328a35ff0d22152a95d38b806707ccc246e4
BLAKE2b-256 checksum
How to use checksums
0f3f9bc7ebc1c74529884e84d3d00f912d481a6a9420e9ab4a15171a59c0434f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.14

Release files / bundlebase-0.13.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Download URL bundlebase-0.13.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Size 66.2 MB
Tags CPython 3.13 Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums
b2d256c3b288ead61f5a35f4d3fae831e668053b5bf16257c5bba73d53de84f1
BLAKE2b-256 checksum
How to use checksums
fb6ed987a5b5299ef70883f9a6eec7a6df3c0746bce9f08eb95f9e32c6d15ae6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.14

Release files / bundlebase-0.13.0-cp313-cp313-macosx_11_0_arm64.whl

Download URL bundlebase-0.13.0-cp313-cp313-macosx_11_0_arm64.whl
Size 60.4 MB
Tags CPython 3.13 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
ef33ddd359e2d7d8a94f065b1a2f2a1a4599cc0c1062822ff21aac203a9e8541
BLAKE2b-256 checksum
How to use checksums
eaf4508caf38796e01989001cb310bb19323196cd82f3812c4d1af35a5a977dc
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.14

Security for bundlebase 0.13.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page