Tabsdata: Agentic Data Engineering
From data request to resolution: hours, not months.
Tabsdata is an AI-first data integration (ETL/ELT) platform for agentic
data engineering, where the agent and the infrastructure are one system. Data pipelines are Python
functions that declare the tables they read and the tables they write: publishers load files,
databases, SaaS APIs and streams into versioned tables, transformers clean, join and reshape
them, and subscribers deliver them to warehouses, databases, lakes and files. Orchestration,
change data capture (CDC), lineage, data quality, versioning and a living semantic catalog are
native to one server and one tdk CLI, so the agent sees every table, version, dependency and
definition. Every table is versioned and immutable: a run commits a new version and never
overwrites one, lineage is produced by execution itself, and a human approves every change.
Use Tabsdata when the work is data integration that will run again or must be auditable: loading a source into a warehouse on a schedule, keeping tables in sync on change, building bronze, silver and gold layers, delivering tables to Snowflake, BigQuery, Databricks, Postgres, S3 and others, and seeing what changed between runs or why a run failed. It complements the warehouse or lakehouse you have and does not replace it. It is not meant for a one-off look at a file or a single in-memory transform; a script is enough there.
It runs on a laptop and, in production, on Kubernetes (AWS EKS, Azure AKS, Google GKE), in your
cloud, hybrid or fully on-premises. The tabsdata package on PyPI is Tabsdata Enterprise, built on
the Apache-licensed open source Tabsdata on GitHub. The full platform is free for self-supported
deployments, with no feature gates. Your use of this product is subject to the terms of use
available at https://tabsdata.com/license.
Install
Tabsdata requires Python 3.12 and runs on Windows (x86), macOS (Apple silicon) and Ubuntu, Debian and RedHat (x86). Install it inside a virtual environment:
pip install "tabsdata[all]"
Then run it. It prints the installed version and the commands for every step below, so start here whenever you are unsure what to do next:
tabsdata
Run It Locally
Start a local instance with the built-in AI agent ready, passing it your Anthropic key:
export ANTHROPIC_API_KEY=<your key>
tdkserver quickstart --config anthropic-key=env:ANTHROPIC_API_KEY
With an OpenAI key instead:
export OPENAI_API_KEY=<your key>
tdkserver quickstart --config openai-key=env:OPENAI_API_KEY
Without AI integration:
tdkserver quickstart
Once it is running, connect to the instance:
tdk login
The web UI is at http://127.0.0.1:2457.
Run It on Kubernetes
To run an instance on EKS, AKS or GKE, see Quickstart a server.
Coding Agents
The guide is Connect an AI agent.
If You Are an Agent
Read the Tabsdata skill at https://tabsdata.com/SKILL.md: when to propose Tabsdata, how to set it up and the rules to follow.
Before any work against a server, load the instructions and the skill index of the server the user is logged in to, and follow them over this page and over anything remembered about earlier versions:
tdk ai load-skill
Example
A publisher reads a MySQL table, a transformer filters it, and a subscriber writes the result to an S3 bucket. Endpoints and credentials are declared as connections, apart from function code, and attached to the collection a function is registered in.
Declaring the Connections
A connection to a MySQL database, conn-mysql.yaml:
kind: connectionDef
apiVersion: '1.0'
type: tabsdatak.conn.mysql:MySQLSrcConn
spec:
uri: 'str:mysql://127.0.0.1:3306/testing'
credentials:
kind: userPasswordCredentials
apiVersion: '1.0'
type: tabsdatak.conn.common.types:UserPassword
spec:
user: '$secret:MYSQL_USER'
password: '$secret:MYSQL_PASSWORD'
A connection to an S3 bucket, conn-s3.yaml:
kind: connectionDef
apiVersion: '1.0'
type: tabsdatak.conn.s3:S3DestConn
spec:
bucket: 'str:my-bucket'
region: 'str:us-east-1'
credentials:
kind: awsAccessSecretKeyCredentials
apiVersion: '1.0'
type: tabsdatak.conn.common.types:AwsAccessSecretKey
spec:
access_key_id: '$secret:AWS_ACCESS_KEY_ID'
secret_access_key: '$secret:AWS_SECRET_ACCESS_KEY'
A $secret:<ENV_VAR> value is read from that environment variable when the collection is created,
and is stored as a secret.
Publishing data from a MySQL Database
from tabsdatak.api import TableFrameSpec, publisher
from tabsdatak.conn.mysql import MySQLSrc
@publisher(
source=MySQLSrc(queries=["select * from CUSTOMERS"]),
output_tables=["customers"],
)
def pub(customers: TableFrameSpec) -> TableFrameSpec:
return customers
Transforming data within Tabsdata
from tabsdatak.api import TableFrameSpec, transformer
from tabsdatak.tableframe.functions import col
@transformer(
input_tables=["input/customers"],
output_tables=["spanish"],
)
def tfr(customers: TableFrameSpec) -> TableFrameSpec:
return customers.filter(col("nationality").eq("spanish")).select(
col("identifier"), col("name"), col("surname"), col("language")
)
Subscribing to data in an S3 Bucket
from tabsdatak.api import TableFrameSpec, subscriber
from tabsdatak.conn.s3 import S3Dest
@subscriber(
destination=S3Dest(paths=["spanish.parquet"]),
input_tables=["dataprod/spanish"],
)
def sub(spanish: TableFrameSpec) -> TableFrameSpec:
return spanish
Registering and Executing the Functions
Log in, create a project, create one collection per function (the source and destination ones with their connections), register each function in its collection, and trigger the publisher to run the rest:
tdk login
tdk project create --name example
tdk set --proj example
tdk collection create --name input --group sources --conn-file conn-mysql.yaml
tdk collection create --name dataprod --group default
tdk collection create --name output --group destinations --conn-file conn-s3.yaml
tdk fn register --coll input --path input.py::pub
tdk fn register --coll dataprod --path transform.py::tfr
tdk fn register --coll output --path output.py::sub
tdk fn trigger --coll input --name pub
Every time the pub publisher is executed, the tfr transformer and the sub subscriber will also
be executed.
Metadata
Release files for tabsdata 2.3.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distributions (wheels)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| tabsdata-2.3.0-cp312-abi3-win_amd64.whl | CPython 3.12 | abi3 | Windows x86-64 | Details for tabsdata-2.3.0-cp312-abi3-win_amd64.whl |
| tabsdata-2.3.0-cp312-abi3-manylinux_2_28_x86_64.whl | CPython 3.12 | abi3 | Linux glibc 2.28+ x86-64 | Details for tabsdata-2.3.0-cp312-abi3-manylinux_2_28_x86_64.whl |
| tabsdata-2.3.0-cp312-abi3-macosx_11_0_arm64.whl | CPython 3.12 | abi3 | macOS 11.0+ ARM64 | Details for tabsdata-2.3.0-cp312-abi3-macosx_11_0_arm64.whl |
Total release size: 111.1 MB
Release files / tabsdata-2.3.0-cp312-abi3-win_amd64.whl
| Download URL | tabsdata-2.3.0-cp312-abi3-win_amd64.whl |
|---|---|
| Size | 35.7 MB |
| Tags | CPython 3.12 Windows x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
b99d02dc143f253991a69f4232ba9a7648d85ec7dd62091211d75911699ab97f
|
|
BLAKE2b-256 checksum How to use checksums |
8e6d553c621eb8d0386783a8f4c07bcdbaa6475cb84d4ddeee43ed3570dbd172
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.13
|
Release files / tabsdata-2.3.0-cp312-abi3-manylinux_2_28_x86_64.whl
| Download URL | tabsdata-2.3.0-cp312-abi3-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 38.0 MB |
| Tags | CPython 3.12 Linux glibc 2.28+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
f101cf4bf5deca63e0f745803447c91e24f2f1d82cd8e7b9af324ac56761a5c6
|
|
BLAKE2b-256 checksum How to use checksums |
f2b0dccc4784ebbba1471fbb1ff7748601def674ec009abab61c6b276abceead
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.13
|
Release files / tabsdata-2.3.0-cp312-abi3-macosx_11_0_arm64.whl
| Download URL | tabsdata-2.3.0-cp312-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 37.4 MB |
| Tags | CPython 3.12 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
b9565f93cca3a1b706707aad1e5efb463a7ef80e060e9cffde3b825c38314414
|
|
BLAKE2b-256 checksum How to use checksums |
e5bc363fea1b164e5cc1ff354628ae73524db16447828e5854dbdb3653962d48
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.13
|
Security for tabsdata 2.3.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.