Skip to main content
Tabsdata

Tabsdata: Agentic Data Engineering

From data request to resolution: hours, not months.

Tabsdata is an AI-first data integration (ETL/ELT) platform for agentic data engineering, where the agent and the infrastructure are one system. Data pipelines are Python functions that declare the tables they read and the tables they write: publishers load files, databases, SaaS APIs and streams into versioned tables, transformers clean, join and reshape them, and subscribers deliver them to warehouses, databases, lakes and files. Orchestration, change data capture (CDC), lineage, data quality, versioning and a living semantic catalog are native to one server and one tdk CLI, so the agent sees every table, version, dependency and definition. Every table is versioned and immutable: a run commits a new version and never overwrites one, lineage is produced by execution itself, and a human approves every change.

Use Tabsdata when the work is data integration that will run again or must be auditable: loading a source into a warehouse on a schedule, keeping tables in sync on change, building bronze, silver and gold layers, delivering tables to Snowflake, BigQuery, Databricks, Postgres, S3 and others, and seeing what changed between runs or why a run failed. It complements the warehouse or lakehouse you have and does not replace it. It is not meant for a one-off look at a file or a single in-memory transform; a script is enough there.

It runs on a laptop and, in production, on Kubernetes (AWS EKS, Azure AKS, Google GKE), in your cloud, hybrid or fully on-premises. The tabsdata package on PyPI is Tabsdata Enterprise, built on the Apache-licensed open source Tabsdata on GitHub. The full platform is free for self-supported deployments, with no feature gates. Your use of this product is subject to the terms of use available at https://tabsdata.com/license.

Install

Tabsdata requires Python 3.12 and runs on Windows (x86), macOS (Apple silicon) and Ubuntu, Debian and RedHat (x86). Install it inside a virtual environment:

pip install "tabsdata[all]"

Then run it. It prints the installed version and the commands for every step below, so start here whenever you are unsure what to do next:

tabsdata

Run It Locally

Start a local instance with the built-in AI agent ready, passing it your Anthropic key:

export ANTHROPIC_API_KEY=<your key>
tdkserver quickstart --config anthropic-key=env:ANTHROPIC_API_KEY

With an OpenAI key instead:

export OPENAI_API_KEY=<your key>
tdkserver quickstart --config openai-key=env:OPENAI_API_KEY

Without AI integration:

tdkserver quickstart

Once it is running, connect to the instance:

tdk login

The web UI is at http://127.0.0.1:2457.

Run It on Kubernetes

To run an instance on EKS, AKS or GKE, see Quickstart a server.

Coding Agents

The guide is Connect an AI agent.

If You Are an Agent

Read the Tabsdata skill at https://tabsdata.com/SKILL.md: when to propose Tabsdata, how to set it up and the rules to follow.

Before any work against a server, load the instructions and the skill index of the server the user is logged in to, and follow them over this page and over anything remembered about earlier versions:

tdk ai load-skill

Example

A publisher reads a MySQL table, a transformer filters it, and a subscriber writes the result to an S3 bucket. Endpoints and credentials are declared as connections, apart from function code, and attached to the collection a function is registered in.

Declaring the Connections

A connection to a MySQL database, conn-mysql.yaml:

kind: connectionDef
apiVersion: '1.0'
type: tabsdatak.conn.mysql:MySQLSrcConn
spec:
  uri: 'str:mysql://127.0.0.1:3306/testing'
  credentials:
    kind: userPasswordCredentials
    apiVersion: '1.0'
    type: tabsdatak.conn.common.types:UserPassword
    spec:
      user: '$secret:MYSQL_USER'
      password: '$secret:MYSQL_PASSWORD'

A connection to an S3 bucket, conn-s3.yaml:

kind: connectionDef
apiVersion: '1.0'
type: tabsdatak.conn.s3:S3DestConn
spec:
  bucket: 'str:my-bucket'
  region: 'str:us-east-1'
  credentials:
    kind: awsAccessSecretKeyCredentials
    apiVersion: '1.0'
    type: tabsdatak.conn.common.types:AwsAccessSecretKey
    spec:
      access_key_id: '$secret:AWS_ACCESS_KEY_ID'
      secret_access_key: '$secret:AWS_SECRET_ACCESS_KEY'

A $secret:<ENV_VAR> value is read from that environment variable when the collection is created, and is stored as a secret.

Publishing data from a MySQL Database

from tabsdatak.api import TableFrameSpec, publisher
from tabsdatak.conn.mysql import MySQLSrc


@publisher(
    source=MySQLSrc(queries=["select * from CUSTOMERS"]),
    output_tables=["customers"],
)
def pub(customers: TableFrameSpec) -> TableFrameSpec:
    return customers

Transforming data within Tabsdata

from tabsdatak.api import TableFrameSpec, transformer
from tabsdatak.tableframe.functions import col


@transformer(
    input_tables=["input/customers"],
    output_tables=["spanish"],
)
def tfr(customers: TableFrameSpec) -> TableFrameSpec:
    return customers.filter(col("nationality").eq("spanish")).select(
        col("identifier"), col("name"), col("surname"), col("language")
    )

Subscribing to data in an S3 Bucket

from tabsdatak.api import TableFrameSpec, subscriber
from tabsdatak.conn.s3 import S3Dest


@subscriber(
    destination=S3Dest(paths=["spanish.parquet"]),
    input_tables=["dataprod/spanish"],
)
def sub(spanish: TableFrameSpec) -> TableFrameSpec:
    return spanish

Registering and Executing the Functions

Log in, create a project, create one collection per function (the source and destination ones with their connections), register each function in its collection, and trigger the publisher to run the rest:

tdk login

tdk project create --name example
tdk set --proj example

tdk collection create --name input --group sources --conn-file conn-mysql.yaml
tdk collection create --name dataprod --group default
tdk collection create --name output --group destinations --conn-file conn-s3.yaml

tdk fn register --coll input --path input.py::pub
tdk fn register --coll dataprod --path transform.py::tfr
tdk fn register --coll output --path output.py::sub

tdk fn trigger --coll input --name pub

Every time the pub publisher is executed, the tfr transformer and the sub subscriber will also be executed.

Metadata

Release files for tabsdata 2.3.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distributions (wheels)

Table of built distributions (wheels) for tabsdata 2.3.0
File Interpreter ABI Platform Actions
tabsdata-2.3.0-cp312-abi3-win_amd64.whl CPython 3.12 abi3 Windows x86-64 Details for tabsdata-2.3.0-cp312-abi3-win_amd64.whl
tabsdata-2.3.0-cp312-abi3-manylinux_2_28_x86_64.whl CPython 3.12 abi3 Linux glibc 2.28+ x86-64 Details for tabsdata-2.3.0-cp312-abi3-manylinux_2_28_x86_64.whl
tabsdata-2.3.0-cp312-abi3-macosx_11_0_arm64.whl CPython 3.12 abi3 macOS 11.0+ ARM64 Details for tabsdata-2.3.0-cp312-abi3-macosx_11_0_arm64.whl

Total release size: 111.1 MB

Release files / tabsdata-2.3.0-cp312-abi3-win_amd64.whl

Download URL tabsdata-2.3.0-cp312-abi3-win_amd64.whl
Size 35.7 MB
Tags CPython 3.12 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
b99d02dc143f253991a69f4232ba9a7648d85ec7dd62091211d75911699ab97f
BLAKE2b-256 checksum
How to use checksums
8e6d553c621eb8d0386783a8f4c07bcdbaa6475cb84d4ddeee43ed3570dbd172
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.13

Release files / tabsdata-2.3.0-cp312-abi3-manylinux_2_28_x86_64.whl

Download URL tabsdata-2.3.0-cp312-abi3-manylinux_2_28_x86_64.whl
Size 38.0 MB
Tags CPython 3.12 Linux glibc 2.28+ x86-64 abi3
SHA-256 checksum
How to use checksums
f101cf4bf5deca63e0f745803447c91e24f2f1d82cd8e7b9af324ac56761a5c6
BLAKE2b-256 checksum
How to use checksums
f2b0dccc4784ebbba1471fbb1ff7748601def674ec009abab61c6b276abceead
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.13

Release files / tabsdata-2.3.0-cp312-abi3-macosx_11_0_arm64.whl

Download URL tabsdata-2.3.0-cp312-abi3-macosx_11_0_arm64.whl
Size 37.4 MB
Tags CPython 3.12 abi3 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
b9565f93cca3a1b706707aad1e5efb463a7ef80e060e9cffde3b825c38314414
BLAKE2b-256 checksum
How to use checksums
e5bc363fea1b164e5cc1ff354628ae73524db16447828e5854dbdb3653962d48
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.13

Security for tabsdata 2.3.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

2.3.0 This release

3 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page