tabsdata 0.9.1

Tabsdata is a publish-subscribe (pub/sub) server for tables.

Tabsdata | User Guide | API Reference

Tabsdata Pub/Sub for Tables

Tabsdata is a publish-subscribe (pub/sub) server for tables.

Tabsdata has connectors to publish and subscribe tables from local files, S3, Azure Storage, MySQL/MariaDB, Oracle, PostgreSQL. It also provides a Connector Plugin API to write custom connectors.

Tables can populated with external data or using data from other tables already existing in the Tabsdata server.

Tables can be manipulated using a TableFrame API (internally Tabsdata uses Polars) that enables selection, filtering, aggregation and joins operations.

For more details refer to Tabsdata Getting Started (latest) or the Tabsdata User Guide (latest).

Installation

Supported platforms:

• Windows (x86 - latest)
• OSX (Apple silicon/x86 - latest)
• Ubuntu, Debian & RedHat - (x86/ARM - latest)

pip install tabsdata

This version (0.9.1) Documentation

• User Guide
• API Reference

How Does Tabsdata Work?

The following snippets show how to publish and subscribe to tables in Tabsdata.

Publishing data from a MySQL Database

@td.publisher(
    td.MySQLSource(
        "mysql://127.0.0.1:3306/testing",
        ["select * from CUSTOMERS"],
        td.UserPasswordCredentials("admin", td.EnvironmentSecret("DB_PASWORD"))
    ),
    tables=["customers"]
)
def customers_publisher(customers: td.TableFrame) -> td.TableFrame:
    return customers

Subscribing, transforming and publishing data within Tabsdata

@td.transformer(
    input_tables=["persons"],
    output_tables=["spanish"]
)
def tfr(persons: td.TableFrame):
    return persons.filter(td.col("nationality").eq("spanish")).select(
        ["identifier", "name", "surname", "language"]
    )

Subscribing to data in an S3 Bucket

@td.subscriber(
    "spanish",
    td.S3Destination(
        "s3://my_bucket/spanish.parquet",
        td.S3AccessKeyCredentials(
            td.EnvironmentSecret("AWS_ACCESS_KEY_ID"),
            td.EnvironmentSecret("AWS_SECRET_KEY")
        )
    ),
)
def sub(spanish: td.TableFrame):
    return spanish

Executing the Publisher

To publish data to Tabsdata run the following command:

$ td fn trigger --collection examples --name pub

Every time the pub publisher is executed, the tfr transformer and the sub subscriber will also be executed.