PyDataRheo

PyDataRheo moves data between hundreds of systems, from plain Python.

Pick a source, read from it, and land the records in DuckDB, Postgres, Snowflake, BigQuery, MotherDuck, or any destination connector. No orchestrator, no cluster, no YAML pipeline definition. It is a library, so it runs wherever your Python runs: a script, a notebook, a Lambda, an Airflow or Dagster task.

PyDataRheo runs Airbyte-protocol connectors, which is what gives it several hundred sources and destinations on day one.

Install

pip install pydatarheo

The import package is datarheo; the distribution on PyPI is pydatarheo.

Quick start

import datarheo as dr

source = dr.get_source(
    "source-faker",
    config={"count": 5_000},
    install_if_missing=True,
)
source.check()
source.select_all_streams()

result = source.read()  # Lands in a local DuckDB cache by default.

for record in result["users"].records:
    print(record)

Read into a dataframe:

df = result["users"].to_pandas()

Write to a real warehouse instead of the default local cache:

from datarheo.caches import SnowflakeCache

cache = SnowflakeCache(
    account="my-account",
    username="my-user",
    password=dr.get_secret("SNOWFLAKE_PASSWORD"),
    database="my-db",
    warehouse="my-wh",
    role="my-role",
)
source.read(cache=cache)

Command line

These console scripts are installed:

pydr --help
pydr validate --connector=source-faker

Connector installation

Declarative (YAML) sources

Declarative sources are downloaded as a single YAML manifest and executed directly. These have the fastest install times, because there is nothing to build.

Python sources

When a connector publishes a Python package, PyDataRheo installs it automatically. Installation uses uv rather than pip, which is both much faster and able to install a connector under a different Python version than the one PyDataRheo itself is running on. Set DATARHEO_NO_UV=true to fall back to pip.

To pin a connector to a specific Python version:

source = dr.get_source("source-faker", use_python="3.10.17")

Docker

Pass docker_image=True to get_source() or get_destination() to run the connector from its published image instead, or docker_image="my-org/my-image:tag" to pick an exact image. Docker is the most reproducible option, because every dependency is locked inside the image, and it is required for Java-based destinations.

Configuration

Every environment variable PyDataRheo reads is prefixed DATARHEO_. The most useful ones:

An environment still carrying the older AIRBYTE_-prefixed names keeps working: each AIRBYTE_FOO is read as DATARHEO_FOO unless the DATARHEO_ name is also set, in which case the explicit DATARHEO_ value wins.

Telemetry

Off by default. PyDataRheo sends nothing anywhere unless you set DATARHEO_TRACKING_KEY to a Segment write key that you own. Setting DO_NOT_TRACK, or enabling offline mode, disables reporting outright.

Documentation

The full API reference is generated from the source with poe docs-generate and published at vrahad-analytics.github.io/pydatarheo.

Contributing

See the Contributors Guide.

Frequently asked questions

Is PyDataRheo a replacement for a data platform? No. It is a library. It has no orchestration, scheduling, alerting, or pipeline monitoring. Run it inside whatever scheduler you already use.

What is the cache? Is it a destination? Effectively yes: it is a built-in destination implementation backed by a SQL engine. We call it a cache to keep it distinct from the destination connectors, which are separate executables.

Does it work with Airflow, Dagster, Prefect, or Snowpark? Yes. It is an ordinary Python dependency with no background services.

Can I build a normal ETL pipeline with it? Yes. Choose the cache type that matches where the data should land, such as SnowflakeCache for Snowflake.

Can it run a connector from a local directory? Yes. Any connector exposing a CLI works, and connectors already on PATH are found by name.

License and provenance

PyDataRheo is released under the MIT license and is a rebranded derivative of PyAirbyte, which is also MIT licensed. See LICENSE and NOTICE.

Changelog

See the GitHub Releases page.