Datatailr empowers your team to streamline analytics and data workflows from idea to production without infrastructure hurdles.

What is Datatailr?

Datatailr is a platform that simplifies the process of building and deploying data applications.

It makes it easier to run and maintain large-scale data processing and analytics workloads.

What is this package?

This is the Python package for Datatailr, which allows you to interact with the Datatailr platform.

It provides the tools to build, deploy, and manage batch jobs, data pipelines, services and analytics applications.

Datatailr manages the underlying infrastructure so your applications can be deployed in an easy, secure and scalable way.

Installation

Installing the Python package

Install the Datatailr Python package:

pip install --user datatailr

Or, if using a virtual environment:

pip install datatailr

On Windows without administrator access, prefer pip install --user — it installs to your user profile and does not require elevation.

If the native dt executable is not on PATH, the next Python start shows a reminder to run datatailr setup-cli.

Testing the installation

import datatailr

print(datatailr.__version__)
print(datatailr.__provider__)

Remote CLI (optional)

If you install the package outside the Datatailr platform, you can enable the remote dt CLI:

python -m datatailr setup-cli

python -m datatailr login prompts interactively for the base URL, username and password. To skip the prompts (for CI or scripted setups), set all three of the following environment variables before running python -m datatailr login:

export DATATAILR_BASE_URL=https://your-datatailr-instance
export DATATAILR_USER_NAME=your-username
export DATATAILR_USER_PASSWORD=your-password
python -m datatailr login

When all three are set, DATATAILR_BASE_URL takes precedence over the --url flag. The resulting session is saved to ~/.dt/remote_client/remote_client.cfg, so the env vars are only needed for the login step.

After datatailr login, you can print the OIDC cookie line for scripts or HTTP clients:

datatailr export-auth
eval "$(datatailr export-auth --shell)"   # sets DATATAILR_OIDC_HEADER (sh/bash/zsh)

For fish:

eval (datatailr export-auth --fish)   # sets DATATAILR_OIDC_HEADER

To remove the saved JWT locally (keeps the base URL for the next login):

datatailr logout

After logout, remote dt commands, export-auth, and Python SDK calls that need authentication will fail until you run datatailr login again.

From Python (after datatailr login), read the same session at runtime:

from datatailr import (
    get_remote_http_headers,
    get_remote_oidc_cookie_line,
    get_remote_oidc_jwt,
    load_remote_client_config,
)

cfg = load_remote_client_config()
print(cfg.base_url)

token = get_remote_oidc_jwt()
line = get_remote_oidc_cookie_line()  # X-Datatailr-Oidc-Data=<jwt>

import requests
requests.get(f"{cfg.base_url}/api/user/ls", headers=get_remote_http_headers())

Example usage:

dt job ls
dt user ls
dt job save path/to/local/file.json

Notes:

Remote CLI configuration inside a virtual environment only applies inside that environment.
The remote CLI cannot be installed inside Datatailr containers; the native CLI is used there.

AI Agent Skills

The package includes agent skills that teach AI coding assistants (Cursor, Claude Code, Codex, Copilot, etc.) how to work with the Datatailr platform. Inside Datatailr workstations, skills are available automatically. On your local machine, run:

datatailr setup-skills

Quickstart

The following example shows how to create a simple data pipeline using the Datatailr Python package.

from datatailr import workflow, task

@task()
def func_no_args() -> str:
    return "no_args"


@task()
def func_with_args(a: int, b: float) -> str:
    return f"args: {a}, {b}"

@workflow(name="MY test DAG")
def my_workflow():
    for n in range(2):
        res1 = func_no_args().alias(f"func_{n}")
        res2 = func_with_args(1, res1).alias(f"func_with_args_{n}")
my_workflow(local_run=True)

Running this code will create a graph of jobs and execute it. Each node on the graph represents a job, which in turn is a call to a function decorated with @task().

Since this is a local run then the execution of each node will happen sequentially in the same process.

To take advantage of the datatailr platform and execute the graph at scale, you can run it using the job scheduler as presented in the next section.

Budgets (spend reporting and limiting)

Jobs can be assigned to a budget for spend reporting and optional cost limiting. The Python SDK exposes the same operations as dt cost CLI command.

If not specified, all jobs are assigned to the default budget which is available to all users and has no limit by default. Admins can set a limit on the default budget, but it cannot be removed.

from datatailr import ACL, Budget, Group, Permission, User

# List budgets visible to the current user
for b in Budget.ls():
    print(b.name, b.budget_usd, b.usage_usd, b.usage_percentage, b.prevent_overflow)

# Load one budget by name
b = Budget("my_budget")

# Create / update / remove (available to admins only)
Budget.add("my_budget", 50000.0, prevent_overflow=True)
Budget.update("my_budget", amount=75000.0, prevent_overflow=False)
Budget.remove("my_budget")

# Permissions (ACL):
# read = see limit and usage.
# operate = assign jobs to this budget
# Creating and deleting budgets, updating limits and ACLs are admin-only operations.
acl = ACL(
    {
        Permission.READ: [User("alice"), Group("analysts")],
        Permission.OPERATE: [Group("developers")],
    }
)
Budget.set_acl("my_budget", acl)
Budget.add_acl("my_budget", acl)
Budget.remove_acl("my_budget", acl)
Budget.set_acl("my_budget", None)  # replace ACL with {}

Execution at Scale

To execute the graph at scale, you can use the Datatailr job scheduler. This allows you to run your jobs in parallel, taking advantage of the underlying infrastructure.

You will first need to separate your function definitions from the DAG definition. This means you should define your functions as a separate module, which can be imported into the DAG definition.

# my_module.py

from datatailr import task

@task()
def func_no_args() -> str:
    return "no_args"


@task()
def func_with_args(a: int, b: float) -> str:
    return f"args: {a}, {b}"

To use these functions in a batch job, you just need to import them and run in a DAG context:

from my_module import func_no_args, func_with_args
from datatailr import workflow

@workflow(name="MY test DAG")
def my_workflow():
    for n in range(2):
        res1 = func_no_args().alias(f"func_{n}")
        res2 = func_with_args(1, res1).alias(f"func_with_args_{n}")

schedule = Schedule(at_hours=0)
my_workflow(schedule=schedule)

This will submit the entire workflow for execution, and the scheduler will take care of running the jobs in parallel and managing the resources. The workflow in the example above will be scheduled to run daily at 00:00.

Distributed scikit-learn / joblib

Any code that parallelizes with joblib — most notably scikit-learn — can run its parallel workload on Datatailr worker containers instead of local processes. No service definitions or workflow authoring required:

import joblib
import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import RandomizedSearchCV
from sklearn.svm import SVC

from datatailr.joblib import register_datatailr

digits = load_digits()
param_space = {
    "C": np.logspace(-6, 6, 30),
    "gamma": np.logspace(-8, 8, 30),
}
search = RandomizedSearchCV(SVC(kernel="rbf"), param_space, cv=5, n_iter=300)

register_datatailr()
with joblib.parallel_backend("datatailr", n_jobs=16):
    search.fit(digits.data, digits.target)

The first Parallel call inside the context starts an ephemeral pool of worker containers (using your workstation's image, so library versions always match) plus a small broker service. When that Parallel finishes, the pool is torn down; a later Parallel in the same process starts a fresh pool. Process exit and shutdown_pools() remain as safety nets.

For systematic performance evaluation, see benchmarks/joblib_sklearn/README.md.

Backend options are passed alongside n_jobs, for example joblib.parallel_backend("datatailr", n_jobs=16, worker_memory="4g", worker_cpu=2). See datatailr.joblib.DatatailrBackend for the full list.

Requirements: run from a Datatailr workstation (or a remote SDK session), and have joblib and cloudpickle installed. To tear the pool down early, call datatailr.joblib.shutdown_pools().

Visit our website for more!

Release files for datatailr 0.1.133

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for datatailr 0.1.133
File	Size	Uploaded
datatailr-0.1.133.tar.gz	277.4 kB	Sep 25, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for datatailr 0.1.133
File	Interpreter	ABI	Platform	Reset
datatailr-0.1.133-py3-none-any.whl 350.0 kB Sep 25, 2026	Python 3	none	any	Details

Total release size: 627.3 kB

Release files / datatailr-0.1.133.tar.gz

Download URL	datatailr-0.1.133.tar.gz
Size	277.4 kB
Tags	Source
SHA-256 checksum How to use checksums	`0246966e3bed86b688a15295ace39b5758c1dfb1f89c675c09d3427b99741409`
BLAKE2b-256 checksum How to use checksums	`45c0092eb06ff56cd22f2beeac3370b195c3dc906618d7c079ad66f4ceafd014`
Upload date	Sep 25, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.12.14`

Release files / datatailr-0.1.133-py3-none-any.whl

Download URL	datatailr-0.1.133-py3-none-any.whl
Size	350.0 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`05fc255103dca3657746941dfec197a171c4062e418d02708d7a8e07bc8a4b2c`
BLAKE2b-256 checksum How to use checksums	`c073d88e964737b0e0d2e7647c659500db76dd6737b2cbd77132953f06e61de5`
Upload date	Sep 25, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.12.14`