Skip to main content

AutoData Python Client

Official Python SDK for the AutoData ML data preparation pipeline API.

Installation

pip install datatoolpack

Or install from source:

git clone https://github.com/datatoolpack/datatoolpack
cd datatoolpack
pip install .

Quick Start

from datatoolpack import AutoDataClient

with AutoDataClient(api_key="dtpk_YOUR_API_KEY") as client:
    result = client.process(
        file_path="data.csv",
        target_columns=["price"],
        output_rows=10000,
    )
    print(result["files"])

Get your API key from the AutoData dashboard → API Keys tab.


Authentication

with AutoDataClient(api_key="dtpk_YOUR_API_KEY") as client:
    ...

Access Code (passcode)

If you have an access code instead of an API key:

with AutoDataClient(passcode="123456789012") as client:
    ...

Environment Variables

Set environment variables to avoid hardcoding credentials:

export AUTODATA_API_KEY="dtpk_YOUR_API_KEY"
# or
export AUTODATA_PASSCODE="123456789012"
# optional
export AUTODATA_BASE_URL="https://autodata.datatoolpack.com"
# No need to pass api_key — reads from AUTODATA_API_KEY automatically
with AutoDataClient() as client:
    result = client.process("data.csv", target_columns="price")

Supported File Formats

Format Extensions
CSV .csv
Excel .xlsx, .xls
Parquet .parquet
JSON .json
Feather .feather
ORC .orc

Reference

AutoDataClient(api_key, passcode, base_url, timeout, max_retries)

Parameter Type Default Env Variable Description
api_key str None AUTODATA_API_KEY API key starting with dtpk_
passcode str None AUTODATA_PASSCODE Access code (alternative to api_key)
base_url str "https://autodata.datatoolpack.com" AUTODATA_BASE_URL Server URL (no trailing slash)
timeout int 120 Request timeout in seconds
max_retries int 3 Auto-retries for 429 / 502 / 503 / 504

Either api_key or passcode is required (via argument or env variable).

Implements context manager — use with with to auto-close connections:

with AutoDataClient(api_key="dtpk_...") as client:
    ...
# session is closed automatically

Or close manually:

client = AutoDataClient(api_key="dtpk_...")
# ... use client ...
client.close()

Core Pipeline

client.process(...) — Upload & run pipeline

result = client.process(
    file_path="data.csv",           # Path to input file (CSV, XLSX, Parquet, ...)
    target_columns=["price"],       # y-column(s) for ML
    output_rows=10000,              # Target row count in the synthetic output (default 10000)
    tools={                         # Toggle pipeline steps (all optional)
        "anomaly": False,           # Anomaly detection (off by default)
        "dtc": True,                # Data Type Conversion
        "mdh": True,                # Missing Data Handler
        "dor": False,               # Dimensionality Reduction (off by default)
        "cds": True,                # Column Scaling
        "dsm": True,                # Data Split Manager
        "dsg": True,                # Synthetic Data Generator
    },
    advanced_params={               # Fine-grained parameters (all optional)
        "excluded_columns": ["id"], # Columns to drop before processing
        "text_mode": 0,             # 0=none, 1=neural, 2=tfidf
        "text_cleaning": True,      # Clean text before encoding
        "zscore_limit": 3.0,        # Z-score outlier threshold
        "dsg_mode": "copula",       # "copula" or "gan"
        "similarity_p": 95,         # Similarity percentile for DSG
    },
    wait=True,                      # Block until complete (default True)
    poll_interval=2,                # Status poll interval in seconds
    download_path="./outputs/",     # Where to save files (default auto)
    auto_download=True,             # Set False to skip download when wait=True
    output_preferences=["dsg.csv"], # Which files to download (default all)
    compressed=True,                # Download as ZIP (default True)
)

Returns a dict:

{
    "session_id": "abc123...",
    "status": "completed",
    "files": [
        {"name": "dsg.csv", "url": "/download/.../dsg.csv", "size": 2097152, "description": "..."},
        {"name": "dsm_train.csv", ...},
    ],
    "row_count": 10000,
    "duration_seconds": 42.1,
}

Set wait=False to get back immediately with just session_id and status:

result = client.process(file_path="data.csv", target_columns="price", wait=False)
session_id = result["session_id"]

client.get_status(session_id) — Poll progress

status = client.get_status(session_id)
# {
#   "status": "running",           # queued | running | completed | error | cancelled
#   "message": "Running MDH...",
#   "current_step": 3,
#   "total_steps": 6,
#   "progress_percent": 50,
#   "duration_seconds": 15.3,
# }

client.get_result(session_id) — Fetch completed results

result = client.get_result(session_id)
# {"status": "completed", "files": [...], "row_count": ..., "duration_seconds": ...}

client.wait_for_completion(session_id, poll_interval) — Block until done

result = client.wait_for_completion(session_id, poll_interval=3)

Prints live progress to stdout. Raises AutoDataError if processing fails.


client.cancel(session_id) — Cancel a running job

cancelled = client.cancel(session_id)  # True if acknowledged

client.retry_session(session_id) — Retry a failed pipeline

Retry from the last checkpoint — no need to re-upload or re-configure:

result = client.retry_session(session_id="failed-session-id")
# {'success': True, 'session_id': '...', 'message': 'Retrying from last checkpoint'}

client.download_results(session_id, ...) — Download output files

path = client.download_results(
    session_id,
    download_path="./my_outputs/",      # Directory to save into
    output_preferences=["dsg.csv"],     # Specific files only (None = all)
    compressed=True,                    # ZIP download (default) or individual files
)
print(f"Saved to {path}")

client.download_file(url, output_path) — Download a single file

client.download_file("/api/v1/download/abc123.../dsg.csv", "dsg.csv")

Feature Selection (F4)

Rank a completed session's columns by predictive importance — without re-running the pipeline. Handy for trimming wide datasets before training.

client.recommend_features(session_id, target_columns, methods, top_k)

ranking = client.recommend_features(
    session_id="abc123...",            # a completed session you own
    target_columns=["price"],          # rank features against these target(s)
    methods=["mutual_information"],    # one or more methods (see below); default ["mutual_information"]
    top_k=20,                          # number of top features to return (default 20)
)
# {
#   "ranking":    [{"feature": "sqft", "score": 0.81}, ...],   # combined across methods
#   "per_method": {"mutual_information": [...], "shap_importance": [...]},
#   "warnings":   [...],
# }

Available methods: variance_threshold, correlation_pruning, mutual_information, rfe, shap_importance, pca. Pass several to blend their rankings, e.g. methods=["mutual_information", "shap_importance"]. (SHAP-based ranking uses a fast LightGBM model on the server; it degrades gracefully to mutual_information if those libraries are unavailable.)


Inference & Retraining (v0.11.0+)

Apply a completed session's fitted transforms to new data — the missing half of the ML lifecycle: train once, then score fresh production rows with exactly the same encoding, imputation and scaling.

client.infer(...) — Score new data

Lightweight pipeline (Anomaly Detection → DTC → MDH → CDS). Outputs features.csv (+ y_columns.csv if targets are present in the new file).

result = client.infer(
    original_session_id="5d3bc0f0-...",   # a completed training session you own
    file_path="new_rows.csv",             # columns must match the training data
    download_path="./scored",             # optional: save outputs locally
    return_dataframes=True,               # optional: get pandas DataFrames back
)
features_df = result["dataframes"]["features"]

client.retrain(...) — Refresh a training set with new rows

Full pipeline including Split → CDS → DSM → DSG; produces retraining_output.csv (features + targets combined).

result = client.retrain(
    original_session_id="5d3bc0f0-...",
    file_path="new_rows.csv",
    run_dsg=True, output_rows=20000,      # synthetic augmentation target
    download_path="./retrained",
)

Both calls are synchronous (the response arrives when processing ends; default per-request timeout is ≥ 900 s — raise timeout= for very large files), require the original session to be completed and owned by you, and are free of credit charges in v1.


API Keys & Usage

client.list_keys() — List API keys

keys = client.list_keys()
# [{"id": "...", "name": "My Key", "prefix": "dtpk_abc123", "created_at": "..."}]

client.get_usage() — Usage statistics

usage = client.get_usage()
# {
#   "daily_credits_used": 500,
#   "daily_credit_limit": 10000,
#   "daily_remaining": 9500,
#   "lifetime_credits_used": 12340,
#   "lifetime_credit_limit": 1000000,
#   "lifetime_remaining": 987660,
#   "daily_request_count": 3,
#   "last_used_at": "2026-04-12T10:30:00Z",
# }

Connectors

Read data directly from databases and cloud storage instead of uploading files.

Supported connector types

Type Description Required secrets
sql PostgreSQL, MySQL, SQL Server connection_string
snowflake Snowflake Data Cloud account, user, password, database, schema, warehouse
bigquery Google BigQuery credentials_json, project_id, dataset
mongodb MongoDB / Atlas connection_string, database
s3 Amazon S3 bucket, access_key_id, secret_access_key, region
gcs Google Cloud Storage bucket, credentials_json
databricks Databricks Lakehouse host, token, catalog, schema
delta Delta Lake path (+ cloud credentials if remote)
fabric Microsoft Fabric workspace_id, lakehouse_id, tenant_id, client_id, client_secret
kafka Apache Kafka bootstrap_servers, topic, group_id
kinesis Amazon Kinesis stream_name, region, access_key_id, secret_access_key

More connectors: AutoData supports 30+ source types. Beyond the common ones above you can also pass oracle, redshift, cassandra, dynamodb, elasticsearch, clickhouse, timescaledb, synapse, azure_blob, adls, sap_hana, mqtt, opcua, influxdb, pi_web, salesforce, hubspot, stripe, netsuite, google_sheets, ga4, sharepoint, and http/rest as connector_type. See Dashboard → Documentation → Connector Data Sources for each type's required secrets fields.

Quick connector example

with AutoDataClient(api_key="dtpk_...") as client:
    # Test the connection
    client.test_connector("sql", secrets={
        "connection_string": "postgresql://user:pass@host/db"
    })

    # List tables
    tables = client.discover("sql", secrets={
        "connection_string": "postgresql://user:pass@host/db"
    })
    print(tables)  # ["users", "orders", ...]

    # Preview columns
    info = client.preview("sql", table="orders", secrets={
        "connection_string": "postgresql://user:pass@host/db"
    })
    print(info["columns"])  # ["id", "price", "date", ...]

    # Run the full pipeline from the connector
    result = client.process_from_connector(
        connector_type="sql",
        table="orders",
        target_columns=["price"],
        secrets={"connection_string": "postgresql://user:pass@host/db"},
        output_rows=10000,
    )
    print(result["files"])

Connector examples by type

Snowflake:

secrets = {
    "account": "abc123.us-east-1",
    "user": "analyst",
    "password": "...",
    "database": "PROD",
    "schema": "PUBLIC",
    "warehouse": "COMPUTE_WH",
}
tables = client.discover("snowflake", secrets=secrets)
result = client.process_from_connector("snowflake", table="ORDERS",
    target_columns=["AMOUNT"], secrets=secrets)

BigQuery:

secrets = {
    "credentials_json": '{"type":"service_account",...}',
    "project_id": "my-project",
    "dataset": "analytics",
}
tables = client.discover("bigquery", secrets=secrets)

Amazon S3:

secrets = {
    "bucket": "my-data-lake",
    "access_key_id": "AKIA...",
    "secret_access_key": "...",
    "region": "us-east-1",
}
result = client.process_from_connector("s3", table="data/sales.csv",
    target_columns=["revenue"], secrets=secrets, output_rows=50000)

MongoDB:

secrets = {
    "connection_string": "mongodb+srv://user:pass@cluster.mongodb.net",
    "database": "analytics",
}
tables = client.discover("mongodb", secrets=secrets)

client.test_connector(connector_type, secrets, credential_id) — Test connection

result = client.test_connector("s3", secrets={...})
# {"success": True, "message": "Connection successful"}

client.discover(connector_type, ...) — List tables / files

tables = client.discover("sql", secrets={...})
# ["users", "orders", "transactions"]

client.preview(connector_type, table, ...) — Preview columns & row count

info = client.preview("sql", table="orders", secrets={...})
# {"success": True, "columns": ["id", "price", ...], "row_count": 50000}

Supports optional custom_query parameter for SQL-based connectors:

info = client.preview("sql", table="orders", custom_query="SELECT * FROM orders WHERE date > '2025-01-01'", secrets={...})

client.process_from_connector(...) — Run pipeline from connector

result = client.process_from_connector(
    connector_type="sql",
    table="orders",
    target_columns=["price"],
    secrets={"connection_string": "postgresql://..."},
    output_rows=10000,
    incremental=True,          # Only fetch new rows since last sync
    type_casts={"age": "int"}, # Override column types before processing
    wait=True,
    download_path="./outputs/",
)

Supports all the same options as client.process(): tools, advanced_params, wait, poll_interval, auto_download, output_preferences, compressed.

client.write_output(session_id, ...) — Write results to a target

client.write_output(
    session_id="abc123...",
    connector_type="sql",
    table_name="ml_prepared_data",
    secrets={"connection_string": "postgresql://..."},
    output_stage="dsg",        # which pipeline output to write
    if_exists="replace",       # "replace", "append", "fail", or "merge"
    merge_keys=["id"],         # column names for merge matching (when if_exists='merge')
)
# {"success": True, "rows_written": 10000, "table": "ml_prepared_data"}

Credential Management

Save credentials on the server so you don't need to pass secrets every time:

# Save a new credential
cred = client.save_credential(
    name="Production DB",
    connector_type="sql",
    secrets={"connection_string": "postgresql://..."},
)
cred_id = cred["id"]

# Use credential_id instead of secrets
tables = client.discover("sql", credential_id=cred_id)
result = client.process_from_connector("sql", table="orders",
    target_columns=["price"], credential_id=cred_id)

# List saved credentials (secrets are never exposed)
creds = client.list_credentials()

# Update a credential
client.update_credential(cred_id, name="Prod DB v2", secrets={...})

# Delete a credential
client.delete_credential(cred_id)

Schema Mapping

Auto-suggest and apply column mappings between source and target schemas:

client.suggest_mapping(source_columns, target_columns) — Auto-suggest mapping

mapping = client.suggest_mapping(
    source_columns=["customer_name", "order_amt", "order_dt"],
    target_columns=["name", "amount", "date"],
    threshold=0.6,  # minimum similarity score (0-1)
)
# {
#   "mapping": {"customer_name": "name", "order_amt": "amount", "order_dt": "date"},
#   "scores": {"customer_name": 0.85, "order_amt": 0.72, "order_dt": 0.68}
# }

client.apply_mapping(session_id, mapping) — Rename columns

result = client.apply_mapping(
    session_id="abc123...",
    mapping={"old_col_name": "new_col_name", ...},
    output_stage="dsg",  # which pipeline stage to apply to
)
# {"columns": ["new_col_name", ...], "preview": [...]}

Quality Alerts

Monitor pipeline metrics and get notified when thresholds are breached:

# Create a quality alert rule
rule = client.create_quality_alert(
    name="High row loss",
    metric="row_loss_pct",    # row_loss_pct, null_pct, column_drop_count, duration_seconds
    operator=">",             # >, <, >=, <=, ==
    threshold=10.0,
    severity="critical",      # warning or critical
    stage="mdh",              # optional: anomaly, dtc, mdh, cds, dsm, dsg
)

# List rules
rules = client.list_quality_alerts()

# Update a rule
client.update_quality_alert(rule_id="...", threshold=15.0, severity="warning")

# Get fired alert events
events = client.get_alert_events(session_id="optional-filter")

# Delete a rule
client.delete_quality_alert(rule_id="...")

Available metrics:

Metric Description
row_loss_pct Percentage of rows lost in a step
null_pct Percentage of null values remaining
column_drop_count Number of columns dropped
duration_seconds Time taken by a pipeline step

Sync Watermarks

Track incremental sync progress for connector-based pipelines:

# List all watermarks
watermarks = client.list_watermarks()
# [{"id": "wm-123", "connector_type": "sql", "table_name": "orders",
#   "watermark_column": "updated_at", "last_value": "2026-04-12T00:00:00Z"}]

# Reset a watermark (re-sync from beginning)
client.reset_watermark(watermark_id="wm-123")

Scheduled Runs

Automate recurring pipeline executions:

# Create an interval-based schedule (every 24 hours)
schedule = client.create_scheduled_run(
    name="Daily ETL",
    schedule_type="interval",
    interval_minutes=1440,
    connector_type="snowflake",
    table="orders",
    credential_id="cred-id",
    y_columns=["target"],
    output_rows=50000,
)

# Create with cron expression (weekdays at 8 AM)
schedule = client.create_scheduled_run(
    name="Weekday ETL",
    schedule_type="cron",
    cron_expression="0 8 * * 1-5",
    connector_type="snowflake",
    table="orders",
    credential_id="cred-id",
    y_columns=["target"],
    tools={"anomaly": True, "dsg": False},  # customize pipeline steps
)

# List all schedules
schedules = client.list_scheduled_runs()

# Enable/disable a schedule
client.toggle_scheduled_run(run_id="run-id")

# Trigger immediately (ignore schedule)
client.run_scheduled_now(run_id="run-id")

# Delete a schedule
client.delete_scheduled_run(run_id="run-id")

Triggers (F7)

Auto-start a pipeline when an external event fires — an inbound webhook, a connector watermark advancing, another pipeline completing, or a quality alert. Triggers are feature-flagged on the server (FEATURE_FLAG_TRIGGERS); if the flag is off, these endpoints return 404.

client.create_trigger(name, trigger_type, config, target_pipeline_config, enabled)

trigger = client.create_trigger(
    name="Reprocess on new S3 batch",
    trigger_type="inbound_webhook",   # inbound_webhook | watermark_advance | pipeline_completion | quality_alert
    config={},                         # type-specific (see table below)
    target_pipeline_config={           # the pipeline to run when the trigger fires
        "source_type": "s3",
        "credential_id": "cred-uuid",
        "table": "incoming/sales.csv",
        "target_columns": ["revenue"],
        "output_rows": 10000,
    },
    enabled=True,
)
# For inbound_webhook the secret is returned ONLY once — save it now:
print(trigger["webhook_secret"], trigger["inbound_url"])

Trigger types & their config:

trigger_type config keys
inbound_webhook optional {"hmac_key": "..."} — require HMAC-SHA256-signed requests
watermark_advance {"credential_id", "table", "watermark_column", "min_advance"}
pipeline_completion {} — chain off another pipeline finishing
quality_alert {} — fire when a quality-alert rule trips
# List / fetch / update / delete
triggers = client.list_triggers()
t        = client.get_trigger(trigger_id)
client.update_trigger(trigger_id, enabled=False)   # name | enabled | config | target_pipeline_config
client.delete_trigger(trigger_id)

# Dry-run: validate config + target WITHOUT firing
report = client.test_trigger(trigger_id)
# {"would_fire": True, "checks": {...}}

Firing an inbound webhook from your own system:

curl -X POST "https://autodata.datatoolpack.com/trigger/inbound/<webhook_secret>" \
     -H "Content-Type: application/json" \
     -d '{"any": "payload"}'
# If you set an hmac_key, also sign the raw body:
#   X-Signature: sha256=<hex HMAC-SHA256 of the request body>

Inbound firing is rate-limited and HMAC-verified server-side; repeated failures auto-disable the trigger.


Folder Listeners

Automatically trigger pipelines when new files appear in cloud storage:

# Create an S3 folder listener
listener = client.create_listener(
    name="S3 Ingest",
    source_type="s3",            # s3, gcs, azure_blob, local, sftp
    watch_path="s3://bucket/incoming/",
    credential_id="cred-id",
    y_columns=["target"],
    pipeline_config={"enable_dsg": False},
    output_rows=10000,
)

# List all listeners
listeners = client.list_listeners()

# Update a listener
client.update_listener(listener_id="...", watch_path="s3://bucket/new-path/", enabled=False)

# Delete a listener
client.delete_listener(listener_id="...")

Supported source types: s3, gcs, azure_blob, local, sftp


SFTP Upload

Upload files to the AutoData server via SFTP for processing:

# Get SFTP server connection info
info = client.sftp_info()
# {"host": "...", "port": 22, "username": "..."}

# Manage SFTP credentials
cred = client.create_sftp_credential(name="My Upload Key")
# {"id": "...", "username": "...", "password": "..."} (password shown only once)

creds = client.list_sftp_credentials()
client.delete_sftp_credential(credential_id="...")

Spark (Large Data)

For datasets too large for pandas, AutoData can use PySpark on the server:

client.spark_status() — Check Spark availability

status = client.spark_status()
# {"available": True, "spark_version": "3.5.0", ...}

client.spark_read(file_path, sample_rows) — Read large files

info = client.spark_read(
    file_path="/data/huge_dataset.parquet",
    sample_rows=20,
)
# {"row_count": 50000000, "columns": [...], "dtypes": {...}, "sample": [...]}

client.spark_transform(...) — Apply Spark transformations

result = client.spark_transform(
    file_path="/data/huge_dataset.parquet",
    output_path="/data/transformed.csv",
    operations=[
        {"op": "cast", "type_casts": {"age": "int"}},
        {"op": "drop_nulls", "subset": ["col1"]},
        {"op": "fill_nulls", "strategy": "mean"},
        {"op": "normalize", "method": "minmax"},
        {"op": "sample", "n": 50000},
    ],
    output_format="csv",  # csv or parquet
)
# {"row_count": 50000, "columns": [...]}

Worker Status

Check the processing worker fleet:

status = client.worker_status()
# {"backend": "local", "active_jobs": 2, "queue_size": 0, ...}

Error Handling

All API errors raise AutoDataError:

from datatoolpack import AutoDataClient, AutoDataError

with AutoDataClient() as client:
    try:
        result = client.process("data.csv", target_columns="price")
    except AutoDataError as e:
        print(f"API error {e.status_code}: {e}")
    except FileNotFoundError as e:
        print(f"File not found: {e}")
    except ValueError as e:
        print(f"Invalid input: {e}")  # e.g. unsupported file format

AutoDataError attributes:

  • str(e) — human-readable error message from the server
  • e.status_code — HTTP status code (e.g. 401, 429, 500), or None for non-HTTP errors

Transient errors (429, 502, 503, 504) are automatically retried up to max_retries times with exponential back-off.


Advanced Example: Non-blocking with manual polling

import time
from datatoolpack import AutoDataClient, AutoDataError

with AutoDataClient() as client:
    # Start job without blocking
    job = client.process(
        "large_dataset.csv",
        target_columns=["churn"],
        wait=False,
    )
    session_id = job["session_id"]
    print(f"Job started: {session_id}")

    # Poll manually
    while True:
        status = client.get_status(session_id)
        print(f"  {status['progress_percent']}% — {status['message']}")
        if status["status"] == "completed":
            break
        elif status["status"] in ("error", "cancelled"):
            raise AutoDataError(f"Job {status['status']}: {status['message']}")
        time.sleep(5)

    # Download results
    path = client.download_results(session_id, download_path="./outputs/")
    print(f"Results saved to {path}")

Sessions, Preferences, Webhooks (v0.12.0+)

The whole dashboard surface is now reachable from the SDK — sessions, parameter profiles, scheduled runs, webhooks, templates and sharing all live under /api/v1.

Sessions

for s in client.list_sessions():
    print(s["session_id"], s["status"])

client.rename_session(session_id, "BBVA credit — August")
detail = client.get_session(session_id)      # incl. the parameters it ran with

Dry-run validation

Check how the server reads a file before spending a run on it:

info = client.validate("new_data.csv")
print(info["columns"], info["row_count"])

Preference profiles

Four groups — outputs, advanced, anomaly, completion:

prefs = client.get_preferences("outputs")
prefs["dsg_output"]["disk"] = False
client.set_preferences("outputs", prefs)
client.reset_preferences("anomaly")          # back to server defaults

Webhooks — get told when a job finishes, instead of polling

wh = client.create_webhook(
    url="https://your-app.example.com/hooks/autodata",
    events=["session.completed", "session.failed"],
    secret="a-long-random-string",           # sign & verify: see note below
)
client.test_webhook(wh["webhook"]["id"])     # confirm your endpoint accepts it
client.webhook_deliveries(wh["webhook"]["id"])

Always set secret and verify the signature on your side — without it, anyone who learns your URL can forge a "your job finished" callback.

Templates and sharing

client.create_template("credit-monthly", config={...})
client.share_session(session_id, shared_with_id="user-b", permission="read",
                     expires_at="2026-12-31T00:00:00Z")

Give public share links an expires_at — a link without one never stops working.

Account & credits

client.account_profile()   # credit balance, role, limits (whole account)
client.account_usage()     # usage across every key, broken down by tool
client.get_usage()         # just the key you're authenticating with

Credits & cost estimation (v0.12.1+)

Runs are billed by data volume and by which stages you enable, scaled by how many rows you ask the synthesiser to produce. Cost isn't guessable from the outside, so you can price a run before committing to it:

quote = client.estimate(
    input_size_mb=250,
    input_rows=1_000_000,
    output_rows=2_000_000,      # 2x synthesis -> 2x the price
    tools={"anomaly": True, "dtc": True, "mdh": True,
           "cds": True, "dsm": True, "dsg": True},
)
print(quote["total_cost"], quote["tool_costs"])

estimate() runs nothing and charges nothing. Pass session_type="inference" to price a scoring call, which is charged at a reduced rate compared to a full pipeline run.

After a run, get_result() tells you what it actually cost:

result = client.get_result(session_id)
result["credits_charged"]      # what this run cost
result["credits_remaining"]    # balance afterwards
result.get("credit_warning")   # present once the balance goes negative

A negative balance does not stop your jobs — they keep running and the warning is there so you can top up before it becomes a conversation. Check the account-wide picture any time with client.account_profile().


Data residency — keeping data away from external LLMs (v0.12.1+)

Three pipeline stages can send data to a language model: parameter optimization (your project description and row/column counts), anomaly detection (column names plus up to ten sample values per column) and data completion/validation (identifiable record content). Everything else — encoding, imputation, scaling, splitting, synthesis — is local computation.

If your data must not leave your infrastructure, turn the calls off:

result = client.process(
    file_path="customers.csv",
    target_columns="churn",
    advanced_params={"llm_enabled": False},   # no stage contacts a model
)

Anomaly detection still runs using local heuristics; completion/validation is skipped, since it has no local equivalent.

Fail instead of degrading. By default a stage that can't reach a model falls back to heuristics — convenient, but silent. If you would rather know:

advanced_params={"strict_llm": True}          # raise instead of degrading

Proof, per run. Every result carries an audit block, so "did our data leave?" is answerable rather than assumed:

result["llm"]
# {
#   "llm_enabled": False,
#   "llm_used": False,                  # the auditable claim
#   "endpoint_host": "llm.internal.example",
#   "endpoint_is_public": False,        # flags api.x.ai, api.openai.com, ...
#   "stages": {
#     "anomaly_detection": {"llm_used": False, "reason": "disabled by policy"},
#     "dcv":               {"llm_used": False, "reason": "disabled by policy"},
#     "grok_optimizer":    {"llm_used": False, "reason": "not requested"}
#   }
# }

Every stage that didn't call out records why, so a fallback is never silent. Runs from before this feature have no llm block at all — that reads as "unknown", not "clean".

Operators can also point the whole deployment at their own OpenAI-compatible endpoint (GROQ_BASE_URL) or disable LLM calls server-wide (LLM_ENABLED=0), which overrides whatever a caller asks for.


Complete Method Reference

Sessions & Preferences (v0.12.0+)

Method Description
list_sessions() List the account's sessions
get_session(session_id) Full detail incl. run parameters
rename_session(session_id, name) Set a display name
set_session_retraining(session_id, enabled) Mark as a retraining base
list_session_outputs(session_id) In-memory outputs for a session
validate(file_path) Dry-run a CSV without starting a pipeline
get_preferences(kind) / set_preferences(kind, values) / reset_preferences(kind) outputs, advanced, anomaly, completion
account_profile() / account_usage() Credit balance and account-wide usage
estimate(input_size_mb=..., ...) Price a run before starting it (charges nothing)

Webhooks, Templates & Sharing (v0.12.0+)

Method Description
list_webhooks() / create_webhook(url, ...) / get_webhook(id) Webhook CRUD
update_webhook(id, **fields) / delete_webhook(id)
test_webhook(id) / webhook_deliveries(id) Send a test delivery; inspect attempts
list_templates() / create_template(name, config, ...) Reusable pipeline configs
update_template(id, **fields) / delete_template(id)
share_session(session_id, ...) / unshare_session(share_id) Share a session or mint a link
list_shared_sessions() / list_received_sessions() Shares you made / received

Core Pipeline

Method Description
process(file_path, target_columns, ...) Upload a file and run the pipeline
get_status(session_id) Poll processing progress
get_result(session_id) Get completed session results
wait_for_completion(session_id) Block until session finishes
cancel(session_id) Cancel a running job
retry_session(session_id) Retry a failed session from checkpoint
download_results(session_id, ...) Download result files
download_file(url, output_path) Download a single file

API Keys & Usage

Method Description
list_keys() List all API keys for the account
get_usage() Get credit usage statistics

Feature Selection

Method Description
recommend_features(session_id, target_columns, ...) Rank a completed session's features (F4)

Connectors

Method Description
test_connector(connector_type, ...) Test connectivity to a data source
discover(connector_type, ...) List tables/files in a data source
preview(connector_type, table, ...) Preview columns and row count
process_from_connector(connector_type, table, ...) Run pipeline from a connector source
write_output(session_id, connector_type, ...) Write results to a database/storage target

Credentials

Method Description
list_credentials() List saved credentials
save_credential(name, connector_type, secrets) Save a new credential
update_credential(credential_id, ...) Update a saved credential
delete_credential(credential_id) Delete a saved credential

Schema Mapping

Method Description
suggest_mapping(source_columns, target_columns) Auto-suggest column mapping
apply_mapping(session_id, mapping) Rename columns using a mapping

Quality Alerts

Method Description
list_quality_alerts() List quality alert rules
create_quality_alert(name, metric, operator, threshold, ...) Create an alert rule
update_quality_alert(rule_id, ...) Update an alert rule
delete_quality_alert(rule_id) Delete an alert rule
get_alert_events(session_id=None) List fired alert events

Sync Watermarks

Method Description
list_watermarks() List all sync watermarks
reset_watermark(watermark_id) Reset a watermark to re-sync

Scheduled Runs

Method Description
list_scheduled_runs() List all scheduled runs
create_scheduled_run(name, schedule_type, ...) Create a scheduled run
delete_scheduled_run(run_id) Delete a scheduled run
toggle_scheduled_run(run_id) Enable/disable a scheduled run
run_scheduled_now(run_id) Trigger a scheduled run immediately

Triggers

Method Description
list_triggers() List your triggers
create_trigger(name, trigger_type, ...) Create an event trigger (F7)
get_trigger(trigger_id) Fetch one trigger
update_trigger(trigger_id, ...) Update a trigger
delete_trigger(trigger_id) Delete a trigger
test_trigger(trigger_id) Dry-run a trigger without firing

Folder Listeners

Method Description
list_listeners() List folder listeners
create_listener(name, source_type, watch_path, ...) Create a folder listener
update_listener(listener_id, ...) Update a folder listener
delete_listener(listener_id) Delete a folder listener

SFTP

Method Description
sftp_info() Get SFTP server connection info
list_sftp_credentials() List SFTP credentials
create_sftp_credential(name) Create SFTP credential
delete_sftp_credential(credential_id) Delete SFTP credential

Spark (Large Data)

Method Description
spark_status() Check if Spark is available
spark_read(file_path, sample_rows) Read a large file via Spark
spark_transform(file_path, output_path, operations, ...) Apply Spark transformations

Worker Status

Method Description
worker_status() Get worker fleet status

Requirements

  • Python >= 3.8
  • requests >= 2.25.0

License

MIT

Metadata

Release files for datatoolpack 0.12.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for datatoolpack 0.12.1
File Size Uploaded Provenance Actions
datatoolpack-0.12.1.tar.gz 60.2 kB Without provenance Details for datatoolpack-0.12.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for datatoolpack 0.12.1
File Interpreter ABI Platform Actions
datatoolpack-0.12.1-py3-none-any.whl Python 3 none any Details for datatoolpack-0.12.1-py3-none-any.whl

Total release size: 96.7 kB

Release files / datatoolpack-0.12.1.tar.gz

Download URL datatoolpack-0.12.1.tar.gz
Size 60.2 kB
Tags Source
SHA-256 checksum
How to use checksums
75de7049ea10d3ac4ffdd43cef4c73d4385bfdf1a93f14e9ae56ea6af4e3868d
BLAKE2b-256 checksum
How to use checksums
22cd4de0945ea746722412befa24295f36da4be451cc4a2605cf2b3f9097da0b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.13.9

Release files / datatoolpack-0.12.1-py3-none-any.whl

Download URL datatoolpack-0.12.1-py3-none-any.whl
Size 36.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
949ccaa5b3aec6771f9c3ada56f0e74ddf7f755be85f5026cad45f9039de065e
BLAKE2b-256 checksum
How to use checksums
5a81e07f6e11c832ddadac53caaa8606ce1833fe29150354c92414763845f45d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.13.9

Security for datatoolpack 0.12.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page