AutoData Python Client
Official Python SDK for the AutoData ML data preparation pipeline API.
Installation
pip install datatoolpack
Or install from source:
git clone https://github.com/datatoolpack/datatoolpack
cd datatoolpack
pip install .
Quick Start
from datatoolpack import AutoDataClient
with AutoDataClient(api_key="dtpk_YOUR_API_KEY") as client:
result = client.process(
file_path="data.csv",
target_columns=["price"],
output_rows=10000,
)
print(result["files"])
Get your API key from the AutoData dashboard → API Keys tab.
Authentication
API Key (recommended)
with AutoDataClient(api_key="dtpk_YOUR_API_KEY") as client:
...
Access Code (passcode)
If you have an access code instead of an API key:
with AutoDataClient(passcode="123456789012") as client:
...
Environment Variables
Set environment variables to avoid hardcoding credentials:
export AUTODATA_API_KEY="dtpk_YOUR_API_KEY"
# or
export AUTODATA_PASSCODE="123456789012"
# optional
export AUTODATA_BASE_URL="https://autodata.datatoolpack.com"
# No need to pass api_key — reads from AUTODATA_API_KEY automatically
with AutoDataClient() as client:
result = client.process("data.csv", target_columns="price")
Supported File Formats
| Format | Extensions |
|---|---|
| CSV | .csv |
| Excel | .xlsx, .xls |
| Parquet | .parquet |
| JSON | .json |
| Feather | .feather |
| ORC | .orc |
Reference
AutoDataClient(api_key, passcode, base_url, timeout, max_retries)
| Parameter | Type | Default | Env Variable | Description |
|---|---|---|---|---|
api_key |
str |
None |
AUTODATA_API_KEY |
API key starting with dtpk_ |
passcode |
str |
None |
AUTODATA_PASSCODE |
Access code (alternative to api_key) |
base_url |
str |
"https://autodata.datatoolpack.com" |
AUTODATA_BASE_URL |
Server URL (no trailing slash) |
timeout |
int |
120 |
Request timeout in seconds | |
max_retries |
int |
3 |
Auto-retries for 429 / 502 / 503 / 504 |
Either api_key or passcode is required (via argument or env variable).
Implements context manager — use with with to auto-close connections:
with AutoDataClient(api_key="dtpk_...") as client:
...
# session is closed automatically
Or close manually:
client = AutoDataClient(api_key="dtpk_...")
# ... use client ...
client.close()
Core Pipeline
client.process(...) — Upload & run pipeline
result = client.process(
file_path="data.csv", # Path to input file (CSV, XLSX, Parquet, ...)
target_columns=["price"], # y-column(s) for ML
output_rows=10000, # Target row count in the synthetic output (default 10000)
tools={ # Toggle pipeline steps (all optional)
"anomaly": False, # Anomaly detection (off by default)
"dtc": True, # Data Type Conversion
"mdh": True, # Missing Data Handler
"dor": False, # Dimensionality Reduction (off by default)
"cds": True, # Column Scaling
"dsm": True, # Data Split Manager
"dsg": True, # Synthetic Data Generator
},
advanced_params={ # Fine-grained parameters (all optional)
"excluded_columns": ["id"], # Columns to drop before processing
"text_mode": 0, # 0=none, 1=neural, 2=tfidf
"text_cleaning": True, # Clean text before encoding
"zscore_limit": 3.0, # Z-score outlier threshold
"dsg_mode": "copula", # "copula" or "gan"
"similarity_p": 95, # Similarity percentile for DSG
},
wait=True, # Block until complete (default True)
poll_interval=2, # Status poll interval in seconds
download_path="./outputs/", # Where to save files (default auto)
auto_download=True, # Set False to skip download when wait=True
output_preferences=["dsg.csv"], # Which files to download (default all)
compressed=True, # Download as ZIP (default True)
)
Returns a dict:
{
"session_id": "abc123...",
"status": "completed",
"files": [
{"name": "dsg.csv", "url": "/download/.../dsg.csv", "size": 2097152, "description": "..."},
{"name": "dsm_train.csv", ...},
],
"row_count": 10000,
"duration_seconds": 42.1,
}
Set wait=False to get back immediately with just session_id and status:
result = client.process(file_path="data.csv", target_columns="price", wait=False)
session_id = result["session_id"]
client.get_status(session_id) — Poll progress
status = client.get_status(session_id)
# {
# "status": "running", # queued | running | completed | error | cancelled
# "message": "Running MDH...",
# "current_step": 3,
# "total_steps": 6,
# "progress_percent": 50,
# "duration_seconds": 15.3,
# }
client.get_result(session_id) — Fetch completed results
result = client.get_result(session_id)
# {"status": "completed", "files": [...], "row_count": ..., "duration_seconds": ...}
client.wait_for_completion(session_id, poll_interval) — Block until done
result = client.wait_for_completion(session_id, poll_interval=3)
Prints live progress to stdout. Raises AutoDataError if processing fails.
client.cancel(session_id) — Cancel a running job
cancelled = client.cancel(session_id) # True if acknowledged
client.retry_session(session_id) — Retry a failed pipeline
Retry from the last checkpoint — no need to re-upload or re-configure:
result = client.retry_session(session_id="failed-session-id")
# {'success': True, 'session_id': '...', 'message': 'Retrying from last checkpoint'}
client.download_results(session_id, ...) — Download output files
path = client.download_results(
session_id,
download_path="./my_outputs/", # Directory to save into
output_preferences=["dsg.csv"], # Specific files only (None = all)
compressed=True, # ZIP download (default) or individual files
)
print(f"Saved to {path}")
client.download_file(url, output_path) — Download a single file
client.download_file("/api/v1/download/abc123.../dsg.csv", "dsg.csv")
Feature Selection (F4)
Rank a completed session's columns by predictive importance — without re-running the pipeline. Handy for trimming wide datasets before training.
client.recommend_features(session_id, target_columns, methods, top_k)
ranking = client.recommend_features(
session_id="abc123...", # a completed session you own
target_columns=["price"], # rank features against these target(s)
methods=["mutual_information"], # one or more methods (see below); default ["mutual_information"]
top_k=20, # number of top features to return (default 20)
)
# {
# "ranking": [{"feature": "sqft", "score": 0.81}, ...], # combined across methods
# "per_method": {"mutual_information": [...], "shap_importance": [...]},
# "warnings": [...],
# }
Available methods: variance_threshold, correlation_pruning, mutual_information, rfe, shap_importance, pca. Pass several to blend their rankings, e.g. methods=["mutual_information", "shap_importance"]. (SHAP-based ranking uses a fast LightGBM model on the server; it degrades gracefully to mutual_information if those libraries are unavailable.)
Inference & Retraining (v0.11.0+)
Apply a completed session's fitted transforms to new data — the missing half of the ML lifecycle: train once, then score fresh production rows with exactly the same encoding, imputation and scaling.
client.infer(...) — Score new data
Lightweight pipeline (Anomaly Detection → DTC → MDH → CDS). Outputs
features.csv (+ y_columns.csv if targets are present in the new file).
result = client.infer(
original_session_id="5d3bc0f0-...", # a completed training session you own
file_path="new_rows.csv", # columns must match the training data
download_path="./scored", # optional: save outputs locally
return_dataframes=True, # optional: get pandas DataFrames back
)
features_df = result["dataframes"]["features"]
client.retrain(...) — Refresh a training set with new rows
Full pipeline including Split → CDS → DSM → DSG; produces
retraining_output.csv (features + targets combined).
result = client.retrain(
original_session_id="5d3bc0f0-...",
file_path="new_rows.csv",
run_dsg=True, output_rows=20000, # synthetic augmentation target
download_path="./retrained",
)
Both calls are synchronous (the response arrives when processing ends;
default per-request timeout is ≥ 900 s — raise timeout= for very large
files), require the original session to be completed and owned by you,
and are free of credit charges in v1.
API Keys & Usage
client.list_keys() — List API keys
keys = client.list_keys()
# [{"id": "...", "name": "My Key", "prefix": "dtpk_abc123", "created_at": "..."}]
client.get_usage() — Usage statistics
usage = client.get_usage()
# {
# "daily_credits_used": 500,
# "daily_credit_limit": 10000,
# "daily_remaining": 9500,
# "lifetime_credits_used": 12340,
# "lifetime_credit_limit": 1000000,
# "lifetime_remaining": 987660,
# "daily_request_count": 3,
# "last_used_at": "2026-04-12T10:30:00Z",
# }
Connectors
Read data directly from databases and cloud storage instead of uploading files.
Supported connector types
| Type | Description | Required secrets |
|---|---|---|
sql |
PostgreSQL, MySQL, SQL Server | connection_string |
snowflake |
Snowflake Data Cloud | account, user, password, database, schema, warehouse |
bigquery |
Google BigQuery | credentials_json, project_id, dataset |
mongodb |
MongoDB / Atlas | connection_string, database |
s3 |
Amazon S3 | bucket, access_key_id, secret_access_key, region |
gcs |
Google Cloud Storage | bucket, credentials_json |
databricks |
Databricks Lakehouse | host, token, catalog, schema |
delta |
Delta Lake | path (+ cloud credentials if remote) |
fabric |
Microsoft Fabric | workspace_id, lakehouse_id, tenant_id, client_id, client_secret |
kafka |
Apache Kafka | bootstrap_servers, topic, group_id |
kinesis |
Amazon Kinesis | stream_name, region, access_key_id, secret_access_key |
More connectors: AutoData supports 30+ source types. Beyond the common ones above you can also pass
oracle,redshift,cassandra,dynamodb,elasticsearch,clickhouse,timescaledb,synapse,azure_blob,adls,sap_hana,mqtt,opcua,influxdb,pi_web,salesforce,hubspot,stripe,netsuite,google_sheets,ga4,sharepoint, andhttp/restasconnector_type. See Dashboard → Documentation → Connector Data Sources for each type's requiredsecretsfields.
Quick connector example
with AutoDataClient(api_key="dtpk_...") as client:
# Test the connection
client.test_connector("sql", secrets={
"connection_string": "postgresql://user:pass@host/db"
})
# List tables
tables = client.discover("sql", secrets={
"connection_string": "postgresql://user:pass@host/db"
})
print(tables) # ["users", "orders", ...]
# Preview columns
info = client.preview("sql", table="orders", secrets={
"connection_string": "postgresql://user:pass@host/db"
})
print(info["columns"]) # ["id", "price", "date", ...]
# Run the full pipeline from the connector
result = client.process_from_connector(
connector_type="sql",
table="orders",
target_columns=["price"],
secrets={"connection_string": "postgresql://user:pass@host/db"},
output_rows=10000,
)
print(result["files"])
Connector examples by type
Snowflake:
secrets = {
"account": "abc123.us-east-1",
"user": "analyst",
"password": "...",
"database": "PROD",
"schema": "PUBLIC",
"warehouse": "COMPUTE_WH",
}
tables = client.discover("snowflake", secrets=secrets)
result = client.process_from_connector("snowflake", table="ORDERS",
target_columns=["AMOUNT"], secrets=secrets)
BigQuery:
secrets = {
"credentials_json": '{"type":"service_account",...}',
"project_id": "my-project",
"dataset": "analytics",
}
tables = client.discover("bigquery", secrets=secrets)
Amazon S3:
secrets = {
"bucket": "my-data-lake",
"access_key_id": "AKIA...",
"secret_access_key": "...",
"region": "us-east-1",
}
result = client.process_from_connector("s3", table="data/sales.csv",
target_columns=["revenue"], secrets=secrets, output_rows=50000)
MongoDB:
secrets = {
"connection_string": "mongodb+srv://user:pass@cluster.mongodb.net",
"database": "analytics",
}
tables = client.discover("mongodb", secrets=secrets)
client.test_connector(connector_type, secrets, credential_id) — Test connection
result = client.test_connector("s3", secrets={...})
# {"success": True, "message": "Connection successful"}
client.discover(connector_type, ...) — List tables / files
tables = client.discover("sql", secrets={...})
# ["users", "orders", "transactions"]
client.preview(connector_type, table, ...) — Preview columns & row count
info = client.preview("sql", table="orders", secrets={...})
# {"success": True, "columns": ["id", "price", ...], "row_count": 50000}
Supports optional custom_query parameter for SQL-based connectors:
info = client.preview("sql", table="orders", custom_query="SELECT * FROM orders WHERE date > '2025-01-01'", secrets={...})
client.process_from_connector(...) — Run pipeline from connector
result = client.process_from_connector(
connector_type="sql",
table="orders",
target_columns=["price"],
secrets={"connection_string": "postgresql://..."},
output_rows=10000,
incremental=True, # Only fetch new rows since last sync
type_casts={"age": "int"}, # Override column types before processing
wait=True,
download_path="./outputs/",
)
Supports all the same options as client.process(): tools, advanced_params, wait, poll_interval, auto_download, output_preferences, compressed.
client.write_output(session_id, ...) — Write results to a target
client.write_output(
session_id="abc123...",
connector_type="sql",
table_name="ml_prepared_data",
secrets={"connection_string": "postgresql://..."},
output_stage="dsg", # which pipeline output to write
if_exists="replace", # "replace", "append", "fail", or "merge"
merge_keys=["id"], # column names for merge matching (when if_exists='merge')
)
# {"success": True, "rows_written": 10000, "table": "ml_prepared_data"}
Credential Management
Save credentials on the server so you don't need to pass secrets every time:
# Save a new credential
cred = client.save_credential(
name="Production DB",
connector_type="sql",
secrets={"connection_string": "postgresql://..."},
)
cred_id = cred["id"]
# Use credential_id instead of secrets
tables = client.discover("sql", credential_id=cred_id)
result = client.process_from_connector("sql", table="orders",
target_columns=["price"], credential_id=cred_id)
# List saved credentials (secrets are never exposed)
creds = client.list_credentials()
# Update a credential
client.update_credential(cred_id, name="Prod DB v2", secrets={...})
# Delete a credential
client.delete_credential(cred_id)
Schema Mapping
Auto-suggest and apply column mappings between source and target schemas:
client.suggest_mapping(source_columns, target_columns) — Auto-suggest mapping
mapping = client.suggest_mapping(
source_columns=["customer_name", "order_amt", "order_dt"],
target_columns=["name", "amount", "date"],
threshold=0.6, # minimum similarity score (0-1)
)
# {
# "mapping": {"customer_name": "name", "order_amt": "amount", "order_dt": "date"},
# "scores": {"customer_name": 0.85, "order_amt": 0.72, "order_dt": 0.68}
# }
client.apply_mapping(session_id, mapping) — Rename columns
result = client.apply_mapping(
session_id="abc123...",
mapping={"old_col_name": "new_col_name", ...},
output_stage="dsg", # which pipeline stage to apply to
)
# {"columns": ["new_col_name", ...], "preview": [...]}
Quality Alerts
Monitor pipeline metrics and get notified when thresholds are breached:
# Create a quality alert rule
rule = client.create_quality_alert(
name="High row loss",
metric="row_loss_pct", # row_loss_pct, null_pct, column_drop_count, duration_seconds
operator=">", # >, <, >=, <=, ==
threshold=10.0,
severity="critical", # warning or critical
stage="mdh", # optional: anomaly, dtc, mdh, cds, dsm, dsg
)
# List rules
rules = client.list_quality_alerts()
# Update a rule
client.update_quality_alert(rule_id="...", threshold=15.0, severity="warning")
# Get fired alert events
events = client.get_alert_events(session_id="optional-filter")
# Delete a rule
client.delete_quality_alert(rule_id="...")
Available metrics:
| Metric | Description |
|---|---|
row_loss_pct |
Percentage of rows lost in a step |
null_pct |
Percentage of null values remaining |
column_drop_count |
Number of columns dropped |
duration_seconds |
Time taken by a pipeline step |
Sync Watermarks
Track incremental sync progress for connector-based pipelines:
# List all watermarks
watermarks = client.list_watermarks()
# [{"id": "wm-123", "connector_type": "sql", "table_name": "orders",
# "watermark_column": "updated_at", "last_value": "2026-04-12T00:00:00Z"}]
# Reset a watermark (re-sync from beginning)
client.reset_watermark(watermark_id="wm-123")
Scheduled Runs
Automate recurring pipeline executions:
# Create an interval-based schedule (every 24 hours)
schedule = client.create_scheduled_run(
name="Daily ETL",
schedule_type="interval",
interval_minutes=1440,
connector_type="snowflake",
table="orders",
credential_id="cred-id",
y_columns=["target"],
output_rows=50000,
)
# Create with cron expression (weekdays at 8 AM)
schedule = client.create_scheduled_run(
name="Weekday ETL",
schedule_type="cron",
cron_expression="0 8 * * 1-5",
connector_type="snowflake",
table="orders",
credential_id="cred-id",
y_columns=["target"],
tools={"anomaly": True, "dsg": False}, # customize pipeline steps
)
# List all schedules
schedules = client.list_scheduled_runs()
# Enable/disable a schedule
client.toggle_scheduled_run(run_id="run-id")
# Trigger immediately (ignore schedule)
client.run_scheduled_now(run_id="run-id")
# Delete a schedule
client.delete_scheduled_run(run_id="run-id")
Triggers (F7)
Auto-start a pipeline when an external event fires — an inbound webhook, a connector watermark advancing, another pipeline completing, or a quality alert. Triggers are feature-flagged on the server (FEATURE_FLAG_TRIGGERS); if the flag is off, these endpoints return 404.
client.create_trigger(name, trigger_type, config, target_pipeline_config, enabled)
trigger = client.create_trigger(
name="Reprocess on new S3 batch",
trigger_type="inbound_webhook", # inbound_webhook | watermark_advance | pipeline_completion | quality_alert
config={}, # type-specific (see table below)
target_pipeline_config={ # the pipeline to run when the trigger fires
"source_type": "s3",
"credential_id": "cred-uuid",
"table": "incoming/sales.csv",
"target_columns": ["revenue"],
"output_rows": 10000,
},
enabled=True,
)
# For inbound_webhook the secret is returned ONLY once — save it now:
print(trigger["webhook_secret"], trigger["inbound_url"])
Trigger types & their config:
trigger_type |
config keys |
|---|---|
inbound_webhook |
optional {"hmac_key": "..."} — require HMAC-SHA256-signed requests |
watermark_advance |
{"credential_id", "table", "watermark_column", "min_advance"} |
pipeline_completion |
{} — chain off another pipeline finishing |
quality_alert |
{} — fire when a quality-alert rule trips |
# List / fetch / update / delete
triggers = client.list_triggers()
t = client.get_trigger(trigger_id)
client.update_trigger(trigger_id, enabled=False) # name | enabled | config | target_pipeline_config
client.delete_trigger(trigger_id)
# Dry-run: validate config + target WITHOUT firing
report = client.test_trigger(trigger_id)
# {"would_fire": True, "checks": {...}}
Firing an inbound webhook from your own system:
curl -X POST "https://autodata.datatoolpack.com/trigger/inbound/<webhook_secret>" \
-H "Content-Type: application/json" \
-d '{"any": "payload"}'
# If you set an hmac_key, also sign the raw body:
# X-Signature: sha256=<hex HMAC-SHA256 of the request body>
Inbound firing is rate-limited and HMAC-verified server-side; repeated failures auto-disable the trigger.
Folder Listeners
Automatically trigger pipelines when new files appear in cloud storage:
# Create an S3 folder listener
listener = client.create_listener(
name="S3 Ingest",
source_type="s3", # s3, gcs, azure_blob, local, sftp
watch_path="s3://bucket/incoming/",
credential_id="cred-id",
y_columns=["target"],
pipeline_config={"enable_dsg": False},
output_rows=10000,
)
# List all listeners
listeners = client.list_listeners()
# Update a listener
client.update_listener(listener_id="...", watch_path="s3://bucket/new-path/", enabled=False)
# Delete a listener
client.delete_listener(listener_id="...")
Supported source types: s3, gcs, azure_blob, local, sftp
SFTP Upload
Upload files to the AutoData server via SFTP for processing:
# Get SFTP server connection info
info = client.sftp_info()
# {"host": "...", "port": 22, "username": "..."}
# Manage SFTP credentials
cred = client.create_sftp_credential(name="My Upload Key")
# {"id": "...", "username": "...", "password": "..."} (password shown only once)
creds = client.list_sftp_credentials()
client.delete_sftp_credential(credential_id="...")
Spark (Large Data)
For datasets too large for pandas, AutoData can use PySpark on the server:
client.spark_status() — Check Spark availability
status = client.spark_status()
# {"available": True, "spark_version": "3.5.0", ...}
client.spark_read(file_path, sample_rows) — Read large files
info = client.spark_read(
file_path="/data/huge_dataset.parquet",
sample_rows=20,
)
# {"row_count": 50000000, "columns": [...], "dtypes": {...}, "sample": [...]}
client.spark_transform(...) — Apply Spark transformations
result = client.spark_transform(
file_path="/data/huge_dataset.parquet",
output_path="/data/transformed.csv",
operations=[
{"op": "cast", "type_casts": {"age": "int"}},
{"op": "drop_nulls", "subset": ["col1"]},
{"op": "fill_nulls", "strategy": "mean"},
{"op": "normalize", "method": "minmax"},
{"op": "sample", "n": 50000},
],
output_format="csv", # csv or parquet
)
# {"row_count": 50000, "columns": [...]}
Worker Status
Check the processing worker fleet:
status = client.worker_status()
# {"backend": "local", "active_jobs": 2, "queue_size": 0, ...}
Error Handling
All API errors raise AutoDataError:
from datatoolpack import AutoDataClient, AutoDataError
with AutoDataClient() as client:
try:
result = client.process("data.csv", target_columns="price")
except AutoDataError as e:
print(f"API error {e.status_code}: {e}")
except FileNotFoundError as e:
print(f"File not found: {e}")
except ValueError as e:
print(f"Invalid input: {e}") # e.g. unsupported file format
AutoDataError attributes:
str(e)— human-readable error message from the servere.status_code— HTTP status code (e.g.401,429,500), orNonefor non-HTTP errors
Transient errors (429, 502, 503, 504) are automatically retried up to max_retries times with exponential back-off.
Advanced Example: Non-blocking with manual polling
import time
from datatoolpack import AutoDataClient, AutoDataError
with AutoDataClient() as client:
# Start job without blocking
job = client.process(
"large_dataset.csv",
target_columns=["churn"],
wait=False,
)
session_id = job["session_id"]
print(f"Job started: {session_id}")
# Poll manually
while True:
status = client.get_status(session_id)
print(f" {status['progress_percent']}% — {status['message']}")
if status["status"] == "completed":
break
elif status["status"] in ("error", "cancelled"):
raise AutoDataError(f"Job {status['status']}: {status['message']}")
time.sleep(5)
# Download results
path = client.download_results(session_id, download_path="./outputs/")
print(f"Results saved to {path}")
Sessions, Preferences, Webhooks (v0.12.0+)
The whole dashboard surface is now reachable from the SDK — sessions,
parameter profiles, scheduled runs, webhooks, templates and sharing all live
under /api/v1.
Sessions
for s in client.list_sessions():
print(s["session_id"], s["status"])
client.rename_session(session_id, "BBVA credit — August")
detail = client.get_session(session_id) # incl. the parameters it ran with
Dry-run validation
Check how the server reads a file before spending a run on it:
info = client.validate("new_data.csv")
print(info["columns"], info["row_count"])
Preference profiles
Four groups — outputs, advanced, anomaly, completion:
prefs = client.get_preferences("outputs")
prefs["dsg_output"]["disk"] = False
client.set_preferences("outputs", prefs)
client.reset_preferences("anomaly") # back to server defaults
Webhooks — get told when a job finishes, instead of polling
wh = client.create_webhook(
url="https://your-app.example.com/hooks/autodata",
events=["session.completed", "session.failed"],
secret="a-long-random-string", # sign & verify: see note below
)
client.test_webhook(wh["webhook"]["id"]) # confirm your endpoint accepts it
client.webhook_deliveries(wh["webhook"]["id"])
Always set secret and verify the signature on your side — without it,
anyone who learns your URL can forge a "your job finished" callback.
Templates and sharing
client.create_template("credit-monthly", config={...})
client.share_session(session_id, shared_with_id="user-b", permission="read",
expires_at="2026-12-31T00:00:00Z")
Give public share links an expires_at — a link without one never stops
working.
Account & credits
client.account_profile() # credit balance, role, limits (whole account)
client.account_usage() # usage across every key, broken down by tool
client.get_usage() # just the key you're authenticating with
Credits & cost estimation (v0.12.1+)
Runs are billed by data volume and by which stages you enable, scaled by how many rows you ask the synthesiser to produce. Cost isn't guessable from the outside, so you can price a run before committing to it:
quote = client.estimate(
input_size_mb=250,
input_rows=1_000_000,
output_rows=2_000_000, # 2x synthesis -> 2x the price
tools={"anomaly": True, "dtc": True, "mdh": True,
"cds": True, "dsm": True, "dsg": True},
)
print(quote["total_cost"], quote["tool_costs"])
estimate() runs nothing and charges nothing. Pass
session_type="inference" to price a scoring call, which is charged at a
reduced rate compared to a full pipeline run.
After a run, get_result() tells you what it actually cost:
result = client.get_result(session_id)
result["credits_charged"] # what this run cost
result["credits_remaining"] # balance afterwards
result.get("credit_warning") # present once the balance goes negative
A negative balance does not stop your jobs — they keep running and the
warning is there so you can top up before it becomes a conversation. Check the
account-wide picture any time with client.account_profile().
Data residency — keeping data away from external LLMs (v0.12.1+)
Three pipeline stages can send data to a language model: parameter optimization (your project description and row/column counts), anomaly detection (column names plus up to ten sample values per column) and data completion/validation (identifiable record content). Everything else — encoding, imputation, scaling, splitting, synthesis — is local computation.
If your data must not leave your infrastructure, turn the calls off:
result = client.process(
file_path="customers.csv",
target_columns="churn",
advanced_params={"llm_enabled": False}, # no stage contacts a model
)
Anomaly detection still runs using local heuristics; completion/validation is skipped, since it has no local equivalent.
Fail instead of degrading. By default a stage that can't reach a model falls back to heuristics — convenient, but silent. If you would rather know:
advanced_params={"strict_llm": True} # raise instead of degrading
Proof, per run. Every result carries an audit block, so "did our data leave?" is answerable rather than assumed:
result["llm"]
# {
# "llm_enabled": False,
# "llm_used": False, # the auditable claim
# "endpoint_host": "llm.internal.example",
# "endpoint_is_public": False, # flags api.x.ai, api.openai.com, ...
# "stages": {
# "anomaly_detection": {"llm_used": False, "reason": "disabled by policy"},
# "dcv": {"llm_used": False, "reason": "disabled by policy"},
# "grok_optimizer": {"llm_used": False, "reason": "not requested"}
# }
# }
Every stage that didn't call out records why, so a fallback is never silent.
Runs from before this feature have no llm block at all — that reads as
"unknown", not "clean".
Operators can also point the whole deployment at their own OpenAI-compatible
endpoint (GROQ_BASE_URL) or disable LLM calls server-wide
(LLM_ENABLED=0), which overrides whatever a caller asks for.
Complete Method Reference
Sessions & Preferences (v0.12.0+)
| Method | Description |
|---|---|
list_sessions() |
List the account's sessions |
get_session(session_id) |
Full detail incl. run parameters |
rename_session(session_id, name) |
Set a display name |
set_session_retraining(session_id, enabled) |
Mark as a retraining base |
list_session_outputs(session_id) |
In-memory outputs for a session |
validate(file_path) |
Dry-run a CSV without starting a pipeline |
get_preferences(kind) / set_preferences(kind, values) / reset_preferences(kind) |
outputs, advanced, anomaly, completion |
account_profile() / account_usage() |
Credit balance and account-wide usage |
estimate(input_size_mb=..., ...) |
Price a run before starting it (charges nothing) |
Webhooks, Templates & Sharing (v0.12.0+)
| Method | Description |
|---|---|
list_webhooks() / create_webhook(url, ...) / get_webhook(id) |
Webhook CRUD |
update_webhook(id, **fields) / delete_webhook(id) |
|
test_webhook(id) / webhook_deliveries(id) |
Send a test delivery; inspect attempts |
list_templates() / create_template(name, config, ...) |
Reusable pipeline configs |
update_template(id, **fields) / delete_template(id) |
|
share_session(session_id, ...) / unshare_session(share_id) |
Share a session or mint a link |
list_shared_sessions() / list_received_sessions() |
Shares you made / received |
Core Pipeline
| Method | Description |
|---|---|
process(file_path, target_columns, ...) |
Upload a file and run the pipeline |
get_status(session_id) |
Poll processing progress |
get_result(session_id) |
Get completed session results |
wait_for_completion(session_id) |
Block until session finishes |
cancel(session_id) |
Cancel a running job |
retry_session(session_id) |
Retry a failed session from checkpoint |
download_results(session_id, ...) |
Download result files |
download_file(url, output_path) |
Download a single file |
API Keys & Usage
| Method | Description |
|---|---|
list_keys() |
List all API keys for the account |
get_usage() |
Get credit usage statistics |
Feature Selection
| Method | Description |
|---|---|
recommend_features(session_id, target_columns, ...) |
Rank a completed session's features (F4) |
Connectors
| Method | Description |
|---|---|
test_connector(connector_type, ...) |
Test connectivity to a data source |
discover(connector_type, ...) |
List tables/files in a data source |
preview(connector_type, table, ...) |
Preview columns and row count |
process_from_connector(connector_type, table, ...) |
Run pipeline from a connector source |
write_output(session_id, connector_type, ...) |
Write results to a database/storage target |
Credentials
| Method | Description |
|---|---|
list_credentials() |
List saved credentials |
save_credential(name, connector_type, secrets) |
Save a new credential |
update_credential(credential_id, ...) |
Update a saved credential |
delete_credential(credential_id) |
Delete a saved credential |
Schema Mapping
| Method | Description |
|---|---|
suggest_mapping(source_columns, target_columns) |
Auto-suggest column mapping |
apply_mapping(session_id, mapping) |
Rename columns using a mapping |
Quality Alerts
| Method | Description |
|---|---|
list_quality_alerts() |
List quality alert rules |
create_quality_alert(name, metric, operator, threshold, ...) |
Create an alert rule |
update_quality_alert(rule_id, ...) |
Update an alert rule |
delete_quality_alert(rule_id) |
Delete an alert rule |
get_alert_events(session_id=None) |
List fired alert events |
Sync Watermarks
| Method | Description |
|---|---|
list_watermarks() |
List all sync watermarks |
reset_watermark(watermark_id) |
Reset a watermark to re-sync |
Scheduled Runs
| Method | Description |
|---|---|
list_scheduled_runs() |
List all scheduled runs |
create_scheduled_run(name, schedule_type, ...) |
Create a scheduled run |
delete_scheduled_run(run_id) |
Delete a scheduled run |
toggle_scheduled_run(run_id) |
Enable/disable a scheduled run |
run_scheduled_now(run_id) |
Trigger a scheduled run immediately |
Triggers
| Method | Description |
|---|---|
list_triggers() |
List your triggers |
create_trigger(name, trigger_type, ...) |
Create an event trigger (F7) |
get_trigger(trigger_id) |
Fetch one trigger |
update_trigger(trigger_id, ...) |
Update a trigger |
delete_trigger(trigger_id) |
Delete a trigger |
test_trigger(trigger_id) |
Dry-run a trigger without firing |
Folder Listeners
| Method | Description |
|---|---|
list_listeners() |
List folder listeners |
create_listener(name, source_type, watch_path, ...) |
Create a folder listener |
update_listener(listener_id, ...) |
Update a folder listener |
delete_listener(listener_id) |
Delete a folder listener |
SFTP
| Method | Description |
|---|---|
sftp_info() |
Get SFTP server connection info |
list_sftp_credentials() |
List SFTP credentials |
create_sftp_credential(name) |
Create SFTP credential |
delete_sftp_credential(credential_id) |
Delete SFTP credential |
Spark (Large Data)
| Method | Description |
|---|---|
spark_status() |
Check if Spark is available |
spark_read(file_path, sample_rows) |
Read a large file via Spark |
spark_transform(file_path, output_path, operations, ...) |
Apply Spark transformations |
Worker Status
| Method | Description |
|---|---|
worker_status() |
Get worker fleet status |
Requirements
- Python >= 3.8
requests>= 2.25.0
License
MIT
Metadata
Release files for datatoolpack 0.12.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| datatoolpack-0.12.1.tar.gz | 60.2 kB | Without provenance | Details for datatoolpack-0.12.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| datatoolpack-0.12.1-py3-none-any.whl | Python 3 | none | any | Details for datatoolpack-0.12.1-py3-none-any.whl |
Total release size: 96.7 kB
Release files / datatoolpack-0.12.1.tar.gz
| Download URL | datatoolpack-0.12.1.tar.gz |
|---|---|
| Size | 60.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
75de7049ea10d3ac4ffdd43cef4c73d4385bfdf1a93f14e9ae56ea6af4e3868d
|
|
BLAKE2b-256 checksum How to use checksums |
22cd4de0945ea746722412befa24295f36da4be451cc4a2605cf2b3f9097da0b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.13.9
|
Release files / datatoolpack-0.12.1-py3-none-any.whl
| Download URL | datatoolpack-0.12.1-py3-none-any.whl |
|---|---|
| Size | 36.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
949ccaa5b3aec6771f9c3ada56f0e74ddf7f755be85f5026cad45f9039de065e
|
|
BLAKE2b-256 checksum How to use checksums |
5a81e07f6e11c832ddadac53caaa8606ce1833fe29150354c92414763845f45d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.13.9
|
Security for datatoolpack 0.12.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.