daggydata

Upload datasets to Daggy from a script, a notebook or cron, with an API key: one file, several files, a hive-partitioned folder, or a pandas or polars DataFrame straight from memory, up to 25 GB. CSV, TSV, JSON and Parquet are supported. Bytes go straight to storage in parallel parts, interrupted uploads resume where they stopped, and only requests is needed, no cloud credentials.

pip install daggydata

The package installs the daggy command and imports as daggy.

Create an API key on the API keys page of the Daggy app, then:

export DAGGY_API_KEY=daggy_prod_...
daggy upload sales ./sales.parquet
daggy upload events ./events/ --overwrite --description "Clickstream"

import daggy  # reads DAGGY_API_KEY

ds = daggy.upload("sales", "./sales.parquet")
print(ds.name, ds.num_obs, "rows")

What gets uploaded

• A file is uploaded under its name, with anything outside [A-Za-z0-9_.-] replaced (Q3 sales.csv → Q3_sales.csv).
• A folder becomes one dataset. Hive folders like region=west/ become columns. Marker files (_SUCCESS, .crc, _temporary/) are skipped.
• Formats: .parquet .pq .csv .tsv .txt .json .ndjson .jsonl, one format per upload, uncompressed.
• --overwrite (overwrite=True) replaces an existing dataset; without it, an existing name is refused. Appends are not supported.

From a DataFrame

pip install "daggydata[pandas]"   # adds pyarrow; polars needs nothing extra

ds = daggy.upload("sales", df)   # pandas or polars DataFrame, or a pyarrow Table

Any object with __arrow_c_stream__ (DuckDB relations, ibis tables, ...) works too, with pyarrow installed. The table is written to a temporary Parquet file (it needs free space in $TMPDIR), uploaded, and deleted. A pandas index is kept as a column only when every level is named (df.set_index("date") keeps date; the leftover integer index of a filtered frame is dropped).

A table can't be resumed. If an upload fails, call daggy.upload again, with overwrite=True if the dataset was already created.

Resuming

The session id is printed (logged, from Python) before any data is sent. If a run is killed or a part keeps failing, run it again with the id; only the missing parts are sent:

daggy upload sales ./sales.parquet --resume <session_id>

try:
    daggy.upload("sales", "./sales.parquet")
except daggy.UploadError as e:
    print(e.code, e.session_id, e.final_state)
    # final_state None or "uploading": daggy.upload(..., resume=e.session_id)

UploadError has status, code, message, session_id and final_state (None for a refused request or a failed part, otherwise failed, aborted or uploading). From Python, progress goes to the daggy logger: logging.basicConfig(level=logging.INFO) shows it.

Options

CLI exit codes: 0 done; 1 the upload ended failed/aborted or needs --resume; 2 bad usage, a refused request or a part that kept failing; 130 interrupted.

License

MIT