daggydata

Upload datasets to Daggy from a script, a notebook or cron, with an API key: one file, several files, or a hive-partitioned folder, up to 25 GB. Bytes go straight to storage in parallel parts; only requests is needed, no cloud credentials.

pip install daggydata

The package installs the daggy command and imports as daggy.

Create an API key on the API keys page of the Daggy app, then:

export DAGGY_API_KEY=daggy_prod_...
daggy upload sales ./sales.parquet
daggy upload events ./events/ --overwrite --description "Clickstream"

import daggy  # reads DAGGY_API_KEY

ds = daggy.upload("sales", "./sales.parquet")
print(ds.name, ds.num_obs, "rows")

What gets uploaded

• A file is uploaded under its name, with anything outside [A-Za-z0-9_.-] replaced (Q3 sales.csv → Q3_sales.csv).
• A folder becomes one dataset. Hive folders like region=west/ become columns. Marker files (_SUCCESS, .crc, _temporary/) are skipped.
• Formats: .parquet .pq .csv .tsv .txt .json .ndjson .jsonl, one format per upload, uncompressed.
• --overwrite (overwrite=True) replaces an existing dataset; without it, an existing name is refused. Appends are not supported.

Resuming

The session id is printed (logged, from Python) before any data is sent. If a run is killed or a part keeps failing, run it again with the id; only the missing parts are sent:

daggy upload sales ./sales.parquet --resume <session_id>

try:
    daggy.upload("sales", "./sales.parquet")
except daggy.UploadError as e:
    print(e.code, e.session_id, e.final_state)
    # final_state None or "uploading": daggy.upload(..., resume=e.session_id)

UploadError has status, code, message, session_id and final_state (None for a refused request or a failed part, otherwise failed, aborted or uploading). From Python, progress goes to the daggy logger: logging.basicConfig(level=logging.INFO) shows it.

Options

CLI exit codes: 0 done; 1 the upload ended failed/aborted or needs --resume; 2 bad usage, a refused request or a part that kept failing; 130 interrupted.

License

MIT