agnue-push

CLI and Python SDK for uploading sensor data to the AgNUE platform.

Step-by-step guide (install, configure, first upload, scheduling with cron or Windows Task Scheduler, troubleshooting): Automatic uploads with agnue-push.

Installation

pip install agnue-push

Configuration

When your site is approved, you receive an agnue.toml file by email. Place it in the directory you run agnue from — the SDK reads ./agnue.toml, or pass --config PATH.

You can also configure via environment variables: AGNUE_PLATFORM_URL, AGNUE_CLIENT_ID, AGNUE_CLIENT_SECRET, and optionally AGNUE_KEYCLOAK_URL (defaults to <platform origin>/realms/agnue) and AGNUE_S3_*. Precedence: CLI flags / explicit args > env vars > agnue.toml.

CLI

# Upload a single file
agnue upload data.csv --data-source DK-Foo-SOIL-001

# Upload a directory
agnue upload ./sensor-exports/ --data-source DK-Foo-SOIL-001

# Upload with glob, excluding backups
agnue upload "*.csv" --data-source DK-Foo-SOIL-001 --exclude "*_backup*"

# Several patterns; a leading ! excludes (single-quote it in bash)
agnue upload "*.csv" "*.txt" '!*.locked' --data-source DK-Foo-SOIL-001

# Scheduled runs: send everything, delete what arrived, retry the rest next run
agnue upload "*.dat" '!*.locked' --data-source DK-Foo-SOIL-001 --delete-after-upload

The --data-source ID follows the format CC-SSS-DOMAIN-SEQ (e.g. DK-Foo-SOIL-001).

Compression: text files (.csv, .txt, .dat, .tsv, .json) are gzip-compressed before upload; the platform keeps the original content. Pass --no-compress (or upload(..., compress=False)) to send files as-is.

Deleting after upload: pass --delete-after-upload (or upload(..., delete_after_upload=True)) to delete each local file once the platform or S3 confirms it has it. A file that changed after it was read or can't be deleted is kept and reported, but a write landing in the instant before deletion is lost, so leave out the file the logger is still writing to ('!*.locked', or have the logger start a new file e.g. every 30 minutes and exclude the current one); a file only queued locally in agnue-staging/ is kept too. Run the same command again later and those are sent (or recognised) and deleted then.

Fallback behaviour: if the platform API is unreachable after 3 attempts, files are uploaded directly to S3 only if the resolved S3 settings are complete (from the [s3] block, AGNUE_S3_* env vars or --s3-* flags), otherwise copied to a local agnue-staging/ folder. In both cases upload() returns None.

Registering data sources (for AI agents)

agnue sources registers data sources from JSON spec files, with the site client in agnue.toml. It is built for coding agents; agnue skill --install installs the instructions for Claude Code (agnue skill prints them).

agnue sources list                                 # this site's sources and status
agnue sources template --data-type LAB_ANALYSIS > specs/nitrate.json
agnue sources validate specs/*.json                # server dry run, nothing saved
agnue sources register specs/*.json                # writes the assigned ID into each spec
agnue domains | badm-variables nitrate | locations | credentials list
agnue api GET api/v1/data-sources/domains          # any endpoint, authenticated

Output is JSON. A failed spec doesn't stop the rest; the exit code is 1 if any failed. Registration stops at DRAFT: an operator approves the schema. These commands are never loaded by agnue upload.

Python API

from agnue_push import AgNUEClient

client = AgNUEClient.from_config()  # reads ./agnue.toml or env vars
# client = AgNUEClient.from_config("/path/to/agnue.toml")

# Upload a file — returns ingestion_id, or None after the direct-S3 or local-staging fallback
ingestion_id = client.upload("data.csv", data_source_id="DK-Foo-SOIL-001")

# Re-uploading the same file is safe: the platform recognises it and returns the original ingestion_id

Pass explicit config instead of a file:

client = AgNUEClient.from_config(
    platform_url="https://nuedata.eu",
    client_id="site-DK-Foo",
    client_secret="...",
)

Changelog

Unreleased

Added

• agnue sources template | validate | register | list: register data sources from JSON spec files, meant for AI coding agents. validate is a server dry run; register uploads the sample (or stores the ZentraCloud credential), sends an optional help request, and writes the assigned ID back into the spec so re-runs are safe. Needs a platform that assigns data source IDs (domain in the create request).
• Lookups agnue domains, agnue badm-variables, agnue locations, agnue credentials list, and agnue api METHOD PATH [--data @body.json] for any other endpoint.
• agnue skill [--install] prints the agent instructions shipped in the package, or installs them to ~/.claude/skills/agnue-cli/SKILL.md.
• --delete-after-upload / upload(..., delete_after_upload=True) deletes the local file once the platform or S3 confirms it has it. Never deletes a file that was only queued locally, or one seen to change after it was read. A write in the instant before deletion is not caught, so exclude files the logger is still writing to.
• agnue upload --config PATH reads a config file other than ./agnue.toml, like the sources commands.
• agnue upload takes several paths or patterns; a pattern starting with ! excludes matching files (same as --exclude). A file matched by more than one pattern is sent once.

0.4.0

Breaking

• Removed Parquet conversion and the convert argument of upload(). It guessed each file's layout and could change values (headerless files lost their first row, some files failed to upload). Files are now sent exactly as they are. --no-convert and pip install 'agnue-push[convert]' still work but do nothing.
• Requires a platform that accepts compressed uploads (rolled out before this release).

Added

• Text files (.csv, .txt, .dat, .tsv, .json) are gzip-compressed before upload, typically 4–25× smaller. The platform stores the original content. Turn it off with --no-compress or upload(..., compress=False).

0.3.0

Breaking

• Removed AgNUEClient.verify_upload() and AgNUEClient.list_uploads(). They could only see files sent through the direct-S3 fallback, not files sent through the platform. To check whether a file arrived, upload it again: the platform recognises it and returns the original ingestion ID.