agnue-push

CLI and Python SDK for uploading sensor data to the AgNUE platform.

Step-by-step guide (install, configure, first upload, scheduling with cron or Windows Task Scheduler, troubleshooting): Automatic uploads with agnue-push.

Installation

pip install agnue-push

Configuration

When your site is approved, you receive an agnue.toml file by email. Place it in the directory you run agnue from — the SDK reads ./agnue.toml.

You can also configure via environment variables: AGNUE_PLATFORM_URL, AGNUE_CLIENT_ID, AGNUE_CLIENT_SECRET, and optionally AGNUE_KEYCLOAK_URL (defaults to <platform origin>/realms/agnue) and AGNUE_S3_*. Precedence: CLI flags / explicit args > env vars > agnue.toml.

CLI

# Upload a single file
agnue upload data.csv --data-source DK-Foo-SOIL-001

# Upload a directory
agnue upload ./sensor-exports/ --data-source DK-Foo-SOIL-001

# Upload with glob, excluding backups
agnue upload "*.csv" --data-source DK-Foo-SOIL-001 --exclude "*_backup*"

The --data-source ID follows the format CC-SSS-DOMAIN-SEQ (e.g. DK-Foo-SOIL-001).

Compression: text files (.csv, .txt, .dat, .tsv, .json) are gzip-compressed before upload; the platform keeps the original content. Pass --no-compress (or upload(..., compress=False)) to send files as-is.

Fallback behaviour: if the platform API is unreachable after 3 attempts, files are uploaded directly to S3 only if the resolved S3 settings are complete (from the [s3] block, AGNUE_S3_* env vars or --s3-* flags), otherwise copied to a local agnue-staging/ folder. In both cases upload() returns None.

Python API

from agnue_push import AgNUEClient

client = AgNUEClient.from_config()  # reads ./agnue.toml or env vars
# client = AgNUEClient.from_config("/path/to/agnue.toml")

# Upload a file — returns ingestion_id, or None after the direct-S3 or local-staging fallback
ingestion_id = client.upload("data.csv", data_source_id="DK-Foo-SOIL-001")

# Re-uploading the same file is safe: the platform recognises it and returns the original ingestion_id

Pass explicit config instead of a file:

client = AgNUEClient.from_config(
    platform_url="https://nuedata.eu",
    client_id="site-DK-Foo",
    client_secret="...",
)

Changelog

0.4.0

Breaking

• Removed Parquet conversion and the convert argument of upload(). It guessed each file's layout and could change values (headerless files lost their first row, some files failed to upload). Files are now sent exactly as they are. --no-convert and pip install 'agnue-push[convert]' still work but do nothing.
• Requires a platform that accepts compressed uploads (rolled out before this release).

Added

• Text files (.csv, .txt, .dat, .tsv, .json) are gzip-compressed before upload, typically 4–25× smaller. The platform stores the original content. Turn it off with --no-compress or upload(..., compress=False).

0.3.0

Breaking

• Removed AgNUEClient.verify_upload() and AgNUEClient.list_uploads(). They could only see files sent through the direct-S3 fallback, not files sent through the platform. To check whether a file arrived, upload it again: the platform recognises it and returns the original ingestion ID.