datus-s3-plugin

A Datus plugin for browsing and moving S3 data from datus s3 ... — ls/stat/cat/head, cp/sync/mv/rm, presigned URLs, and S3 Select (SQL over CSV/JSON/Parquet). Backed by boto3.

pip install datus-s3-plugin

Requires datus-agent >= 0.3.8 — the system-prompt template uses the config_mutable render-context variable (older versions skip the whole prompt section).

Configuration

Profiles live under agent.plugins.s3.<profile> in Datus' agent.yml:

agent:
  plugins:
    s3:
      prod:
        default: true
        region: us-east-1
        bucket: my-data-lake        # optional default bucket
        kms_key_id: arn:aws:kms:...:key/abc   # optional SSE-KMS on writes
        endpoint_url: http://minio:9000       # optional (MinIO/S3-compatible)
        # credentials: standard AWS chain, or profile / keys / role_arn

Alibaba OSS is supported through its S3-compatible endpoint with compatibility: aliyun-oss, signature_version: s3v4, and addressing_style: virtual. AWS-only S3 Select and SSE-KMS are intentionally disabled in this mode.

Commands

datus s3 ls s3://my-lake/events/ -r
datus s3 select s3://my-lake/events/day.csv --format csv --header --sql "select count(*) from s3object"
datus s3 cp ./out.parquet s3://my-lake/exports/
datus s3 rm s3://my-lake/tmp/ -r -y

Exit codes

0 success · 1 runtime/API error · 2 usage (also: rm without -y when not interactive) · 3 config error.

Development

uv run --package datus-s3-plugin pytest datus-s3-plugin

Never imports datus; declares the plugin contract in datus-plugin.yml and registers the s3 entry point in datus.plugins. Shared boto3 plumbing lives in datus-aws-common. Bundled skills: s3 and s3-setup.

Agent bash permissions

Read/list/preview/select/presign run everywhere; cp/sync/mv are routine (confirmed under normal, auto under auto); rm always requires confirmation. User rules in agent.yml always win.