S3 Storage PaaS Plugin for MetaPaaS

A production-ready S3-compatible object storage service (RustFS + MinIO) packaged as a MetaPaaS plugin.

What is This?

This repository provides the s3aas plugin: a complete, installable PaaS service for Exordos MetaPaaS. Instead of deploying separate control-plane infrastructure for S3 storage, this plugin integrates with a shared MetaPaaS runtime.

Key components:

• CP (Control Plane): REST API for managing S3 instances, buckets, users, policies, access keys (Python)
• DP (Data Plane): RustFS-based S3-compatible object storage VM (Packer-built)
• Integration: Automatic installation + scaling via MetaPaaS orchestration

Quick Start

Prerequisites

• Running exordos_core deployment
• Running exordos_metapaas deployment (shares one control-plane)
• Local build tools: exordos CLI, packer, Python 3.10+

Build

make build \
  REPOSITORY=http://10.20.0.1:8080/exordos-elements \
  INDEX_URL=http://10.20.0.1:8080/simple/

Produces:

• output/dist/exordos_paas_s3-*.whl — Control-plane package
• output/exordos-metapaas-s3-dp.raw.zst — Data-plane VM image
• output/manifests/s3aas.yaml — Element manifest for Exordos Core

Publish to Index

make publish-wheel
# Copies .whl to /srv/exordos-local-repo/simple/

Install to Exordos Core

exordos -e http://10.20.0.2:11010 \
  -u admin -p <admin-password> \
  ee install metapaas --version 0.0.7 --repository http://10.20.0.1:8080/exordos-elements

exordos -e http://10.20.0.2:11010 \
  -u admin -p <admin-password> \
  ee install s3aas --version 0.0.1 --repository http://10.20.0.1:8080/exordos-elements

Wait for both elements to become ACTIVE.

Create an S3 Instance

curl -X POST http://metapaas-cp:8080/v1/types/s3/instances \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer <token>' \
  -d '{
    "name": "s3-data",
    "version": "0.0.1",
    "bucket_name": "data-bucket",
    "encryption": "aes256",
    "versioning": true
  }'

Response (example):

{
  "id": "s3-instance-uuid",
  "name": "s3-data",
  "status": "CREATING",
  "bucket_name": "data-bucket",
  "encryption": "aes256",
  "versioning": true,
  "nodes": [
    {
      "id": "node-uuid",
      "ip": "10.20.0.21",
      "port": 9000
    }
  ]
}

Wait for status to become ACTIVE, then access via boto3:

import boto3

s3 = boto3.client(
    "s3",
    endpoint_url="http://10.20.0.21:9000",
    aws_access_key_id="<access-key>",
    aws_secret_access_key="<secret-key>",
    region_name="us-east-1",
)

s3.put_object(Bucket="data-bucket", Key="test.txt", Body=b"hello")
obj = s3.get_object(Bucket="data-bucket", Key="test.txt")
print(obj["Body"].read())  # b'hello'

API Reference

Instances

Instance Fields

Buckets

Users

Access Keys

Testing

Unit Tests

make test
# or: tox -e py312

Lint & Type Checking

make lint      # ruff check
make format    # ruff format
make typecheck # mypy

Functional Tests

Requires live exordos_core + exordos_metapaas deployment:

EXORDOS_ENDPOINT=http://10.20.0.2:80/api/core \
EXORDOS_USERNAME=admin \
EXORDOS_PASSWORD=<pass> \
EXORDOS_S3_CP_URL=http://10.20.0.2:80/api/metapaas \
make functional

To build that environment from scratch — what the build workflow does on a throwaway runner:

exordos compute hypervisors init --connection-uri qemu+tcp://10.20.0.1/system
exordos bootstrap -i latest -f -m core --pool-agent-placement local \
  --admin-password <admin-pass> --cidr 10.20.0.0/22
exordos e e install metapaas
exordos e e install s3aas          # add -v <version> to pin a build

Wait for both elements to report ACTIVE (exordos ee l -o json -f name=s3aas) and the suite has everything it needs. To test a build of this working tree rather than the published element, make build it and exordos push it to a repository the core can reach first.

Architecture

Control Plane (exordos_paas_s3/)

• models.py — SQLAlchemy: S3Instance, Bucket, User, Policy, AccessKey, Version
• controllers.py — REST endpoints with field-level IAM permissions
• iam_config.py — IAM roles: owner, operator, viewer
• migrations/ — Database schema migrations (per-paas independent)
• tests/ — Unit + functional test suites

Data Plane (exordos/s3-dp/)

• packer.pkr.hcl — Builds RustFS-based S3 VM image
• conf/ — Configuration templates (rendered per-instance)
• systemd/ — Service units (rustfs, minio-gateway, etc.)

Manifest & Build

• exordos/exordos.yaml — Build config (orchestrates CP wheel + DP image + manifest)
• exordos/manifests/s3aas.yaml.j2 — Jinja2 element manifest (deployed to exordos_core)

CI/CD

• .github/workflows/tests.yaml — Unit tests on Python 3.10/3.12/3.13
• .github/workflows/func_tests.yaml — Full E2E: bootstrap core → build → install → test

Repository Structure

.
├── exordos_paas_s3/              # Control-plane Python package
│   ├── models.py                 # SQLAlchemy ORM models
│   ├── controllers.py            # REST API controllers
│   ├── iam_config.py             # IAM setup
│   ├── migrations/               # Database migrations
│   └── tests/
│       ├── test_*.py             # Unit tests
│       └── functional/           # E2E tests
├── exordos/
│   ├── exordos.yaml              # Build config
│   ├── manifests/
│   │   └── s3aas.yaml.j2         # Element manifest template
│   └── s3-dp/
│       └── packer.pkr.hcl        # Data-plane image config
├── pyproject.toml                # Project metadata + dependencies
├── tox.ini                       # Test automation
├── Makefile                      # Build targets
├── .github/workflows/            # CI/CD pipelines
├── README.md                     # This file
└── output/                       # Build artifacts (git-ignored)
    ├── dist/                     # Python wheel
    ├── manifests/                # Rendered manifests
    └── images/                   # Packer-built images

Configuration

Build Manifest Variables

Control how the element image and manifest are rendered:

exordos build \
  --manifest-var repository=http://custom-repo.local/exordos-elements \
  --manifest-var index_url=http://custom-repo.local/simple/

These override defaults in exordos/exordos.yaml.

Instance Configuration

When creating an S3 instance, specify:

{
  "name": "my-s3",
  "version": "0.0.1",
  "bucket_name": "default-bucket",
  "encryption": "aes256",
  "versioning": true,
  "mfa_delete": false
}

Configuration is applied to all data-plane nodes in the instance.

Distributed Instances

kind is single_node by default. A distributed instance runs one RustFS cluster over nodes_number nodes (4 to 16, one data disk each):

{
  "name": "my-s3-cluster",
  "kind": "distributed",
  "nodes_number": 4,
  "parity": 1,
  "cpu": 2,
  "ram": 4096,
  "disk_size": 100,
  "version": "/v1/types/s3/versions/<uuid>"
}

• parity is the erasure-coding parity (RUSTFS_STORAGE_CLASS_STANDARD=EC:<parity>), at most nodes_number / 2. Leave it out to keep the RustFS default for the set size (2 for 4–5 nodes, 3 for 6–7, 4 for 8 and more).
• kind, nodes_number and parity are set on create only; cpu and ram of a distributed instance can't be changed either. disk_size can grow.
• Every node serves the whole S3 API; ipsv4 lists them in endpoint order.
• IAM and bucket changes are applied by the first node (the first address in ipsv4) and stall while that node is down; data keeps being served by the rest.

See docs/clustering.md for how a cluster is laid out, what its status means and how it is operated.

See docs/monitoring.md for the metrics the nodes send to the observability element and how to query how full the disks are.

Troubleshooting

A recursive listing stops early

RustFS before 1.0.1-preview.11 (seen on 1.0.0-beta.4, 1.0.0 and 1.0.1-preview.10) ends a flat (recursive, no delimiter) ListObjectsV2 early and still answers IsTruncated=false when two directories share a name prefix, such as a DBaaS backup directory <id>/ next to <id>-rollbacks/: in keys - sorts before /. On the stand a bucket of 104,137 objects listed as 5,000; two such directories of 1,200 keys each list as 2,000. Upgrading the element to a version on RustFS 1.0.1-preview.11 or later fixes it; until then, list with the / delimiter, directory by directory. test_listing.py checks it.

Instance stuck in CREATING

Check PluginReconciler logs on metapaas-cp:

exordos -e http://10.20.0.2:11010 -u admin -p <pass> \
  cn exec metapaas-cp -- \
  journalctl -u metapaas-plugin-reconciler -f

Also check orchestration logs:

exordos -e http://10.20.0.2:11010 -u admin -p <pass> \
  cn exec metapaas-cp -- \
  tail -f /var/log/orch-api.log

Cannot connect to S3 API

Verify data-plane node is ACTIVE and reachable:

# Get node IP from instance
curl -s http://metapaas-cp:8080/v1/types/s3/instances/<id> \
  -H 'Authorization: Bearer <token>' | jq '.nodes[].ip'

# Test connectivity
telnet <ip> 9000

# Check S3 health
curl -s http://<ip>:9000/health

Permissions error (403 Forbidden)

Ensure test user has owner role in METAPAAS_PROJECT_ID:

# Query user's roles
curl -s http://10.20.0.2:11010/v1/iam/projects/4d657461-0000-0000-0000-000000000002/roles \
  -u admin:<pass> | jq '.items[] | select(.user_id=="<your-uuid>")'

Development Guidelines

Adding New Endpoints

1. Add model field to models.py
2. Add controller method + permissions to controllers.py
3. Add migration in migrations/ with UUID name
4. Test in tests/functional/

Read-Only Fields

Fields like status, id, created_at, nodes, kind are read-only and auto-populated. Omit them from CREATE payloads.

IAM Model

Permissions are bound to METAPAAS_PROJECT_ID, not instance project. Users must have explicit role grant in metapaas project to access any S3 instance.

References

• MetaPaaS Platform: ../exordos_metapaas/
• How to Build New PaaS: ../exordos_metapaas/HOW_TO_BUILD_NEW_PAAS.md
• Mail-aas Blueprint: ../exordos_mail/ (reference plugin, simpler than s3aas)
• Exordos Core: https://github.com/exordos/exordos_core

License

Proprietary — Exordos