s3cd 0.0.1

S3 Content Delivery tool

s3cd

S3 Content Delivery tool - CLI for uploading and copying static files/artifacts to S3

Features

• 📦 Upload directories recursively to S3 buckets maintaining structure
• 🔄 Copy objects between S3 buckets without local download (server-side copy)
• 🚀 Multipart upload support for large files
• 📝 Automatic __info.json generation with GitLab CI/CD metadata
• 🔐 Flexible authentication via environment variables or CLI flags
• 🌐 Support for S3-compatible storage (MinIO, Yandex Object Storage, etc.)
• 🎯 Path-style and virtual-hosted-style S3 addressing

Quick Start

# Using uvx (no installation needed)
uvx s3cd upload my-bucket production/v1.0 ./dist

# Or using uvxt
uv tool install uvxt
uvxt s3cd upload my-bucket production/v1.0 ./dist

Installation

# Using pip
pip install s3cd

# Using uv
uv tool install s3cd

Usage

Upload Command

Upload a local directory to an S3 bucket:

s3cd upload <bucket> <target> <source_dir>

Example:

# Upload ./dist directory to s3://my-bucket/production/v1.0/
s3cd upload my-bucket production/v1.0 ./dist

# With custom S3 endpoint (e.g., MinIO)
s3cd upload my-bucket artifacts/build-123 ./build \
  --s3-endpoint https://minio.example.com \
  --s3-region us-east-1 \
  --s3-addressing-style path

# With verbose output
s3cd upload my-bucket staging/latest ./dist -v

The upload command:

• Recursively uploads all files from source_dir to s3://bucket/target/
• Preserves directory structure
• Automatically uses multipart upload for large files
• Generates __info.json with deployment metadata

Copy Command

Copy objects from one S3 location to another (server-side copy):

s3cd copy <from_bucket> <from_target> <to_bucket> <to_target>

Example:

# Copy within same bucket
s3cd copy my-bucket staging/latest my-bucket production/v1.0

# Copy between different buckets
s3cd copy source-bucket releases/v1.0 backup-bucket archives/v1.0

# With verbose output
s3cd copy my-bucket temp/build-123 my-bucket backups/build-123 -v

The copy command:

• Performs server-side copy (no local download)
• Preserves directory structure
• Copies all objects including __info.json

Configuration

Authentication

S3CD supports two ways to configure credentials (CLI flags take priority):

1. Environment Variables

export S3CD_S3_ACCESS_KEY_ID="your-access-key"
export S3CD_S3_SECRET_ACCESS_KEY="your-secret-key"
export S3CD_S3_REGION="us-east-1"
export S3CD_S3_ENDPOINT="https://s3.amazonaws.com"
export S3CD_S3_ADDRESSING_STYLE="virtual"  # or "path"

2. CLI Flags (Higher Priority)

s3cd upload my-bucket target ./source \
  --s3-access-key-id "your-access-key" \
  --s3-secret-access-key "your-secret-key" \
  --s3-region "us-east-1" \
  --s3-endpoint "https://s3.amazonaws.com" \
  --s3-addressing-style virtual

S3 Addressing Styles

• virtual (default): https://bucket.endpoint/key
• path: https://endpoint/bucket/key

Use path-style for S3-compatible services like MinIO.

__info.json Metadata

Each upload automatically generates __info.json containing:

GitLab CI/CD Variables (if available)

• CI_PROJECT_NAME, CI_PROJECT_PATH, CI_PROJECT_URL
• CI_COMMIT_SHA, CI_COMMIT_SHORT_SHA, CI_COMMIT_REF_NAME, CI_COMMIT_BRANCH
• CI_PIPELINE_ID, CI_PIPELINE_URL
• CI_JOB_ID, CI_JOB_URL
• CI_RUNNER_DESCRIPTION, CI_RUNNER_TAGS
• CI_ENVIRONMENT_NAME, CI_ENVIRONMENT_URL

Deployment Information

• DEPLOY_TIMESTAMP - UTC timestamp in ISO8601 format
• SOURCE_DIR - Local source directory path
• TARGET_PATH - Target path in S3 bucket
• S3_BUCKET - Bucket name
• S3_ENDPOINT - S3 endpoint URL
• S3_REGION - S3 region
• S3_ADDRESSING_STYLE - Addressing style used

Example __info.json:

{
  "CI_PROJECT_NAME": "my-frontend",
  "CI_PROJECT_PATH": "mygroup/my-frontend",
  "CI_COMMIT_SHA": "a1b2c3d4e5f6",
  "CI_COMMIT_SHORT_SHA": "a1b2c3d",
  "CI_PIPELINE_ID": "12345",
  "DEPLOY_TIMESTAMP": "2026-01-18T15:00:00+00:00",
  "SOURCE_DIR": "./dist",
  "TARGET_PATH": "production/v1.0",
  "S3_BUCKET": "my-bucket",
  "S3_ENDPOINT": "https://s3.amazonaws.com",
  "S3_REGION": "us-east-1",
  "S3_ADDRESSING_STYLE": "virtual"
}

GitLab CI/CD Integration

Example .gitlab-ci.yml:

deploy:
  stage: deploy
  image: python:3.11
  script:
    - pip install s3cd
    # Upload build artifacts
    - s3cd upload my-bucket "production/${CI_COMMIT_SHORT_SHA}" ./dist
  environment:
    name: production
  only:
    - main

backup:
  stage: backup
  image: python:3.11
  script:
    - pip install s3cd
    # Create backup copy
    - s3cd copy my-bucket "production/latest" my-bucket "backups/$(date +%Y%m%d)"
  when: manual

Use Cases

1. Frontend Deployment: Upload built static files to S3 for CDN distribution
2. Artifact Archival: Store build artifacts with versioning
3. Environment Promotion: Copy artifacts between staging and production
4. Backup & Restore: Create backup copies of important S3 objects
5. Multi-region Distribution: Copy objects to different regional buckets

Development

Prerequisites

# Clone the repository
git clone https://gitlab.com/rocshers/python/s3cd.git
cd s3cd

# Install development dependencies
make install

Testing

# Format code
make format

# Run tests
make test

Contributing

Issue Tracker: https://gitlab.com/rocshers/python/s3cd/-/issues
Source Code: https://gitlab.com/rocshers/python/s3cd

License

MIT License - see LICENSE file for details

Author

Aleksei Marusich [e-mail]