s3-pypi-server 0.3.1

Deploy and manage a private PyPI server on AWS using S3, API Gateway, CloudFront, and Lambda. Supports PEP 503 package hosting, twine uploads, LDAP/AD and API key authentication, KMS encryption, and VPC placement — all from a single CLI.

s3-pypi-server

A private PyPI server backed by AWS S3, API Gateway, and CloudFront. Upload Python packages to S3 and install them with pip using a PEP 503 compliant simple repository interface. Supports optional KMS encryption, API Gateway authorization (LDAP/AD + API keys), and VPC placement for Lambda functions.

Architecture

pip install ──▶ CloudFront ──▶ API Gateway ──▶ S3 Bucket
                 (cache)        (REST API)     (storage)
                                    │
                              ┌─────┴─────┐
                              │ Authorizer │ (optional)
                              │  Lambda    │
                              └─────┬─────┘
                                    │
                         ┌──────────┼──────────┐
                         ▼                     ▼
                    DynamoDB              Secrets Manager
                   (API keys)           (LDAP config)

• S3 stores distribution files under packages/{name}/ and HTML index pages under simple/.
• API Gateway maps /simple/ URL paths to S3 objects, serving HTML indexes and binary downloads.
• CloudFront caches responses and provides HTTPS with TLS 1.2+.
• Lambda Authorizer (optional) validates Bearer tokens against DynamoDB or Basic Auth against LDAP/AD.
• CLI uploads packages to S3, manages API keys, and configures LDAP secrets.

See docs/design.md for detailed architecture documentation.

Quickstart

1. Deploy the infrastructure

s3pypi deploy --stack-name my-pypi

This deploys the CloudFormation stack (S3 bucket, API Gateway, CloudFront distribution) and automatically saves the stack outputs to your CLI config. The bucket and table names are randomly generated by CloudFormation.

2. Upload a package

# Build your package
python -m build

# Save your defaults (one-time setup)
s3pypi configure --bucket <bucket-name> --cloudfront-distribution-id <distribution-id>

# Upload to your private PyPI
s3pypi upload dist/my_package-1.0.0-py3-none-any.whl

3. Install from your private PyPI

pip install my-package --index-url https://<cloudfront-domain>/simple/

Deploy with security features

s3pypi deploy --stack-name my-pypi \
  --enable-kms-encryption true \
  --enable-authorizer true \
  --vpc-id vpc-abc123 \
  --subnet-ids subnet-111,subnet-222

After deploying with the authorizer enabled, configure LDAP and create API keys:

# Configure LDAP
s3pypi configure-ldap \
  --secret-arn <from stack output> \
  --host ldap.example.com \
  --bind-user "cn=admin,dc=example,dc=com" \
  --bind-password "secret" \
  --entitlement-group "cn=pypi-users,ou=groups,dc=example,dc=com"

# Create an API key for CI/CD
s3pypi apikey --table-name <from stack output> create --description "CI pipeline"

CLI Usage

s3pypi deploy --stack-name NAME [--profile P] [--region R] [Key=Value ...]
s3pypi configure [--bucket B] [--cloudfront-distribution-id ID] [--api-key-table-name T] [--ldap-secret-arn ARN]
s3pypi upload <dist_file> [--bucket B] [--cloudfront-distribution-id ID]
s3pypi apikey [--table-name T] <create|list|get|delete|update> [options]
s3pypi configure-ldap --host H --bind-user U --bind-password P --entitlement-group G [--write-entitlement-group WG] [--secret-arn ARN]

deploy

Deploy or update the CloudFormation stack. Outputs are automatically saved to the CLI config.

When run without --stack-name, enters interactive mode and prompts for all values (press Enter to accept template defaults):

$ s3pypi deploy
Stack name (required): my-pypi
AWS region [us-east-1]:
AWS profile []:
Stack name prefix [s3-pypi]:
Cache TTL (seconds) [300]:
Custom domain name:
ACM certificate ARN:
Enable KMS encryption (true/false) [false]: true
Enable authorizer (true/false) [false]: true
Subnet IDs (comma-separated):
VPC ID:
Deploying stack 'my-pypi' in us-east-1...

Or pass flags directly:

s3pypi deploy --stack-name my-pypi --enable-authorizer true --enable-kms-encryption true

Argument	Required	Description
--stack-name	No*	CloudFormation stack name (prompted if not provided)
--profile	No	AWS CLI / boto3 named profile
--region	No	AWS region (default: us-east-1)
--stack-name-prefix	No	Resource naming prefix (default: s3-pypi)
--cache-ttl	No	CloudFront cache TTL in seconds (default: 300)
--domain-name	No	Custom domain for CloudFront
--acm-certificate-arn	No	ACM certificate ARN for custom domain
--enable-kms-encryption	No	true or false (default: false)
--enable-authorizer	No	true or false (default: false)
--subnet-ids	No	Comma-separated subnet IDs for VPC
--vpc-id	No	VPC ID for Lambda placement

* If not provided, interactive mode prompts for all values.

configure

Save default settings so you don't need to pass flags on every command. Settings are stored in ~/.s3pypi/config.json.

When run without any flags, configure enters interactive mode and prompts for each value:

$ s3pypi configure
S3 bucket name: my-pypi-bucket
CloudFront distribution ID: E1234567890
DynamoDB API key table name: s3-pypi-api-keys
Secrets Manager LDAP secret ARN: arn:aws:secretsmanager:us-east-1:123:secret:s3-pypi-ldap-config

Current values are shown in brackets — press Enter to keep them unchanged:

$ s3pypi configure
S3 bucket name [my-pypi-bucket]:
CloudFront distribution ID [E1234567890]: E9999999999
DynamoDB API key table name [s3-pypi-api-keys]:
Secrets Manager LDAP secret ARN [arn:...]:

You can also pass flags directly to skip prompts:

s3pypi configure \
  --bucket my-pypi-bucket \
  --cloudfront-distribution-id E1234567890 \
  --api-key-table-name s3-pypi-api-keys \
  --ldap-secret-arn arn:aws:secretsmanager:us-east-1:123:secret:s3-pypi-ldap-config

upload

Argument	Required	Description
dist_file	Yes	Path to .whl or .tar.gz distribution file
--bucket	No*	S3 bucket name. Falls back to configured value.
--cloudfront-distribution-id	No	CloudFront distribution ID to invalidate.

* Required if not previously saved via s3pypi configure.

apikey

Manage API keys in the DynamoDB table. Keys use the __token__ username convention (same as PyPI).

# Create a read-only key
s3pypi apikey create --description "CI pipeline"

# Create a key with write access (for twine uploads)
s3pypi apikey create --description "Publisher" --access read/write

# List all keys
s3pypi apikey list

# Get details of a key
s3pypi apikey get <key-value>

# Update access level
s3pypi apikey update <key-value> --access read/write

# Delete a key
s3pypi apikey delete <key-value>

Flag	Required	Description
--table-name	No*	DynamoDB table name. Falls back to configured value.
--description	No	Description for create action.
--access	No	Access level: read (default) or read/write. Used with create and update.

configure-ldap

Set or update the LDAP/AD configuration in Secrets Manager.

s3pypi configure-ldap \
  --host ldap.example.com \
  --bind-user "cn=admin,dc=example,dc=com" \
  --bind-password "secret" \
  --entitlement-group "cn=pypi-readers,ou=groups,dc=example,dc=com" \
  --write-entitlement-group "cn=pypi-writers,ou=groups,dc=example,dc=com"

Flag	Required	Description
--host	Yes	LDAP/AD server hostname
--bind-user	Yes	DN or username for LDAP bind
--bind-password	Yes	Password for LDAP bind
--entitlement-group	Yes	DN of the read entitlement group
--write-entitlement-group	No	DN of the write entitlement group (for uploads)
--secret-arn	No*	Secrets Manager ARN. Falls back to configured value.

Exit codes: 0 success, 1 runtime error, 2 argument error.

CloudFormation Parameters

Parameter	Default	Description
StackNamePrefix	s3-pypi	Prefix for resource naming
CacheTTL	300	CloudFront default cache TTL in seconds
DomainName	(empty)	Optional custom domain (e.g. pypi.example.org)
AcmCertificateArn	(empty)	ACM certificate ARN for the custom domain
EnableKMSEncryption	false	Create a CMK and encrypt all resources at rest
EnableAuthorizer	false	Add Lambda authorizer to API Gateway
SubnetIds	(empty)	Comma-separated subnet IDs for Lambda VPC placement
VpcId	(empty)	VPC ID for Lambda VPC placement

Custom domain

To use a custom domain like pypi.example.org:

1. Request (or import) an ACM certificate in us-east-1 for your domain.
2. Deploy with both parameters:

./deploy.sh my-pypi DomainName=pypi.example.org AcmCertificateArn=arn:aws:acm:us-east-1:123456789012:certificate/abc-123

3. Create a CNAME or alias DNS record pointing pypi.example.org to the CloudFront distribution domain.

VPC deployment

When your LDAP/AD server is on a private network, deploy Lambdas inside the VPC:

s3pypi deploy --stack-name my-pypi \
  --enable-authorizer true \
  --vpc-id vpc-abc123 \
  --subnet-ids subnet-111,subnet-222

This creates a Security Group allowing all outbound traffic and places the authorizer Lambda in the specified subnets.

Authentication

When EnableAuthorizer=true, the API Gateway requires authentication on all routes. Authentication uses Basic Auth with two modes:

API keys (__token__ username)

For CI/CD pipelines and automated systems, use __token__ as the username and the API key as the password (same convention as PyPI):

pip install my-package \
  --index-url https://__token__:<api-key>@<endpoint>/simple/

LDAP/AD (corporate credentials)

For developers using corporate credentials:

pip install my-package \
  --index-url https://user:password@<endpoint>/simple/

Publishing with twine

To upload packages using twine, create an API key with read/write access and configure ~/.pypirc:

[distutils]
index-servers = private

[private]
repository = https://<endpoint>/simple/
username = __token__
password = <api-key-with-readwrite-access>

Then upload:

twine upload --repository private dist/*

Access levels

• read — Can install packages (GET requests only)
• read/write — Can install and upload packages (GET and POST requests)

LDAP users in entitlement_group get read access. Users in write_entitlement_group get read/write access.

Development

Setup

python -m venv .venv
source .venv/bin/activate
pip install -e ".[test]"

Running tests

pytest                                    # Full test suite
pytest --cov=s3pypi --cov-fail-under=80   # With coverage
pytest -m smoke                           # Smoke tests only

Linting

pylint s3pypi/
bandit -r s3pypi/

License

MIT — see LICENSE for details.