HEA Server AWS Accounts Microservice

Comprehensive Oncology Data and Engineering Shared Resource (CODE), Huntsman Cancer Institute, Salt Lake City, UT

The HEA Server AWS Accounts Microservice is It manages the AWS account information details.

Version 1.20.0

• Bumped heaserver version to 1.71.1.
• /accounts and /accounts/{id} now reach out to other microservices that are registered as resources to fetch account subclasses.
• Restored returning projects in search results as projects. Requires heaserver-folders-aws-s3 version 1.24.0 or greater for its /volumes/{volume_id}/awss3projects/filter endpoint.

Version 1.19.1

• Use time.monotonic for determining how long something ran.
• Bumped heaserver version to 1.64.0.

Version 1.19.0

• Removed the /awsaccounts/{id}/search/{id} endpoint.
• Cleaned up the search code.

Version 1.18.17

• More improved exception handling in heaserver.account.service.search_awsaccounts.

Version 1.18.16

• Improved exception handling in heaserver.account.service.search_awsaccounts.

Version 1.18.15

• Ensure context_dependent_object_path is populated in the search results.

Version 1.18.14

• Ensure id_in_index is set when receiving s3 events to insert into the OpenSearch index.

Version 1.18.13

• Bumped heaserver version to 1.62.9.
• Update existing index records again.

Version 1.18.12

• Bumped heaserver version to 1.62.7.
• Refresh the OpenSearch index after an insert.

Version 1.18.11

• Bumped heaserver version to 1.62.6.

Version 1.18.10

• Bumped heaserver version to 1.62.5.

Version 1.18.9

• Added logging.

Version 1.18.8

• Bumped heaserver version to 1.62.4.
• Prevent insertion of duplicate search entries.

Version 1.18.7

• Replaced incorrect self link for the search results endpoint with hea-actual.

Version 1.18.6

• Bumped heaserver version to 1.62.3.

Version 1.18.5

• Bumped heaserver version to 1.62.2.

Version 1.18.4

• Exit s3event processing when there are no more messages to process.

Version 1.18.3

• Bumped heaserver version to 1.61.3.
• Fixed volume_id not defined error.

Version 1.18.2

• Fixed sporadic tuple unpacking error.

Version 1.18.1

• Bumped heaserver version to 1.61.1.

Version 1.18.0

• Bumped heaserver version to 1.61.0.
• Use the /peopleviews and /groupviews endpoints for dropdowns.

Version 1.17.4

• Improved logging when there is an error in the internal/s3event endpoint.

Version 1.17.3

• Bumped heaserver version to 1.59.0.

Version 1.17.2

• Bumped heaserver version to 1.58.0.
• Don't treat limit query parameter as a filter parameter.

Version 1.17.1

• Don't treat return_attr query parameter as a filter parameter.

Version 1.17.0

• Bumped heaserver version to 1.55.0.
• Sorting account views now works.
• Dropdowns are now sorted.
• Request only required attributes for form template dropdowns.

Version 1.16.0

• Changed AWS account display name to AWS {account number}.
• Bumped heaserver version to 1.51.0.

Version 1.15.1

• Bumped heaserver version to 1.50.0.

Version 1.15.0

• Bumped heaserver version to 1.49.0.
• Allow filtering account view queries by their attributes.

Version 1.14.0

• Bumped heaserver version to 1.48.0.
• Added support for encrypted RabbitMQ password.

Version 1.13.1

• Bumped heaserver version to 1.47.1 for a bug fix.

Version 1.13.0

• Bumped heaserver version to 1.45.1.
• New startup.py module to initialize logging before importing any third-party libraries.
• New optional -e/--env command-line argument for setting the runtime environment (development, staging, or production). Logging is now configured based on this setting, unless the older -l/--logging argument is provided. The default value is development.
• Logs are now scrubbed.
• Added log files to .gitignore.

Version 1.12.7

• Catching all s3 events in else that aren't permanent delete or delete marker created so they recorded in opensearch.

Version 1.12.6

• Bug fix in logic for crud operations in opensearch based off of s3 events.

Version 1.12.5

• Bumped heaserver version to 1.43.1 to address fixes for search.

Version 1.12.4

• Document which REST APIs require an Authorization header.
• Bumped heaserver version to 1.43.0.

Version 1.12.3

• Bumped heaserver version to 1.41.2 to address issue getting AWS account-level collaborator info.

Version 1.12.2

• Makes search more reliable and now enforcing the current version to be the most recent event from sqs.

Version 1.12.1

• Write a type property for AWS account collaborators in the AWS account properties template.

Version 1.12.0

• Bumped heaserver version to 1.41.1.
• Added the new collaborator info to the CJ templates.
• Silenced repeating log message when OpenSearch is not configured.

Version 1.11.1

• Changed the activity code OpenSearch index updates to hea-update-part to reflect that it's part of an object update, and it's a lot of detail that is probably not very interesting to users.
• Bumped heaserver version to 1.37.2 and heaobject version to 1.28.1.

Version 1.11.0

• Accounts now sends events to AWS Folders to allow for handling of those delete events to update opensearch.

Version 1.10.3

• Ask AWS for permission to create buckets when deciding whether or not to offer bucket creation in an account.

Version 1.10.2

• Give all users VIEWER permissions to accounts because there's no way to create nor edit nor delete them yet.

Version 1.10.1

• Bumped heaserver version to 1.32.0.
• Added instructions and require cb-* name when creating a new bucket (cb-* name is not enforced server-side, it's just passed to the client to enforce).

Version 1.10.0

• We now mark account objects with the new hea-container, hea-self-container, and hea-actual-container rel values.

Version 1.9.4

• Bumped heaserver version to 1.30.1 to address a potential issue with erroneously assigning CREATOR permissions.

Version 1.9.3

• Bumped heaserver version to 1.30.0.
• Gracefully handle when S3 event listening is not configured.

Version 1.9.2

• Fixed context_dependent_object_path for paths with projects and optimized speed for generating paths.
• Search now filters out aws search items with delete markers

Version 1.9.0

• Added support for group permissions.

Version 1.8.5

• Fixed search error when bucket_items called not having authorization header.
• Errors are now logged for sqs message ingestion.

Version 1.8.4

• Fix for handling standard s3 event notification messages and ones that pass through sns.

Version 1.8.3

• Removed search item from context menu.

Version 1.8.2

• Upgraded heaserver dependency for bug fix getting temporary AWS credentials.

Version 1.8.1

• Upgrading HEA Server dependency for bug fixes in Search.

Version 1.8.0

• Added context path for all search items for navigating functionality.

Version 1.7.3

• Minor bug fix for handling None with 'in' Operator.

Version 1.7.1

• Background task now sends fewer sqs request due to doing bulk delete.
• Better exception handling, aborting background task if issues with message ingestion.

Version 1.7.0

• Adds the scheduled background task for pulling from our sqs queue and ingesting into opensearch
• Improves search endpoint to include permissions and includes values actual_object_*

Version 1.6.0

• Improved error message when call to create a new bucket fails.
• Repointed action to calculate usage to volumes/{volume_id}/awss3storage. Requires heaserver-folders-aws-s3 version 1.9.0 or greater.

Version 1.5.1

• Performance improvements.

Version 1.5.0

• Added support for Python 3.12.

Version 1.4.0

• Removed integration tests because they all overlapped with the unit tests.
• Accept the data query parameter for get requests for a speed boost.

Version 1.3.9

• Dependency upgrades for compatibility with heaserver-keychain 1.5.0.

Version 1.3.8

• Fixed another /accounts regression when no account ids are passed as query parameters.

Version 1.3.7

• Stop returning all accounts with an accessible volume; only return the requested accounts.

Version 1.3.6

• When retrieving /accounts, skip volumes with a null account_id. Fixes crash retrieving accounts.

Version 1.3.5

• Pull /accounts from volume data rather than trying to get account info from AWS. Helps with performance for users who have access to lots of accounts.

Version 1.3.4

• Caching optimizations.

Version 1.3.3

• Fixed new bucket link.

Version 1.3.2

• Fixed a caching issue when listing accounts.
• Additional input validation.

Version 1.3.1

• Avoid timeouts loading accounts, which sporadically caused accounts not to be returned.

Version 1.3.0

• Present accurate account permissions.
• Added caching for getting accounts list.

Version 1.2.0

• New /accounts endpoint that returns heaobject.account.AccountView objects.

Version 1.1.3

• Another attempt at fixing the crash regression.

Version 1.1.2

• Fixed crash regression.

Version 1.1.1

• Fixed new bucket form submission.

Version 1.1.0

• Removed the PUT and DELETE account calls because neither works.

Version 1.0.8

• Addressed occasional slowdown getting one account.

Version 1.0.7

• Addressed occasional slowdown getting accounts.

Version 1.0.6

• Display type display name in properties card, and return the type display name from GET calls.

Version 1.0.5

• Improved performance getting accounts.

Version 1.0.4

• Corrected caching issue.

Version 1.0.3

• Improved error handling when the user lacks authorization for some AWS account information.

Version 1.0.2

• Avoid a 500 error when retrieving accounts when attempting to access a suspended account.
• AWS account info requests return more complete information.
• Omit shares from the properties template.

Version 1.0.1

• Improved performance.

Version 1

Initial release.

Runtime requirements

• Python 3.10, 3.11, or 3.12.

Account builder Lambda

lambdas/AccountBuilderLambda.py is an AWS Lambda function for bootstrapping an AWS Organizations member account and its HEA S3 storage. It can create an organization member account, or configure an existing member account, and then:

• configures the member account's OIDC provider and the aws-Manager.Role and aws-Member.Role roles;
• creates or updates an S3 bucket with encryption, public-access blocking, versioning, inventory, lifecycle rules, and object-change notifications;
• sends object notifications directly to an SQS queue in us-east-1, or through an SNS topic in other regions; and
• creates EventBridge rules in the four supported regions (us-east-1, us-east-2, us-west-1, and us-west-2) that forward S3 bucket creation and deletion events to an event bus in the organization management account.

This is an administrative bootstrap function, not part of the accounts microservice process. Deploy it in the AWS Organizations management account. The function assumes OrganizationAccountAccessRole in each member account and uses the Lambda execution role's temporary credentials. Do not configure AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, or AWS_SESSION_TOKEN yourself; Lambda supplies these reserved environment variables.

Prerequisites

Before installing the function:

1. Create the destination S3 inventory bucket in the management account.
2. Create the two customer-managed policies named aws-Manager.RolePolicy and aws-Member.RolePolicy in the management account. The Lambda copies their current default policy documents into each member account.
3. Create an SQS queue in us-east-1. For buckets outside us-east-1, also create an SNS topic in the bucket's region and arrange for that topic to deliver to the SQS queue used by HEA's S3-event ingestion.
4. Create the destination EventBridge custom event bus in us-east-1 in the management account. Add an event-bus resource policy that accepts events from the member accounts; the Lambda creates an EventBridgeToTargetBusRole identity policy in each member account. Enable CloudTrail management events for S3 if the EventBridge rules are expected to receive CreateBucket and DeleteBucket API calls.
5. For an existing member account, ensure that OrganizationAccountAccessRole exists, trusts the management account, and has enough permissions to administer IAM, S3, and EventBridge. AWS Organizations creates this role automatically when this Lambda creates a new account. Service control policies can still restrict what the role can do.

The Lambda execution role needs the standard CloudWatch Logs permissions plus permission to perform the following management-account operations. Restrict the resource ARNs where the AWS API supports it:

organizations:CreateAccount
organizations:DescribeAccount
organizations:DescribeCreateAccountStatus
sts:GetCallerIdentity
sts:AssumeRole                           on arn:aws:iam::*:role/OrganizationAccountAccessRole
iam:GetPolicy
iam:GetPolicyVersion                    on the two source managed policies
s3:GetBucketPolicy
s3:PutBucketPolicy                      on the inventory bucket
sqs:GetQueueAttributes
sqs:SetQueueAttributes                  on the us-east-1 event queue
sns:GetTopicAttributes
sns:SetTopicAttributes                  on the regional event topics

AWS documents the special considerations and minimum permission for creating an Organizations member account.

Build and install

The source imports jwt, which is supplied by the PyJWT distribution and is not included in the Lambda Python runtime. Build a zip archive with the dependency and put both the dependency and AccountBuilderLambda.py at the root of the archive, as required for a Python Lambda zip deployment:

Linux:

mkdir -p build/account-builder-lambda
python -m pip install --target build/account-builder-lambda PyJWT
cp lambdas/AccountBuilderLambda.py build/account-builder-lambda/
(cd build/account-builder-lambda && zip -r ../account-builder-lambda.zip .)

Windows PowerShell:

New-Item -ItemType Directory -Force build\account-builder-lambda
py -m pip install --target build\account-builder-lambda PyJWT
Copy-Item lambdas\AccountBuilderLambda.py build\account-builder-lambda\
Compress-Archive -Path build\account-builder-lambda\* `
    -DestinationPath build\account-builder-lambda.zip -Force

In IAM, create an execution role whose trust policy allows lambda.amazonaws.com to assume it. Attach AWSLambdaBasicExecutionRole for logging and a customer-managed policy containing the permissions listed above. Then create the function in us-east-1, replacing the example account ID and role name. The function constructs its SQS queue URL and destination EventBridge ARN in us-east-1:

Linux:

aws lambda create-function \
    --region us-east-1 \
    --function-name AccountBuilderLambda \
    --runtime python3.12 \
    --architectures x86_64 \
    --handler AccountBuilderLambda.lambda_handler \
    --role arn:aws:iam::111122223333:role/hea-account-builder-lambda \
    --zip-file fileb://build/account-builder-lambda.zip \
    --timeout 900 \
    --memory-size 256

Windows PowerShell:

aws lambda create-function `
    --region us-east-1 `
    --function-name AccountBuilderLambda `
    --runtime python3.12 `
    --architectures x86_64 `
    --handler AccountBuilderLambda.lambda_handler `
    --role arn:aws:iam::111122223333:role/hea-account-builder-lambda `
    --zip-file fileb://build/account-builder-lambda.zip `
    --timeout 900 `
    --memory-size 256

The 15-minute timeout accommodates the polling performed while AWS Organizations creates an account and while the new account's S3 service and OrganizationAccountAccessRole become available. If the function already exists, deploy a new archive with:

Linux:

aws lambda update-function-code \
    --region us-east-1 \
    --function-name AccountBuilderLambda \
    --zip-file fileb://build/account-builder-lambda.zip

Windows PowerShell:

aws lambda update-function-code `
    --region us-east-1 `
    --function-name AccountBuilderLambda `
    --zip-file fileb://build/account-builder-lambda.zip

The equivalent console installation is: create a Python 3.12 function from scratch, select the execution role described above, upload build/account-builder-lambda.zip, set the handler to AccountBuilderLambda.lambda_handler, and set the timeout to 15 minutes. See AWS's Python handler naming rules for why the module and function names form that handler value.

Invoke the function

Invoke the function synchronously with a JSON object. These keys must always be present because the handler indexes them directly: token, accountid, accountname, bucketname, accountemail, oidcproviderurl, and oidcthumbprint. Pass an empty accountid to create a new member account; in that case accountemail must be a unique AWS account email. If accountid is populated, the existing account is configured and accountemail is ignored.

The normally required configuration keys are inventorybucket, sqsname, snsname, region, and eventbridgename. snsname is only used outside us-east-1. The orgid key is currently accepted but unused. Bucket names must comply with the AWS S3 bucket naming rules.

For example, the following event configures an existing account and creates or updates its bucket:

{
    "token": "<trusted JWT containing a sub claim>",
    "accountid": "444455556666",
    "accountname": "research-a",
    "accountemail": "",
    "bucketname": "cb-research-a-data",
    "oidcproviderurl": "id.example.org/realms/hea",
    "oidcthumbprint": "0123456789abcdef0123456789abcdef01234567",
    "mode": "create",
    "exclude": "",
    "lifecycletype": "moderate",
    "inventorybucket": "hea-inventory-reports",
    "sqsname": "hea-s3-events",
    "snsname": "",
    "region": "us-east-1",
    "eventbridgename": "hea-s3-bucket-events"
}

Save the event as account-builder-event.json and invoke it:

Linux:

aws lambda invoke \
    --region us-east-1 \
    --function-name AccountBuilderLambda \
    --cli-binary-format raw-in-base64-out \
    --payload file://account-builder-event.json \
    account-builder-response.json

Windows PowerShell:

aws lambda invoke `
    --region us-east-1 `
    --function-name AccountBuilderLambda `
    --cli-binary-format raw-in-base64-out `
    --payload file://account-builder-event.json `
    account-builder-response.json

Supported modes are create, update, delete, and delete_all. Both create and update run the same configuration path. delete removes an empty bucket; delete_all first deletes its objects and versions. Supported lifecycletype values are aggressive, moderate, and minimal; the default for a newly created bucket is aggressive. Set exclude to lifecycles to avoid applying the requested archival lifecycle to an existing bucket.

The function decodes token with signature verification disabled and uses only its sub claim as the STS role-session name. Consequently, do not expose this Lambda through a public Function URL or unauthenticated API Gateway. Grant lambda:InvokeFunction only to the trusted HEA caller that has already authenticated the token. A successful invocation returns statusCode 200 and a JSON string in body; handled errors return statusCode 500 in the same envelope.

Account creation controller

scripts/AccountCreationController.py is a local command-line batch client for the account builder Lambda. It obtains a Keycloak access token with the resource-owner password grant, uses the token to assume an organization-administrator role through sts:AssumeRoleWithWebIdentity, and invokes the AccountBuilderLambda function synchronously in us-east-1. It supports creating, updating, and deleting the accounts and buckets described by a tab-separated input file. Its info mode instead inventories buckets in the accounts under one AWS Organizations organizational unit.

The Lambda function name is currently hard-coded as AccountBuilderLambda. Install the Lambda under that name, as in the preceding instructions, or change FunctionName in the controller before running it.

Install the controller

Run the controller from a trusted administrative workstation with network access to Keycloak and the AWS APIs. It requires Python 3.10 or newer and the following packages in addition to the standard library:

Linux:

python -m venv .venv-account-controller
source .venv-account-controller/bin/activate
python -m pip install aiohttp boto3 python-dotenv PyJWT

Windows PowerShell:

py -m venv .venv-account-controller
.\.venv-account-controller\Scripts\Activate.ps1
py -m pip install aiohttp boto3 python-dotenv PyJWT

The source also imports collect_manifest_checksums from find_manifest_checksums, although it does not currently call that function. This repository does not include find_manifest_checksums.py; provide that internal module in scripts, install it from your organization's trusted package source, or remove the unused import before running the controller. Without the module, Python stops at import time.

Create scripts/.env. The controller resolves this file relative to the script, including when it is bundled as an executable:

ADMIN_ROLE=OrganizationAccountAccessRole
KEYLOAK_URL=https://id.example.org/realms/hea/protocol/openid-connect/token
ORG_ADMIN_ROLE_ARN=arn:aws:iam::111122223333:role/hea-organization-admin
CLOUD_TEMPLATE_BUCKET_NAME=hea-cloud-templates
THUMBPRINT=0123456789abcdef0123456789abcdef01234567
OIDC_PROVIDER_URL=id.example.org/realms/hea
INVENTORY_BUCKET=hea-inventory
SQS_QUEUE_NAME=hea-s3-events
SQS_TOPIC_NAME=hea-s3-events
EVENT_BRIDGE_NAME=hea-s3-bucket-events
ORG_ID=o-exampleorgid
ENVIRONMENT=dev

The variables have the following meanings:

For example, an INVENTORY_BUCKET of hea-inventory, a region of us-west-2, and an environment of dev produce hea-inventory-uswest2-dev. Production omits the final environment suffix. Create all resulting inventory buckets before using the controller.

The IAM OIDC provider and trust policy for ORG_ADMIN_ROLE_ARN must accept the Keycloak issuer, the core-browser audience, and sts:AssumeRoleWithWebIdentity. The role needs lambda:InvokeFunction on the AccountBuilderLambda function. For info mode it also needs these Organizations permissions:

organizations:ListRoots
organizations:ListOrganizationalUnitsForParent
organizations:ListAccountsForParent
organizations:DescribeAccount

Each member account's ADMIN_ROLE must trust the same web identity and allow s3:ListAllMyBuckets and s3:GetBucketLocation for info mode. Applicable IAM policies and AWS Organizations service control policies must also permit the operations.

The controller decodes the access token without locally verifying its signature, and its application-level admin-role check is currently commented out. AWS STS still evaluates the token while assuming the configured roles, so the role trust policies are the effective authorization boundary. Do not use a broadly trusted organization-administrator role.

Input files

Input files are TSV files without a header row. Every bucket name must follow the AWS S3 bucket naming rules. The columns depend on the mode:

When a row omits region, the value of --region is used; its default is us-east-1. For example, a create file could contain:

cb-research-a-data	research-a@example.org	research-a	us-east-1
cb-research-b-data	research-b@example.org	research-b	us-west-2

Alternatively, use --account_email as an email template and provide two-column rows:

cb-research-a-data	research-a
cb-research-b-data	research-b

The controller inserts --account_email_count immediately before the @ sign and increments it for each row. For example, --account_email research@example.org --account_email_count 10 generates research10@example.org, then research11@example.org.

An update or delete file could contain:

cb-research-a-data	research-a	444455556666	us-east-1

Run the controller

Username, password, output file, mode, and an input file are needed for account-changing operations. The following example creates every account in create-accounts.tsv:

Linux:

python scripts/AccountCreationController.py \
    --username '<keycloak-username>' \
    --password '<keycloak-password>' \
    --output_file created-accounts.tsv \
    --mode create \
    --account_params_file create-accounts.tsv \
    --lifecycletype moderate

Windows PowerShell:

py scripts\AccountCreationController.py `
    --username '<keycloak-username>' `
    --password '<keycloak-password>' `
    --output_file created-accounts.tsv `
    --mode create `
    --account_params_file create-accounts.tsv `
    --lifecycletype moderate

Passing a password on the command line can expose it in shell history and the local process list. Run the controller only on a controlled workstation, prevent the history entry where the shell supports that, and use a short-lived administrative credential.

Use update, delete, or delete_all with the corresponding TSV layout. delete can remove only an empty bucket. delete_all deletes all objects and versions first and therefore causes unrecoverable data loss; the controller asks for confirmation twice and displays the affected bucket names before invoking the Lambda.

To list buckets for the accounts directly beneath an organizational unit, use info. This mode does not use an account parameters file:

Linux:

python scripts/AccountCreationController.py \
    --username '<keycloak-username>' \
    --password '<keycloak-password>' \
    --output_file account-buckets.tsv \
    --mode info \
    --org_unit 'Core Browser'

Windows PowerShell:

py scripts\AccountCreationController.py `
    --username '<keycloak-username>' `
    --password '<keycloak-password>' `
    --output_file account-buckets.tsv `
    --mode info `
    --org_unit 'Core Browser'

The output is a headerless TSV file with bucket_name, account_name, account_id, and region, in that order. This is also the input layout accepted by update, delete, and delete_all. Use --version to print the controller version without performing an AWS action.

Development environment

Build requirements

• Any development environment is fine.
• On Windows, you also will need:
  • Build Tools for Visual Studio 2019, found at https://visualstudio.microsoft.com/downloads/. Select the C++ tools.
  • git, found at https://git-scm.com/download/win.
• On Mac, Xcode or the command line developer tools is required, found in the Apple Store app.
• Python 3.10, 3.11, or 3.12: Download and install it from https://www.python.org, and select the options to install for all users and add Python to your environment variables. The install for all users option will help keep you from accidentally installing packages into your Python installation's site-packages directory instead of to your virtualenv environment, described below.
• Create a virtualenv environment using the python -m venv <venv_directory> command, substituting <venv_directory> with the directory name of your virtual environment. Run source <venv_directory>/bin/activate (or <venv_directory>/Scripts/activate on Windows) to activate the virtual environment. You will need to activate the virtualenv every time before starting work, or your IDE may be able to do this for you automatically. Note that PyCharm will do this for you, but you have to create a new Terminal panel after you newly configure a project with your virtualenv.
• From the project's root directory, and using the activated virtualenv, run pip install wheel followed by pip install -r requirements_dev.txt. Do NOT run python setup.py develop. It will break your environment.

Running tests

Run tests with the pytest command from the project root directory. To improve performance, run tests in multiple processes with pytest -n auto.

Testing using Swagger

Run python ./run-swaggerui.py and open up http://locahost:8080/docs in your web browser to get a UI for making REST API calls.

Versioning

Use semantic versioning as described in https://packaging.python.org/guides/distributing-packages-using-setuptools/#choosing-a-versioning-scheme. In addition, while development is underway, the version should be the next version number suffixed by .dev.

Version tags in git

Version tags should follow the format heaserver-awsaccounts-<version>, for example, heaserver-awsaccounts-1.0.0.

Uploading to an index server

The following instructions assume separate stable and staging indexes. Numbered releases, including alphas and betas, go into the stable index. Snapshots of works in progress go into the staging index. Thus, use staging to upload numbered releases, verify the uploaded packages, and then upload to stable.

From the project's root directory:

1. For numbered releases, remove .dev from the version number in setup.py, tag it in git to indicate a release, and commit to version control. Skip this step for developer snapshot releases.
2. Run python setup.py clean --all sdist bdist_wheel to create the artifacts.
3. Run twine upload -r <repository> dist/<wheel-filename> dist/<tarball-filename> to upload to the repository. The repository name has to be defined in a twine configuration file such as $HOME/.pypirc.
4. For numbered releases, increment the version number in setup.py, append .dev to it, and commit to version control with a commit message like, "Prepare for next development iteration."