compose-runner

Python package to execute meta-analyses created using neurosynth compose and NiMARE as the meta-analysis execution engine.

AWS Deployment

This repository includes an AWS CDK application that turns compose-runner into a serverless batch pipeline using Step Functions, AWS Lambda, and ECS Fargate. The deployed architecture works like this:

ComposeRunnerSubmit (Lambda Function URL) accepts HTTP requests, validates the meta-analysis payload, and starts a Step Functions execution. The response is immediate and returns both a durable job_id (the execution ARN) and the artifact_prefix used for S3 and log correlation.
A Standard state machine runs a single Fargate task (compose_runner.ecs_task) and waits for completion. The container downloads inputs, executes the meta-analysis on up to 4 vCPU / 30 GiB of memory, uploads artifacts to S3, and writes metadata.json into the same prefix.
ComposeRunnerStatus (Lambda Function URL) wraps DescribeExecution, merges metadata from S3, and exposes a simple status endpoint suitable for polling.
ComposeRunnerLogPoller streams the ECS CloudWatch Logs for a given artifact_prefix, while ComposeRunnerResultsFetcher returns presigned URLs for stored artifacts.

Create a virtual environment and install the CDK dependencies:

cd infra/cdk
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

(One-time per account/region) bootstrap the CDK environment:
```
cdk bootstrap
```
Deploy the stack (supplying the compose-runner version you want baked into the images):
```
cdk deploy \
  -c composeRunnerVersion=$(hatch version) \
  -c resultsPrefix=compose-runner/results \
  -c taskCpu=4096 \
  -c taskMemoryMiB=30720
```
Pass -c resultsBucketName=<bucket> to use an existing S3 bucket, or omit it to let the stack create and retain a dedicated bucket. Additional knobs:

-c stateMachineTimeoutSeconds=32400 to control the max wall clock per run
-c submitTimeoutSeconds / -c statusTimeoutSeconds / -c pollTimeoutSeconds to tune Lambda timeouts
-c taskEphemeralStorageGiB if the default 21 GiB scratch volume is insufficient

The deployment builds both the Lambda image (aws_lambda/Dockerfile) and the Fargate task image (Dockerfile), provisions the Step Functions state machine, and configures a public VPC so each task has outbound internet access. The CloudFormation outputs list the HTTPS endpoints for submission, status, logs, and artifact retrieval, alongside the Step Functions ARN.

Metadata

Release files for compose-runner 0.14.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for compose-runner 0.14.3
File	Size	Uploaded
compose_runner-0.14.3.tar.gz	51.6 MB	Sep 2, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for compose-runner 0.14.3
File	Interpreter	ABI	Platform	Reset
compose_runner-0.14.3-py2.py3-none-any.whl 51.7 MB Sep 2, 2026	Python 2, Python 3	none	any	Details

Total release size: 103.2 MB

Release files / compose_runner-0.14.3.tar.gz

Download URL	compose_runner-0.14.3.tar.gz
Size	51.6 MB
Tags	Source
SHA-256 checksum How to use checksums	`034dbf381b7dda4b0424389abb2127b3a977d3ff729d8aed0708c6cd01d03621`
BLAKE2b-256 checksum How to use checksums	`f1e087a243833c8639d0ec3f804bbd767d18e515a7c41ae043053e030174b2c1`
Upload date	Sep 2, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Release files / compose_runner-0.14.3-py2.py3-none-any.whl

Download URL	compose_runner-0.14.3-py2.py3-none-any.whl
Size	51.7 MB
Tags	Python 2 Python 3
SHA-256 checksum How to use checksums	`0822680501a8fc7fc98bf190f53393d55d717e5b51f345329a1086e6b48723c7`
BLAKE2b-256 checksum How to use checksums	`08d0dfee8af06a7868a1cc09def8beeed4f96d9c5e82a964b3f6391cd377d5c5`
Upload date	Sep 2, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.13.14`