medallion-pipeline 0.2.13

A composable scraper pipeline built around the medallion architecture (bronze / silver / gold).

Medallion

A scraper platform running hundreds of processing steps in Python mandates minimizing maintenance cost for updating and deploying processing steps, and guaranteeing availability of the output data for internal users in 0.5 seconds or less, especially in the case of a thundering herd causing an explosion in the amount of data to process at any point in time.

Medallion is a Python library that reduces the code developers need to change to update processing steps to the mere business logic. It deploys scrapers as microservices, isolating deployments and allowing for seamless re-routing of data traffic after deployment. It exposes the output of each processing step directly as a stream of data that internal users can opt–in for near-instant data availability. The output of each step is also written to disk for asynchronous inspection. Medallion manages thundering herds by design, because the processing–step microservices are inherently stateless, allowing you to scale horizontally as needed.

Why Medallion

Building a scraper platform that stays cheap to maintain and serves data in 0.5 seconds or less usually means gluing together a message broker, a worker framework, an archival sink, a schema layer, a local emulator, and a pile of deployment YAML. Medallion collapses that stack into a single config-driven library so the only thing you actually write is your business logic.

• Only the business logic changes. Updating a processing step means editing one extract() or transform() method. Queues, storage, wiring, schemas, and deployment artifacts are derived from config.yml — there is no plumbing to touch when you ship a change.

• Sub-second availability of raw source data. Every step exposes its output as a live stream that internal users can subscribe to directly. Internal users don't have to wait for the entire pipeline to finish to get access to the data, if they prefer instant availability instead of more complex queries on the final output. This also means that if a downstream step is slow or broken, the raw source data is still available to internal users without having to wait for the entire pipeline to be fixed.

• Thundering herds handled by design. Processing-step microservices are stateless by construction, so a sudden explosion in input volume is absorbed by scaling steps horizontally rather than by tuning or babysitting the system.

• Every intermediate stream is archived automatically. A Store is attached to each queue with no code on your part, writing output to disk (or a bucket) for asynchronous inspection and replay — without requiring a replay-capable broker. This doubles as a precise replay tool: a developer who changes one step can re-process exactly the same archived input without re-running the entire pipeline — avoiding the risk of costly re-processing or that upstream changes or non-determinism produce different input.

• Typed pipelines, end to end. Queues are typed and every processor's In/ Out must match the queues it reads from and writes to, so topology mistakes are caught as early as possible.

• Isolated, independently deployable steps. Each extractor and transformer runs as its own microservice, so deployments are isolated and data traffic can be re-routed seamlessly after a deploy.

• One config, local and prod parity. From a single config.yml, Medallion generates a runnable Docker Compose cluster, debug configurations for VS Code, and a Google Cloud Run fleet of services with Pub/Sub as broker and storage bucket — the same topology you run locally is the one you deploy. If you prefer a different cloud provider than GCP, you can easily swap it, you only need to implement a handful of functions for the Queue and BlobStore interfaces for your provider of choice.

In total, Medallion replaces what is typically 8–10 separate components plus their glue code.

Installation

pip install medallion-pipeline

The distribution is published as medallion-pipeline; the Python import name and CLI tool name is medallion. Python 3.12+ required.

Core concepts

• Extractor — produces the initial data and sends to a Queue. You implement a class that inherits from BaseExtractor[Out] and implement extract(self) -> Iterable[Out].
• Transformer — consumes the previous step's output from a Queue and sends transformed output to a different queue. Subclass BaseTransformer[In, Out] and implement transform(self, data: Iterable[In]) -> Iterable[Out]. If you need streaming, you need to inherit from BaseStreamingTransformer[In, Out] and implement transform_one(self, data: In) -> Out instead.
• Queues – Queues are typed and processors (Extractor or Transformer) must match the queue's type in their corresponding In and/or Out types. The library supports a mock-in-memory queue and GCP Pub/Sub. To roll your own, implement the Queue interface.
• Store – is attached to each queue automatically – you don't need to write any code to do that. A Store reads from a queue and stores the data. The library has built-in support for storage on a local disc or GCP Storage Bucket. To roll your own, implement the BlobStore interface.
• Orchestration – you tie everything together in a config.yml file. Here you define data types, queues, extractors and transformers, and which queues the processors read and write to.

Usage

Start a new project

Create boilerplate for your new project and set up an example file structure.

medallion start MY_NEW_PROJECT_NAME

Install dependencies and create a virtual environment.

poetry install

Your business logic implementation

The start command creates an example extractor and transformer for you to get started. You can modify these files or create new ones as needed. If you add new extractors or transformers, make sure to make them available to the config.yml file by importing them in the src/__init__.py file.

More examples can be found in the examples folder.

Create debugging configurations for VS Code

Medallion creates debugging configurations for your scraper pipelines, which you can run in the VS Code debugger. The start command already creates working configurations for the example extractor and transformer, so you can start debugging right away. Re-run the command after making changes to your config.yml file to update the debugging configurations with your new pipelines.

The command automatically validates your config.yml file before generating the debugging configurations.

Inside your project, run:

medallion vscode

Or alternatively, run the command Configure VS Code Debugger from the command palette.

Example result (your's may vary):

The last few configurations starting with Run Pipeline: are your scraper pipelines, defined in config.yml. Run them to launch a debugger for your scraper.

After making changes to your config.yml, run the command Configure VS Code Debugger to validate it and re-generate the ./launch/config.json file with updated pipelines.

Configure Docker workspace

Medallion can generate a docker-compose.yml file from your config.yml to run your pipelines in an isolated environment. Each service runs in its own container, and you can attach a debugger to any of them.

The startcommand already generates an innitial docker-compose.yml file for you to get started. Re-run the below command after making changes to your config.yml file to update the docker-compose.yml file with your new pipelines.

Run the debug configuration Generate docker compose from config.yml or execute this command:

poetry run python -m medallion.configure_entrypoint.generate_docker_compose

This validates your config.yml and generates a docker-compose.yml and a docker-compose.debug.yml

The docker-compose file will:

1. Run an isolated GCP Pub/Sub emulator host as microservice
2. Run a microservice for each transformer from your config.yml file
3. Run a Store microservice for each queue defined in your config.yml file. The storage type is mounted to your local folder .medallion-root, which is meant to re-use the data folder you use for local runs.
4. Run a microservice for each extractor, that listens on port 8001, 8002, ..., 8000+n for n extractors.

To use it, run:

EXTRACTOR_API_KEY=YOUR_API_KEY docker compose up [--build]

To trigger an extractor, run:

curl --location --request POST 'http://0.0.0.0:EXTRACTOR_PORT' \
--header 'X-Extractor-Api-Key: YOUR_API_KEY'

Debugging inside of Docker container

To run a debugger in a docker container, have a look at the docker-compose.debug.yml file generated together with docker-compose.yml. It picks the first extractor from the compose by default. Modify the service name to debug any other service from your docker-compose.yml.

Run:

docker compose -f docker-compose.yml -f docker-compose.debug.yml up [--build]

Once all containers are ready and their logs indicate that they're listening, run the debugging configuration Attach to docker service.

Configure GCP Deployment

medallion generates a Google Cloud Run fleet of services from your config.yml, using Pub/Sub as the broker and a storage bucket for each queue's Store — the same topology defined for local runs.

... Documentation coming soon ...

Run an Extractor as HTTP Server

If you need to debug the extractor HTTP server alone, which usually only runs inside docker or in production, you may want to use this entrypoint instead of running the entire fleet in docker.

EXTRACTOR_CLASS=MY_EXTRACTOR poetry run uvicorn medallion.run.extractor:app --host 0.0.0.0 --port 8080

Alternatively, run the debug configuration Run extractor as HTTP server and set the environment variable EXTRACTOR_CLASS to the extractor you want to run.

Contributing

See CONTRIBUTING.md for developer notes and the release process.