Skip to main content
This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (pypi.python.org).
Help us improve Python packaging - Donate today!

A micro service framework for data pipelines, providingscheduling, retrying, and error reporting.

Project Description

Mettle is a framework for managing extract/transform/load (ETL) jobs. ETL processes present a number of problems that Mettle is designed to solve:

License

License is indicated in the project metadata (typically one or more of the Trove classifiers). For more details, see this explanation.

Description

  • Jobs need to be run at specific times. Sometimes they need to be triggered by the completion of other jobs. Mettle supports scheduling both time-based and trigger-based jobs.
  • Various people in an organization need to be able to see job schedules and the state of recent runs. Naive scripts running on cron jobs, scattered amongst a large number of servers, create a serious problem with visibility. Mettle solves this by centralizing the job scheduling, state reporting, and log viewing.
  • Sometimes jobs fail because of temporary problems somewhere (a flaky network, a too-full disk). Mettle will automatically retry jobs to deal with this.
  • Sometimes jobs fail and will not be able to succeed until the job has been reconfigured (a changed password on a database, for example). Mettle makes it easy to manually re-launch a job after such issues have been resolved.
  • If you try to solve the above problems by centralizing all your ETL execution, you quickly run into a problem of proliferating dependencies. A centralized ETL service can become hard to develop and hard to deploy because all those dependencies (libraries, external APIs, external databases) introduce more instability. Mettle is designed to isolate those dependencies into separate ETL services, so instability in one ETL doesn’t impact any others.

We picked the name “Mettle” because:

  • It’s got the letters E, T, and L in it.
  • It means “ability to continue despite difficulties”.
  • It sounds like “metal”, which is solid.

Mettle is comprised of several components:

  • Web UI. Features:
    • Configure schedules for pipelines.
    • Display past jobs, both successful and failed.
    • Display currently-executing jobs, with live status updates and streaming logs.
    • Manually launch jobs.
  • Timer: Reads pipeline schedules from the database and sends out RabbitMQ messages when pipelines need to be kicked off.
  • Dispatcher: Records which jobs are being executed by which workers, and their eventual success or failure.
  • Logger: Receives log messages sent from ETL Services over RabbitMQ, and saves them to Postgres.
  • ETL Services: Implement the actual business logic and systems integration to move data between systems.

Mettle uses Postgres to store state, and RabbitMQ for inter-process communication.

Release History

Release History

This version
History Node

0.7.13

History Node

0.7.12

History Node

0.6.1

Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
mettle-0.7.13-py2.py3-none-any.whl (343.1 kB) Copy SHA256 Checksum SHA256 py2.py3 Wheel Oct 30, 2017
mettle-0.7.13.tar.gz (306.4 kB) Copy SHA256 Checksum SHA256 Source Oct 30, 2017

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting