Skip to main content

Lineage and tracing for ML pipelines

Project description

mltrace

mltrace Documentation Status PyPI License

mltrace is a lightweight, open-source Python tool to get "bolt-on" observability in ML pipelines. It offers the following:

  • interface to define data and ML tests for components in pipelines
  • coarse-grained lineage and tracing
  • Python API to log versions of data and pipeline components
  • database to store information about component runs
  • UI and CLI to show the trace of steps in a pipeline taken to produce an output, flag outputs for review, and identify what steps of the pipeline to investigate first when debugging

mltrace is designed specifically for Agile or multidisciplinary teams collaborating on machine learning or complex data pipelines. The prototype is very lofi, but this README contains instructions on how to run the prototype on your machine if you are interested in developing. For general usage instructions, please see the official documentation. The accompanying blog post can be found here.

screenshot

Quickstart

You should have Docker installed on your machine. To get started, you will need to do 3 things:

  1. Set up the database and Flask server
  2. Run some pipelines with logging
  3. Launch the UI

If you are interested in learning about specific mltrace concepts, please read this page in the official docs.

Database setup (server-side)

We use Postgres-backed SQLAlchemy. Assuming you have Docker installed, you can run the following commands from the root directory after cloning the most recent release:

docker-compose build
docker-compose up [-d]

And then to tear down the containers, you can run docker-compose down.

Run pipelines (client-side)

To use the logging functions in dev mode, you will need to install various dependencies:

pip install -r requirements.txt
pip install -e .

Next, you will need to set the database URI. It is recommended to use environment variables for this. To set the database address, set the DB_SERVER variable:

export DB_SERVER=<SERVER'S IP ADDRESS>

where <SERVER'S IP ADDRESS> is either the IP address of a remote machine or localhost if running locally. If, when you set up the server, you changed the URI in docker-compose.yaml, you can set the DB_URI variable (which represents the entire database URI) client-side instead of DB_SERVER.

The files in the examples folder contain sample scripts you can run. For instance, if you run examples/industry_ml.py, you might get an output like:

> python examples/industry_ml.py
Final output id: aafknvtoag

And if you trace this output in the UI (trace aafknvtoag), you will get:

screenshot

You can also look at examples for ways to integrate mltrace into your ML pipelines, or check out the official documentation.

Launch UI (client-side)

If you ran docker-compose up from the root directory, you can just navigate to the server's IP address at port 8080 (or localhost:8080) in your browser. To launch a dev version of the UI, navigate to ./mltrace/server/ui and execute yarn install then yarn start. It should be served at localhost:3000. The UI is based on create-react-app and blueprintjs. Here's an example of what tracing an output would give:

screenshot

Commands supported in the UI

Command Description
recent Shows recent component runs, also the home page
history COMPONENT_NAME Shows history of runs for the component name. Defaults to 10 runs. Can specify number of runs by appending a positive integer to the command, like history etl 15
inspect COMPONENT_RUN_ID Shows info for that component run ID
trace OUTPUT_ID Shows a trace of steps for the output ID
tag TAG_NAME Shows all components with the tag name
flag OUTPUT_ID Flags an output ID for further review. Necessary to see any results from the review command.
unflag OUTPUT_ID Unflags an output ID. Removes this output ID from any results from the review command.
review Shows a list of output IDs flagged for review and the common component runs involved in producing the output IDs. The component runs are sorted from most frequently occurring to least frequently occurring.

Launch without UI (client-side)

If you want to launch database and api containers without the UI, you will run docker-compose docker-compose-not-ui.yml up from the root directory. If running correctly, you should see nothing displayed in the server's IP address at port 8080 (or localhost:8080) but the database and API service should work unaffectedly:

  1. change directory to the level where docker-compose-not-ui.yml exist (root directory)
  2. run docker-compose -f docker-compose-not-ui.yml build to build the image (skip this step if image already exist)
  3. run docker-compose -f docker-compose-not-ui.yml up to bring up the service

Using the CLI for querying

The following commands are supported via CLI:

  • history
  • recent
  • trace
  • flag
  • unflag
  • review

You can execute mltrace --help in your shell for usage instructions, or you can execute mltrace command --help for usage instructions for a specific command.

Future directions

The following projects are in the immediate roadmap:

  • API to log from any type of file, not just a Python file
  • Prometheus integrations to monitor component output distributions
  • Support for finer-grained lineage (at the record level)

Contributing

Anyone is welcome to contribute, and your contribution is greatly appreciated! Feel free to either create issues or pull requests to address issues.

  1. Fork the repo
  2. Create your branch (git checkout -b YOUR_GITHUB_USERNAME/somefeature)
  3. Make changes and add files to the commit (git add .)
  4. Commit your changes (git commit -m 'Add something')
  5. Push to your branch (git push origin YOUR_GITHUB_USERNAME/somefeature)
  6. Make a pull request

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

mltrace-0.17.tar.gz (32.7 kB view details)

Uploaded Source

Built Distribution

mltrace-0.17-py3-none-any.whl (39.8 kB view details)

Uploaded Python 3

File details

Details for the file mltrace-0.17.tar.gz.

File metadata

  • Download URL: mltrace-0.17.tar.gz
  • Upload date:
  • Size: 32.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.6.1 pkginfo/1.7.0 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.61.2 CPython/3.9.6

File hashes

Hashes for mltrace-0.17.tar.gz
Algorithm Hash digest
SHA256 bf2532e1b68a4617d431bd58cd81c2d857c67a8324d318721ba37270a7376e0b
MD5 3ac6634d3f49a9db7664a06d3179f3e9
BLAKE2b-256 88db81b05b366f39abc60ac51526a75f63b7b53272883bc5e5b7454f2cf87587

See more details on using hashes here.

File details

Details for the file mltrace-0.17-py3-none-any.whl.

File metadata

  • Download URL: mltrace-0.17-py3-none-any.whl
  • Upload date:
  • Size: 39.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.6.1 pkginfo/1.7.0 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.61.2 CPython/3.9.6

File hashes

Hashes for mltrace-0.17-py3-none-any.whl
Algorithm Hash digest
SHA256 bae35818758b77d7cc877267ad68f5cf3c5544da1d5a7572afe613f29bf55d66
MD5 e2326c8b0daaf580489390be9138f270
BLAKE2b-256 c78f72d109812fb80c367859c4fddb0ab5f57bf1a2352a852e74756e6ac8f868

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page