Skip to main content

MLSpec helper library to making using metadata in ML workflows easier

Project description

ML Spec Library

The purpose of MLSpec Lib is to provide a library that can easily read and write schema-tized metadata related to the steps in an ML Pipeline.

The original set of schema was drawn from here - - as a set of illustrative data for initial specs.

If you had a sample pipeline that did the following:

  • Read data
  • Transformed the data in some way and saved it to a new directory
  • Trained a model based on the data
  • Wrote the results of the training to a new directory
  • Converted the model to a training format
  • Created a serving endpoint for the model

At each step, you would want to use first class python objects to drive the workflow, and, ideally, you would want to read/write metadata about what you were doing to have a permenant record.

This is what MLSpecLib is designed to do:

  • Read the metadata in
  • Provide a first class Python object for use
  • Allow validation of the Python object according to a schema
  • Write the object to disk in a readable YAML format

To see it in action, go to the Sample Notebook.

*** NOTE: The Sample Notebook is illustrative only! It does not actually execute any actual training and the schema are just for examples. ***

  • Getting Started *
# Go into the directory and install all packages
pip3 install -r requirements.txt

# Change into the notebook directory and install jupyter
pip3 install jupyter

# Start a notebook server
jupyter notebook

Now go to the URL that jupyter gives you, and run through the Sample Notebook. It should provide examples of how to read and write full objects from disk using native feeling Python.

TODO:

  • Support better typing / flexibility in lists (including allowed)
  • Real world examples of usage - more notebooks

Library structure and tools are derived from the work of Kenneth Reitz:

  • Learn more <http://www.kennethreitz.org/essays/repository-structure-and-python>_.
  • If you want to learn more about setup.py files, check out this repository <https://github.com/kennethreitz/setup.py>_.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

mlspeclib-0.0.8.linux-x86_64.tar.gz (28.9 kB view details)

Uploaded Source

Built Distribution

mlspeclib-0.0.8-py3-none-any.whl (46.7 kB view details)

Uploaded Python 3

File details

Details for the file mlspeclib-0.0.8.linux-x86_64.tar.gz.

File metadata

  • Download URL: mlspeclib-0.0.8.linux-x86_64.tar.gz
  • Upload date:
  • Size: 28.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.3.0 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.6.9

File hashes

Hashes for mlspeclib-0.0.8.linux-x86_64.tar.gz
Algorithm Hash digest
SHA256 c27b20f3746cd016a3ca877b77d3342782274bf28fc2bfb60ff31b399925fdb3
MD5 097795b7ead82e7c270abd9d1184fe96
BLAKE2b-256 0493f7642b181d10bba6838519edb2fd249b53af0d91f31f1c1a673d88d10d4f

See more details on using hashes here.

File details

Details for the file mlspeclib-0.0.8-py3-none-any.whl.

File metadata

  • Download URL: mlspeclib-0.0.8-py3-none-any.whl
  • Upload date:
  • Size: 46.7 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.3.0 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.6.9

File hashes

Hashes for mlspeclib-0.0.8-py3-none-any.whl
Algorithm Hash digest
SHA256 f487a5d8e0dce8142c89e5c3429029d02713002b4907a65c3b128546516fb341
MD5 655da6d14204f058b05304f142233f56
BLAKE2b-256 cc5f5be0580cb86027b466f3320ea6c41236c3970a579c45decd76db81339031

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page