Metaflow Checkpoint

Imagine running a machine learning training job or any data processing task that takes hours or even days to complete. In such scenarios, you don't want failures or collaboration complexities to force you to start over and lose all the progress made. This is where Metaflow's new decorators—@checkpoint, @model, and @huggingface_hub—come into play. These decorators are specifically designed to address these challenges by simplifying checkpointing, model management, and efficient loading of external models, ensuring that your long-running jobs can be resumed seamlessly after a failure and that models and checkpoints are properly versioned in multi-user environments.

This repository introduces three new decorators for Metaflow that address these challenges:

  • @checkpoint: Simplifies saving and reloading checkpoints within your Metaflow flows.
  • @huggingface_hub: Enables efficient loading and caching of large models from Hugging Face Hub.
  • @model: Allows for easy saving and loading of models created during your Metaflow flows.

Examples for these decorators can be found in this repository.

Features

@checkpoint Decorator

The @checkpoint decorator alleviates the pain points associated with saving and reloading the state of your program (a Metaflow @step) in Metaflow flows. It also handles version control in multi-user settings by isolating checkpoints per user and run. Whether it's a checkpoint created by a machine learning model or intermediate data required in case of crashes, this decorator simplifies state management and failure recovery.

  • Checkpointing: Save the state of your @step at designated points.
  • Seamless Recovery: Restart your job from the last checkpoint upon retries without any manual intervention.
  • User Isolation: Checkpoints are managed per user to prevent overwriting in collaborative environments.
  • Ease of Use: Minimal code changes required to implement checkpointing.

@huggingface_hub Decorator

The @huggingface_hub decorator allows you to load large models from Hugging Face Hub and cache them for increased performance benefits. It also ensures that models are versioned and managed appropriately in multi-user environments.

  • Efficient Model Loading: Load models on-the-fly from Hugging Face Hub.
  • Caching Mechanism: Cache models locally to avoid redundant downloads.
  • Version Control: Manages different versions of models to prevent conflicts.
  • Integration with Metaflow: Easily incorporate models across your Metaflow flows.

@model Decorator

The @model decorator provides a trivial way to save and load models/checkpoints created as part of your Metaflow flow.

  • Simplified Model Loading: Automatically load models based on references and identifiers created by decorators such as @model/@checkpoint/@huggingface_hub.
  • Model Identity: Associates a uniquie identity to models so that there is clear distinction between different versions making it easy to track their lineage.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.2.13)

2 maintainers

PyPI data
Data sourced directly from PyPI's database.
Avatar for oavdeev from gravatar.com oavdeev Avatar for valay-ob from gravatar.com valay-ob

Credits

Author: Valay Dave

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

metaflow_checkpoint-0.2.11.tar.gz (88.1 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

metaflow_checkpoint-0.2.11-py2.py3-none-any.whl (113.2 kB view details)

Uploaded Python 2Python 3

File details

Details for the file metaflow_checkpoint-0.2.11.tar.gz.

File metadata

  • Download URL: metaflow_checkpoint-0.2.11.tar.gz
  • Upload date:
  • Size: 88.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for metaflow_checkpoint-0.2.11.tar.gz
Algorithm Hash digest
SHA256 93bb55170eb17fb5bbcf7a16e18bf4b0c65f43bddce7a21b46936f973d1c14d4
MD5 2c10fffe28632407045ae4bbc3184c23
BLAKE2b-256 cfd94e5815e6faf11c9ba795d50a6b836ea2c8995ba9ec8df817dd4ad905e28e

See more details on using hashes here.

Provenance

The following attestation bundles were made for metaflow_checkpoint-0.2.11.tar.gz:

Publisher: publish.yaml on outerbounds/metaflow-checkpoint

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file metaflow_checkpoint-0.2.11-py2.py3-none-any.whl.

File metadata

File hashes

Hashes for metaflow_checkpoint-0.2.11-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 8afb77c9ef32c9e55a4c63611e69a34b203c5e319c9402e743f72854721db640
MD5 190906d277445f34bcc62fed808c7960
BLAKE2b-256 2c116c3c7e98f0c6fcc609ea7fc9f903cffb2a9ad9cacddd503f57458a4762e9

See more details on using hashes here.

Provenance

The following attestation bundles were made for metaflow_checkpoint-0.2.11-py2.py3-none-any.whl:

Publisher: publish.yaml on outerbounds/metaflow-checkpoint

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.2.13

2 files

0.2.12

2 files

This release

0.2.11 This release

2 files

0.2.10

2 files

0.2.8

2 files

0.2.7

2 files

0.2.6

2 files

0.2.5

2 files

0.2.4

2 files

0.2.3

2 files

0.2.2

2 files

0.2.1

2 files

0.2.0

2 files

0.1.9

2 files

Pre-release

0.1.8.1rc0

2 files

0.1.8

2 files

0.1.7

2 files

0.1.6

2 files

0.1.5

2 files

0.1.4

2 files

0.1.3

2 files

0.1.2

2 files

0.1.1

2 files

0.1.0

2 files

0.0.13

2 files

0.0.12

2 files

0.0.11

2 files

0.0.10

2 files

0.0.9

2 files

0.0.8

2 files

0.0.7

2 files

0.0.6

2 files

0.0.5

2 files

0.0.4

2 files

0.0.3

2 files

0.0.2

2 files

0.0.1

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page