Skip to main content

No project description provided

Project description

nshsnap

nshsnap is a Python library for creating and managing snapshots of Python projects and environments. It's particularly useful for scenarios where you need to preserve the exact state of your code and dependencies, such as when running machine learning training jobs on cluster systems like SLURM.

Table of Contents

Motivation

When running long-running jobs or experiments, especially in distributed environments, it's critical to maintain consistency in your codebase. Small changes in your project files can lead to crashes or inconsistent results in ongoing runs. nshsnap addresses this issue by allowing you to take a full snapshot of your main development package, preserving the code state at that moment.

This tool was originally created to solve issues encountered when running ML training jobs on SLURM. It enables you to:

  1. Capture the current state of your project
  2. Ensure reproducibility across multiple runs
  3. Isolate changes in your development environment from active jobs

Installation

You can install nshsnap using pip:

pip install nshsnap

Usage

Programmatic Usage

Here's a basic example of how to use nshsnap in your Python code:

from nshsnap import snapshot

# Create a snapshot with default settings
snapshot_info = snapshot()

# Or, create a snapshot with custom configuration
snapshot_info = snapshot(
    modules=["my_project", "my_other_module"],
    editable_modules=True
)

print(f"Snapshot created at: {snapshot_info.snapshot_dir}")
print(f"Modules included: {', '.join(snapshot_info.modules)}")

Command Line Usage

nshsnap also provides a command-line interface:

# Snapshot all editable packages in the current environment
nshsnap --editables

# Snapshot specific modules
nshsnap --modules my_project my_other_module

# Specify a custom snapshot directory
nshsnap --editables --dir /path/to/snapshot/directory

# Get help
nshsnap --help

Activating and Using Snapshots

After creating a snapshot, all you need to do is prepend the snapshot directory to your PYTHONPATH to activate the snapshot environment:

export PYTHONPATH=/path/to/snapshot:$PYTHONPATH

You can also activate or execute commands within the snapshot environment using our helper scripts:

# Activate the snapshot environment
source /path/to/snapshot/.bin/activate

# Execute a command within the snapshot environment
/path/to/snapshot/.bin/execute python my_script.py

Features

  • Snapshot editable packages and specified modules
  • Preserve exact state of code and dependencies
  • Easy activation and execution within snapshot environments
  • Integration with version control systems (respects .gitignore)
  • Metadata storage for snapshot information

Requirements

  • Python 3.10+
  • git
  • rsync

Contributing

Contributions to nshsnap are welcome! Please feel free to submit a Pull Request.

License

MIT License

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

nshsnap-0.4.3.tar.gz (13.8 kB view details)

Uploaded Source

Built Distribution

nshsnap-0.4.3-py3-none-any.whl (15.6 kB view details)

Uploaded Python 3

File details

Details for the file nshsnap-0.4.3.tar.gz.

File metadata

  • Download URL: nshsnap-0.4.3.tar.gz
  • Upload date:
  • Size: 13.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.8.3 CPython/3.10.12 Linux/6.8.0-39-generic

File hashes

Hashes for nshsnap-0.4.3.tar.gz
Algorithm Hash digest
SHA256 ae3cf849b5df802b7d427486f18e24833978fbaad2ca33cc9b37f2cb4328598d
MD5 0f6e04b4ccc63f0f3a6373f8e087a74e
BLAKE2b-256 bac45ca3fe7ab4b53307df6d66f71107bf0798878a92b683c8595dd083ca2fbf

See more details on using hashes here.

File details

Details for the file nshsnap-0.4.3-py3-none-any.whl.

File metadata

  • Download URL: nshsnap-0.4.3-py3-none-any.whl
  • Upload date:
  • Size: 15.6 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.8.3 CPython/3.10.12 Linux/6.8.0-39-generic

File hashes

Hashes for nshsnap-0.4.3-py3-none-any.whl
Algorithm Hash digest
SHA256 6ff8570d5208dbcb8ced8b4cf7e452c607300858895576362a5f07b699bb7652
MD5 4bee0f6a6033bc2548077fa1f3c5310f
BLAKE2b-256 cd6f4cf2bb31ff6aa13b3f29588812e9b631d6d5bed1239d089a3bc5485f882e

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page