A utility to run Jupyter Notebook code on a remote server using Docker.
Project description
PyStributed: Remote Execution Utility
This package allows users to mark specific code cells in a Jupyter Notebook for remote execution. Once marked, the code will be packaged into a Docker container, sent to a specified remote server for execution, and the results will be fetched back to the local machine.
Installation
- Extract the contents of
pystributed.zip
to a location on your machine. - Navigate to the directory and run
pip install .
to install the package.
Prerequisites
- Docker installed on both local and remote machines.
- SSH access to the remote machine.
- PyTorch and Transformers libraries if you are using them in your code.
Configuration
Before using the package, you need to set up some configurations in config.py
:
DOCKER_IMAGE_NAME
: Name of the Docker image that will be created.DOCKER_REGISTRY
: Docker registry where the image will be pushed.REMOTE_SERVER
: SSH-compatible address of your remote server (e.g.,user@remote_server_ip
).REMOTE_WORKDIR
: Working directory on the remote server where results will be stored.USER_CODE_PATH
: Temporary path on the local machine where the user's code will be saved before packaging.
Usage
- In your Jupyter Notebook, import the package:
import pystributed.main as runner
- Use the
%%save_for_remote
magic command to mark the code cell you want to run remotely:
%%save_for_remote
# Your code here
# For example:
import torch
model = torch.load('my_model.pth')
result = model(some_data)
- After marking the desired code cell, call the main function from the package to execute the process:
runner.main()
Under the Hood
The package works in the following sequence:
- The code cell marked with
%%save_for_remote
is saved to a Python script (user_code.py
by default). - A Docker image is built with the user's code and necessary dependencies.
- The Docker image is pushed to the specified Docker registry.
- The package SSHs into the specified remote server, pulls the Docker image, and runs it.
- Once the code execution is complete on the remote server, the results are fetched and saved to the local machine.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
pystributed-0.1.4.tar.gz
(3.0 kB
view hashes)
Built Distribution
Close
Hashes for pystributed-0.1.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e8801b2bcf9a1ff971dac78ae4d7d5ece0753c430754eedacb70088199b189a0 |
|
MD5 | 25b5098c3ce91149e1912910214311ee |
|
BLAKE2b-256 | de1db665be47c2090732571f05fd63447db273988f711fae1c763102e5e5c2c4 |