A wrapper around magic-wormhole and joblib to send pickled objects across the internet.
Project description
MagicPickle
magicpickle
allows you to transfer pickled representations of objects between local and remote instances of scripts, providing a near-seamless way to write code which both accesses data stored remotely and visualizes it locally. This avoids the need to:
- store, load, and sync intermediate data representations between local and remote machines
- use X11 forwarding/VNC with noticable latency
Internally, magicpickle
uses joblib
to pickle and unpickle objects, and magic-wormhole
to transfer the pickled data between local and remote instances of a script.
Note that magicpickle
assumes that each mp.save
is associated with a single mp.load
in the same script; it assumes that both local and remote instances have the same control flow.
Installation
pip install magicpickle
Usage
Check the docstrings in src/magicpickle.py
for more information. Example use:
from magicpickle import MagicPickle
with MagicPickle(MY_LOCAL_HOSTNAME) as mp: # or MagicPickle(func_that_returns_true_if_local)
if mp.is_remote: # or mp.is_local
mp.save("hello")
else:
print(mp.load())
Tmux
tmux_magicpickle.py
is a script that scrapes your panes and automatically enters the magic-wormhole
code for you. Add the following to your ~/.tmux.conf
to use it:
bind-key g run-shell "python3 PATH_TO/tmux_magicpickle.py"
to add the prefix + g
binding.
Gotchas
To allow the pickling of lambda functions, prefix your script with
import dill as pickle
To allow the loading of pickled CUDA tensors onto a CPU, prefix your script with
import torch
# https://stackoverflow.com/a/78399538/10702372
torch.serialization.register_package(0, lambda x: x.device.type, lambda x, _: x.cpu())
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for magicpickle-0.0.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 606951235b513cd486df26cce2221239e8246b001de202f089e7fddf8bd7bcbd |
|
MD5 | eb40364192b96db292a329501b947086 |
|
BLAKE2b-256 | 3e9864676c73cabbb474f7bbde69334e8bbca6e39ef2bb78ccc97a85e77d120f |