Run Jupyter cells in AWS Lambda for massively parallel experimentation
Project description
Eigensheep lets you easily run cells in Jupyter Notebooks on AWS Lambda with massive parallelism. You can instantly provision and run your code on 1000 different tiny VMs by simply prefixing a cell with %%eigensheep -n 1000
.
Getting Started
Open up your Terminal and install eigensheep
with pip
pip3 install eigensheep
Open a Jupyter notebook with jupyter notebook
and create a new Python 3 notebook. Run the following code in a cell:
import eigensheep
Follow the on-screen instructions to configure AWS credentials. Eigensheep uses AWS CloudFormation so you only need to a few clicks to get started.
Once Eigensheep is set up, you can run any code on Lambda by prefixing the cell with %%eigensheep
. You can include dependencies from pip
by typing %%eigensheep requests numpy
. You can invoke a cell multiple times concurrently with %%eigensheep -n 100
.
Acknowledgements
This library was written by Kevin Kwok and Guillermo Webster. It is based on Jupyter/IPython, tqdm
, boto3
, and countless Stackoverflow answers.
If you're interested in this project, you should also check out PyWren by Eric Jonas, and ExCamera from Sadjad Fouladi, et al.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for eigensheep-0.0.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 13dae8326e905c33f4a8ce61b032812012ea765fdd7d362c96fc3f2b27d42f4a |
|
MD5 | 0e20d367c75fa4e0e61ad6640cdd24d6 |
|
BLAKE2b-256 | 5e39d30abb1e0c61c34fe2e52f516326961bb9472ccffbb1e2380ba93dac9407 |