Write maintainable, production-ready pipelines using Jupyter or your favorite text editor. Develop locally, deploy to the cloud.
Project description
Join our community | Newsletter | Contact us | Docs | Blog | Website | YouTube
Ploomber is the fastest way to build data pipelines ⚡️. Use your favorite editor (Jupyter, VSCode, PyCharm) to develop interactively and deploy ☁️ without code changes (Kubernetes, Airflow, AWS Batch, and SLURM). Do you have legacy notebooks? Refactor them into modular pipelines with a single command.
Installation
Compatible with Python 3.6 and higher.
Install with pip
:
pip install ploomber
Or with conda
:
conda install ploomber -c conda-forge
Getting started
Open a hosted JupyterLab instance:
Run an example locally:
# ML pipeline example
ploomber examples -n templates/ml-basic -o ml-basic
cd ml-basic
# install dependencies
pip install -r requirements.txt
# run pipeline
ploomber build
You just ran a Ploomber pipeline! 🎉
Check out the output/nb.html
report with model results!
The pipeline.yaml
contains the pipeline declaration. Feel free to modify any of the tasks, then call ploomber build
again to update the results (Note: if using VSCode or PyCharm, execute ploomber nb -i
before editing the files).
What's next?
Ready to migrate your project? Click here.
Do you want to learn more? Check out the introductory tutorial.
Run more examples.
Community
Main Features
⚡️ Get started quickly
A simple YAML API to get started quickly, a powerful Python API for total flexibility.
https://user-images.githubusercontent.com/989250/150660813-fc289c6c-0ed5-432d-b6df-063ce98c0093.mp4
⏱ Shorter development cycles
Automatically cache your pipeline’s previous results and only re-compute tasks that have changed since your last execution.
https://user-images.githubusercontent.com/989250/150660820-9a3a0abd-5904-492b-97ff-5494285dfebf.mp4
☁️ Deploy anywhere
Run as a shell script in a single machine or distributively in Kubernetes, Airflow, AWS Batch, or SLURM.
https://user-images.githubusercontent.com/989250/150660830-3f81c9a2-5392-49e5-976d-cb8a38441ecb.mp4
📙 Automated migration from legacy notebooks
Bring your old monolithic notebooks, and we’ll automatically convert them into maintainable, modular pipelines.
https://user-images.githubusercontent.com/989250/150660840-b0c12f85-504c-4233-8c3d-6724d291f1aa.mp4
I want to migrate my notebook.
Resources
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for ploomber-0.19.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 10c6f04385ab6dd436d5683cc68db743550031d1db42312f56ab1b58e4a02685 |
|
MD5 | de7e92fe42e4f6872dde1ead015241ef |
|
BLAKE2b-256 | c4197d26fc43f6ba45d969284c7b6a1edde8a4db174aacf98df30733a353b7b1 |