ploomber 0.17.1

Write maintainable, production-ready pipelines using Jupyter or your favorite text editor. Develop locally, deploy to the cloud.

Join our community | Newsletter | Contact us | Docs | Blog | Website | YouTube

Ploomber is the fastest way to build data pipelines ⚡️. Use your favorite editor (Jupyter, VSCode, PyCharm) to develop interactively and deploy ☁️ without code changes (Kubernetes, Airflow, AWS Batch, and SLURM). Do you have legacy notebooks? Refactor them into modular pipelines with a single command.

Installation

Compatible with Python 3.6 and higher.

Install with pip:

pip install ploomber

Or with conda:

conda install ploomber -c conda-forge

Getting started

Open a hosted JupyterLab instance:

Run an example locally:

# ML pipeline example
ploomber examples -n templates/ml-basic -o ml-basic
cd ml-basic

# install dependencies
pip install -r requirements.txt

# run pipeline
ploomber build

You just ran a Ploomber pipeline! 🎉

Check out the output/nb.html report with model results!

The pipeline.yaml contains the pipeline declaration. Feel free to modify any of the tasks, then call ploomber build again to update the results (Note: if using VSCode or PyCharm, execute ploomber nb -i before editing the files).

What's next?

Ready to migrate your project? Click here.

Do you want to learn more? Check out the introductory tutorial.

Run more examples.

Community

• Join us on Slack
• Newsletter
• YouTube
• Contact the development team

Main Features

⚡️ Get started quickly

A simple YAML API to get started quickly, a powerful Python API for total flexibility.

https://user-images.githubusercontent.com/989250/150660813-fc289c6c-0ed5-432d-b6df-063ce98c0093.mp4

⏱ Shorter development cycles

Automatically cache your pipeline’s previous results and only re-compute tasks that have changed since your last execution.

https://user-images.githubusercontent.com/989250/150660820-9a3a0abd-5904-492b-97ff-5494285dfebf.mp4

☁️ Deploy anywhere

Run as a shell script in a single machine or distributively in Kubernetes, Airflow, AWS Batch, or SLURM.

https://user-images.githubusercontent.com/989250/150660830-3f81c9a2-5392-49e5-976d-cb8a38441ecb.mp4

📙 Automated migration from legacy notebooks

Bring your old monolithic notebooks, and we’ll automatically convert them into maintainable, modular pipelines.

https://user-images.githubusercontent.com/989250/150660840-b0c12f85-504c-4233-8c3d-6724d291f1aa.mp4

I want to migrate my notebook.

Show me a demo.

Resources

• Documentation
• PyData Chicago talk (covers motivation and demo)
• Develop and deploy an ML pipeline in 30 minutes (EuroPython 2021)
• Guest blog post on the official Jupyter blog
• Examples (Machine Learning pipeline, ETL, among others)
• Blog
• Comparison with other tools
• More videos