Mage is a tool for building and deploying data pipelines.
Project description
Intro
Mage is an open-source tool for building and running data pipelines that transform your data.
Here is a sample data pipeline defined across 3 files:
# data_loaders/load_data_from_file.py
@data_loader
def load_csv_from_file():
return pd.read_csv('default_repo/titanic.csv')
# transformers/select_columns.py
@transformer
def select_columns_from_df(df, *args):
return df[['Age', 'Fare', 'Survived']]
# data_exporters/export_to_file.py
@data_exporter
def export_titanic_data_to_disk(df) -> None:
df.to_csv('default_repo/titanic_transformed.csv')
What the data pipeline looks like in the UI:
New? We recommend reading about blocks and learning from a hands-on tutorial.
Table of contents
🏃♀️ Quick start
Install Mage using Docker or pip
:
Using Docker
Create a new project and launch tool (change demo_project
to any other name if you want):
docker run -it -p 6789:6789 -v $(pwd):/home/src \
mageai/mageai mage start demo_project
Follow the guide if you want to use PySpark kernel in your notebook.
Using pip
1. Install Mage
pip install mage-ai
For additional packages (e.g. spark
, postgres
, etc), please see Installing extra packages.
If you run into errors, please see Install errors.
2. Create new project and launch tool (change demo_project
to any other name if you want):
mage start demo_project
Open tool in browser
Open http://localhost:6789 in your browser and build a pipeline.
🎮 Demo
Live demo
Try a hosted version of the tool here: http://demo.mage.ai.
WARNING
The live demo is public, please don’t save anything sensitive.
Demo video (2 min)
Click the image to play video
👩🏫 Tutorials
- Train model on Titanic dataset
- Load data from API, transform it, and export it to PostgreSQL
- Integrate Mage into an existing Airflow project
🔮 Features
Read more here.
🏔️ Core design principles
Every user experience and technical design decision adheres to these principles.
- Easy developer experience
- Engineering best practices built-in
- Data is a first-class citizen
- Scaling made simple
Read more here.
🛸 Core abstractions
These are the fundamental concepts that Mage uses to operate.
Read more here.
📚 Documentation
Read more here.
🙋♀️ Contributing
Check out the 🎁 contributing guide to get started by setting up your development environment and exploring the code base.
🧙 Community
We love the community of Magers (/ˈmājər/
);
a group of mages who help each other realize their full potential!
To live chat with the Mage team and community, please join the free Mage Slack channel.
For real-time news and fun memes, check out the Mage Twitter.
To report bugs or add your awesome code for others to enjoy, visit GitHub.
🪪 License
See the LICENSE file for licensing information.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.