Mage is a tool for building and deploying data pipelines.
Project description
🧙 Mage
Mage is an open-source data pipeline tool for transforming and integrating data.
Here is a sample data pipeline defined across 3 files:
# data_loaders/load_data_from_file.py
@data_loader
def load_csv_from_file():
return pd.read_csv('default_repo/titanic.csv')
# transformers/select_columns.py
@transformer
def select_columns_from_df(df, *args):
return df[['Age', 'Fare', 'Survived']]
# data_exporters/export_to_file.py
@data_exporter
def export_titanic_data_to_disk(df) -> None:
df.to_csv('default_repo/titanic_transformed.csv')
What the data pipeline looks like in the UI:
New? We recommend reading about blocks and learning from a hands-on tutorial.
Table of contents
🏃♀️ Quick start
Install Mage:
Using Docker
Create a new project and launch tool (change demo_project
to any other name if you want):
docker run -it -p 6789:6789 -v $(pwd):/home/src \
mageai/mageai mage start demo_project
Want to use Spark or other integrations? Read more about integrations.
Using pip
or conda
1. Install Mage
pip install mage-ai
or
conda install -c conda-forge mage-ai
For additional packages (e.g. spark
, postgres
, etc), please see Installing extra packages.
If you run into errors, please see Install errors.
2. Create new project and launch tool (change demo_project
to any other name if you want):
mage start demo_project
Open tool in browser
Open http://localhost:6789 in your browser and build a pipeline.
🎮 Demo
Live demo
Try a hosted version of the tool here: https://demo.mage.ai/.
WARNING
The live demo is public, please don’t save anything sensitive.
Demo video (2 min)
Click the image to play video
👩🏫 Tutorials
- Train model on Titanic dataset
- Load data from API, transform it, and export it to PostgreSQL
- Integrate Mage into an existing Airflow project
🔮 Features
Read more here.
📚 Documentation
Read more here.
🏔️ Core design principles
Every user experience and technical design decision adheres to these principles.
- Easy developer experience
- Engineering best practices built-in
- Data is a first-class citizen
- Scaling made simple
Read more here.
🛸 Core abstractions
These are the fundamental concepts that Mage uses to operate.
Read more here.
🙋♀️ Contributing
Check out the 🎁 contributing guide to get started by setting up your development environment and exploring the code base.
🤔 Frequently Asked Questions (FAQs)
Check out our FAQ page to find answers to some of our most asked questions.
🧙 Community
We love the community of Magers (/ˈmājər/
);
a group of mages who help each other realize their full potential!
To live chat with the Mage team and community:
For real-time news, fun memes, data engineering topics, and more, join us on:
🪪 License
See the LICENSE file for licensing information.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.