mage-ai 0.3.4

Mage is a tool for building and deploying data pipelines.

Intro

Mage is an open-source tool for building and deploying data pipelines.

Here is a sample data pipeline defined across 3 files:

# data_loaders/load_data_from_file.py
import pandas as pd


@data_loader
def load_data():
    return pd.read_csv('default_repo/titanic.csv')

# transformers/select_columns.py
@transformer
def transform_df(df, *args):
    return df[['Age', 'Fare', 'Survived']]

# data_exporters/export_to_file.py
@data_exporter
def export_data(df) -> None:
    df.to_csv('default_repo/titanic_transformed.csv')

What the data pipeline looks like in the UI:

New? We recommend reading about blocks and learning from a hands-on tutorial.

Table of contents

1. Quick start
2. Demo
3. Tutorials
4. Features
5. Documentation
6. Contributing
7. Community

🏃‍♀️ Quick start

Install Mage using Docker or pip:

Using Docker

Create a new project and launch tool (change demo_project to any other name if you want):

docker run -it -p 6789:6789 -v $(pwd):/home/src \
  mageai/mageai mage start demo_project

Open http://localhost:6789 in your browser and build a pipeline.

_{Follow the guide if you want to use PySpark kernel in your notebook.}

Using pip

1. Install Mage

pip install mage-ai

_{For additional packages (e.g. spark, postgres, etc), please see Installing extra packages.}

_{If you run into errors, please see Install errors.}

2. Create new project and launch tool (change demo_project to any other name if you want):

mage start demo_project

Open http://localhost:6789 in your browser and build a pipeline.

🎮 Demo

Live demo

Try a hosted version of the tool here: http://demo.mage.ai.

WARNING

The live demo is public, please don’t save anything sensitive.

Demo video (2 min)

_{Click the image to play video}

👩‍🏫 Tutorials

• Train model on Titanic dataset
• Load data from API, transform it, and export it to PostgreSQL
• Integrate Mage into an existing Airflow project

🔮 Features

• Overview
• Roadmap
• Changelog

📚 Documentation

Read more docs here.

🙋‍♀️ Contributing

We welcome all contributions to Mage; from small UI enhancements to brand new cleaning actions. We love seeing community members level up and give people power-ups!

Check out the 🎁 contributing guide to get started by setting up your development environment and exploring the code base.

Got questions? Live chat with us in Slack

Anything you contribute, the Mage team and community will maintain. We’re in it together!

🧙 Community

We love the community of Magers (/ˈmājər/); a group of mages who help each other realize their full potential!

To live chat with the Mage team and community, please join the free Mage Slack channel.

For real-time news and fun memes, check out the Mage Twitter.

To report bugs or add your awesome code for others to enjoy, visit GitHub.

🪪 License

See the LICENSE file for licensing information.