mage-ai 0.2.10

Mage is a notebook for building and deploying data pipelines.

Intro

Mage is an open-source notebook for building and deploying data pipelines.

A data pipeline defined across 3 different files:

# load_data_from_file.py
from pandas import DataFrame
import pandas as pd


@data_loader
def load_data() -> DataFrame:
    return pd.read_csv('default_repo/titanic.csv')

# select_columns.py
from pandas import DataFrame


@transformer
def transform_df(df: DataFrame, *args) -> DataFrame:
    return df[['Age', 'Fare', 'Survived']]

# export_to_file.py
from pandas import DataFrame


@data_exporter
def export_data(df: DataFrame) -> None:
    return df.to_csv('default_repo/titanic_transformed.csv')

What the data pipeline looks like in the UI:

New? We recommend reading about blocks and learning from a hands-on tutorial.

Table of contents

1. Quick start
2. Tutorials
3. Features
4. Contributing
5. Community
6. Documentation

🏃‍♀️ Quick start

You can install Mage using Docker or pip:

Using Docker

Create a new project and launch tool (change demo_project to any other name if you want):

docker run -it -p 6789:6789 -v $(pwd):/home/src \
  mageai/mageai mage start demo_project

Open http://localhost:6789 in your browser and build a pipeline.

Follow the guide if you want to use PySpark kernel in your notebook.

Using pip

1. Install Mage

pip install mage-ai

Mage also has the following extras:

• spark: to use Spark in your Mage pipeline
• bigquery: to connect to BigQuery for data import or export
• hdf5: to process HDF5 file format
• postgres: to connect to PostgreSQL for data import or export
• redshift: to connect to Redshift for data import or export
• s3: to connect to S3 for data import or export
• snowflake: to connect to Snowflake for data import or export
• all: to install all of the above to use all functionalities

pip install "mage-ai[spark]"

You may need to install development libraries for MIT Kerberos to use some Mage features. On Ubuntu, this can be installed as:

apt install libkrb5-dev

2. Create new project and launch tool

(change demo_project to any other name if you want):

mage start demo_project

Open http://localhost:6789 in your browser and build a pipeline.

👩‍🏫 Tutorials

• Train model on Titanic dataset
• Load data from API, transform it, and export it to PostgreSQL
• Integrate Mage into an existing Airflow project
• [Coming soon] How to visualize your data with charts

🔮 Features

1. Data centric editor
2. Production ready code
3. Extensible

1. Data centric editor

An interactive coding experience designed for preparing data to train ML models.

Visualize the impact of your code every time you load, clean, and transform data.

2. Production ready code

No more writing throw away code or trying to turn notebooks into scripts.

Each block (aka cell) in this editor is a modular file that can be tested, reused, and chained together to create an executable data pipeline locally or in any environment.

Read more about blocks and how they work.

Run your data pipeline end-to-end using the command line function: $ mage run [project] [pipeline]

You can run your pipeline in production environments with the orchestration tools

• Airflow
• Prefect
• Dagster (Tutorial coming soon)

3. Extensible

Easily add new functionality directly in the source code or through plug-ins (coming soon).

Adding new API endpoints (Tornado), transformations (Python, PySpark, SQL), and charts (using React) is easy to do (tutorial coming soon).

New features and changelog

Check out what’s new here.

🙋‍♀️ Contributing

We welcome all contributions to Mage; from small UI enhancements to brand new cleaning actions. We love seeing community members level up and give people power-ups!

Check out the 🎁 contributing guide to get started by setting up your development environment and exploring the code base.

Got questions? Live chat with us in Slack

Anything you contribute, the Mage team and community will maintain. We’re in it together!

🧙 Community

We love the community of Magers (/ˈmājər/); a group of mages who help each other realize their full potential!

To live chat with the Mage team and community, please join the free Mage Slack channel.

For real-time news and fun memes, check out the Mage Twitter.

To report bugs or add your awesome code for others to enjoy, visit GitHub.

🪪 License

See the LICENSE file for licensing information.