Skip to main content

Mage is a tool for building and deploying data pipelines.

Project description

Fire mage

PyPi mage-ai License Join Slack

Intro

Mage is an open-source tool for building and running data pipelines that transform your data.


Here is a sample data pipeline defined across 3 files:

# data_loaders/load_data_from_file.py
@data_loader
def load_csv_from_file():
    return pd.read_csv('default_repo/titanic.csv')
# transformers/select_columns.py
@transformer
def select_columns_from_df(df, *args):
    return df[['Age', 'Fare', 'Survived']]
# data_exporters/export_to_file.py
@data_exporter
def export_titanic_data_to_disk(df) -> None:
    df.to_csv('default_repo/titanic_transformed.csv')

What the data pipeline looks like in the UI:

data pipeline overview

New? We recommend reading about blocks and learning from a hands-on tutorial.


Join us on Slack

Table of contents

  1. Quick start
  2. Demo
  3. Tutorials
  4. Core design principles
  5. Features
  6. Documentation

🏃‍♀️ Quick start

Install Mage using Docker or pip:

Using Docker

Create a new project and launch tool (change demo_project to any other name if you want):

docker run -it -p 6789:6789 -v $(pwd):/home/src \
  mageai/mageai mage start demo_project

Follow the guide if you want to use PySpark kernel in your notebook.

Using pip

1. Install Mage
pip install mage-ai

For additional packages (e.g. spark, postgres, etc), please see Installing extra packages.

If you run into errors, please see Install errors.

2. Create new project and launch tool (change demo_project to any other name if you want):
mage start demo_project

Open tool in browser

Open http://localhost:6789 in your browser and build a pipeline.


🎮 Demo

Live demo

Try a hosted version of the tool here: http://demo.mage.ai.

WARNING

The live demo is public, please don’t save anything sensitive.

Demo video (2 min)

Mage quick start demo

Click the image to play video


👩‍🏫 Tutorials


🏔️ Core design principles

Every user experience and technical design decision adheres to these principles.

  1. Easy developer experience
  2. Engineering best practices built-in
  3. Data is a first-class citizen
  4. Scaling made simple

Read more here.


🛸 Core abstractions

These are the fundamental concepts that Mage uses to operate.

Read more here.


🔮 Features

Read more here.

Mage


📚 Documentation

Read more here.


🙋‍♀️ Contributing

Check out the 🎁 contributing guide to get started by setting up your development environment and exploring the code base.


🧙 Community

We love the community of Magers (/ˈmājər/); a group of mages who help each other realize their full potential!

To live chat with the Mage team and community, please join the free Mage Slack Slack channel.

Join us on Slack

For real-time news and fun memes, check out the Mage Twitter Twitter.

To report bugs or add your awesome code for others to enjoy, visit GitHub.


🪪 License

See the LICENSE file for licensing information.

Water mage casting spell

Project details


Release history Release notifications | RSS feed

This version

0.4.3

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

mage-ai-0.4.3.tar.gz (6.2 MB view hashes)

Uploaded Source

Built Distribution

mage_ai-0.4.3-py3-none-any.whl (6.4 MB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page