meerschaum 0.0.53

Create and Manage Pipes with Meerschaum

Build Pipes with Meerschaum

Meerschaum is a platform for quickly creating and managing time-series data streams called Pipes. With Meerschaum, you can have a data visualization stack running in minutes.

Please visit https://meerschaum.io for setup, usage, and troubleshooting information. You can read a brief overview of the project below.

Table of Contents

• Disclaimer
• Setup
  • Requirements
  • Installation
  • Quickstart
  • Create Visualizations with Grafana
• FAQ
  • What are Pipes, exactly?
  • I broke my configuration. How do I start over?
  • I can't open the Meerschaum shell!
  • Connectors don't work for <database flavor>!
• Thank you

Disclaimer

Meerschaum is undergoing active development and is still an alpha release. Expect to encounter bugs, and please open issues when you find them! Meerschaum is distributed with no warranty, so use with care!

Meerschaum was built using powerful open source software like TimescaleDB, Grafana, pandas, and more. Check the Thank You section below for more information about dependencies.

Setup

Requirements

Make sure you have Docker and Docker Compose installed.

If you are running Linux, search your repositories for docker.io or run this script:

curl https://get.docker.com | sh

If you're on Windows or MacOS, install Docker Desktop.

You can install Docker Compose with pip:

pip install docker-compose

Installation

To install the full version of Meerschaum, install with pip:

pip install meerschaum[full]

Quickstart

Launch the Meerschaum Shell

To jump into the mrsm shell, run the command:

mrsm

All standard actions can either be executed from within the mrsm shell or on the command line directly. For example, the command

$ mrsm show version

is equivalent to

$ mrsm
> show version

Start the Stack

To start the Meerschaum stack in the background (using docker-compose -d), run the command

mrsm stack [-d]

Add a Connector

If you want to create a Pipe for remote data, you first need to define a Connector. Run this command to open the configuration YAML file with your text editor:

mrsm edit config

For this example, let's add a connector called exampledb. Add a new entry under the sql section:

meerschaum:
  connectors:
    sql:
      exampledb:
        username: myuser
        password: mypass
        flavor: postgresql
        host: myserverhostname
        database: mydb

Note that for sqlite connectors, only the database parameter is needed (path to the .sqlite file).

Register a Pipe

Pipes are defined by three keys:

1. Connection (keys for a connector in the format type:label; e.g. sql:exampledb for the Connector added above)
2. Metric (label for the contents of the Pipe's data, such as power or temperature)
3. Location (optional; None if omitted)

To register a Pipe, run this command:

mrsm register pipes -C sql:exampledb -M mydata

This will create a Pipe with the connector we added above (sql:exampledb) and the metric mydata. The location for this Pipe is None.

Next, we need to edit the parameters for this pipe. Run this command to open your text editor:

mrsm edit pipes -C sql:exampledb -M mydata

For this example, edit the YAML file to look something like this:

columns:
  datetime: mydtcolumn
  id: myidcolumn
fetch:
  definition: SELECT * FROM mytable

This information describes the remote dataset we will add into the Pipe. The columns defined will be indexed, and the definition is executed on the remote server to get the data.

Add Data to a Pipe

If you defined the parameters described above, just run the command below to fetch new data and sync the Pipe:

mrsm sync pipes

If you want to add an existing dataframe to a Pipe, run pipe.sync(df) to append the dataframe to the Pipe's table. You can launch into a Python shell with mrsm python:

>>> import meerschaum as mrsm
>>> pipe = mrsm.Pipe('sql:exampledb', 'mydata')
>>> 
>>> if not pipe.columns: ### if pipe has not been registered, you can define columns here
>>>   pipe.columns = {'datetime' : 'mydtcolumn', 'id' : 'myidcolumn'}
>>> 
>>> import pandas as pd
>>> df = pd.read_csv('mydata.csv')
>>> 
>>> pipe.sync(df)

In this case, the Pipe has created and indexed the table sql_exampledb_mydata on the Meerschaum TimescaleDB database.

Create Visualizations with Grafana

Grafana is included in the Meerschaum stack, pre-configured with the Meerschaum TimescaleDB database. Open a web browser and navigate to http://localhost:3000 and log in to Grafana with username admin, password admin.

FAQ

What are Pipes, exactly?

Pipes are built from TimescaleDB Hypertables, which are PostgreSQL tables with special datetime indexing and functions. In future releases of Meerschaum, Pipes may be created from continous aggregate views.

I broke my configuration. How do I start over?

The command to reset your configuration files is:

bootstrap config

If that doesn't work, just remove the folder ~/.config/meerschaum (%AppData%/Meerschaum on Windows).

To completely remove the stack and its data, run this command:

stack down [-v]

which is equivalent to docker-compose down -v.

I can't open the Meerschaum shell!

You can invoke mrsm directly with python -m meerschaum. Check that your PATH includes packages installed by pip, such as ~/.local/bin.

How do I turn off the emoji / colors / I'm running Windows?

Open the configuration file with mrsm edit config and search for the key formatting under the system section. From there, you can turn off emoji (unicode: false) or colors (ansi: false).

Connectors don't work for <database flavor>!

Although Connectors should work with any database flavor supported by sqlalchemy Engines, it is difficult to test against many database flavors. When bugs are encountered, please open an issue and describe your configuration!

As of now, there is (limited) support for the following database flavors:

• PostgreSQL / TimescaleDB
• MySQL / MariaDB
• MSSQL
• Oracle SQL (limited support, testing needed)
• SQLite

Thank you

I want to give my sincere thanks to the developers of the following projects:

• Docker
• Pandas
• SQLAlchemy
• FastAPI
• Uvicorn
• pprintpp
• CascaDict
• pyvim
• Colorama
• more_termcolor
• SQL CLI Tools