This release is a pre-release and may not be stable for production use.
| PyPI | GitHub | Info | Stats |
|---|---|---|---|
 |
 |
 |
 |
 |
 |
 |
 |
What is Meerschaum?
Meerschaum is an ETL framework for time-series data. You define pipes — named data streams — and Meerschaum keeps them in sync: it fetches only the new or changed rows, deduplicates and upserts them, manages the schema, and handles scheduling, serving, and storage.
Write a few lines of fetch logic; Meerschaum handles the rest of the pipeline. No more copy/pasting ETL scripts, hand-rolling incremental windows, or babysitting cron jobs. Drop it into an existing stack or stand up a full database-and-dashboard stack in minutes.
import meerschaum as mrsm
pipe = mrsm.Pipe('plugin:noaa', 'weather', 'atl', instance='sql:local')
pipe.sync() ### Pulls only what's new since the last sync.
Features
- ⚡️ Incremental by default — the sync engine fetches only new or changed rows and concurrently updates many streams at once. Duplicate rows are ignored; rows with existing keys are updated.
- 📊 Built for data scientists and analysts — integrate with Pandas, Grafana, and friends; persist DataFrames and always get the latest data. Skip pandas overhead and read rows as plain dicts with
Pipe.get_docs(). - 🗄️ Production-ready, batteries included — one-click deploy a TimescaleDB + Grafana stack, serve data org-wide via
FastAPI(uvicorn/gunicorn), and secure API instances with scoped auth tokens. Supports PostGIS geometry (incl. ESRI CRS) for geospatial pipelines. - 💼 Jobs and scheduling — run any command as a background job with
-d. Built-in scheduler handles cron and interval schedules — nocrontaborsystemdsetup. Execute locally, viasystemd, or remotely on an API instance with--executor-keys. - 🔌 Easily expandable — ingest any source with a simple plugin: just return a DataFrame. Add any function as a command, define parent/child pipe relationships for composable SQL pipelines, or embed Meerschaum via its Python API.
- ✨ Tailored for your experience — a rich CLI that's surprisingly enjoyable, a web dashboard for the graphically inclined, and connectors for SQL, API, Valkey, and custom backends.
- 🧳 Portable from the start —
$MRSM_ROOT_DIRemulates multiple installations and groups instances. The base wheel has no required Python dependencies (the native core is inside it; anything optional installs into a virtual environment), and it'suv-compatible:uv tool install meerschaum.
Want to learn more?
Find a wealth of information at meerschaum.io, or read up on Meerschaum in the wild:
- Interview featured in Console 100 - The Open Source Newsletter
- A Data Scientist's Guide to Fetching COVID-19 Data in 2022 (Towards Data Science)
- Time-Series ETL with Meerschaum (Towards Data Science)
- How I automatically extract my M1 Finance transactions
Installation
For a more thorough setup guide, visit the Getting Started page at meerschaum.io.
TL;DR
pip install meerschaum # or `uv tool install meerschaum[api]`
mrsm stack up -d
mrsm bootstrap pipes
Meerschaum 5 is a Rust core with a Python SDK. While it is in release candidates, install it with pip install --pre meerschaum; for a standalone mrsm binary run cargo install meerschaum --locked (Rust 1.95 or newer; add --version 5.0.0-rc.1 for the candidate). Upgrading from 4.x? Read the migration guide.
Usage
Visit meerschaum.io for setup, usage, and troubleshooting information. You can find technical documentation at docs.meerschaum.io.
CLI
### Install the NOAA weather plugin.
mrsm install plugin noaa
### Register a new pipe to the built-in SQLite DB.
### You can instead run `bootstrap pipe` for a wizard.
### Enter 'KATL' for Atlanta when prompted.
mrsm register pipe -c plugin:noaa -m weather -l atl -i sql:local
### Pull data and create the table "plugin_noaa_weather_atl".
mrsm sync pipes -l atl -i sql:local
Python API
import meerschaum as mrsm
pipe = mrsm.Pipe(
'foo', 'bar',
instance = 'sql:local', ### Built-in SQLite DB.
columns = {'datetime': 'dt', 'id': 'id'},
)
### Sync a DataFrame (or list of dicts) — creates the table on first run.
pipe.sync([{'dt': '2024-07-01', 'id': 1, 'val': 10}])
### Duplicates are ignored; rows with existing keys are updated.
pipe.sync([{'dt': '2024-07-01', 'id': 1, 'val': 100}])
assert pipe.get_rowcount() == 1
### Read back as a DataFrame, filtered by time range and params.
df = pipe.get_data(begin='2024-01-01', end='2025-01-01', params={'id': [1]})
### Or skip pandas and read plain dicts.
docs = pipe.get_docs(params={'id': [1]})
### [{'dt': datetime(2024, 7, 1), 'id': 1, 'val': 100}]
For composable in-database SQL pipelines (reference inheritance and {{ Pipe(...) }} table resolution), see the SQL pipes guide.
Plugins
Ingest any source by returning rows from a fetch function — Meerschaum handles the rest:
# ~/.config/meerschaum/plugins/example.py
__version__ = '1.0.0'
required = ['requests']
def register(pipe, **kw):
return {'columns': {'datetime': 'dt', 'id': 'id'}}
def fetch(pipe, begin=None, end=None, **kw):
import requests
rows = requests.get('https://api.example.com/data').json()
return rows ### list of dicts or a Pandas DataFrame
Support Meerschaum's Development
For consulting services and to support Meerschaum's development, please considering sponsoring me on GitHub sponsors.
Additionally, you can always buy me a coffee☕!
License
Copyright 2020-2026 Bennett Meares
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
1 maintainer
Metadata
1 maintainer
Release files for meerschaum 5.0.0rc1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| meerschaum-5.0.0rc1.tar.gz | 5.0 MB | Details |
Built distributions (wheels)
| File | Reset | |||
|---|---|---|---|---|
| meerschaum-5.0.0rc1-cp314-cp314t-manylinux_2_28_x86_64.whl | CPython 3.14 | CPython 3.14 free-threading | Linux glibc 2.28+ x86-64 | Details |
| meerschaum-5.0.0rc1-cp311-abi3-win_amd64.whl | CPython 3.11 | abi3 | Windows x86-64 | Details |
| meerschaum-5.0.0rc1-cp311-abi3-manylinux_2_28_x86_64.whl | CPython 3.11 | abi3 | Linux glibc 2.28+ x86-64 | Details |
| meerschaum-5.0.0rc1-cp311-abi3-manylinux_2_28_aarch64.whl | CPython 3.11 | abi3 | Linux glibc 2.28+ ARM64 | Details |
| meerschaum-5.0.0rc1-cp311-abi3-macosx_11_0_arm64.whl | CPython 3.11 | abi3 | macOS 11.0+ ARM64 | Details |
Total release size: 175.6 MB
Release files / meerschaum-5.0.0rc1.tar.gz
| Download URL | meerschaum-5.0.0rc1.tar.gz |
|---|---|
| Size | 5.0 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
3b998a82fa14fadfccf6befda2a69c20807ce783f6a6841247fc59562d374b8e
|
|
BLAKE2b-256 checksum How to use checksums |
9d5a6e3a260cb6be55e9a25154b333ff78572a89c11ef7649b85f02d9ca38b74
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.8
|
Release files / meerschaum-5.0.0rc1-cp314-cp314t-manylinux_2_28_x86_64.whl
| Download URL | meerschaum-5.0.0rc1-cp314-cp314t-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 34.7 MB |
| Tags | CPython 3.14 CPython 3.14 free-threading Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
fffc1f59f4992e50d834aeeab0f0ccedb26c922823774828cd4f066ddc9d6dbc
|
|
BLAKE2b-256 checksum How to use checksums |
ab8db1746fb5951a8d40a1f5095ce50be9c0ff241e0588034a53199e1fa50f55
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.8
|
Release files / meerschaum-5.0.0rc1-cp311-abi3-win_amd64.whl
| Download URL | meerschaum-5.0.0rc1-cp311-abi3-win_amd64.whl |
|---|---|
| Size | 37.4 MB |
| Tags | CPython 3.11 Windows x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
9be1f35180479285669e985ff29eb60ef5e2172fde567c53fca1105b203393ee
|
|
BLAKE2b-256 checksum How to use checksums |
5cc9a9e0a151be774e94dc9025c369774664aecf7a561d890f9a9c1db7b8f5b6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.8
|
Release files / meerschaum-5.0.0rc1-cp311-abi3-manylinux_2_28_x86_64.whl
| Download URL | meerschaum-5.0.0rc1-cp311-abi3-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 34.8 MB |
| Tags | CPython 3.11 Linux glibc 2.28+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
87006f6f47df7db01862291fe62a7fcd34f6cd3ae0f9c886dda74920640d2ced
|
|
BLAKE2b-256 checksum How to use checksums |
10b2bc4305ff86cf38b24f2764cff6ccd60290467d41b5623b5ca3641ff7e19d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.8
|
Release files / meerschaum-5.0.0rc1-cp311-abi3-manylinux_2_28_aarch64.whl
| Download URL | meerschaum-5.0.0rc1-cp311-abi3-manylinux_2_28_aarch64.whl |
|---|---|
| Size | 32.4 MB |
| Tags | CPython 3.11 Linux glibc 2.28+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
3b8759d60b44035705101c115b30b1e8287c577716fc3de08fbfe022041d97ca
|
|
BLAKE2b-256 checksum How to use checksums |
afa5b64354d96295dd2c16747c8085b2941156328c6b405fe97c8f0a3ce80d0f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.8
|
Release files / meerschaum-5.0.0rc1-cp311-abi3-macosx_11_0_arm64.whl
| Download URL | meerschaum-5.0.0rc1-cp311-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 31.3 MB |
| Tags | CPython 3.11 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
6279ec7a7b0caca70b87ebf69650eec9d57accf1180f9d19b602f811281aff54
|
|
BLAKE2b-256 checksum How to use checksums |
d34538de6b3fa36c543ada0b352699ef25e9da146e860c2219753742582e0347
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.8
|
