mq2db 1.0.0

Collect MQ messages and save them into database.

mq2db

Dump data from a message queue into a database.

Note: mq2db currently uses ZeroMQ sockets as its message-queue backend.

mq2db connects to one or more configured MQ endpoints, parses each received message with a loader, and writes the resulting rows into SQLAlchemy-supported databases.

Installation

After the package is published to PyPI:

python -m pip install mq2db

For development from this repository:

python -m pip install -e '.[test]'

Usage

mq2db path/to/config.yaml

If the mq2db configuration is nested under a larger YAML file, pass the section:

mq2db path/to/config.yaml --section mq2db.specific.setting

Configuration

Example configuration is here: example.yaml. More examples can also be found in the tests directory.

The top-level mq2db configuration contains a targets mapping. Each target defines:

• ZeroMQ socket settings, such as type, method, address, topic, and receive method.
• A loader class, such as mq2db.DictLoader, mq2db.JSONLoader, mq2db.TextLoader, mq2db.CSVLoader, or a custom importable class.
• Database settings, including SQLAlchemy URL, table columns, keys, indices, automatic timestamp/raw columns, and write interval.

Development

python -m pip install -e '.[dev]'
pytest
python -m build
twine check dist/*

Release

Releases use semantic version tags prefixed with v, such as v1.0.0. Pushing a matching tag publishes the package to PyPI through GitHub Actions:

git tag v1.0.0
git push origin v1.0.0

License

mq2db is distributed under the MIT License. See LICENSE.