Skip to main content

dsinfluxlogger

Log PubSubDataMessages (see datastreamcorelib) to InfluxDB, this is a pretty quick and dirty implementation and definitely not optimal. Supports only InfluxDB < 2.0 (ie 1.8) due to aioinflux not supporting 2.0.

For optimal write performance we should add a plugin system that allows one to convert the payload from the PubSubDataMessages into line-protocol decorated custom classes https://aioinflux.readthedocs.io/en/stable/usage.html#writing-user-defined-class-objects

The quick-and-dirty optimization is to batch writes into pandas dataframes, which has the not insignificant drawback of adding pandas/numpy to our requirements.

Docker

For more controlled deployments and to get rid of “works on my computer” -syndrome, we always make sure our software works under docker.

It’s also a quick way to get started with a standard development environment.

SSH agent forwarding

We need buildkit:

export DOCKER_BUILDKIT=1

And also the exact way for forwarding agent to running instance is different on OSX:

export DOCKER_SSHAGENT="-v /run/host-services/ssh-auth.sock:/run/host-services/ssh-auth.sock -e SSH_AUTH_SOCK=/run/host-services/ssh-auth.sock"

and Linux:

export DOCKER_SSHAGENT="-v $SSH_AUTH_SOCK:$SSH_AUTH_SOCK -e SSH_AUTH_SOCK"

Creating a development container

Build image, create container and start it:

docker build --ssh default --target devel_shell -t dsinfluxlogger:devel_shell .
docker create --name dsinfluxlogger_devel -p 58770:58770 -v `pwd`":/app" -it -v /tmp:/tmp `echo $DOCKER_SSHAGENT` dsinfluxlogger:devel_shell
docker start -i dsinfluxlogger_devel

pre-commit considerations

If working in Docker instead of native env you need to run the pre-commit checks in docker too:

docker exec -i dsinfluxlogger_devel /bin/bash -c "pre-commit install"
docker exec -i dsinfluxlogger_devel /bin/bash -c "pre-commit run --all-files"

You need to have the container running, see above. Or alternatively use the docker run syntax but using the running container is faster:

docker run -it --rm -v `pwd`":/app" dsinfluxlogger:devel_shell -c "pre-commit run --all-files"

Test suite

You can use the devel shell to run py.test when doing development, for CI use the “tox” target in the Dockerfile:

docker build --ssh default --target tox -t dsinfluxlogger:tox .
docker run -it --rm -v `pwd`":/app" `echo $DOCKER_SSHAGENT` dsinfluxlogger:tox

Production docker

There’s a “production” target as well for running the application (change “myconfig.toml” for config file):

docker build --ssh default --target production -t dsinfluxlogger:latest .
docker run -it --name dsinfluxlogger -v myconfig.toml:/app/config.toml -p 58770:58770 -it -v /tmp:/tmp `echo $DOCKER_SSHAGENT` dsinfluxlogger:latest

Local Development

TLDR:

  • Create and activate a Python 3.8 virtualenv (assuming virtualenvwrapper):

    mkvirtualenv -p `which python3.8` my_virtualenv
  • change to a branch:

    git checkout -b my_branch
  • install Poetry: https://python-poetry.org/docs/#installation

  • Install project deps and pre-commit hooks:

    poetry install
    pre-commit install
    pre-commit run --all-files
  • Ready to go, try the following:

    dsinfluxlogger --defaultconfig >config.toml
    dsinfluxlogger -vv config.toml

Remember to activate your virtualenv whenever working on the repo, this is needed because pylint and mypy pre-commit hooks use the “system” python for now (because reasons).

Running “pre-commit run –all-files” and “py.test -v” regularly during development and especially before committing will save you some headache.

Metadata

Release files for dsinfluxlogger 0.5.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for dsinfluxlogger 0.5.0
File Size Uploaded Provenance Actions
dsinfluxlogger-0.5.0.tar.gz 6.9 kB Without provenance Details for dsinfluxlogger-0.5.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for dsinfluxlogger 0.5.0
File Interpreter ABI Platform Actions
dsinfluxlogger-0.5.0-py3-none-any.whl Python 3 none any Details for dsinfluxlogger-0.5.0-py3-none-any.whl

Total release size: 14.3 kB

Release files / dsinfluxlogger-0.5.0.tar.gz

Download URL dsinfluxlogger-0.5.0.tar.gz
Size 6.9 kB
Tags Source
SHA-256 checksum
How to use checksums
85e33aa507db73aa11cc15434f2b78633e04b80f801cec18119b5b9f630474aa
BLAKE2b-256 checksum
How to use checksums
00cc292fc495780da5b8dc1a83775cecdf80a332e11029435bb7ec5ca9227335
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/3.5.0 importlib_metadata/4.8.2 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12

Release files / dsinfluxlogger-0.5.0-py3-none-any.whl

Download URL dsinfluxlogger-0.5.0-py3-none-any.whl
Size 7.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
5347fbff81916c79dd2bedb5c98bc458d9bbdacb845784c7c8efd98d52a17be4
BLAKE2b-256 checksum
How to use checksums
ecebb169f8f682489624256d4d9f5a4f0fa7b748f90a16775dc346d0a2fe9fd5
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/3.5.0 importlib_metadata/4.8.2 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12

Security for dsinfluxlogger 0.5.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page