Skip to main content

AsyncIO eventloop helpers and Abstract Base Classes for making services that use ZMQ nice, easy and DRY

Usage

Use the CookieCutter template at https://gitlab.com/advian-oss/python-datastreamserviceapp_template

You can also take a look at src/datastreamservicelib/console.py for some very simple test examples.

Docker

For more controlled deployments and to get rid of “works on my computer” -syndrome, we always make sure our software works under docker.

It’s also a quick way to get started with a standard development environment.

SSH agent forwarding

We need buildkit:

export DOCKER_BUILDKIT=1

And also the exact way for forwarding agent to running instance is different on OSX:

export DOCKER_SSHAGENT="-v /run/host-services/ssh-auth.sock:/run/host-services/ssh-auth.sock -e SSH_AUTH_SOCK=/run/host-services/ssh-auth.sock"

and Linux:

export DOCKER_SSHAGENT="-v $SSH_AUTH_SOCK:$SSH_AUTH_SOCK -e SSH_AUTH_SOCK"

Creating a development container

Build image, create container and start it:

docker build --ssh default --target devel_shell -t datastreamservicelib:devel_shell .
docker create --name datastreamservicelib_devel -v `pwd`":/app" -it -v /tmp:/tmp `echo $DOCKER_SSHAGENT` datastreamservicelib:devel_shell
docker start -i datastreamservicelib_devel

pre-commit considerations

If working in Docker instead of native env you need to run the pre-commit checks in docker too:

docker exec -i datastreamservicelib_devel /bin/bash -c "pre-commit install"
docker exec -i datastreamservicelib_devel /bin/bash -c "pre-commit run --all-files"

You need to have the container running, see above. Or alternatively use the docker run syntax but using the running container is faster:

docker run -it --rm -v `pwd`":/app" datastreamservicelib:devel_shell -c "pre-commit run --all-files"

Test suite

You can use the devel shell to run py.test when doing development, for CI use the “tox” target in the Dockerfile:

docker build --ssh default --target tox -t datastreamservicelib:tox .
docker run -it --rm -v `pwd`":/app" `echo $DOCKER_SSHAGENT` datastreamservicelib:tox

Production docker

There’s a “production” target as well for quick running of testsubscriber and testpublisher

docker build –ssh default –target production -t datastreamservicelib:latest . docker run –rm -it -v /tmp:/tmp datastreamservicelib:latest testpublisher -s ipc:///tmp/test_pub.sock -t foo docker run –rm -it -v /tmp:/tmp datastreamservicelib:latest testsubscriber -s ipc:///tmp/test_pub.sock -t foo

Note that on non-linux platforms the IPC sockets may not as expected between host and container over volume mounts.

Local Development

TLDR:

  • Create and activate a Python 3.9 virtualenv (assuming virtualenvwrapper):

    mkvirtualenv -p `which python3.9` my_virtualenv
  • change to a branch:

    git checkout -b my_branch
  • install Poetry: https://python-poetry.org/docs/#installation

  • Install project deps and pre-commit hooks:

    poetry install
    pre-commit install
    pre-commit run --all-files
  • Ready to go, try the following:

    testpublisher --help
    testsubscriber --help

Use Python 3.9 for development since it’s the lowest supported version so you don’t accidentally use features only available in higher versions and then have to re-do everything when CI tests fail on 3.9.

Remember to activate your virtualenv whenever working on the repo, this is needed because pylint and mypy pre-commit hooks use the “system” python for now (because reasons).

Running “pre-commit run –all-files” and “py.test -v” regularly during development and especially before committing will save you some headache.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

datastreamservicelib-1.15.0.tar.gz (12.8 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

datastreamservicelib-1.15.0-py3-none-any.whl (14.7 kB view details)

Uploaded Python 3

File details

Details for the file datastreamservicelib-1.15.0.tar.gz.

File metadata

  • Download URL: datastreamservicelib-1.15.0.tar.gz
  • Upload date:
  • Size: 12.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/7.0.0 CPython/3.14.7

File hashes

Hashes for datastreamservicelib-1.15.0.tar.gz
Algorithm Hash digest
SHA256 4483e9682ffceae1f9760226fdc6cecb406666ddac1cb164e592ff89e99854c9
MD5 5b0ddada1e4f7754d2880067afb5c2e6
BLAKE2b-256 73f70379d9b3b7081a0fef36dc778395a12e807697ad0d86edf61f2b0aedc81f

See more details on using hashes here.

File details

Details for the file datastreamservicelib-1.15.0-py3-none-any.whl.

File metadata

File hashes

Hashes for datastreamservicelib-1.15.0-py3-none-any.whl
Algorithm Hash digest
SHA256 a5af68533db7ba55a864e84045dd6305ae9ef26d2e676131ffedea5273bd9473
MD5 3d628e3de7a8dbe43643c1c823fd4299
BLAKE2b-256 65d40401fff1fe07ae8930f20a818e39dd6827f3ae29dad5b377aca31b1135d7

See more details on using hashes here.

Release history Release notifications | RSS feed

This release

1.15.0 This release

2 files

1.14.0

2 files

1.13.0

2 files

1.12.2

2 files

1.12.1

2 files

1.12.0

2 files

1.11.0

2 files

1.10.1

2 files

1.10.0

2 files

1.9.1

2 files

1.8.3

2 files

1.8.2

2 files

1.8.1

1 file

1.8.0

1 file

1.7.0

1 file

1.6.0

1 file

1.5.2

1 file

1.5.1

1 file

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page