Pytest Streaming

Augmenting and simplifying the flow of testing streaming applications including pulsar, google pubsub, kafka, and nats.

Documentation

PyTest streaming documentation can be found here.

Description

This PyTest plugin makes it easy to test streaming functionality (unit, integration, test, etc) through seamless and easy to use decorators. The plugin currently supports bootstrapping of resources for the following technologies

1. Apache Pulsar
2. Google PubSub
3. [Coming soon] Kafka
4. [Coming soon] Nats
5. [Coming soon] Kinesis

The only setup the user needs to take is to have these running locally (ideally via Docker). Docker compose files are provided for each of these, with formal documentation on how to setup everything easily in CI/CD as well.

Quick start

To get started, you should reference the extensive documentation page found here.

For a quick start, you can follow these steps

Install pytest-streaming

pip install pytest-streaming

Utilize the decorator for the streaming technology of your choice

If you're using docker compose, adding this for pulsar will get you up and running end to end (docker-compose.yml)

services:
  pulsar:
    image: apachepulsar/pulsar:latest
    container_name: pulsar
    ports: ['6650:6650', '8080:8080']
    environment:
    - PULSAR_STANDALONE_ENABLED=true
    - allowAutoTopicCreation=true
    command: [bin/pulsar, standalone]

Now you can run: docker compose up -d to have pulsar locally bootstrapped.

Lastly, you can simply create test specific topics

class TestPulsarProducer:
    @pytest.mark.pulsar(topics=["test-topic1", "test-topic2"])
    def test_pubsub_marker_topic_creation_base(self) -> None:
        # these pulsar topics are now available and completely clean
        ...

Everything will be created and cleaned up for you by default on every test run. Each decorator has a suite of customizable features. Read the documentation to see other parameters you can use and how you can create topics for global use (integration tests).

Adding to your CI/CD

Running this in CI/CD is as simple and ensuring pulsar (or other streaming choice) is running in the network. For the example above, adding this step to your GitHub action will yield all the setup required

- name: run docker-compose
    run: |
    docker compose up -d
    sleep 5s

Examples

A suite of examples which are provided and ran within the test suite directly to verify functionality. These can be found under the examples directory at the root level of this project.