pspf 0.1.0a1

Python Stream Processing Framework

Python Stream Processing Framework (PSPF)

PSPF is a lightweight, high-performance stream processing framework for Python. It provides Kafka-like semantics (partitions, offsets, consumer groups, exactly-once processing) without requiring Kafka or JVM infrastructure.

It is designed for building event-driven applications, event sourcing systems, and data pipelines that need to be robust, replayable, and easy to deploy.

Note: While the Quick Start below uses Valkey (Redis), PSPF is backend-agnostic! You can also use Kafka, or the local-only Memory and File backends for testing without Docker.

Key Features

• Valkey-Based Streams: Production-ready stream processing using Valkey (or Redis) as the message broker.
• Kafka-like Semantics: Partitions, consumer groups, and high-performance throughput.
• Exactly-Once Semantics: Idempotent processing with built-in deduplication.
• Partitioned Concurrency: Automatic load balancing across consumer instances.
• Built-in Observability: Prometheus metrics, Grafana dashboards, and Admin API.
• Native Event Log (Preview): Lightweight file-based log for local-only use cases.

Installation

pip install .

Quick Start: User Signups

PSPF makes it easy to handle high-volume event streams. Here is a simple processor that handles user signups.

1. Requirements

Ensure you have Valkey (or Redis) running:

docker run -d -p 6379:6379 valkey/valkey:latest

2. Run the Demo

The included valkey_demo.py example demonstrates a producer and consumer running together.

python examples/valkey_demo.py

3. Verification

You will see logs showing events being produced and then consumed with their respective offsets.

How It Works

1. Producers emit events to a Stream using a specific Backend (e.g., ValkeyStreamBackend).
2. Workers consume from the stream in Consumer Groups, sharing the load across multiple instances.
3. Processors handle message batches, providing built-in retries, deduplication, and dead-letter routing.
4. Observability is baked in; every processed message updates metrics and traces.

Project Structure

pspf/
├── connectors/   # Backend implementations (Valkey, Kafka, Memory)
├── processing/   # Core logic (BatchProcessor, DLO, Retries)
├── state/        # Stateful processing backends (SQLite, etc.)
└── stream.py     # Main Stream facade

Check out the Tutorial for a deeper dive.

Documentation

• Architecture Guide
• Observability Guide
• Roadmap
• Contributing

About the Author

PSPF™ was designed and developed by Joseph Hall.

I built this framework to solve specific challenges in stream processing—focusing on low-latency data handling, simplified deployments without JVM overhead, and ease of use. While my current professional background is in high-volume customer success and sales, my passion lies in engineering robust, scalable Python tools.

🚀 Let's Connect

I am currently looking for my next challenge in Software Engineering, Data Engineering, or DevOps. If you’re looking for a developer who understands both clean code and how to communicate with stakeholders, let’s talk!

• GitHub: https://github.com/JosephMattHall
• Email: [e-mail]