Muffin-Kafka
Muffin-Kafka is an Apache Kafka integration plugin
for the Muffin web framework, built on top of aiokafka.
Features
- Async Kafka integration using
aiokafka - Single or batch message consumption — stream messages one-by-one or read in batches
- Per-topic task model — each topic is consumed in an isolated asyncio task
- Simple handler registration using
@plugin.handle_topics(...) - Manual or auto-commit support, custom group IDs
- Producer support via
send - Built-in monitoring with offsets, lag, and poll delay
- Healthcheck manage command for liveness probes and observability
- Optional error handler via
@plugin.handle_error(...)
Requirements
- Python ≥ 3.11
- Muffin ≥ 0.71
- Kafka cluster or broker (local or cloud)
Installation
pip install muffin-kafka
Usage
from muffin import Application
from muffin_kafka import Kafka
app = Application("example")
# Initialize the plugin with config options
kafka = Kafka(app, bootstrap_servers="localhost:9092", produce=True, listen=True)
Registering Handlers
Use @kafka.handle_topics(...) to register a handler for specific Kafka topics:
@kafka.handle_topics("events.user", "events.auth")
async def handle_event(message):
data = message.value.decode()
print("Received:", data)
You can also register a global error handler:
@kafka.handle_error
async def on_error(exc):
print("Kafka error:", exc)
Sending Messages
You can send messages to Kafka topics using the send method:
# Send a message
await kafka.send("events.user", {"action": "signup"}, key="user123")
Listening (kafka-listen)
Start consuming messages using the kafka-listen management command:
# Listen to all registered handlers
muffin myapp kafka-listen
# Listen to specific topics only
muffin myapp kafka-listen events.user events.order
# Override group id for this run
muffin myapp kafka-listen --group-id=workers-v2
# Enable monitoring with custom interval
muffin myapp kafka-listen --monitor --monitor-interval=30
# Batch mode (process messages in batches)
muffin myapp kafka-listen --batch-size=100
CLI Options:
| Option | Description |
|---|---|
topics |
Specific topics to listen (optional, defaults to all registered handlers) |
--group-id |
Override consumer group ID |
--monitor |
Enable built-in monitoring |
--monitor-interval |
Monitoring interval in seconds (default: 60) |
--batch-size |
Read messages in batches (uses getmany()) |
Healthcheck
Run the kafka-healthcheck management command to check consumer lag:
muffin myapp kafka-healthcheck
Or programmatically via Muffin's manage command:
# Check specific topics
await app.manage.commands["kafka-healthcheck"]("events.user", max_lag=1000)
Monitoring
If monitor=True is passed, the plugin will log:
- Committed offsets
- Latest offsets
- Poll timestamps
- Per-partition lag and delay
This data can be extended for Prometheus/Grafana metrics or alerting.
Configuration Options
You can pass configuration options either as keyword arguments to the plugin:
kafka = Kafka(app, bootstrap_servers="localhost:9092", produce=True)
Or set them via Muffin's config system (e.g. .env, YAML):
"KAFKA_BOOTSTRAP_SERVERS": "localhost:9092",
"KAFKA_PRODUCE": True,
Supported Options
| Option | Type | Default | Description |
|---|---|---|---|
bootstrap_servers |
str |
"localhost:9092" |
Kafka broker connection string |
group_id |
str |
None |
Kafka consumer group ID |
client_id |
str |
"muffin" |
Kafka client ID |
produce |
bool |
False |
Enable Kafka producer |
listen |
bool |
True |
Enable consumers (message listening) |
monitor |
bool |
False |
Enable internal consumer monitor |
batch_size |
int |
None |
Read messages in batches (uses getmany()) |
setup_commands |
bool |
True |
Register management commands (kafka-healthcheck, kafka-listen) |
monitor_interval |
int |
60 |
Monitor frequency in seconds |
auto_offset_reset |
str |
"earliest" |
Where to start if no committed offset |
enable_auto_commit |
bool |
False |
Auto-commit offsets after each message/batch. When False, the plugin commits manually after all handlers succeed. |
max_poll_records |
int |
None |
Max records to poll in one batch |
request_timeout_ms |
int |
30000 |
Request timeout |
retry_backoff_ms |
int |
1000 |
Retry interval on failure |
security_protocol |
str |
"PLAINTEXT" |
Kafka protocol (SSL, SASL_PLAINTEXT, …) |
sasl_mechanism |
str |
"PLAIN" |
SASL auth mechanism |
sasl_plain_username |
str |
None |
SASL auth user |
sasl_plain_password |
str |
None |
SASL auth password |
ssl_cafile |
str |
None |
Path to trusted CA certs |
Bug Tracker
Found a bug or have a feature request? Please open an issue at: https://github.com/klen/muffin-kafka/issues
Contributing
Pull requests are welcome! Development happens here: https://github.com/klen/muffin-kafka
License
MIT – See LICENSE for full details.
Metadata
Release files for muffin-kafka 3.1.6
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| muffin_kafka-3.1.6.tar.gz | 10.2 kB | Without provenance | Details for muffin_kafka-3.1.6.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| muffin_kafka-3.1.6-py3-none-any.whl | Python 3 | none | any | Details for muffin_kafka-3.1.6-py3-none-any.whl |
Total release size: 22.6 kB
Release files / muffin_kafka-3.1.6.tar.gz
| Download URL | muffin_kafka-3.1.6.tar.gz |
|---|---|
| Size | 10.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ae259592ce299178b817bbed5429906f52392f46d8a21e74842a697e0f8aa1ff
|
|
BLAKE2b-256 checksum How to use checksums |
aebd1507b3ad34cf61780b62ebb5f9a822bdd12bf0e005cf73f6a94a94e7c2a4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / muffin_kafka-3.1.6-py3-none-any.whl
| Download URL | muffin_kafka-3.1.6-py3-none-any.whl |
|---|---|
| Size | 12.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
6fa7220df19b6200ddef82190443bd3d1cad0b64899085d09dde07b6194cbcb1
|
|
BLAKE2b-256 checksum How to use checksums |
7346ebdb0ef708f8a39c3e5c0129ce3c32cc02b8d411bcdcb7ed399a2e517ddf
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for muffin-kafka 3.1.6
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.