Skip to main content

Muffin-Kafka

Muffin-Kafka is an Apache Kafka integration plugin for the Muffin web framework, built on top of aiokafka.

Tests Status PYPI Version Python Versions


Features

  • Async Kafka integration using aiokafka
  • Single or batch message consumption — stream messages one-by-one or read in batches
  • Per-topic task model — each topic is consumed in an isolated asyncio task
  • Simple handler registration using @plugin.handle_topics(...)
  • Manual or auto-commit support, custom group IDs
  • Producer support via send
  • Built-in monitoring with offsets, lag, and poll delay
  • Healthcheck manage command for liveness probes and observability
  • Optional error handler via @plugin.handle_error(...)

Requirements

  • Python ≥ 3.11
  • Muffin ≥ 0.71
  • Kafka cluster or broker (local or cloud)

Installation

pip install muffin-kafka

Usage

    from muffin import Application
    from muffin_kafka import Kafka

    app = Application("example")

    # Initialize the plugin with config options
    kafka = Kafka(app, bootstrap_servers="localhost:9092", produce=True, listen=True)

Registering Handlers

Use @kafka.handle_topics(...) to register a handler for specific Kafka topics:

    @kafka.handle_topics("events.user", "events.auth")
    async def handle_event(message):
        data = message.value.decode()
        print("Received:", data)

You can also register a global error handler:

    @kafka.handle_error
    async def on_error(exc):
        print("Kafka error:", exc)

Sending Messages

You can send messages to Kafka topics using the send method:

    # Send a message
    await kafka.send("events.user", {"action": "signup"}, key="user123")

Listening (kafka-listen)

Start consuming messages using the kafka-listen management command:

    # Listen to all registered handlers
    muffin myapp kafka-listen

    # Listen to specific topics only
    muffin myapp kafka-listen events.user events.order

    # Override group id for this run
    muffin myapp kafka-listen --group-id=workers-v2

    # Enable monitoring with custom interval
    muffin myapp kafka-listen --monitor --monitor-interval=30

    # Batch mode (process messages in batches)
    muffin myapp kafka-listen --batch-size=100

CLI Options:

Option Description
topics Specific topics to listen (optional, defaults to all registered handlers)
--group-id Override consumer group ID
--monitor Enable built-in monitoring
--monitor-interval Monitoring interval in seconds (default: 60)
--batch-size Read messages in batches (uses getmany())

Healthcheck

Run the kafka-healthcheck management command to check consumer lag:

    muffin myapp kafka-healthcheck

Or programmatically via Muffin's manage command:

    # Check specific topics
    await app.manage.commands["kafka-healthcheck"]("events.user", max_lag=1000)

Monitoring

If monitor=True is passed, the plugin will log:

  • Committed offsets
  • Latest offsets
  • Poll timestamps
  • Per-partition lag and delay

This data can be extended for Prometheus/Grafana metrics or alerting.

Configuration Options

You can pass configuration options either as keyword arguments to the plugin:

kafka = Kafka(app, bootstrap_servers="localhost:9092", produce=True)

Or set them via Muffin's config system (e.g. .env, YAML):

"KAFKA_BOOTSTRAP_SERVERS": "localhost:9092",
"KAFKA_PRODUCE": True,

Supported Options

Option Type Default Description
bootstrap_servers str "localhost:9092" Kafka broker connection string
group_id str None Kafka consumer group ID
client_id str "muffin" Kafka client ID
produce bool False Enable Kafka producer
listen bool True Enable consumers (message listening)
monitor bool False Enable internal consumer monitor
batch_size int None Read messages in batches (uses getmany())
setup_commands bool True Register management commands (kafka-healthcheck, kafka-listen)
monitor_interval int 60 Monitor frequency in seconds
auto_offset_reset str "earliest" Where to start if no committed offset
enable_auto_commit bool False Auto-commit offsets after each message/batch. When False, the plugin commits manually after all handlers succeed.
max_poll_records int None Max records to poll in one batch
request_timeout_ms int 30000 Request timeout
retry_backoff_ms int 1000 Retry interval on failure
security_protocol str "PLAINTEXT" Kafka protocol (SSL, SASL_PLAINTEXT, …)
sasl_mechanism str "PLAIN" SASL auth mechanism
sasl_plain_username str None SASL auth user
sasl_plain_password str None SASL auth password
ssl_cafile str None Path to trusted CA certs

Bug Tracker

Found a bug or have a feature request? Please open an issue at: https://github.com/klen/muffin-kafka/issues


Contributing

Pull requests are welcome! Development happens here: https://github.com/klen/muffin-kafka


License

MIT – See LICENSE for full details.

Metadata

Release files for muffin-kafka 3.1.6

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for muffin-kafka 3.1.6
File Size Uploaded Provenance Actions
muffin_kafka-3.1.6.tar.gz 10.2 kB Without provenance Details for muffin_kafka-3.1.6.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for muffin-kafka 3.1.6
File Interpreter ABI Platform Actions
muffin_kafka-3.1.6-py3-none-any.whl Python 3 none any Details for muffin_kafka-3.1.6-py3-none-any.whl

Total release size: 22.6 kB

Release files / muffin_kafka-3.1.6.tar.gz

Download URL muffin_kafka-3.1.6.tar.gz
Size 10.2 kB
Tags Source
SHA-256 checksum
How to use checksums
ae259592ce299178b817bbed5429906f52392f46d8a21e74842a697e0f8aa1ff
BLAKE2b-256 checksum
How to use checksums
aebd1507b3ad34cf61780b62ebb5f9a822bdd12bf0e005cf73f6a94a94e7c2a4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / muffin_kafka-3.1.6-py3-none-any.whl

Download URL muffin_kafka-3.1.6-py3-none-any.whl
Size 12.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
6fa7220df19b6200ddef82190443bd3d1cad0b64899085d09dde07b6194cbcb1
BLAKE2b-256 checksum
How to use checksums
7346ebdb0ef708f8a39c3e5c0129ce3c32cc02b8d411bcdcb7ed399a2e517ddf
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for muffin-kafka 3.1.6

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

3.1.6 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page