kafka-agent-bridge

kafka-agent-bridge is a small Python library for developers who already know LangGraph or CrewAI and want Kafka to be the event bus without hand-rolling consumers, retries, idempotency, result publishing, and dead-letter handling. It fills the space between heavyweight streaming-agent platforms and raw framework glue code.

Quickstart

pip install -e '.[langgraph,redis,dev]'
docker compose up

from kafka_agent_bridge import KafkaAgentBridge
from kafka_agent_bridge.adapters import LangGraphAdapter

adapter = LangGraphAdapter(graph=compiled_graph)

bridge = KafkaAgentBridge(
    bootstrap_servers="localhost:9092",
    consumer_group="reports-agent",
    idempotency_backend="redis://localhost:6379",
)
bridge.register(
    topic="reports.requested",
    handler=adapter,
    adapter="langgraph",
    output_topic="reports.completed",
    dlq_topic="reports.dlq",
)
await bridge.start()

Run the included LangGraph example:

make dev
make example-fraud

Kafka UI is available at http://localhost:8080.

Architecture

Kafka topic
    |
    v
KafkaAgentBridge
    |-- JsonDeserializer -> EventEnvelope
    |-- AgentRouter -> topic glob, X-Agent-Type, predicate
    |-- IdempotencyGuard -> Redis or in-memory TTL cache
    |-- LangGraphAdapter / CrewAIAdapter / custom handler
    |-- KafkaResultProducer -> output topic
    |
    +-- DeadLetterQueue -> *.dlq after retries

LangGraph graphs can use KafkaCheckpointer:
thread_id -> compacted Kafka topic agent.checkpoints

What Is Included

• KafkaAgentBridge async runtime with multiple topic registrations.
• AgentRouter with glob, header, and predicate routing.
• KafkaResultProducer with JSON serialization and configurable batching.
• EventEnvelope Pydantic model for consumed messages.
• Redis-backed or in-memory idempotency guard.
• Retry and dead-letter publishing.
• LangGraphAdapter, CrewAIAdapter, and KafkaCheckpointer.
• Docker Compose with Kafka, Redis, and Kafka UI.

Avro and Schema Registry support are intentionally stubbed for v0.2.

Comparison

The intent is not to hide Kafka forever. The bridge makes the default path small, then leaves explicit escape hatches for teams that need partitioning, offset, batching, security, or consumer-group tuning.

Local Development

python -m venv venv
venv/bin/pip install -e '.[all,dev]'
make test
make lint

Contributing

Issues and pull requests are welcome. For now, keep contributions focused on small, well-tested improvements: adapter behavior, Kafka operational safety, examples, and documentation. Please include tests for behavior changes and keep public APIs typed and documented.