redis-stream-logger 0.1.2

High-performance Redis Stream logging handler with Cython compilation

Redis Stream Logger (Cython-compiled)

High-performance Redis Stream logging handler compiled with Cython for production use.

Features

✨ Cython Compiled — C-extension for 10-50x performance boost
⚡ Non-blocking — Short socket timeouts, best-effort delivery
📊 Structured Logs — JSON payloads with labels and context
🔄 Stream-based — Redis Streams for log ordering and consumer groups
🔁 Optional Fallback — File-based fallback on Redis failure
📈 Metrics — Prometheus-compatible metrics via loki_worker

Installation

From Wheel (Pre-compiled, fastest)

pip install dist/redis_stream_logger-0.1.0-cp312-cp312-linux_x86_64.whl

From Source

pip install dist/redis_stream_logger-0.1.0.tar.gz

From Package Index

pip install redis-stream-logger

Quick Start

from redis_stream_logger import get_logger, RedisAppLogger

# Create logger
logger = RedisAppLogger(get_logger(
    name="myapp",
    redis_url="redis://localhost:6379/0",
    default_labels={"app": "myapp", "env": "prod"}
))

# Log with labels and context
logger.info(
    "User login successful",
    labels={"component": "auth", "action": "login"},
    ctx={"user_id": 12345, "ip": "192.168.1.1"}
)

logger.error(
    "Database connection failed",
    labels={"component": "db"},
    ctx={"error": "connection timeout", "retry_count": 3}
)

Log Format

Logs are stored as compact JSON in Redis Stream:

{
  "ts": "1761796229212145492",
  "level": "info",
  "msg": "User login successful",
  "labels": {
    "app": "myapp",
    "env": "prod",
    "component": "auth",
    "action": "login"
  },
  "ctx": {
    "user_id": 12345,
    "ip": "192.168.1.1"
  }
}

API Reference

get_logger(name, redis_url, default_labels, enable_fallback_file, level)

Configure and return a logger with RedisStreamHandler.

Parameters:

• name (str) — Logger name
• redis_url (str) — Redis connection URL (default: redis://localhost:6379/0)
• default_labels (dict) — Low-cardinality labels attached to all logs
• enable_fallback_file (bool) — Enable file fallback on Redis failure
• fallback_path (str) — Path to fallback log file
• level (int) — Logging level (default: logging.INFO)

Returns: logging.Logger with RedisStreamHandler

RedisAppLogger(logger)

Thin wrapper to standardize passing labels and context.

Methods:

• debug(msg, labels=None, ctx=None)
• info(msg, labels=None, ctx=None)
• warning(msg, labels=None, ctx=None)
• error(msg, labels=None, ctx=None)
• exception(msg, labels=None, ctx=None) — Includes stack trace

Performance

Cython vs Pure Python:

• Compilation overhead: ~5ms first call
• Per-call speedup: 10-50x faster JSON serialization
• Memory: ~2.5MB compiled module

Recommended Settings:

• BATCH_SIZE: 1000 (per batch to Loki)
• FLUSH_INTERVAL_MS: 250ms
• timeout_ms: 5ms (non-blocking)

Integration with loki_worker

The companion Go worker (loki_worker) reads from Redis Stream and pushes to Loki:

# Build worker
cd loki_worker && go build -o loki_worker

# Run with environment variables
export REDIS_STREAM_KEY="logs:stream"
export LOKI_URL="https://loki.example.com/loki/api/v1/push"
export LOKI_BASIC_USER="admin"
./loki_worker

Check health:

curl http://localhost:9090/readyz    # Readiness
curl http://localhost:9090/metrics   # Prometheus metrics

Building from Source

Requirements

• Python 3.7+
• Cython >= 0.29
• GCC/Clang compiler

Build Steps

# Install build dependencies
pip install -r requirements.txt
pip install cython wheel build

# Compile to C extension
python setup.py build_ext --inplace

# Build wheel (for your Python version)
python -m build --wheel

# Build source distribution
python -m build --sdist

Rebuild for Different Python Version

# Python 3.10
/usr/bin/python3.10 setup.py build_ext --inplace --force
/usr/bin/python3.10 -m build --wheel

Testing

python quickstart.py
python usage_examples.py

Troubleshooting

"ModuleNotFoundError: No module named 'redis'"

pip install redis>=4.0.0

Logs not reaching Redis

• Check Redis connection: redis-cli ping
• Check fallback log: /var/log/app.log
• Monitor worker: curl http://localhost:9090/metrics

Performance Issues

• Reduce timeout_ms for async scenarios
• Increase BATCH_SIZE in loki_worker
• Check Redis memory usage: redis-cli info memory

License

MIT

Author

TechMaju — [e-mail]