Caducus

Caducus helps operations teams understand what is going wrong right now across logs, alerts, dead-letter queues, and other operational event streams.

It is a CLI-first system for collecting timestamped operational events, normalizing them into a canonical schema, storing them as plain JSON, and using semantic reinforcement memory to surface recurring patterns, fresh anomalies, and just-in-time context during incidents.

Why Caducus Exists

Operational signals are scattered across many systems:

• CloudWatch logs
• alerting systems
• dead-letter queues
• notifications and incident messages

Each source captures part of the truth, but not the whole picture. Caducus is intended to bring those signals together into one stream of timestamped event records that can be analyzed as a living memory of operational behavior.

The goal is not just to search historical data. The goal is to create a radar for what looks unusual, active, or important now.

How It Works

Caducus is designed around a simple flow:

1. Collect operational events from source systems.
2. Normalize them into canonical event records with text, timestamps, source identity, and generalized metadata.
3. Persist them as JSON files in a Virtuus-backed folder structure.
4. Analyze event groups using Biblicus reinforcement memory.
5. Surface patterns, anomalies, and context for operators.

This keeps the system inspectable and composable. The underlying data lives in plain folders, not inside a black-box database.

CLI-First MVP

The initial product is a CLI utility.

The MVP is focused on a coherent end-to-end flow:

• collect events from operational sources
• store them in a canonical schema
• run analysis over selected event groups
• inspect recent events and analysis outputs from the command line

Initial source areas for the MVP are:

• CloudWatch Logs
• SQS dead-letter queues
• one alert source

Configuration is intended to be layered through YAML, environment variables, and CLI overrides. Caducus will own collection and orchestration while allowing Biblicus-related analysis settings to flow through the Caducus configuration tree without duplicating Biblicus's schema.

Architecture At A Glance

Caducus is intentionally thin:

• Caducus handles collection, normalization, orchestration, and CLI workflows.
• Virtuus provides file-backed JSON storage and retrieval.
• Biblicus provides semantic reinforcement-memory analysis.

flowchart LR
    sources[OpsSources] --> caducus[Caducus]
    caducus --> events[CanonicalEvents]
    events --> virtuus[VirtuusStorage]
    caducus --> biblicus[BiblicusAnalysis]
    biblicus --> radar[OpsRadar]

Running the demo

Real HDFS data uses component-derived group IDs: each log row’s component becomes hdfs-demo:<component> (e.g. hdfs-demo:dfs.DataNode$DataXceiver). You must use a group ID that exists in your ingested data.

Quick demo (small fixture, no download)

pip install -e ".[reinforcement-memory]"
caducus demo run --input tests/fixtures/demo_hdfs_sample.csv --group-id "hdfs-demo:DataNode" --data-dir /tmp/caducus-demo

The fixture has components DataNode and NameNode, so valid group IDs are hdfs-demo:DataNode and hdfs-demo:NameNode.

Full demo on real HDFS data

1. Install optional deps (Biblicus reinforcement-memory and the datasets library for the download script):

   pip install -e ".[reinforcement-memory]"
   pip install datasets
   

2. Download a subset of the HDFS_v1 dataset:

   python scripts/download_hdfs_demo.py --output demo_data/hdfs_sample.csv --max-rows 10000
   

3. Ingest and list available groups (group IDs come from the CSV component column):

   caducus demo ingest --input demo_data/hdfs_sample.csv --data-dir ./caducus-data
   caducus groups --data-dir ./caducus-data
   

4. Run analysis for one of the listed group IDs:

   caducus analyze --group-id "hdfs-demo:dfs.DataNode$DataXceiver" --data-dir ./caducus-data
   

   Or do ingest and analyze in one step (use a group ID that exists in the CSV):

   caducus demo run --input demo_data/hdfs_sample.csv --group-id "hdfs-demo:dfs.DataNode$DataXceiver" --data-dir ./caducus-data
   

Releases

Caducus uses python-semantic-release with Conventional Commits.

Use commit messages like:

• feat: add CloudWatch collector checkpointing
• fix: quote group IDs containing dollar signs in docs
• feat!: change canonical event schema

Release behavior:

• feat: triggers a minor release
• fix: triggers a patch release
• feat!: or a BREAKING CHANGE: footer triggers a major release

The release workflow lives in .github/workflows/release.yml and runs on pushes to main. It will:

1. Determine the next version from commit messages.
2. Update project.version in pyproject.toml and src/caducus/__init__.py.
3. Generate CHANGELOG.md, create a tag, and create a GitHub Release.
4. Publish the built distributions to PyPI.

PyPI publishing is configured for GitHub Actions trusted publishing. Before the first live release, configure the caducus project on PyPI to trust this repository's release.yml workflow.

Roadmap

Caducus is intended to grow beyond the initial CLI foundation over time.

Planned directions include:

• broader source integrations across operational systems
• deeper analysis of concepts and entities derived from operational activity
• richer incident context and root-cause workflows
• a future web UI and embeddable components for other applications

Repository Direction

This repository is being built outside-in. Product definition and behavior specifications come first, followed by the minimum implementation needed to satisfy them.