eaveswatch
Logs, deploys, config and connections, joined to the code that made them, for an AI agent to set up and review. It's designed for hobkin, and works with any agent harness that meets its reviewer contract.
Scanners look at one thing at a time: a diff, a dependency, a log line. The interesting problems are joins: since Tuesday's config change that error is 40× an hour; the box listens on a port nobody mentioned. eaveswatch gathers what's needed to see those and hands it to a reviewer agent. The agent works out what's worth collecting and how to read it. It gets a bar for what counts as a finding, and a ledger that traces every conclusion back to its evidence.
-
No hand-written log config. The reviewer agent runs
discover:- which services run here, and what they run
- which files each one has open for writing
- which ports they listen on and talk to
- which config files they point at
- who writes to the journal
It asks you what you care about, then writes an interpreter spec: JSON that names the sources and says how to read them. That means multi-line entries, timestamps, levels, placeholders for what varies, noise to drop, where versions come from, and which config files to watch.
interpret testreplays the real recent lines through it;interpret applymakes it the active, numbered version. -
No model per line, and no code from the model. The eave runs the active spec's patterns in plain code. Patterns that backtrack forever are caught in testing.
-
Secrets never leave, whatever the spec says. A built-in redaction floor always runs first. It catches private keys, AWS key ids, JWTs, bearer tokens,
password=/token=values,PS_TOKEN/JSESSIONID,user:pass@in URLs and long base64 blobs. A spec can add patterns but never remove them. Credential files and folders (~/.ssh,~/.aws, key files,~/.hobkins...) are refused as sources. -
Same module everywhere, one standard-library Python file (3.9+), three roles:
| Role | Where | What |
|---|---|---|
finial |
one box, all of it | eave, ridge and reviewer together; files to S3 (the archive) and straight to the reviewer |
eave |
every host of a fleet | its interpreter's sources, inventory, flows → the store; answers asks |
ridge |
a fleet's hub | the store → a data folder for the reviewer; asks → the store |
(A finial is the ornament at the very top of a single roof.)
A finial: one box, watched from the inside
As the same user the watched services run as (with those services as user systemd units), run:
pipx install 'eaveswatch[hobkin]' # with hobkin, the reviewer it's designed for
eaveswatch https://openrouter.ai/api/v1 anthropic/claude-sonnet-4.5 sk-or-... # the reviewer's model API
Running as that user lets discovery see the services' own open files, sockets and environment, with no root and no extra capabilities.
The one command asks you three things:
- Which S3 bucket to use, showing the ones it can see.
- Where alerts go: an email address. It sends through the box's
sendmail, or through an SMTP relay if the box has none, and offers a test message. Leave it blank and alerts are kept in the data folder for now. - Which folder a web server here already serves, for the findings page.
Anyone who can reach that site sees the findings. Leave it blank and the
page is in
~/eaveswatch/ridge/report/either way.
Everything else is decided for you:
- Region: the bucket's own.
- Credentials: the instance profile.
- Host: the hostname.
- Config:
~/.config/eaveswatch/finial.json. - State:
~/eaveswatch. It isn't under a dot-folder on purpose: hobkin's safe mode refuses write folders inside~/.something, because those are ways out of the sandbox.
It runs check, installs and starts eaveswatch as a user service of whoever
ran it, turns on lingering (so it runs with nobody logged in), installs the
reviewer's instructions as a hobkin chore, and hands over to hobkin. You end up
in a chat with the reviewer. Run it again later and it reuses the same setup.
--reviewer none sets everything up without starting an agent, for a harness
of your own.
Options:
--bucket NAME,--alert-email ADDR(with--smtp host:portfor a relay),--publish-dir DIR: answers, so nothing is asked--store s3://bucket/prefix, or a folder to try it without S3--env,--host--no-service,--no-chat--force(replace an existing setup)
The reviewer then:
- runs
discover - tells you in a few lines what it found ("a PeopleSoft app server domain HRDEV, a PIA on 8000...")
- proposes what to collect, watch and archive, and what should alert you at once, with a recommendation for each
- asks what you care about
- writes, tests and applies the interpreter, and tells you what's now collected
Discovery runs again every hour. What changed (new units, programs, listeners, logs written but not collected) comes in its digest. It fixes and narrows the interpreter on its own, and asks you before adding a source.
A system of several nodes (an autoscaling group)
Every instance runs the same thing. There's no hub to deploy, and any node can drop out at any moment without the system losing anything, because what the system knows lives in the store:
PREFIX/systems/system=ps-hr-app/
config.json the shared config and the reviewer's model API (never its key)
interpreter/vN.json every version: a node creates vN only if no other node has (the loser is told)
charter.md the reviewer's REVIEW.md
ledger/ findings events, one object each, never overwritten: every node folds the same events
alerts/ sent/ fired alerts; notes already sent (once for the system, however many nodes saw it)
nodes/HOST.json heartbeats lease/ridge.json who does the ridge duty
jobs/ID.json review jobs state/map.json the merged map, for every node's page
- The ridge is a duty, not a box. One node holds a lease (S3's
conditional writes, renewed every tick, taken over when it runs out). The
ridge duty is plain code with no model:
- merge every node's data into the map
- spot what's new across the fleet: a template seen nowhere before, a deploy, a config change, a new host, new connections
- notice nodes that went silent without saying they were leaving (a warning, sent once, and a job)
- turn all of that into review jobs
- Reviews go where the evidence is. A job names the host its evidence came
from, and that host's reviewer claims it first (it has the logs, the
processes and the code right there). If it doesn't within
job_wait_seconds, because it's dead or busy, any node takes it. Claims are renewed while the reviewer works and run out if its node dies, so the job goes back to the others. Alerts come first: a node gives back a routine review to take one. - Each node's reviewer is woken only by the job it holds (
eaveswatch watch). Idle nodes cost nothing. The reviewer finishes witheaveswatch job done ID --note "...", or gives a job back withjob release ID. - Everything spreads within a minute: findings recorded on any node, dismissals, the interpreter applied anywhere, the charter. Every node renders the findings page from the shared state, so a load balancer can send you to any of them.
- An alert is emailed once for the system: the first node to create its
sent/record sends it. A failed unit is per host; an error storm seen by ten nodes is one email. - Asks between nodes go through the store, and the answer comes back to the reviewer that asked.
- A clean stop (scale-in, shutdown) marks the node as left, gives the lease and its job back, and isn't reported as gone.
Every node boots the same way, with no questions, from the same line in your launch template (examples/asg/user-data.sh). Everything comes in by flag, so feed them from whatever variables or secret store you already use:
eaveswatch boot --store s3://my-eaves-bucket/eaveswatch --system ps-hr-app \
--model-url "$YOUR_MODEL_URL" --model "$YOUR_MODEL" --model-key "$YOUR_MODEL_KEY" \
--alert-email ops@example.gov --publish-dir /var/www/html/eaveswatch
- The system is there already: the node joins it. It writes its own config (its own host), and on its first tick pulls the interpreter, alert rules, charter and findings.
- It isn't: the node creates it from the flags and shares its config (the
model API, alerts, publish folder; never the key). Its reviewer then runs
the setup conversation once, by
notifyas well as chat since nobody's attached at boot. You answer by chatting with it (hobkin chat eaveswatch). - Either way it runs
check(sources that aren't there yet don't stop it: the services may still be starting), installs and starts the user service, and starts the reviewer with the key you gave and no chat. Running it again is safe. --collectormakes a collector-only node, which shares data but never leads or reviews.--reviewer nonebrings your own agent harness. The hobkin adapter keeps the key in hobkin's own key file, readable only by that user.
AWS access is the instance's own: eaveswatch uses whatever credentials the box
has (instance profile, environment, ~/.aws) and only needs the bucket name.
cluster in the config tunes the timings: lease_seconds (180),
claim_seconds (600), job_wait_seconds (120), node_gone_seconds (300),
job_keep_days (7).
Running as a service
eaveswatch runs as a systemd user service of whoever launches it, like
hobkin. That's normally the user the watched services run as. setup and
boot install it; you can manage it yourself too:
eaveswatch --config ~/.config/eaveswatch/finial.json service add # install, start now and at boot
eaveswatch --config ~/.config/eaveswatch/finial.json service status # running? since when; its last lines
eaveswatch --config ~/.config/eaveswatch/finial.json service remove # stop it and take it off boot
- The unit:
~/.config/systemd/user/eaveswatch.service, running this same eaveswatch (its full path) with this config. Another config gets its own unit (eaveswatch-<hash>), so two setups on one box never clash. - Lingering is turned on, so it runs with nobody logged in. If the system
won't allow that without a password, it says so and gives the
loginctl enable-linger USERto run. - Failures: a crash is restarted after 30 seconds. systemd gives up after 5
failures in 10 minutes, so a broken config can't spin. A clean stop
(
systemctl --user stop eaveswatch, scale-in) stays stopped, and the node hands its work back first. - Logs: its own output goes to the journal
(
journalctl --user -u eaveswatch). - Overwriting: a unit that already runs a different eaveswatch or config is
left alone unless you pass
--force.bootreplaces it, since it owns it. - The reviewer is its harness's service. For hobkin that's
hobkin@eaveswatch, whichhobkin listshows.
Alerts and the findings page
- Alerts are fast and plain code. The interpreter's
alertsrules (the reviewer proposes them, you agree) are checked every tick. Each has a cooldown and a severity (pageorwarn). There are three kinds:match: N entries matching a template or level in M minutesunit_failed: a named unit has failed, user or systemsilent: a source has written nothing for M minutes
- When one fires, you get a terse email at once. The reviewer is woken (its watch runs every minute, free when quiet) and follows up within minutes with what it means and what to do.
- Everything for a person goes through one queue: fired alerts, the
reviewer's notes (
eaveswatch notify TEXT, which works from its sandbox), and hobkin's own "I've paused". The loop sends it, at mostper_hour(20) an hour. With no channel set, notes are kept inDATA/notify/unsent/.eaveswatch notify --testsends one now. SNS is the likely next channel. - The findings page is rendered by the loop whenever something changes:
open findings with their evidence, recent alerts, digests, hosts and
versions, and what's collected. It's written to
DATA/report/index.html, and copied to the publish folder if there is one. It's static HTML with everything escaped, because it shows text from logs that others write, and eaveswatch serves nothing itself.
The interpreter
{"sources": [
{"name": "appsrv", "path": "/home/psadm2/psft/pt/8.60/appserv/HRDEV/LOGS/APPSRV_*.LOG", "service": "hr-app",
"multiline": {"start": "^PSAPPSRV\\.\\d+ "}, "placeholders": [{"name": "sql", "pattern": "'[^']*'"}]},
{"name": "pia", "path": "/home/psadm2/psft/pt/8.60/webserv/peoplesoft/servers/PIA/logs/PIA_weblogic.log",
"multiline": {"start": "^####<"}, "level": {"pattern": "^####<[^>]*> <(\\w+)>", "map": {"error": "error", "warning": "warn"}},
"timestamp": {"pattern": "^####<(\\w{3} \\d{1,2}, \\d{4} \\d{1,2}:\\d\\d:\\d\\d,\\d+ [AP]M)", "format": "%b %d, %Y %I:%M:%S,%f %p"}},
{"name": "units", "journald": {"user": true, "units": ["psappsrv.service", "pia.service"]}}],
"services": {"hr-app": {"version": {"file": "/home/psadm2/psft/pt/8.60/peopletools.properties", "pattern": "^toolsrelease=(\\S+)"},
"watch": ["/home/psadm2/psft/pt/8.60/appserv/HRDEV/psappsrv.cfg"]}}}
That's the kind of spec a reviewer writes. None of it ships in eaveswatch.
interpret init starts from generic placeholders (times, ids, addresses,
numbers, quoted strings) and levels, and the reviewer adds the rest:
| Field | What it does |
|---|---|
sources |
path (a glob, so date-rolled logs are followed) or journald (user, units, identifiers) |
| per source | service, multiline.start, timestamp (pattern + strptime format), level (pattern + map), placeholders, drop/keep, raw (archive redacted entries), from_start |
services |
git (a checkout's commit) or version (file + pattern), and watch: config files hashed (never read out); a change shows in the digest |
| top level | placeholders, level, redact (extra patterns), watch, spike, warmup_minutes, max_template, max_samples |
eaveswatch interpret init FILE # the starter
eaveswatch interpret test FILE # real recent lines through it: entries, templates, timestamps, slow patterns
eaveswatch interpret apply FILE --note "why" # the next version, active on the eave's next tick
eaveswatch interpret show | rollback [N]
A source's first read starts from the end of what's already there. A file
that appears later, like tomorrow's APPSRV_1115.LOG, is read from its start.
The store
A folder, or s3://bucket/prefix. Hive-style prefixes, so Athena and Glue can
query the archive as it is:
PREFIX/env=prod/kind=packets/dt=2026-11-14/hour=09/host=ps-hr-01/ps-hr-01_20261114T091502Z_7f3a.jsonl.gz
- Time before host: the ridge lists one hour across every host, and each
host's files are still under its own prefix (
*/host=web-3/*). - The name stands alone: host, UTC stamp (sorts as text), a random suffix.
- Kinds:
packets: templates, counts, samples, new and spikinginventory: versions, watched-file hashes, units, the interpreter version, the spoolflowsraw: redacted entries of sources markedraw, which is the archiverequestsandanswers: asksalerts: what fired, and whycheck: probes, which a lifecycle rule can expire
- Systems' shared state sits beside the data, at
PREFIX/systems/system=NAME/(see "A system of several nodes"). - Nothing is lost when the store is away. Every file is spooled to disk
before the eave saves its place in the logs, then sent oldest first. Past
spool_max_mbthe oldestrawfiles go first, and the gap is reported once the store is back. Delivery is at least once. - Late files keep their hour. Every
late_everyseconds the ridge looks backlate_hours. - No boto3: SigV4 is signed in the standard library. Credentials come from the
environment,
~/.aws/credentials, the ECS task role or the instance role.
A fleet
Every host runs an eave (examples/fleet/): plumbing-only
config, with its interpreter in STATE_DIR/interpreter. The hub runs the ridge
sync loop with the store's credentials, and the reviewer agent has none:
ridge watch and every command it runs read and write only the ridge's data
folder. The ridge never gets
kind=raw.
The reviewer contract
eaveswatch never imports an agent. It talks to its reviewer through commands and files, so any harness that does these four things can be the reviewer:
- Wake on
eaveswatch watchabout once a minute, and act only when it prints something. Silence means there's nothing to do, so it costs nothing. - Follow the instructions printed by
eaveswatch reviewer instructions, filled in for this box. - Reach a person through
eaveswatch notify TEXT, for its own notes and for the harness's own trouble (its model refusing, say). - Run sandboxed: write only the data folder, no network, no credentials. The reviewer reads logs, commit messages and requests that attackers can write. eaveswatch keeps its own side safe (redaction before anything leaves a host, nothing that needs credentials, asks to other hosts from a fixed menu), but the sandbox is the harness's job.
eaveswatch reviewer contract prints all of that as JSON, for this box.
hobkin is built in as the adapter (pip install 'eaveswatch[hobkin]').
eaveswatch reviewer install hobkin writes the instructions as a hobkin chore:
its wake, safe mode writing only the data folder, and its alert command into
notify. setup and boot start it. Another harness gets an adapter class
beside HobkinReviewer (in REVIEWERS), or just uses the contract.
What the reviewer does
The instructions (REVIEWER_MD in eaveswatch.py; eaveswatch reviewer instructions prints them) cover setting up, keeping the interpreter right, and
reviewing:
- The map (
DATA/map/): hosts (what runs, versions, watched hashes), edges (listeners and who talks to whom), templates (hosts, versions, totals). - The digest says what changed:
- new hosts, and hosts gone quiet
- deploys
- config changes and interpreter versions
- what changed on the box (from discovery)
- new connections
- spiking and new templates
- answers to its asks
- Asks go to an eave from a fixed menu, never a shell:
ask HOST lines FP,ask HOST inventory,ask HOST flows. - A finding needs evidence, a failure scenario, an action and a confidence. Without all four it's an observation. Before recording one, a brownie tries to knock it down; at most three are recorded per digest.
- The ledger (
DATA/findings.jsonl) is append-only. The same category and location is the same finding (counted again, not duplicated), a dismissal needs a reason and isn't recorded again, and each finding carries its inputs and the charter version.
eaveswatch finding add < finding.json
eaveswatch finding list [--all]
eaveswatch finding status F-1a2b3c4d dismissed --why "behind the VPN only"
Check a box
eaveswatch --config FILE check, as the service's user, tests exactly what
its role does:
- eave: writes under its own host and lists its asks
- ridge: lists and reads what it syncs
- finial: only writes
It also reports where the credentials came from, the clock against S3's, the active interpreter, every source (globs, permissions, journald), every service's version and the watched files. Then it shows a preview of recent lines as templates, with how many values were redacted. Each failure says what to change, and the exit code is 1 on any failure.
Config
Plumbing only; what to collect is the interpreter's.
| Setting | Default | |
|---|---|---|
role |
eave |
eave, ridge or finial |
env |
prod |
the env= prefix |
host |
hostname | the host= prefix |
system |
host | the name a system's nodes share (an ASG's): its shared state in the store |
cluster |
a system's timings: lease_seconds, claim_seconds, job_wait_seconds, node_gone_seconds, job_keep_days |
|
store |
a folder, or s3://bucket/prefix |
|
region, endpoint |
$AWS_REGION |
S3 region; an S3-compatible endpoint (path-style) |
state_dir |
/var/lib/eaveswatch |
offsets, cursors, baselines, the spool; the ridge's data default |
interpreter |
the interpreter folder; default RIDGE_DATA/interpreter for a finial (the reviewer writes it), else STATE_DIR/interpreter |
|
every |
60 |
seconds between ticks |
inventory_every |
3600 |
inventory and flows: on change, and at least this often |
discover_every |
3600 |
a finial's look at what runs here |
spool_max_mb |
512 |
the most kept waiting for the store |
notify |
where alerts and notes go: {"email", "sendmail"} or {"email", "smtp": "host:port"}, plus from and per_hour |
|
publish |
{"dir": PATH}: a folder a web server already serves, for the findings page |
|
ridge |
data, envs, silent_after, late_hours (24), late_every (600) |
Tests
python3 tests/test_eaveswatch.py. No AWS and no cost. It covers:
- the redaction floor
- the interpreter: validation, credentials refused, multi-line WebLogic records, levels, timestamps, a catastrophic pattern caught
- discovery of a real process here: its open log and its port
- date-rolled logs
- versions and watched files
- the layout and AWS's own SigV4 test vector
- full round trips on a folder, a fake S3 that checks every signature, and a finial
- the spool through an outage
- check, the ledger and the chore
Metadata
Release files for eaveswatch 0.8.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| eaveswatch-0.8.0.tar.gz | 111.1 kB | With provenance | Details for eaveswatch-0.8.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| eaveswatch-0.8.0-py3-none-any.whl | Python 3 | none | any | Details for eaveswatch-0.8.0-py3-none-any.whl |
Total release size: 192.0 kB
Release files / eaveswatch-0.8.0.tar.gz
| Download URL | eaveswatch-0.8.0.tar.gz |
|---|---|
| Size | 111.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
6f76bbfeb3b7af92d5a0e9bb00b2c537978624ccfae3a71014968a2a33036762
|
|
BLAKE2b-256 checksum How to use checksums |
ff249d92e50330cbb4cee197935cfb3558f04f6b2fbb39b541eefbb903b0f4d9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.
Transparency logRelease files / eaveswatch-0.8.0-py3-none-any.whl
| Download URL | eaveswatch-0.8.0-py3-none-any.whl |
|---|---|
| Size | 80.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
47cc0f4370b2fe70216fef69e6827bafaeda1fe80f1047f668e3cb83028c28ff
|
|
BLAKE2b-256 checksum How to use checksums |
6bb192c881034aa9a75c9565b2e6c1a339c84b89e57a6d6eebb702eb421a11b9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.
Transparency logSecurity for eaveswatch 0.8.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for eaveswatch-0.8.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.