sentry-v2 0.2.5

It's 3am and app-billing.service has died for the fourth time. The journal says "permission denied", ausearch hands you two hundred denials from the last hour, and on the same box a batch job has been chewing through memory since midnight. Was it SELinux? Was it the batch job? You find out by hand, one terminal at a time, while alerts keep coming in.

Sentry-v2 does that legwork for you. It watches cgroup pressure, unit failures and SELinux denials on your Rocky, Alma, RHEL or CentOS Stream 10 hosts, ties each failure to the denials that landed within a configurable window of it, and tells you why it thinks they're linked. Then, if you let it, it does what your runbook would: squeezes the noisy job's memory for five minutes, or freezes it until the disk calms down, and puts everything back when the time is up. Out of the box it does none of that. It starts in dry-run and stays there until you decide it has earned more.

That's a real run of examples/demo/run.sh on a Rocky Linux 10.2 test VM, with both policies in dry-run. Run it yourself on a throwaway VM; it cleans up after itself.

What one incident looks like

Two services were denied the same file within a second of each other. One died. This is the bundle Sentry-v2 wrote for it, trimmed to the parts that matter; the full bundle is unedited apart from the hostname and machine ID.

{
  "severity": "warning",
  "incident": {
    "kind": "unit_failed",
    "target": "demo-app-99ae0869.service",
    "trigger": { "source": "journal", "kind": "failed", "result": "exit-code" },
    "correlations": [
      {
        "avc": {
          "perms": ["read"],
          "scontext": "system_u:system_r:sentry_demo_app_t:s0",
          "tcontext": "unconfined_u:object_r:sentry_demo_secret_t:s0",
          "tclass": "file",
          "comm": "sentry-demo-app",
          "cgroup_at_receipt": "sentry.slice/sentry-test.slice/demo-app-99ae0869.service"
        },
        "confidence": 0.976,
        "reasons": [
          "pid in unit cgroup",
          "scontext type matches unit domain",
          "comm matches ExecStart"
        ]
      }
    ]
  },
  "context": {
    "psi": {
      "sentry.slice/sentry-test.slice": {
        "memory": { "some": { "avg10": 12.15, "avg60": 4.2, "avg300": 3.1 } }
      }
    }
  },
  "decision": {
    "policy": "demo-app-failed",
    "mode": "dry-run",
    "actions": [{ "action": "notify", "status": "dry_run" }]
  }
}

How to read it:

• The linked denial scored 0.976 because three things lined up: the denied process was in the failed unit's cgroup, its SELinux domain is the unit's, and its name matches the unit's ExecStart. The weights and the arithmetic are in Correlation.
• The denial that isn't there: sentry_demo_other_t was denied the same file in the same window, and ausearch shows it. It ran in another unit's cgroup, under another domain and binary, so it scored nothing and stayed out. The integration test test_unrelated_denial_in_window_is_left_out checks exactly this on a real host with SELinux enforcing.
• The pressure in the unit's slice when it failed. Tasks there spent about 12% of the last ten seconds stalled on memory, because a memory hog was running next door.
• The decision. This policy only notifies, and it's in dry-run. The hog's own bundle, incident-memory-pressure.json, shows what the other policy would have done: set MemoryHigh to 80% of the hog's usage and CPUWeight to 50 for five minutes, marked dry_run.

What it does

• Watches cgroup v2 Pressure Stall Information through kernel triggers, systemd unit failures in the journal, and SELinux AVC denials straight from the audit netlink stream.
• Correlates a failure with the denials around it and scores each one, with reasons: the process was in the unit's cgroup, its domain matches, its name matches ExecStart.
• Acts, reversibly: runtime MemoryHigh and CPUWeight throttles, freeze and thaw, or a backup of the unit's files followed by a reload. Every throttle and freeze has an expiry, and every action is journalled before it touches systemd.
• Reports each incident as one redacted JSON bundle, to disk and to an HMAC-signed webhook, RFC 5424 syslog, Splunk HEC or Elasticsearch.
• Ships ready for EL10: an RPM with a hardened unit, an SELinux subpackage, sentry-v2ctl, an optional localhost Prometheus endpoint and a Cockpit page. Runtime dependencies come from BaseOS and AppStream only.

Compatibility

EL9, Debian, Ubuntu and cgroup v1 aren't supported. AlmaLinux, RHEL and CentOS Stream 10 use the same packages, but the test runs above are on Rocky.

Quickstart

Just want to check policies in CI? pip install sentry-v2 and run sentry-v2ctl policy validate.

Download the RPMs from the v0.2.5 release and check them against SHA256SUMS before you install:

$ sha256sum -c SHA256SUMS --ignore-missing

Every file also carries build provenance. If you have the gh CLI, confirm a download came from this repository's release workflow:

$ gh attestation verify <file> --repo jamesconwaydev/sentry-v2

Prefer to build it yourself? See Installation. Then install:

# dnf install ./sentry-v2-0.2.5-*.noarch.rpm ./sentry-v2-selinux-0.2.5-*.noarch.rpm

EL kernels ship with PSI off. If cat /proc/pressure/memory says No such file or directory, turn it on. The guard below only reboots a host that doesn't have PSI yet:

# test -e /proc/pressure/memory || { grubby --update-kernel=ALL --args=psi=1 && reboot; }

Then add a policy of your own and start the daemon:

# cd /etc/sentry-v2/policies.d       # three dry-run examples live here
# cp 20-unit-failure-notify.toml 50-myapp.toml
# sed -i 's/^name = .*/name = "myapp-notify"/' 50-myapp.toml

# sentry-v2ctl policy validate
ok: 4 policies
# systemctl enable --now sentry-v2
# sentry-v2ctl status
sentry-v2 0.2.5  running for 3s  mode dry-run

COLLECTOR  HEALTH
psi        state=ok, triggers=0, restarts=0, last_error=None
journal    state=ok, restarts=0, last_error=None
audit      source=netlink, user_avc_truncated=0, overruns=0, rcvbuf=425984, restarts=0, last_error=None

no active actions

It's healthy when mode says dry-run (until you change it), every collector says state=ok or, for audit, source=netlink, and restarts stays at 0. state=disabled on psi means PSI is still off; source=logfile on audit is a working fallback, explained in Troubleshooting.

Then break something on purpose and read the bundle; the quickstart guide walks through it, and the demo stages a fuller incident on a test VM.

Safety

Sentry-v2 changes running services, so it's built to be boring about it.

Every action except notify needs both daemon.mode and the policy's mode set to enforce, a target that isn't protected, and room under the limits below. Things it never does on its own: restart, stop or kill a service; change SELinux policy or booleans; touch a protected unit, or a slice unless you allow it; keep acting on a unit that needed three actions in half an hour; or change anything at all in dry-run, which is where it starts.

• Dry-run by default. The daemon and every shipped policy start in dry-run, which runs the whole pipeline and records what it would have done. Protection and cooldowns are previewed exactly; limits are checked against actions really in force, so dry-run never counts its own would-be actions.
• Two switches for enforce. An action is enforced only when daemon.mode and the policy's mode both say enforce. sentry-v2ctl pause and the circuit breaker force dry-run on top of that.
• Protected units. SSH, journald, D-Bus, logind, auditd, systemd's own services, the top-level slices, everyone's login sessions and Sentry-v2 itself are never throttled, frozen or reloaded. Add your own with protection.protected. Slices, which reach every unit under them, are off limits too until you set protection.allow_slices.
• Rollback guarantees. Each throttle and freeze is written to disk with the values it will restore before systemd is touched, then undone at expiry, on shutdown, and on the next start after a crash. A throttle you changed by hand in the meantime is left alone. A rollback that keeps failing raises a critical bundle and blocks new actions on that unit.
• Limits. Five active actions, one freeze at a time and 30 minutes per action by default. Three systemd failures in ten minutes trip the breaker for fifteen. A unit acted on three times in half an hour is put on hold until you sentry-v2ctl ack it, because by then it needs a person, not a fourth throttle.

The details, including the worst-case time a stop can take, are in actions and safety.

When it gets it wrong

A linked denial is a lead, not a verdict. Here's where the scoring can mislead you, and what happens when something underneath fails.

• A denial from a process that's already gone. The cgroup is read when the denial arrives. If the process has exited by then, its PID still counts when it was the unit's main process; otherwise only the domain and name can, and a sibling running the same binary in the same domain (two instances of a template unit, say) can score 0.76 against the wrong one. The reasons show it: no pid in unit cgroup. When the cgroup is known and belongs to another unit, the denial is left out.
• A denial storm. On a busy host the daemon can fall seconds behind the audit stream. A failure waits until audit has been read past its window, for up to 30 s; after that it goes out with audit_caught_up: false, and denials may be missing from it.
• Timing. Only denials from 5 s before to 1 s after the failure count. A service that's denied early and dies later needs a wider correlation.before.
• Dropped records. USER_AVC records over 1000 bytes are dropped and counted in user_avc_truncated; denials the kernel drops before the daemon reads them show up as overruns. Both are in sentry-v2ctl status.
• The daemon crashes or is killed. systemd runs sentry-v2 --rollback-only as soon as it exits, which puts back every throttle and freeze still on record, then restarts the daemon after 5 s. If the daemon can't come back, say because its configuration no longer loads, the rollback has already happened. If the rollback itself can't reach systemd, the next start tries again.
• A rollback fails. It's retried 10 times. Then the record is marked rollback_failed, a critical bundle goes to every sink, and no new action touches that unit until you retry with sentry-v2ctl rollback ID.
• systemd keeps refusing. Three failed systemd calls in ten minutes trip the circuit breaker, and everything runs in dry-run for fifteen.
• The host reboots. Throttles are runtime-only, so they're gone with the reboot whatever else happened.

More of this, with the fixes, is in Troubleshooting and Actions and safety.

How it fits together

flowchart LR
    subgraph Collectors
        PSI[PSI triggers]
        J[Journal: unit failures]
        A[Audit: AVC denials]
    end
    PSI & J & A --> C[Correlator]
    C --> P[Policy engine]
    P --> E[Executor<br/>journal, limits, rollback]
    E --> SD[(systemd over D-Bus)]
    E --> S[Sinks<br/>file, webhook, syslog,<br/>Splunk HEC, Elasticsearch]
    E --- CS{{Control socket}}
    CS --> CLI[sentry-v2ctl]
    CS --> CP[Cockpit page]

Documentation

The full index is docs/README.md, and changes are in the changelog.

License

Apache-2.0. Maintained by James Conway (@jamesconwaydev).