CryptoLabs SRE Agent

A lightweight reverse-tunnel agent that connects your server to the CryptoLabs AI SRE investigation platform — no SSH keys, no port forwarding, no firewall changes.

Quick Start

# Install (requires Python 3.10+)
pipx install cryptolabs-sre-agent

# Connect using the token from your CryptoLabs SRE dashboard
# Option 1: token file (recommended — not visible in process list)
echo "YOUR_TOKEN" > /etc/sre-agent.token
chmod 600 /etc/sre-agent.token
sudo $(which sre-agent) connect --token-file /etc/sre-agent.token

# Option 2: environment variable
export SRE_AGENT_TOKEN="YOUR_TOKEN"
sudo $(which sre-agent) connect

# Option 3: CLI flag (token visible in `ps aux` — use for quick testing only)
sudo $(which sre-agent) connect --token YOUR_TOKEN

How it Works

1. Install — pipx install cryptolabs-sre-agent
2. Run — sudo $(which sre-agent) connect --token-file /etc/sre-agent.token
3. The agent opens an outbound WebSocket (wss://) to sre.ai.cryptolabs.co.za
4. The token links this agent to your registered server on the dashboard
5. Commands are sent by the SRE system, executed locally, results returned

Security

Network

• TLS enforced — agent refuses non-wss:// connections in production
• Outbound only — no inbound ports opened on your machine
• Heartbeat timeout — server drops idle connections after 90s of silence

Authentication

• HMAC-SHA256 signed tokens — cryptographically bound to a single server
• Token hashed in storage — only a SHA-256 hash is stored server-side (not the raw token)
• Rate limiting — 10 failed auth attempts per IP triggers a 5-minute lockout
• 10-second auth deadline — connections that don't authenticate are dropped

Execution Safety

• Server-side allowlist — commands are validated by the SRE API's command allowlist before reaching the agent
• Agent-side blocklist — defense-in-depth: the agent independently blocks catastrophic commands (rm -rf /, mkfs, dd to disk, fork bombs, halt/poweroff, writes to /etc/shadow, /root/.ssh/authorized_keys)
• Output truncation — stdout/stderr capped at 1 MB per field
• Concurrent limit — max 10 commands in parallel to prevent resource exhaustion
• Command timeout — every command has a configurable timeout (default 300s)

Token Handling

• --token-file recommended — avoids exposing token in process list
• Environment variable (SRE_AGENT_TOKEN) — alternative to file
• CLI flag (--token) — agent warns when this is used

Revocation

• Deleting the server from the dashboard instantly disconnects the agent
• Token revocation via API (DELETE /agent/token/{server_id})
• Agent auto-shuts down when server sends disconnect message

Approving a fix

A fix that CryptoLabs AI SRE proposes does not run on your server until root on that server approves that exact plan. The agent enforces this itself for every command of a fix run, so a compromised web account (WordPress, the admin API, a stolen agent token) cannot start a fix. Investigation commands are not affected.

sudo sre-agent approvals                       # the fixes waiting for approval on this server
sudo sre-agent approve <incident_id>           # review the plan, type this server's host name
sudo sre-agent approve <incident_id> --override-risk-review   # only if the risk review rejected it

• approve shows the incident, the risk review verdict and summary, and every step with its full command (control and invisible characters are shown as escapes). You confirm by typing this server's own host name. The plan hash is calculated on the server from the steps you were shown, never taken from CryptoLabs.
• If the risk review rejected the plan, approve refuses unless you add --override-risk-review. That decision is yours, on your server.
• The approval is one file, /var/lib/sre-agent/approvals/<incident_id>.json (directory 0700, file 0600). It allows one run of that plan within 2 hours. A changed plan, a second run or an expired approval is refused with exit code -3 and not approved on this box: run sudo sre-agent approve <incident_id>; nothing is started.
• The file is written before CryptoLabs is told. If CryptoLabs refuses the approval (an HTTP 4xx answer), the file is removed again and its message is printed. If CryptoLabs does not answer (a timeout, a connection error or a 5xx), the file stays, because the fix may already be running; run sudo sre-agent approvals to see the status. It expires by itself after 2 hours.
• Needs the agent token in /etc/sre-agent.token (or SRE_AGENT_TOKEN_FILE) and https (the API base is SRE_RELAY_BASE, as for setup). CryptoLabs starts a fix only on a server whose connected agent reports that it enforces this, so update the agent first.

Remote terminal (local consent)

The agent can open interactive root shells and run commands for the machine owner through CryptoLabs (srecli). It is off by default and only you can turn it on, on the machine itself.

sudo sre-agent terminal on      # allow (prints exactly what this allows)
sudo sre-agent terminal off     # refuse
sre-agent terminal status       # prints on/off and, when off, why

• Consent is the file /etc/sre-agent.conf ([terminal] / enabled = true). It counts only if it is a regular file owned by root and not writable by group or others; anything else (missing, wrong owner, unreadable, enabled not true) means off. The agent re-reads it for every request and every 5 seconds during a session; turning it off ends open sessions.
• Each session uses its own outbound wss:// connection; the agent never takes a host or URL from the server. Sessions end when the connection drops, on timeout (PTY: 30 min idle, 8 h maximum), or when you turn consent off. At most 3 run at once.
• Read-only commands run inside a systemd-run sandbox (no writes outside a private /tmp, no signals to other processes, no docker, no kernel/mount/clock calls). Read-only is not confidential: output can contain secrets.
• This switch protects against mistakes and policy, not against a compromised CryptoLabs server, which can already run root commands through the agent. The hard stop is sudo systemctl stop sre-agent.

Protocol: docs/contracts/2026-10-01-terminal-agent-protocol.html.

srecli: the owner's command-line client

srecli runs on your own computer (not on the servers) and reaches the machines you registered with CryptoLabs AI SRE. It never needs root and is installed from the same package as the agent:

pipx install 'cryptolabs-sre-agent[cli]'   # [cli] adds keyring (the macOS Keychain) and segno (the QR code)
srecli login --code 7HQ2M-9PZ4X-C1VB8-NN3KD-6RT0Y   # a one-time code from SSH on wk01, see below
srecli servers                              # hostname, online, terminal setting, adopted, linked, agent version, ready
srecli show venter-h100x4-01                # details of one server (a hostname or a server id)
srecli exec venter-h100x4-01 -- nvidia-smi -L
srecli elevate venter-h100x4-01             # ask for write access; approve it on your phone
srecli exec venter-h100x4-01 -w --timeout 120 -- systemctl restart nginx
srecli connect venter-h100x4-01             # live root shell (needs the same approval); Ctrl-] or Enter ~ . leaves
srecli logout                               # revokes the key in use (SRECLI_KEY first, else the stored key)

• exec SERVER [-w] [--timeout N] -- COMMAND...: the words after -- are joined with single spaces and run as one shell command. Output goes to your stdout and stderr, and piped stdin is streamed. Leaving on purpose decides the status: Ctrl-] in connect is 0, Ctrl-C is 130, SIGTERM is 143, SIGHUP is 129 (Enter ~ . leaves like Ctrl-]), whatever the server sends afterwards. The exit status of everything else is the remote one, as this table says (operator contract, section 8; the rules are applied in this order, and the first that fits decides):

  Status	When
  124	The command timed out, whatever exit code or signal it ended with. Checked first, so a timed-out command never reads as success.
  255	The session was cut short (it ended for any reason but a normal exit: the key or the grant ended, the setting was lowered, the command was closed) and the server's answer would give 0. A cut-short session never reads as success.
  the command's own status	The command ended by itself with exit status N: N.
  128 + N	The command (or its unit) ended by signal N, for example 143 for SIGTERM, which is what a read-only command that was stopped reports.
  255	The server's final answer carries neither an exit status nor a signal: the connection was lost without a final answer, the broker gave up waiting for the agent (a command that ignores SIGTERM for more than 3 s; the reason, for example grant ended, is on stderr), or the agent reported an error.
  126	Refused (reason and message on stderr), for example blocked, no grant, or the terminal is off.
  2	A usage error, or a hostname that two servers share.
  1	Any other local error, including "not logged in", Your key is not valid. Run: srecli login --code CODE (get a code over SSH on wk01), Rate limited, try again later, and a server whose remote terminal is not enabled.

  One status is not in this table, because it is srecli's own and not an answer of the server: 141 (128 + SIGPIPE), when the reader of your output goes away (a broken pipe, for example | head) and the command had not ended by itself (see the reader bullet below).

  A 0 therefore means the command ran to its end and said 0; 143 or 255 after a revoke means it was stopped, not that it succeeded.

• A slow reader never ends the session. A pipe into less, Ctrl-S on a terminal or a slow disk only makes srecli hold the output it cannot write yet: in memory, then in a private temporary file (mode 0600, removed when srecli ends, however it ends), up to 1 GiB, in order. What is held is counted at what it costs (a stream of one-byte frames counts their bookkeeping too), and the limit applies to memory and file together. srecli keeps reading the socket meanwhile, so the connection stays up. Past 1 GiB it sends close and exits 1 with "local output could not keep up". Ctrl-C or SIGTERM still ends it at once.

• A reader that goes away (a broken pipe, for example | head) is treated like SIGPIPE in a local pipeline, with no race on when srecli happened to read the final answer: srecli stops writing, sends close if the command is still running (srecli sends close once, and no more input after it), waits up to 5 s in total for the final answer, then drops the connection without waiting for the close handshake, and exits quietly: with 124 when that answer says the command timed out (a timeout still comes first), with the command's own status when it says the command exited, else with 141 (128 + SIGPIPE). A command that was still running is stopped by the close, so srecli exec box -- dmesg | head -1 gives the command's status only if the command had ended by itself. Any other write failure (no space left on the device, an I/O error) exits 1 with "local output failed".

• connect: Ctrl-], or Enter then ~ then . (as in ssh, for keyboards where Ctrl-] cannot be typed), leaves with 0. The start of the session counts as after Enter, so ~. as the very first keys leaves too. A ~ that does not follow Enter, or that is followed by another character, is sent unchanged, and ~~ sends one ~. On start it prints one line, on stderr: how to leave, and that background jobs end with the session.

• Without -w a command must be on the read-only command list. Write commands and connect need an approved grant for the key you are using, which srecli elevate requests. The grant belongs to that key only, and it is approved on your phone with a code from your authenticator app.

• login --code CODE exchanges a one-time login code for a key. You get the code over SSH on wk01 (docker exec -it sre-api python -m app.terminal_admin login-code <user_id>): it works once, for 10 minutes. Case, dashes and spaces in what you type are ignored; anything that is not 25 characters of the login-code alphabet (0-9 and A-Z without I, L, O and U) exits 2 before any request. srecli login without --code exits 2 and says how to get one. There is no polling and no browser. If the server does not accept the code, srecli exits 1 and says the code is not valid or has expired, and, if you minted it a moment ago, that someone else may have used it first: on wk01 run terminal_admin keys list <user_id> and revoke any key you do not know.

• elevate prints the approval address, a text QR code of it (made locally with segno) and "Approve on your phone: open the link or scan the code, then type the 6-digit code from your authenticator app". The code is drawn for a dark terminal; on a light one, use the link. Without segno (the [cli] extra) it still prints the address, and says so. The address is checked before it is printed: it must be exactly https://sre.ai.cryptolabs.co.za/terminal/ui/grant/<grant id> (the origin is compiled in) with the grant id of the answer, a g_ and 32 lowercase hex characters; anything else exits 1 with "the server sent an unexpected approval address", and nothing is printed or polled. A second elevate from the same key supersedes the first, so srecli only ever shows the request it has just made. If another key of your account has a request waiting, the server answers pending_exists and srecli exits 1, naming that key and when its request closes, with no address: wait, or on wk01 run terminal_admin keys revoke <key_id>. A grant that ends denied or revoked prints its reason (for example superseded, too many wrong codes, tamper alarm). A server that is not adopted, an account with no authenticator and a locked account each exit 1 with what to do.

• Browser: srecli never opens a browser, for an approval or for anything else, and BROWSER is ignored. SRECLI_URL is for tests: it is honoured only when it names a loopback host, any other value exits 2 and nothing is sent, and the approval address is then checked against that loopback origin instead.

• --json on servers and show prints the API answer as JSON. servers shows hostname, online, terminal setting, adopted, linked, agent version, ready; adopted is whether you adopted the server over SSH, and linked is when it was attached to your account.

• Text from the server (hostnames, OS info, labels, messages) has escape sequences, control characters and every invisible or unassigned Unicode character (categories Cf, Co, Cs, Cn: direction marks, zero-width and tag characters) removed, and line breaks turned into spaces, before it is printed. A missing value prints as -. --json keeps the raw values, escaped by JSON.

• An error answer that is not JSON (for example from nginx) is reported as the HTTP status and a short generic text. A refused session upgrade maps like the REST answer: 401 is the key hint, 429 is rate limited, 503 is "not enabled".

• srecli elevate waits for your approval and stops with exit 1 ("the request expired") 30 s after the request's request_expires_at (at most an hour from the request in any case). It retries a 429 or 503 until then; a grant status it does not know is an error.

• srecli never sends a key that is not sre_pk_ plus 43 letters and digits, and ignores proxy environment variables (HTTPS_PROXY and the like) for both REST and the session.

• srecli login checks that the key can be stored before it sends the code, because a login code works once. If storing fails after a key was issued, it revokes that key at once and exits 1; you then need a new code. With a keyring it tries a throwaway entry first and falls back to the file if the keyring refuses. If the key is stored but the older copy in the other store cannot be removed, that is a warning, not a failed login.

• A credential store that exists but cannot be read (macOS Keychain locked, no GUI access, a changed Python binary) is reported as that, with the reason, and exits 1. It is never reported as "not logged in", and srecli login does not fall back to the file while the Keychain may hold a key it cannot read, so one computer never ends up with two live keys. An empty Keychain is still "not logged in".

• A certificate that does not verify is reported with the reason OpenSSL gave, and exits 1. Only when the issuer cannot be found locally (OpenSSL verify codes 2, 19 and 20: an issuer certificate or a self-signed root that is not trusted here) does it say this is probably a local trust-store problem and how to fix it (python.org Python on macOS: run "Install Certificates.command"). A self-signed server certificate (code 18), an expired certificate, a wrong host name or any other reason says that the server's certificate is not valid and not to continue: it is not a problem of your trust store.

• The key is stored in the macOS Keychain (service srecli) when keyring is installed, else in ~/.config/srecli/credentials (mode 0600, with a warning). It is never put on a command line or printed. SRECLI_KEY in the environment takes precedence over a stored key. srecli warns when the key expires within 7 days; get a new login code over SSH and run srecli login --code CODE again to renew.

• srecli logout revokes the key in use: SRECLI_KEY first, then the stored key. It deletes the stored copy only if it is that key, so logging out an assistant's SRECLI_KEY never touches the key you stored for yourself. If the revoke fails, nothing is deleted and you can retry.

Using srecli with an AI coding assistant

• Give the assistant its own key: run srecli login --code with its own login code on a separate account or computer, or set SRECLI_KEY for the assistant's process. A read key can list servers and run read-only commands and nothing else.
• Writes need your approval: the assistant can run srecli elevate, but the grant only starts after you type a code from your authenticator app on your phone, it lasts at most 60 minutes (default 30), and it applies to the assistant's key only, never to your own key.
• Read output can contain secrets (environment variables, config files, logs). Anything srecli exec prints is visible to the assistant. Run srecli logout with the assistant's SRECLI_KEY set when you are done: that revokes the assistant's key and leaves yours.

Environment Variables

Running as a Service

# systemd (Linux) — using token file for security
sudo tee /etc/systemd/system/sre-agent.service << 'EOF'
[Unit]
Description=CryptoLabs SRE Agent
After=network-online.target
Wants=network-online.target

[Service]
ExecStart=/root/.local/bin/sre-agent connect --token-file /etc/sre-agent.token
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl enable --now sre-agent

Support

Need help? Join our Discord: https://discord.gg/7yeHdf5BuC

Development

cd services/sre-agent
pip install -e ".[dev]"
pytest