StateSet Reference Computer Use Agent

Official StateSet reference agent for secure computer-use workflows on Linux/X11.

This repository is the maintained example of how StateSet packages a computer-use agent: a real desktop toolchain, a stable CLI, preset workflow routing, observability hooks, and a local fallback engine for offline iteration.

What This Repository Includes

• Real computer-use primitives for screenshots, mouse movement, clicks, typing, scrolling, and zoomed inspection.
• A packaged CLI surface centered on stateset-cua run and stateset-cua doctor.
• Hosted execution through Anthropic-backed computer-use models.
• Local execution through the NSR engine with --nsr.
• StateSet workflow presets such as auto-close, social-media, linkedin, slack, shopify, and agentic.
• Reference-repo standards for environment templates, security guidance, contribution hygiene, and generated-artifact cleanup.

Reference Principles

This repo is intended to feel like an official reference implementation, not a demo dump.

• Checked-in config uses placeholders only.
• Runtime outputs stay local and out of version control.
• Setup docs match the actual CLI and toolchain.
• Diagnostics reflect the real supported backends.
• Preset agent routing is explicit about which environment variables are required.

Execution Modes

Mode	When to use it	Requirements
stateset-cua run --nsr ...	Local experimentation, no hosted agent config yet	Python, X11 display, xdotool, scrot or gnome-screenshot
stateset-cua run ...	Anthropic-backed computer-use runs	ANTHROPIC_API_KEY plus DEFAULT_* or per-agent *_ORG_ID and *_AGENT_ID
stateset-cua run --cli ...	CLI-only workflows with no GUI actions	Python environment only

Quick Start

1. Install system dependencies

sudo apt update
sudo apt install -y python3 python3-venv xdotool xvfb scrot x11-utils

If you prefer GNOME screenshot tooling, gnome-screenshot also works as the screenshot backend.

2. Create a virtual environment and install the package

git clone https://github.com/stateset/stateset-computer-use-agent.git
cd stateset-computer-use-agent

python3 -m venv venv
source venv/bin/activate
pip install --upgrade pip
pip install -e ".[dev]"

3. Configure the environment

cp .env.example .env

Fill in .env with the values you actually plan to use:

• For local NSR only, DISPLAY is enough once your X11 session is running.
• For hosted Anthropic mode, set ANTHROPIC_API_KEY.
• For StateSet preset agents, set either:
  • DEFAULT_ORG_ID and DEFAULT_AGENT_ID for one shared hosted agent, or
  • per-agent values such as AUTO_CLOSE_ORG_ID and AUTO_CLOSE_AGENT_ID.

4. Start a virtual display

Xvfb :1 -screen 0 1920x1080x24 &
export DISPLAY=:1

5. Validate the workstation

stateset-cua doctor

6. Run the first task

Local NSR mode:

stateset-cua run --nsr "inspect the current desktop and summarize what is actionable"

Hosted Anthropic mode:

stateset-cua run "organize the current desktop workflow and report blockers"

Preset agent example:

stateset-cua run --agent auto-close "close resolved tickets from the last 24 hours"

Common Commands

stateset-cua doctor
AGENT_ENV=production stateset-cua doctor --strict
stateset-cua eval report --policy config/runtime_eval_gate_policy.json
stateset-cua eval release --policy config/runtime_eval_gate_policy.json
stateset-cua eval certify --policy config/runtime_eval_gate_policy.json
stateset-cua eval verify --certification artifacts/release_certification.json
stateset-cua eval baseline promote --from-certification artifacts/release_certification.json --policy config/runtime_eval_gate_policy.json
stateset-cua eval verify --baseline artifacts/runtime_eval_baseline.json
make release-governance
stateset-cua run --dry-run "triage the current workflow"
stateset-cua run --stream "review the open application windows"
stateset-cua run --cli "summarize the current repository status"
stateset-cua run --nsr "inspect the desktop and decide the next action"

In AGENT_ENV=production, the runtime now refuses to start if enterprise controls are unsafe. The only bypass is the explicit break-glass override STATESET_ALLOW_UNSAFE_PRODUCTION_START=1.

For operational readiness, stateset-cua eval release now builds a normalized runtime-eval artifact from benchmark/session evidence, writes the current report, and fails closed when the configured regression gate does not pass.

For audit-ready deployments, stateset-cua eval certify writes a release certification bundle plus a detached .sha256 digest covering the exact config, runtime-eval artifacts, audit-log integrity result, and release gate decision used for the candidate.

After a certified candidate is accepted, stateset-cua eval baseline promote turns that exact certified report into the next regression baseline and records the promotion in immutable audit logs. By default it also verifies the source certification bundle and its detached digest before promotion; bypassing that check requires the explicit flag --allow-unverified-certification.

For post-build control verification, stateset-cua eval verify rechecks the detached digest, manifest hashes, embedded certification hash, audit-log hash chain, and baseline lineage before an operator treats an artifact as trusted.

For automated enforcement, make release-governance runs the full production governance chain through the official CLI: strict doctor checks, release gate, certification, artifact verification, baseline promotion, and promoted-baseline verification.

Hosted Agent Configuration

Preset routing is environment-driven.

• DEFAULT_ORG_ID, DEFAULT_AGENT_ID, and optional DEFAULT_STRIPE_ID provide one shared hosted StateSet agent identity for every preset.
• AUTO_CLOSE_*, SOCIAL_MEDIA_*, STATESET_AGENTIC_*, and the other per-agent variables override the defaults for specific presets.
• If no hosted agent identifiers are configured, preset agents remain unavailable. That is expected for local --nsr usage.

Repository Layout

• agent contains the runtime, tool implementations, routing, observability, and NSR engine.
• cli contains the packaged stateset-cua command surface.
• tests contains unit and end-to-end coverage for the reference surface.
• GETTING_STARTED.md is the detailed setup guide.
• QUICKSTART.md is the shortest path from clone to first run.

Additional Documentation

• GETTING_STARTED.md
• QUICKSTART.md
• CHANGELOG.md
• CONTRIBUTING.md
• SECURITY.md