unix-tutor 0.2.0

A terminal assistant that runs shell commands from natural language using a local LLM

Assignment 4: Student Choice

Quick Install (DMG, macOS)

1. Go to GitHub Releases and download UnixAgent.dmg.
2. Open the DMG and drag UnixAgent.app into Applications.
3. Launch from Applications.
4. If macOS warns the app cannot be verified:

• Right-click UnixAgent.app -> Open -> Open.
• Or go to System Settings -> Privacy & Security -> Open Anyway.

5. On first run, if setup is needed:

• Click Setup Ollama in the app.
• Install Ollama, then run ollama pull qwen2.5:7b in Terminal.
• Return to the app and click Check Ollama.

See the assignment page for full requirements.

Latest Updates

• Switched default local model to qwen2.5:7b across config and setup flow.
• Added first-run Ollama readiness gate in UI (chat input disabled until backend + Ollama are ready).
• Added Guardrails manager in UI:
  • hard guardrails: user-blocked file/folder paths (enforced in tools)
  • soft guardrails: editable assistant guidance notes
• Added explicit history visibility in UI chat: when command history is updated, the executed command is shown.
• Updated confirmation policy:
  • confirmation required for delete_path, create_directory, and write_text_file only when overwriting a populated file
  • removed blanket confirmation for every non-read-only shell command
• Hardened tool routing so sensitive actions do not go through raw shell execution:
  • delete -> delete_path
  • create directory -> create_directory
  • file search -> search_in_files
  • file writes -> write_text_file
• Added stale-state handling guidance in agent behavior so filesystem actions are re-validated with tools each turn.
• Reduced noisy debug output in CLI by making verbose logs opt-in (UNIX_AGENT_LOG_LEVEL=DEBUG).

Guardrails for Beginners (Hard vs Soft)

You can configure guardrails in the desktop app from the Guardrails button.

1. Hard constraints (enforced)

• Add file/folder paths that the agent must not access.
• These are enforced in tools and cannot be bypassed by prompts.
• Use this for sensitive paths inside your workspace.

2. Soft constraints (guidance)

• Add plain-language safety preferences (for example: "always ask before editing config files").
• These are included as assistant guidance, but you should still review actions.

Guardrails are saved in the app support folder and reused across runs.

Assignment 4 Built on Top of UNIX Agent

How a user will install this app (initial architecture)

2-layer installer user experience:

1. App install

• User downloads app DMG.
• Drag-and-drop install into Applications.
• Launches normally like any Mac app.

2. First-run local model setup

• On first launch, app checks:
  • Is Ollama installed?
  • Is Ollama running?
  • Is required model present?
• If anything is missing, show a guided setup screen with one-click actions.

Recommended user flow

1. Welcome screen: “This app runs fully local. No data leaves your machine.”
2. Dependency check:

• Ollama installed: yes/no
• Ollama running: yes/no
• Model available: yes/no

3. One-click fix buttons:

• Install Ollama
• Start Ollama
• Download model (for example qwen2.5:7b)

4. Progress bar while model downloads.
5. Final health check against localhost endpoint.
6. Continue into the app.

What to bundle vs what not to bundle

1. Bundle:

• Your app code, UI/CLI wrapper, configs, guardrails, prompts.
• A first-run setup wizard and diagnostics panel.

2. Not in bundle:

• Ollama runtime inside app binary (usually better to keep as external dependency).
• Large model weights inside DMG (makes distribution huge and updates painful).

Packaging choices

1. DMG installer

• First launch runs dependency checks and setup wizard.

2. Homebrew formula

• Best for developer users
• brew install your-app then first run handles Ollama/model checks.

3. Hybrid

• Offer both DMG for non-technical users and Homebrew for technical users.

What users need to do manually (minimum)

1. Install app (DMG or brew).
2. Approve macOS prompts (network/local permissions if needed).
3. Click “Set up local model” once. After that, everything should be automatic.

Update strategy

1. App updates: small and frequent.
2. Model updates: in-app model manager with “Update model” button.
3. Keep app usable even if user delays model update.

Failure cases you should handle in UI

1. Offline during model download.
2. Not enough disk space.
3. Ollama service not reachable.
4. Wrong model name configured.
5. Slow first token latency during warmup.

Recommended baseline for your project

1. Ship app without model weights.
2. Require Ollama as external dependency.
3. Auto-detect and guide install/start/download on first run.
4. Cache model locally and reuse.
5. Add a Settings page: model name, endpoint, quick health test.

How to push updates to the APP

First time creating an app so this is handy

Make code changes in a branch. Commit + push source code. Merge to main. Rebuild .app and .dmg. Create a new GitHub Release (new version tag) and upload new DMG.

UNIX Tutor Agent

This repository contains an initial draft of a cli based UNIX tutor and agent which can teach beginners UNIX commands, and actually interact with the file system to do most of the basic commands a UNIX cli can do in natural language

What This Draft Includes

• Agent framework usage: OpenAI Agents SDK (openai-agents)
• Chat-based multi-turn CLI (run_agent.py)
• Tool-based interaction with the outside world (filesystem + shell command execution)
• Structured file search tool for readable match results
• SQLite command history logging for every executed command
• OpenAI-compatible API configuration via environment variables (for TritonAI)

Project Structure

• run_agent.py: Entry point for the CLI agent
• run_ui.py: Entry point for the desktop UI
• src/unix_tutor_agent/config.py: Environment/config loading
• src/unix_tutor_agent/agent.py: Agent construction and turn execution
• src/unix_tutor_agent/ui.py: Tkinter desktop UI and backend adapter
• src/unix_tutor_agent/tools.py: Agent tools (run commands, file search, read cwd, history)
• src/unix_tutor_agent/session.py: Workspace-scoped shell session (cd state)
• src/unix_tutor_agent/db.py: SQLite history schema + logging
• .env.example: Example environment config for TritonAI
• .gitignore: Ignores .env, virtual env, and sqlite artifacts

Setup

1. Create and activate a virtual environment:

python3 -m venv .venv
source .venv/bin/activate

2. Install dependencies:

pip install -r requirements.txt

3. Create your .env file:

cp .env.example .env

4. Fill in .env values:

• TRITON_API_KEY: your TritonAI key
• TRITON_BASE_URL: TritonAI OpenAI-compatible endpoint
• TRITON_MODEL: model name to use
• HISTORY_DB_PATH (optional): sqlite file path

Run

PYTHONPATH=src python run_agent.py

Run Desktop UI

PYTHONPATH=src python run_ui.py

Current Behavior

• User can ask in natural language (example: "create a folder called notes and move into it")
• Agent explains command intent and can execute via tool calls
• Supported command set in this draft:
  • pwd, ls, cat, head, tail, mkdir, touch, echo, cd
• File search is available through a dedicated structured tool that returns file, line, and text matches.
• cd state persists across turns inside the agent session
• Every command execution is logged to SQLite (command_history table)

Demo Video

https://youtu.be/cUaGNh2H8Jc

Example Interaction

Turn 1

Input

input - show me current directory

LLM Output

llm - 
Agent> ### 📍 Current Working Directory:
```
/Users/aniruddhafale/Documents/CSE291P/PA3/03-agents-aniruddha-assignment-3
```

---

💡 Try next:
- "Go into sandbox"** → `cd sandbox`
- "List files here"** → `ls`
- "Create a new folder"** → `mkdir myfolder`

What would you like to do? 🚀

Turn 2

Input

input - create a folder new and go into the new folder

LLM Output

llm - 
[CONFIRMATION REQUIRED] The agent wants to run:
  mkdir new
Allow command execution? [y/N] 

Turn 3

Input

input - y 

LLM Output

llm - 
Agent> ✅ Successfully moved into the `new` folder!

---

### 📍 You are now here:
```
/Users/aniruddhafale/Documents/CSE291P/PA3/03-agents-aniruddha-assignment-3/new
```

---

### 💡 Try next:
- **"List files here"** → `ls` (folder is empty for now!)
- **"Create a file here"** → `touch myfile.txt`
- **"Go back to the previous directory"** → `cd ..`

What would you like to do? 🚀

Turn 4

Input

input - go up a folder

LLM Output

llm - 
Agent> ✅ Successfully moved up one directory!

---

### 📍 You are now here:
```
/Users/aniruddhafale/Documents/CSE291P/PA3/03-agents-aniruddha-assignment-3
```

---

💡 **What `cd ..` does:** `..` refers to the **parent directory** — one level up from where you are. It's a quick way to navigate backwards!

### 💡 Try next:
- **"List files here"** → `ls`
- **"Go into sandbox"** → `cd sandbox`
- **"Go into new"** → `cd new`

What would you like to do? 🚀

Turn 5

Input

input - delete the folder "new"

LLM Output

llm-
[CONFIRMATION REQUIRED] The agent wants to delete the directory:
  /Users/aniruddhafale/Documents/CSE291P/PA3/03-agents-aniruddha-assignment-3/new
Allow deletion? [y/N] 

Turn 6

Input

input - y 

LLM Output

llm - 
Agent> ✅ The `new` folder is gone! 🎉

### 📁 Current directory contains:
| Name | Type |
|---|---|
| `design.md` | Markdown document |
| `README.md` | Readme document |
| `requirements.txt` | Python dependencies |
| `run_agent.py` | Python script |
| `eval/` | Folder |
| `sandbox/` | Folder |
| `src/` | Folder |
| `test/` | Folder |

---

### 💡 Try next:
- **"Create a new folder"** → `mkdir myfolder`
- **"Go into sandbox"** → `cd sandbox`
- **"Show me the contents of README.md"** → `cat README.md`

Turn 7

Input

input - go up a directory

LLM Output

llm - 
Agent> ⚠️ **Access Denied!** Going up a directory is **not allowed** as it would take us outside the **workspace root**. This is a safety restriction to protect your system files.

---

### 📍 You remain here:
```
/Users/aniruddhafale/Documents/CSE291P/PA3/03-agents-aniruddha-assignment-3
```

### 💡 You can only navigate **within** the workspace:
- **"Go into sandbox"** → `cd sandbox`
- **"Go into src"** → `cd src`
- **"Go into eval"** → `cd eval`
- **"Go into test"** → `cd test`

Evaluation Harness

The evaluation harness and dataset:

• Dataset: eval/scenarios.json
• Runner: eval/run_eval.py
• LLM candidate generator: eval/generate_llm_candidates.py
• Judge spot-check tool: eval/spot_check_judge.py

Current dataset categories are aligned with the tutor goals:

• Tutoring: explaining commands and clarifying ambiguous requests
• Filesystem manipulation: create, write, copy, move
• Guardrails: confirmation before delete (destructive action), deny to go outside workspace, deny any OS path access
• Robustness: oracle-injection resistance and behavior checks under misleading prompts
• Task completion checks: verify expected end state in filesystem

High-Level Eval Design

The evaluation strategy uses two layers:

• Deterministic oracle checks (primary): These are objective checks in each scenario's checks field, such as expected files created, expected files removed, expected file content, command history constraints, and guardrail outcomes.
• LLM-as-judge (secondary): Used mainly for subjective dimensions like tutoring quality and helpfulness of the response. The judge sees scenario context, trial output, and history, then returns rubric JSON.

Why this design:

• Deterministic checks are trusted for correctness and safety.
• LLM judge extends coverage for qualities that are hard to encode with strict rules.
• Judge verdicts can be checked manually to reduce risk of judge bias.

Where to see natural language inputs and expected behavior:

• Natural language test prompts are in eval/scenarios.json under each scenario's prompt.
• Expected behavior/oracle is in eval/scenarios.json under each scenario's checks.
• Scoring implementation is in eval/run_eval.py (score_trial).

Run Deterministic Eval

Run the default model from .env for k trials per scenario:

python eval/run_eval.py --k 5

By default, eval runs auto-approve confirmation-required actions (y) unless a scenario explicitly defines confirmation_response (for example, deny tests set "n").

To disable this and default to deny, use:

python eval/run_eval.py --k 5 --no-auto-approve-permissions

Run two configurations for comparison:

python eval/run_eval.py --k 5 --models "claude-haiku-4-5,claude-opus-4-6-v1"

Run one specific scenario by id:

python eval/run_eval.py --k 1 --scenario-id create_notes_dir

You can also pass multiple scenario ids by repeating the flag or using comma-separated ids:

python eval/run_eval.py --k 1 --scenario-id create_notes_dir --scenario-id clarify_ambiguous_request
python eval/run_eval.py --k 1 --scenario-id create_notes_dir,clarify_ambiguous_request

The script prints pass@1, pass@k, guardrail_violation_rate, hard_safety_breach_rate, and policy_message_failure_rate, and writes full logs to eval/results/.

Evaluation Set Description

The benchmark dataset is in eval/scenarios.json and currently contains 14 single-turn scenarios across five categories:

• Tutoring (3): explain commands and ask clarifying questions for ambiguous requests.
• Filesystem manipulation (4): create directories/files, write content, copy files, and move files.
• Guardrails (5): confirmation-required deletion, workspace boundary protection, and OS-path blocking.
• Robustness (1): oracle-injection resistance and behavior checks under misleading prompts.
• Task completion check (1): complete a mini task and confirm completion.

Each scenario defines:

• prompt: natural-language input to the agent.
• setup (optional): initial filesystem state.
• checks: deterministic oracle conditions (file existence/content, command prefixes, output tokens, safety assertions).

Scoring is implemented in eval/run_eval.py (score_trial), and all per-trial traces are stored in eval/results/*.json.

Pass@k Results (Two Configurations)

The following results were generated by running:

python eval/run_eval.py --k 5 --models "claude-opus-4-6-v1,api-gpt-oss-120b"

Detailed report file:

For each model, 14 natural language scenarios were evaluated where pass@1 uses 1 trial per scenario, and pass@5 uses 5 trials per scenario.

The evaluation now reports two separate safety metrics: hard_safety_breach_rate (actual execution of blocked or destructive actions) and policy_message_failure_rate (missing required safety wording or confirmations). This distinction clarifies whether a model's output led to a real safety breach or just failed to meet strict policy messaging. Hard-block guardrails are always enforced at the tool layer, so even if a model's response wording is imperfect, dangerous actions are never executed.

=== Metrics ===

Model: claude-sonnet-4-6

• Scenarios: 14
• pass@1: 0.714 (10/14)
• pass@5: 0.786 (11/14)
• guardrail_violation_rate: 0.200
• hard_safety_breach_rate: 0.000
• policy_message_failure_rate: 0.200

Model: api-gpt-oss-120b

• Scenarios: 14
• pass@1: 0.500 (7/14)
• pass@5: 0.571 (8/14)
• guardrail_violation_rate: 0.400
• hard_safety_breach_rate: 0.000
• policy_message_failure_rate: 0.400

The hard-block guardrails are still active at the tool layer, so commands like privilege escalation, system control, and known dangerous system operations are rejected before execution, hardblock commands are never executed and destructive delete behavior still requires user confirmation. In other words, the safety gates that actually prevent risky execution are still in place and still enforce behavior even if model wording varies.

The lower pass@k rates are mainly due to the evaluation’s strictness: a scenario only counts as passed if at least one of the k trials fully meets all correctness and safety criteria, including required output wording.

The lower pass rates and higher guardrail violation rates are mostly coming from eval strictness. The current scoring marks some guardrail scenarios as violations when expected keywords are missing (this is the input guardrail before sending the command to llm) in the response, and with k>1 a scenario can be counted as a violation if any trial fails that wording check. So you can see high violation numbers even when hard safety behavior is still being enforced correctly.

Interpretation:

• pass@k: fraction of scenarios where at least one of k trials passed.
• guardrail_violation_rate: fraction of guardrail scenarios with at least one failing safety-critical trial.

Optional LLM-As-Judge

LLM can be used to generate a scenario and score it

python eval/run_eval.py --k 5 --use-llm-judge --judge-model "claude-opus-4-6-v1"

The judge receives the scenario, output, command history, and dataset context and returns rubric JSON.

Spot-Check Judge Verdicts

After running with --use-llm-judge, manually spot-check a handful of judged trials:

python eval/spot_check_judge.py --results eval/results/eval_results_<timestamp>.json --samples 5

This prints deterministic pass/fail side-by-side with the judge verdict and reason so you can detect judge bias or inconsistencies.

Generate More Candidate Prompts

python eval/generate_llm_candidates.py --n 20

This writes candidate examples to eval/candidate_scenarios.json for manual review before adding to the benchmark.