nyuwayskillsandbox 0.1.0

Dynamic behavioral sandbox for AI agent skill bundles (SKILL.md)

nyuwayagentskillsandbox

Dynamic behavioral sandbox for AI agent skill bundles (SKILL.md).

Skills ship as a folder: a SKILL.md file (YAML frontmatter + prose instructions an agent loads into context) plus bundled scripts the agent may invoke. Static analysis can flag suspicious patterns in that text and code, but it can't tell you what actually happens when an agent uses the skill. nyuwayagentskillsandbox detonates a skill bundle inside a hardened, network-sinkholed Docker container, records every observable behavior, and evaluates the resulting timeline against a bundled ruleset — producing the same kind of verdict (PASS / LOW / MEDIUM / HIGH / CRITICAL) that nyuwaymcpsandbox produces for MCP servers.

$ nyuwayskillsandbox detonate ./malicious-notes-skill --mode fast --allow-network

Verdict:  HIGH  (score 60/100)              Duration: 00:16   Findings: 3
+----------------------- nyuwayagentskillsandbox - Behavioral Analysis --------+
|   Target:  ./malicious-notes-skill                                           |
|     Mode:  fast                                                              |
|  Verdict:  HIGH  (score 60/100)                                              |
| Duration:  00:16                                                             |
| Findings:  3                                                                 |
+------------------------------------------------------------------------------+
                              Behavioral Timeline
+------------------------------------------------------------------------------+
|  Time |   | Event                                                            |
|-------+---+------------------------------------------------------------------|
| 00:00 | . | Container started, skill bundle mounted read-only at /src        |
| 00:01 | . | Script 'scripts_harvest_files' invoked (python, deterministic)   |
| 00:01 | X | filesystem.read  /etc/passwd      [DETECTION: sensitive_read_…]  |
| 00:01 | X | filesystem.read  /root/.ssh/id_rsa [DETECTION: sensitive_read_…] |
| 00:01 | . | Script 'scripts_harvest_files' completed (exit 0, 0.34s)         |
| 00:14 | . | Script 'scripts_organize' invoked (python, deterministic)        |
| 00:14 | X | network.dns_lookup  attacker-telemetry.example.tk               |
|       |   |    [DETECTION: outbound_network_from_script]                     |
|       |   |    [DETECTION: suspicious_dns_tld_from_script]                   |
| 00:15 | . | Script 'scripts_organize' completed (exit 0, 1.0s)               |
| 00:16 | . | Container stopped                                                |
+------------------------------------------------------------------------------+
| X | HIGH   | sensitive_read_outside_bundle    | Script read /etc/passwd, … |
| X | HIGH   | outbound_network_from_script     | DNS/network during script  |
| X | MEDIUM | suspicious_dns_tld_from_script   | .tk TLD lookup             |
+------------------------------------------------------------------------------+

---

How it fits in

nyuwayagentskillsandbox is the dynamic layer of the Nyuway Agent Skill Security toolkit:

Phase	Product	Question
Pre-deployment static	nyuwayskillscanner	What does the code say this skill does?
Pre-deployment dynamic	nyuwayagentskillsandbox	What does this skill actually do when an agent runs it?
MCP server runtime	nyuwaymcpsandbox	What does this MCP server actually do at runtime?

---

Two detonation modes

Mode	What runs	When to use
--mode fast (default)	Every bundled script is invoked once with a synthetic empty argument — no LLM involved	CI pipelines, pre-deploy gates, no API key required
--mode full	The skill's own SKILL.md body is loaded as an LLM's system prompt — exactly as a real agent would — and the LLM is given a test task. The driver observes which scripts the LLM invokes and what they do	Tool-choice and intent-mismatch behaviors the deterministic harness cannot surface

---

Install

git clone https://github.com/Nyuway-Cybersecurity/nyuwayagentskillsandbox
cd nyuwayagentskillsandbox
python -m venv .venv
.venv/Scripts/activate        # Windows
# source .venv/bin/activate   # Linux / macOS
pip install -e .
nyuwayskillsandbox setup      # verifies Docker, pulls base images

---

Quick start

# Fast mode against a local skill directory
nyuwayskillsandbox detonate ./my-skill

# Full mode with a real LLM and a custom test task
nyuwayskillsandbox detonate ./my-skill --mode full --llm claude-sonnet-4-5 \
    --test-task "Summarise this quarterly report."

# Exercise the CLI without Docker or an LLM key
nyuwayskillsandbox detonate ./my-skill --dry-run

# CI gate — non-zero exit on any HIGH or above finding
nyuwayskillsandbox detonate ./my-skill --output json --fail-on high

# SARIF output for GitHub Advanced Security
nyuwayskillsandbox detonate ./my-skill --output sarif > results.sarif

# From GitHub (same source resolver as nyuwaymcpsandbox)
nyuwayskillsandbox detonate github:someuser/some-skill

TARGET may be a local path to a skill directory (or a SKILL.md file directly), github:owner/repo[@ref], npm:package[@version], or pypi:package[@version].

---

Detection rules

Nine bundled rules cover the behaviors most relevant to skill detonation:

Rule	Severity	What it flags
shell_exec_in_script	HIGH	A bundled script spawned a subprocess
outbound_network_from_script	HIGH	Network activity during script execution
sensitive_read_outside_bundle	HIGH	A script read SSH keys, cloud credentials, or other sensitive system paths
post_execution_network_activity	HIGH	Network activity observed after a script completed — a stronger exfil signal than in-execution activity
intent_action_mismatch	HIGH	The LLM requested a script not declared in the bundle (Full mode)
credential_env_access_in_script	MEDIUM	A script read process environment variables
file_write_outside_bundle	MEDIUM	A script wrote outside its own bundle directory
suspicious_dns_tld_from_script	MEDIUM	DNS lookup for a high-risk TLD (.tk, .gq, .xyz, …)
slow_script_execution	LOW	A script ran longer than the configured threshold

Rules are pure YAML and follow the same schema as nyuwaymcpsandbox's detection engine — both exact and regex payload matching with dotted key paths.

---

Verdicts

Score = min(100, sum_of_finding_weights).

Verdict	Score	Action
PASS	0–19	Deploy. No suspicious behaviour observed.
LOW	20–39	Deploy with monitoring.
MEDIUM	40–59	Review before deployment.
HIGH	60–79	Block deployment.
CRITICAL	80–100	Do not deploy.

---

Key flags

Flag	Default	Purpose
--mode fast|full	fast	fast = deterministic harness; full = adds LLM driver
--llm <model>	—	Any litellm model identifier (required for full mode)
--api-key	env NYUWAY_LLM_API_KEY	LLM API key
--test-task	derived from skill description	Prompt given to the LLM in full mode
--allow-network	off	Grant real outbound egress (sinkholed by default)
--script-timeout	30s	Hard timeout per script execution
--post-exec-grace	2.0s	Seconds to observe the container after each script completes
--fail-on low|medium|high|critical	off	Non-zero exit when findings meet or exceed the threshold
--output timeline|json|sarif	timeline	Output format
--env-file	—	Load extra container env vars from a .env-style file
--dry-run	off	In-memory fakes only — no Docker, no LLM needed

---

Output formats

--output timeline   # Rich terminal view (default)
--output json       # Stable JSON schema for scripting / dashboards
--output sarif      # SARIF 2.1.0 for GitHub Advanced Security / VS Code Problems

---

What gets captured

Layer	Mechanism	What lands on the timeline
Container	docker run lifecycle	container.started, container.stopped
Script execution	docker exec (one-shot)	skill.script_invoked, skill.script_completed, skill.slow_script
Filesystem reads	In-process builtins.open patch (Python scripts)	filesystem.read
Filesystem writes	watchdog on the bind-mounted /src	filesystem.write, filesystem.delete
Process	docker container.top() polling	process.spawn, process.exit
Network (DNS)	tcpdump sidecar	network.dns_lookup
Environment	sitecustomize.py shim (Python scripts)	environment.read
LLM driver (full mode)	litellm	skill.llm_decision, skill.intent_mismatch

---

Security defaults

Every detonation runs the skill's scripts inside a container with:

• network_mode='none' — sinkholed; only --allow-network opts in to real egress
• read_only=True root filesystem
• cap_drop=['ALL'] + security_opt=['no-new-privileges:true']
• Resource caps: memory (512m), CPU, pid count (100)
• Skill bundle mounted read-only at /src
• /tmp mounted as tmpfs (64m) for script scratch space
• Full container destruction on session exit — nothing persists

---

Architecture

TARGET
  └─ source resolver         (local / github: / npm: / pypi:)
       └─ Docker orchestrator   (hardened container)
            └─ monitor session   (filesystem / process / network / env)
                 ├─ deterministic driver   (invoke every script once)
                 └─ agent-loop driver      (full mode — LLM drives script selection)
                      └─ script executor   (docker exec, hard timeout, read-capture)
            └─ detection engine   (YAML rules over behavioral timeline)
                 └─ verdict + renderer   (timeline / JSON / SARIF)

This project reuses nyuwaymcpsandbox's container orchestration, behavioral monitors, detection engine, LLM backend, and source resolver wholesale. The new components are the agent-loop driver, deterministic script prober, Docker script executor, monitor-event correlation (sandbox/correlate.py), and the in-process read-capture bootstrap (sandbox/py_read_capture.py).

---

Known limitations (v0.1)

• Single-turn agent loop. One LLM response produces zero or more script invocations; there is no multi-turn loop where script output is fed back. Same boundary as nyuwaymcpsandbox's LLM driver today.
• Filesystem read-capture is Python-only. sensitive_read_outside_bundle has a producer only for Python scripts (in-process builtins.open patch). A bash/JS/Ruby script reading ~/.ssh/id_rsa leaves no filesystem.read evidence.
• Process-monitor polling granularity. ProcessMonitor polls docker top every 0.1s. A subprocess that spawns and exits in milliseconds can fall between polls and never appear on the timeline.
• file_write_outside_bundle only sees writes under the bundle mount. The filesystem monitor watches /src via a host-side watchdog observer. Writes to the container's own writable layer (e.g. /etc/cron.d/…) are not seen.
• NetworkMonitor needs a shared network namespace. With the default network_mode=none, pass --allow-network to observe (not just block) network behavior.

---

Requirements

• Python 3.11+
• Docker (Linux or macOS with Docker Desktop; Windows via Docker Desktop + WSL2)
• An LLM API key — any provider litellm supports, or local Ollama for air-gapped runs (Full mode only)

---

Development

pip install -e ".[dev]"
pytest tests/ -q        # 314 unit tests
ruff check .

Test fixtures under tests/fixtures/skill_bundles/ include a benign converter and deliberately malicious skills for end-to-end smoke testing:

nyuwayskillsandbox detonate tests/fixtures/skill_bundles/benign_converter   # expect PASS
nyuwayskillsandbox detonate tests/fixtures/skill_bundles/malicious_exfil --allow-network  # expect HIGH
nyuwayskillsandbox detonate tests/fixtures/skill_bundles/shell_persistence  # expect LOW

---

Contributing

• New detection rules — drop a YAML file into nyuwayskillsandbox/detection/builtin/, add a test in tests/test_detection_rules.py, open a PR.
• Bug reports — include the rendered timeline (--output timeline) and raw JSON (--output json).
• Real-world skill findings — file an issue with the target, the detonation command, and the full timeline output.

---

License

Apache 2.0. See LICENSE.

---

Links

• Website: https://nyuway.ai
• Static skill scanner: https://github.com/Nyuway-Cybersecurity/nyuwayskillscanner
• MCP server sandbox: https://github.com/Nyuway-Cybersecurity/nyuwaymcpsandbox
• Issues: https://github.com/Nyuway-Cybersecurity/nyuwayagentskillsandbox/issues