MARGIN Arena

MARGIN provides counted resource limits and inspectable records for agents working on external tasks. Its scope is detect, analyze, and understand. It does not supply a new research dataset, train agents, implement interventions, or claim that a particular limit induces reward hacking.

Version 0.2.0b1 is a beta. Choose a local Hugging Face causal language model, attach an external Inspect task or ControlArena setting, and specify each trial's resources. The source setting keeps its dataset, tools, sandbox, and scorer.

Install

Python 3.11 or later is required; the tested environments use Python 3.12. Use Linux or WSL for Docker settings.

pip install "margin-arena[hf,control]==0.2.0b1"

Until the version appears on PyPI, install the public repository:

pip install "margin-arena[hf,control] @ git+https://github.com/CarterYoo/margin-arena.git"

For immutable experiments, replace the branch with a recorded commit. Tested CPU dependency snapshots are in requirements/runtime.txt and requirements/analysis.txt. Install CPU Torch from the official PyTorch index before applying those files. GPU users must select the appropriate official PyTorch build and record their hardware and driver versions.

Five explicit limits

None disables time, tool, or verification limits. Sessions default to one. Each generation sees its remaining counters. Require 0 < report_tokens < tokens; per_call_tokens caps one complete generation. Input tokens are recorded separately. Truncated output is terminal and never executes a partial request. An early final answer or stop is terminal even when resources remain.

Model loading and initial sandbox preparation precede the trial clock. Source scoring and cleanup follow actor execution and are not charged to it. If generation is killed before returning, actual usage is unknown and its outstanding reservation stays in the ledger. Polling and process termination add a small measured delay; an exact millisecond stop is not promised.

Use a ControlArena setting

from margin_arena import Arena, HFModel, Limits
from control_arena.settings.apps.apps_setting import AppsSetting

def main():
    arena = Arena(
        HFModel(name="YOUR_MODEL_REPOSITORY", revision="YOUR_MODEL_COMMIT",
                device="cuda", dtype="bfloat16", context_window=8192),
        Limits(tokens=16000, seconds=600, tool_calls=40, sessions=1,
               per_call_tokens=1024),
        seed=0,
    )
    task = arena.from_control_arena(AppsSetting())
    arena.evaluate(task, out="runs/development_001", limit=1)

if __name__ == "__main__":
    main()

The main guard is required for the separate model process. Weights must fit the selected device. This beta adds no quantization, multi GPU distribution, or automatic backend switching.

APPS has a submission tool and official scorer, but no interactive verification tool. The example therefore does not invent a verification allowance. For that axis, choose a source with an actual verification service and pass verification_tools=("its_tool_name",). Multiple sessions in a mutable environment require an asynchronous reset(state) callback restoring that source's initial state. Missing bindings fail before generation.

ControlArena is used in honest task mode. The actor receives the setting description and main task; hidden solutions and side task metadata stay outside its prompt. This adapter uses the setting lifecycle and official scorer, not ControlArena attack policies or control protocols. Validate each setting, especially tools that call other models or return nontext observations.

margin-eval --setting control_arena.settings.apps.apps_setting:AppsSetting \
  --model YOUR_MODEL_REPOSITORY --revision YOUR_MODEL_COMMIT \
  --device cuda --dtype bfloat16 --context-window 8192 \
  --tokens 16000 --seconds 600 --tool-calls 40 --sessions 1 --per-call-tokens 1024 \
  --limit 1 --out runs/development_001

Other external data and environments

Arena.task(dataset, tools=..., scorer=..., sandbox=..., setup=..., post_submission=..., reset=...) accepts Inspect samples and the source's lifecycle components. --binding module:function loads a trusted installed factory receiving the arena and returning this task.

margin_arena.datasets.from_jsonl and from_huggingface map explicitly selected input, target, and ID columns. Only the input becomes visible. Hub revisions resolve to a commit; row hashes retain provenance. These loaders do not create a repository environment or official benchmark scorer from a text row. Loading SWE bench issue text, for example, is not a SWE bench evaluation without its official checkout, execution, and test bindings. Text CLI modes offer only explicit match or includes scoring.

Each evaluation saves the complete sample snapshot and hash, setting configuration, package versions, limits, seed, and model identity. Preserve external image digests and assets not contained in the samples. Reuse saved samples or pin upstream dataset revisions; moving source references alone are insufficient for reproduction.

Model interface and evidence

The local backend retains ordinary Hugging Face model and tokenizer objects. It applies the model's chat template and requests one strict JSON action or final report. Exact rendered text, input IDs, mask, output IDs, original response, sampling parameters, and tool schema are recorded. It does not silently trim context, repair JSON, remove reasoning, or retry failed responses. Models without a chat template and encoder decoder models are unsupported. Model selection does not establish competence: check the exact output contract on development cases before a larger run.

Every sample has a hash chained events.jsonl, result.json, and optional activation artifacts. Inspect logs retain source scores. Their top level model name is the explicitly unused mockllm/margin_transport_unused transport; real inference runs through the MARGIN backend recorded in configuration.json and each trial. Do not attribute research results to the placeholder.

margin verify-trial runs/development_001/sample_SAMPLE_ID

Verification replays resource admissions and settlements, checks raw token counts, and compares the summary with its journal. It does not rerun external scorers, prove human review, or authenticate against a host that can rewrite all files and hashes. Infrastructure errors stop later sample execution. Malformed output, truncation, task failure, and refusal remain separate outcomes.

Read only interpretation

Set HFModel(capture_modules=("MODEL_SPECIFIC_MODULE",)) or repeat --capture-module. Native PyTorch hooks capture the first prefill forward of the original generation, with exact positions, module names, shapes, dtype, model identity, unique request ID, and safetensors hash. This beta does not save decode step activations or provide activation patching.

These are software checks with an untrained tiny CPU model, not trained model findings or universal architecture support. SAE training preprocessing and intended layer must also match; no general pretrained SAE is supplied. Offline replay can differ due to dtype, kernels, cache, or library versions. Preserve that distinction.

ControlArena 19 requires Hugging Face Hub below 1, whereas TransformerLens 3.9 requires Transformers 5 and Hub 1 or later. Use separate environments. There is intentionally no combined all extra.

python -m venv analysis_env
analysis_env/bin/pip install "margin-arena[hf,analysis,nnsight,lens,sae,captum]==0.2.0b1"

from margin_arena.interpret import load_activations, to_pandas
tensors, metadata = load_activations("runs/development_001/sample_ID/activations/REQUEST_ID.safetensors")
table = to_pandas("runs/development_001/sample_ID/events.jsonl")

Enforcement and interpretation limits

Verification limits count named calls, not commands inside an unrestricted shell. True service execution quotas require an exclusive source service binding. MARGIN rejects unbound quotas but cannot discover every alternative path in third party environments.

Local deadlines terminate the model worker. External tool deadlines cancel the awaiting coroutine and record remote effects as unconfirmed. Remote job cleanup depends on the source sandbox. Tools that secretly invoke other models are unsupported because those tokens are outside this ledger. Validate source properties before claiming exact end to end enforcement.

Resource exhaustion does not prove a model felt pressure. A source score does not automatically label deception or reward hacking. Compare controlled conditions on the same external tasks and inspect actual actions, effects, and independent source scores. This release reports no new model experiment or causal finding.

Development checks

python -m unittest discover -s tests -v
python -m margin_arena build
MARGIN_DOCKER_TESTS=1 python -m unittest discover -s tests -p test_docker.py -v
python scripts/check_external_apps.py --out runs/apps_binding_check

The final command fetches one public APPS row and submits a fixed diagnostic program through the original Docker setting and scorer, without model calls. The three old Artifact Depot tasks remain legacy regression fixtures, described in the archived guide; they are not the beta's research dataset. First party code is MIT licensed; external artifacts retain their own licenses.

See the beta report for actual validation and release status. References: ControlArena, Inspect limits, NNsight, TransformerLens, Captum, SAE Lens.