This repository contains the code for the A3 framework, which uses LLMs to systematically generate synthetic web agent training data by decomposing the annotation process into three roles: Task Designer, Annotator, and Supervisor.

Installation

pip install agent-as-annotators

Or install from source:

git clone https://github.com/McGill-NLP/agent-as-annotators.git
cd agent-as-annotators
pip install -e .

Quick Start: Evaluation

1. Serve a model with vLLM

vllm serve --config configs/vllm/Qwen3.5-9B.yaml

2. Run evaluation

a3-eval --benchmark webarena_test --model A3-qwen3.5-9b

Pipeline: Generating A3-Synth

The A3 pipeline generates synthetic training data in 5 steps:

Step 1: Create personas

python scripts/create_personas.py

Step 2: Generate task intents (via exploration)

# 2a. Build the exploration.tasks.json config that a3-explore consumes.
# One task per (persona, site) pair, written to
# agent_as_annotators/configs/<exploration_model>/exploration.tasks.json so
# that a3-explore picks it up via importlib.resources.
python scripts/generate_exploration_tasks.py -m <exploration-model>

# 2b. Run the exploration agent. Trajectories are saved as agentlab pickles
# under $AGENTLAB_EXP_ROOT/<study_dir>/<task_dir>/step_*.pkl.gz.
a3-explore

# 2c. Extract chat messages from each step pickle into a parallel JSON tree
# at outputs/chat_messages/<study_dir>/<task_dir>/step_*.json.
python scripts/extract_chat_messages.py --find-latest <exploration-model>

# 2d. For each trajectory, randomly sample N steps (default 3, skipping step 0),
# append the TASK_INTENT_PROMPT_TEMPLATE as a final user turn, and write each
# prompt as outputs/task_intents/prompts/<exploration_model>/task_<i>.step_<j>.json.
python scripts/prepare_tasks_intents_prompts.py --find-latest <exploration-model>

# 2e. Send each prepared prompt to the Task Designer LLM. Completions land in
# outputs/task_intents/completions/<exploration_model>/<task_designer_model>/.
python scripts/generate_task_intents.py \
    --exploration-model <exploration-model> \
    --model <task-designer-model>

What exactly gets passed to the Task Designer

One exploration step, not the whole trajectory. Each prepared prompt is the exploration agent's own prompt/response pair at a single sampled step, with one user turn appended. For the released gemini-3-pro-preview run that is exactly four messages:

So the Task Designer sees the AXTree and screenshot of the sampled step only. Earlier steps reach it solely through the agent's # History of interaction with the task: block, which lists past actions and nothing else (## step 0 <action>click('156')</action> …) — no earlier observations, and no multi-turn chat history.

How exploration_step_num is chosen

prepare_tasks_intents_prompts.py globs step_*.json in each trajectory, sorts by step number, drops step_0, and takes a uniform random sample without replacement:

random.seed(seed)                                     # --seed, default 42
sampled = random.sample(step_files, min(num_samples, len(step_files)))

Each sampled step becomes one prompt file task_<task_num>.step_<step_num>.json, and that step_num is what scripts/create_synth_configs.py records as exploration_step_num in the A3-Synth task configs. The released run used the defaults --num_samples 3, --num_intents 2, --seed 42, which is why exploration_step_num in A3-Synth is never 0, is capped at the exploration budget of 20 steps, and appears at most three times per exploration trajectory.

Known wart, kept for reproducibility. A trajectory's terminal step records no agent call, so it extracts to messages: [] and, if sampled, yields a prompt whose only turn is the appended instruction — the Task Designer is asked to write tasks "based on the conversation above" with no conversation above. This affected 381 of 4497 prompts (8.5%) in the released run. The default behaviour is unchanged so that generation reproduces exactly; pass --skip-empty-steps to exclude those steps in new collections.

Step 3: Create A3-Synth task configs

python scripts/create_synth_configs.py

Step 4: Collect trajectories

a3-synth --benchmark a3_synth --model gemini-3-pro

Step 5: Convert to training data

python scripts/convert_trajectories_to_json.py
python scripts/generate_rft_data.py

Training

a3-train --config configs/train/qwen3.5-9b.json

Training uses SFT with FSDP for multi-GPU parallelism. See configs/train/ for hyperparameters and configs/accelerate/ for FSDP configuration.

Reproducing Dataset Statistics

The A3-Synth website-state coverage, instruction-diversity, and frequent-page statistics can be reproduced with a standalone standard-library script:

python3.12 scripts/analyze_a3_synth.py data/A3-Synth/training/train.jsonl

See the methodology and data instructions for the exact definitions and a machine-readable output option.

To reproduce the full-benchmark Wilson confidence intervals and exact paired McNemar tests, install the analysis dependency, place the base and A3 AgentLab study directories under agentlab_results/, and run:

pip install -e '.[analysis]'
python scripts/analyze_evaluation_significance.py \
  --results-root agentlab_results

CLI Commands

Project Structure

agent-as-annotators/
  agent_as_annotators/       # Core package
    cli/                     # CLI entry points (eval, synth, explore, train)
    modeling.py              # Agent model wrapper (vLLM, Gemini, OpenAI)
    prompts/                 # All prompt templates
    judge/                   # Inverted evaluation protocol (Judge module)
    benchmarks/a3_synth/     # A3-Synth benchmark registration
    exploration/             # Exploration task registration
    utils/                   # Utilities
    configs/a3_synth/        # A3-Synth task configurations
  configs/
    model_configs.json       # Model registry
    train/                   # Training hyperparameters
    vllm/                    # vLLM serving configs
    accelerate/              # FSDP configs
  scripts/                   # Data pipeline scripts