dl-faults-sandbox 0.1.0

MER: Minimal Extraction and Reconstruction for deep learning bug reproduction environments.

MER: Minimal Extraction and Reconstruction

MER is an agentic framework for verifying and reconstructing deep learning bug reproduction environments.

Installation

1. Clone the repository:

   git clone <repository_url>
   cd DL-Faults-Sandbox
   

2. Install dependencies: It is recommended to use a virtual environment.

   python3 -m venv venv
   source venv/bin/activate
   pip install -r requirements.txt
   

   Or install the package directly with pip:

   pip install .
   

3. Docker Setup: Ensure Docker is installed and running on your machine, as MER agents utilize Docker containers for environment isolation.

Usage

Running the Orchestrator

The main entry point for the MER system is mer/orchestrator/main.py.

# After `pip install .`
mer

# Run all bug samples under dataset/
python3 -m mer.orchestrator.main

# Run a subset by bug_id
python3 -m mer.orchestrator.main --sample huggingface_diffusers_12975 --sample pyg-team_pytorch_geometric_10548

# Limit number of samples
python3 -m mer.orchestrator.main --limit 2

# Use a custom dataset root
python3 -m mer.orchestrator.main --dataset-root /path/to/dataset

# Enable LangSmith tracing
python3 -m mer.orchestrator.main --langsmith --langsmith-project MER

# Run no-planner ablation
python3 -m mer.orchestrator.main --ablation-profile no_planner_agent

# Run no-data ablation
python3 -m mer.orchestrator.main --ablation-profile no_data_agent

# Run no-determinism ablation
python3 -m mer.orchestrator.main --ablation-profile no_determinism_agent

# Run no-environment ablation
python3 -m mer.orchestrator.main --ablation-profile no_environment_agent

# Combine full-agent and sub-component ablations
python3 -m mer.orchestrator.main \
  --disable-agent planner \
  --disable-component data.llm_terminal_judge \
  --disable-component determinism.profile_llm

For external developers, install the package and use the portable study runner:

pip install dl-faults-sandbox

mer-study-runner \
  --bug-id 035 \
  --codebase /path/to/cloned/repo \
  --bug-report /path/to/bug_report.txt \
  --repro /path/to/reproduce.py \
  --openai-api-key "$OPENAI_API_KEY"

The runner creates the dataset layout MER expects:

.mer-study/
  dataset/
    <bug-id>/
      bug_report.txt
      reproduce.py
      codebase/

By default the codebase is symlinked into the staged dataset. Use --mode copy if you want an isolated copy instead. Use --prepare-only if you only want the staged dataset without invoking MER yet.

For local repository-based testing, the repo also includes:

scripts/setup_and_run_bug.sh --openai-api-key "$OPENAI_API_KEY"

Each bug run now writes exactly one run log file:

• logs/activities/<run_id>/run.log

Additional traceability artifacts are non-log files:

• workflow_logs_manifest (all activity logs)
• workflow_trajectory_json (phase-by-phase trajectory with durations and terminal-check outcomes)
• workflow_errors_json (structured failures and stack traces)
• workflow_langsmith_trace_config_json (trace metadata used for this run)

Batch mode also writes logs/dataset_batch_summary_<timestamp>.json including pointers to these artifacts per bug.

Ablations

Ablation config is fully CLI/JSON-driven:

• --ablation-profile <name>: apply a predefined profile (repeatable)
• --disable-agent <planner|environment|data|determinism>: disable a whole agent
• --disable-component <agent.component>: disable sub-components
• --ablation-config <path.json>: load JSON config with keys profiles, disable_agents, disable_components, notes
• --ablation-json '<json>': inline JSON config

Metadata is persisted for every run and batch:

• workflow_ablation_config_json artifact in each run
• ablation_label and ablation_config_path in per-bug batch result entries
• batch summary includes top-level ablation payload

When environment is disabled, MER bootstraps a barebones CPU sandbox (dl-sandbox:cpu-py310-torch) if needed and passes it through run context so data/determinism still run end-to-end.

Dataset Link Downloading

The data agent now scans both bug_report.* and reproduce_issue.py for dataset links, then downloads into the bug folder when remote artifacts are required.

Provider-aware handling:

• Kaggle links (kaggle.com/datasets/..., kaggle.com/competitions/...) via Kaggle API.
• GitHub links (github.com/..., raw.githubusercontent.com/...) via GitHub endpoints.
• Other links via direct HTTP download.
• requested_format is inferred from URL/query/context; if a ZIP is downloaded and a non-zip format is requested, MER extracts and returns a matching file when possible.

Configuration (environment variables):

• KAGGLE_USERNAME and KAGGLE_KEY for Kaggle API downloads.
• GITHUB_TOKEN (optional but recommended) for GitHub API/authenticated downloads.

Built-in profiles:

• no_planner_agent
• no_environment_agent
• no_data_agent
• no_determinism_agent
• planner_no_llm_heuristics
• planner_no_llm_policy_override
• planner_no_llm
• environment_no_dependency_planner_llm
• environment_no_resource_mocking
• environment_no_gpu_detection
• data_no_bug_static_scan
• data_no_remote_download
• data_no_bug_dir_reconstruction
• data_no_fidelity_iteration
• data_no_repo_context_scan
• data_no_data_role_assessment
• data_no_llm
• determinism_no_profile_llm
• determinism_no_refinement

Legacy/utility profile:

• none

LangSmith

Set LANGSMITH_API_KEY in your environment (or .env) and run with --langsmith. Optional env/flags:

• LANGSMITH_PROJECT / --langsmith-project
• LANGSMITH_ENDPOINT / --langsmith-endpoint

Development

• Agents: Located in mer/agents/. Each agent (local, data, determinism, driver) has its own package.
• Tools: Located in mer/tools/. Generic tools are shared; specific tools are injected by the orchestrator.

Structure

• mer/orchestrator/: Core logic and agent management.
• mer/agents/: Autonomous agents for each phase of the pipeline.
• mer/tools/: Tool definitions (Docker control, Static Analysis, etc.).
• mer/libs/: Native libraries (e.g., libdeterminism.so source).