📖 About The Project

This repository contains the code and experimental setup for the research paper titled "Effects of structure on reasoning in instance-level Self-Discover". The project introduces iSELF-DISCOVER, an instance-level adaptation of the SELF-DISCOVER framework, to empirically evaluate the performance of dynamically generated structured JSON reasoning against its unstructured, natural language counterpart.

Our findings, particularly on benchmarks like MATH, BBH, and a replicated T4D, suggest a consistent advantage for unstructured reasoning plans. This work aims to provide insights into optimal plan generation granularity (instance-level vs. task-level) and the nuanced reliance on structured formats for complex LLM problem-solving.

🚀 Key Features

• Implementation of the iSELF-DISCOVER framework.
• Support for both structured (JSON) and unstructured (natural language) reasoning plan generation and execution.
• Evaluation scripts for benchmarks: BBH, T4D (replicated), and MATH.
• Integration with models like LLaMA-3.1-405B-Instruct and Mistral-Large via APIs.
• Configuration for 0-shot and few-shot (e.g., 5-shot) guidance for plan generation.

🛠️ Getting Started

Prerequisites

• Python (e.g., 3.9+)
• Poetry (for dependency management)
• API Keys:
  • MISTRAL_API_KEY
  • LAMBDA_LABS_API_KEY

Installation

1. Clone the repository:

   git clone https://github.com/anonymous/self-discover.git
   cd self-discover
   

2. Set up your Python virtual environment: It's recommended to create and activate a virtual environment. For example:

   python -m venv .venv
   source .venv/bin/activate  # On Windows: .venv\Scripts\activate
   

3. Install dependencies using Poetry:

   poetry install
   

4. Set up API Keys: Create a .env file in the root of the project with your API keys:

   MISTRAL_API_KEY="YOUR_MISTRAL_API_KEY"
   LAMBDA_LABS_API_KEY="YOUR_LAMBDA_LABS_API_KEY"
   

⚙️ Configuration

Experiments are primarily configured via evals/config.toml.

Key parameters in evals/config.toml:

[MODEL]
# Specifies the model family to use.
# "mistral" will use "mistral-large-2407".
# "llama" will use "llama3.1-405b-instruct-fp8".
model_type = "mistral"

[EVAL]
# batch_size controls the number of instances processed in a single batch.
# This is useful for managing API rate limits or resource usage.
batch_size = 5
# wait_time is the duration (in seconds) to wait if any Exceptions arises during the evaluation process (Can allow breathing room if unexpected API errors occur).
wait_time = 1

🔬 Running Research Experiments

To reproduce the experiments presented in the paper:

1. Ensure your Poetry environment is active: If you haven't already, activate it:

   source .venv/bin/activate # Or your chosen environment activation command
   # Alternatively, you can prefix commands with `poetry run`
   

2. Prepare Log Directory: The evaluation scripts are designed to check for current progress in evals/logs and resume. If you want to run experiments from scratch, ensure the evals/logs directory is deleted or empty.

3. Run evaluation scripts (from the project root directory):

   • To evaluate the iSELF-DISCOVER approach (our proposed method): Use evals/iself_discover_eval.py. Key arguments include:

     • --structured: (Flag, no value) Use structured JSON reasoning. If omitted, defaults to unstructured.
     • --few_shot_examples <N>: Number of few-shot examples to use (e.g., 0 for zero-shot, 5 for five-shot). Defaults to 0.
     • --stream: (Flag, no value) Stream LangGraph steps one by one and log debug messages for each output from every stage. Defaults to False.

     Example (unstructured, 0-shot, run from root):

     python evals/iself_discover_eval.py
     

     Example (structured, 5-shot, with streaming, run from root):

     python evals/iself_discover_eval.py --structured --few_shot_examples 5 --stream
     

   • To evaluate the original SELF-DISCOVER approach (baseline): Use evals/self_discover_eval.py. Key arguments include:

     • --phase <PHASE_VALUE>: (Required for research experiments) Specify the stage of the SELF-DISCOVER framework to run.
       • Use 1 to run Phase I (e.g., task-specific reasoning structure discovery).
       • Use 2 to run Phase II (e.g., solving instances using a discovered structure). While the script might default to running both if this argument is omitted, for research evaluation purposes, you must explicitly specify either 1 or 2.
     • --stream: (Flag, no value) Stream LangGraph steps and log debug messages. Defaults to False.

     Example (running Phase I, no streaming, run from root):

     python evals/self_discover_eval.py --phase 1
     

     Example (running Phase II with streaming, run from root):

     python evals/self_discover_eval.py --phase 2 --stream
     

Datasets

The necessary datasets are available locally in the data/ folder:

• T4D Dataset (data/t4d/t4d.csv): 564 samples of replicated T4D benchmark
• BBH Dataset (data/bbh/bbh.csv): 6,511 samples across 25 BigBench Hard subsets
• MATH Dataset (data/math/math.csv): 200 samples of MATH benchmark subsample

The evaluation scripts are configured to automatically load these datasets sequentially. No manual dataset placement or downloading is required - the scripts will handle loading from the local CSV files using the HuggingFace datasets library.

Expected Output

Experimental results, detailed logs, and any generated reasoning traces will be stored in the evals/logs directory. The structure within this directory should allow for identification of results based on the benchmark, model, and experimental configuration.

📜 License

Distributed under the MIT License. See LICENSE file for more information.

🙏 Acknowledgements

• Authors of the original SELF-DISCOVER paper (Zhou et al., 2024).
• Mistral AI for providing a generous free-tier access.
• The open-source community for tools and libraries that made this work possible.