A Python package to reason by communicating with agents
Project description
ReComA: A Library for Reasoning via Communicating Agents
ReComA is a library designed to enable easy development of solutions for reasoning problems via communicating agents. It is a generalization of the codebase for Decomposed Prompting. The key features of the library:
- A general-purpose framework that implements many existing approaches for reasoning via mutliple agents -- DecomP, ReACT, Least-to-Most, Faithful CoT
- Can be easily extended to use other control flows (e.g., Self-Ask, IRCoT)
- Provides an interactive GUI which includes the entire reasoning trace (with underlying prompts) for easy debugging
- Built-in Best-First Search to explore multiple reasoning traces
- Can be used as a pip-installable library in your own codebase
- Configurable via JSONNET files -- no code change needed for many use cases
Table of Contents
Setup
If you want to directly make changes in this library, set it up using conda
conda create -n recoma python=3.9
conda activate recoma
pip install -r requirements.txt
To install it as a dependency in your own conda environment
pip install -e .
OpenAI Setup
This library relies on the OPENAI_API_KEY
environment variable to call GPT3+ models. Make sure
to set this env. variable
export OPENAI_API_KEY=<key>
Running ReComA
The library can be used to solve complex reasoning tasks in two modes:
Demo/Interactive Mode
python -m recoma.run_inference \
--config configs/inference/letter_cat/decomp.jsonnet \
--output_dir output/letter_cat_decomp/ \
--gradio_demo
This will start an interactive server on http://localhost:7860 for the kth letter concatenation task. Try the following question (no QID/Context needed):
Take the letters at position 3 of the words in "Reasoning via Communicating Agents" and concatenate them using a space.
The library will use text-davinci-002
model with Decomposed Prompting (specified via the input
config file) to answer this question. You can open the collapsed nodes (indicated with ▶) to see
the full execution trace (along with the prompts).
Batch Inference Mode
To use the library to produce predictions for an input file (e.g. the 3rd letter concatenation dataset with 4 words):
python -m recoma.run_inference \
--config configs/inference/letter_cat/decomp.jsonnet \
--output_dir output/letter_cat_decomp/ \
--input datasets/letter_cat/n4_eg100_pos2_space.json
Running this script will populate the output directory with :
predictions.json
: qid-to-prediction mapall_data.jsonl
: Input examples with model predictions and correctness label (using exact match)html_dump/
: Dump of the execution traces for all the examples in HTML formatsource_config.json
: JSON config used to run this experiment (for future reproducibility)
Using ReComA in your work
Using existing agents
If the provided agents are sufficient for your work, you can use this library by just defining the
configuration files and prompts. See examples in the configs/
folder.
Defining a new agent
If you define a new agent (see the models README), you need to load
them when running inference. Assuming your agents are defined under the package my_new_agents_pkg
python -m recoma.run_inference \
--config configs/inference/letter_cat/decomp.jsonnet \
--output_dir output/letter_cat_decomp/ \
--input datasets/letter_cat/n4_eg100_pos2_space.json \
--include_package my_new_agents_pkg
Please reach out if there are any questions or issues.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.