reasoning-benchmarks 1.0.1

Generate argumentation reasoning benchmarks and prompts.

reasoning-benchmarks

The reasoning-benchmarks library can be used to dynamically generate reasoning tasks based on formal argumentation to evaluate reasoning capabilities.

Reasoning puzzles

The following types of reasoning puzzles can be generated:

The following is a reasoning puzzle. Witnesses should always be believed
unless there is believable testimony that they are lying. Now consider the
following facts:

Witness Livia says that the employee arrived on time
Witness Aniyah says that Livia is lying
Witness Tatum says that Aniyah is lying

Final question: can we believe that the employee arrived on time??
End your answer with: "Answer: yes or no".

In addition to the binary reasoning task, we also provide multi-label tasks, where the solution requires the identification of all believable arguments:

The following is a reasoning puzzle. Witnesses should always be believed
unless there is believable testimony that they are not credible. Now consider the
following facts:

Witness Grace says that the password was changed
Witness Amelia says that Grace is not credible
Witness Micah says that Amelia is not credible

Final question: which witnesses can be believed?
End your answer with: "Answer: list of names that can be believed".
E.g. Answer: ['Alice', 'Bob']

Installation

pip install reasoning-benchmarks

Quickstart

from reasoning_benchmarks import generate_benchmark

# Quickly generate a benchmark
benchmark = generate_benchmark()

# Generate a benchmarks with optional parameters:
benchmark = generate_benchmark(
	chain_range=(1, 15), # Generates all unique chain graphs with 1 to 15 arguments 
	star_range=(3, 10), # Generates all unique 'star' graphs with 3 to 10 arguments
	shuffled=True, # Whether the arguments in the prompt should be shuffled or presented in order
	percentage_irrelevant=0.1, # The ratio of prompts that contain irrelevant argument chains
    irrelevant_len_range=(1,5), # A tuple (min_length, max_length) to determine the length of irrelevant chains, where (1, 5) would include irrelevant chains of length 1 to 5.
    irrelevant_num_range=(1,3), # A tuple (min_num, max_num) to determine the number of irrelevant chains, where (1, 3) would include 1 to 3 irrelevant chains.
	attack_words=["not credible", "unreliable", "lying"], # Which attack words are used in the prompt (randomly sampled from the provided list)
	num_variations_per_graph=(1, 1), # The number of variations per chain graph and star graph
	seed=95, # A fixed seed for reproducibility
    output_file='benchmark.csv', #save the benchmark DataFrame to this CSV file.
)

print(benchmark.head())

This allows for the dynamic generation of reasoning puzzles of scaling complexity, ranging from simple to complex.

Argumentation backend

The puzzles are based on argument attack graphs, that can be created using the library as follows:

from reasoning_benchmarks import ArgumentGraph, Argument, plot_argument_graph

# Initialize the graph
graph = ArgumentGraph()

# Create two arguments
a = Argument(0, 'Alice says that the train is late.')
b = Argument(1, 'Bob says that Alice is lying.')

# Add the arguments to the graph
graph.add_argument(a)
graph.add_argument(b)

# Add the attack between argument b and a
graph.add_attack(b, a)

# Determine the values of the arguments 
graph.solve()

# Display the resulting argument graph visually
plot_argument_graph(graph)

Based on this graph, you can generate a prompt using the create_prompt function:

from reasoning_benchmarks import create_prompt

prompt = create_prompt(graph)

Functions are included to quickly generate one, or multiple graphs:

from reasoning_benchmarks import generate_chain, generate_star_graph, generate_all_chains, generate_all_star_graphs

# Generate a chain graph with 3 arguments
chain = generate_chain(3)

# Generate a star graph with one center arguments, with chains of 3, 1, and 4 arguments attacking it
star = generate_star_graph([3, 1, 4])

# Generate all unique chain graphs with between 1 and 25 total number of arguments
chains = generate_all_chains(min_n=1, max_n=25)

# Generate all unique star graphs between 3 and 6 total number of arguments
stars = generate_all_star_graphs(min_n=3, max_n=6)

Development Build

python -m build

The package includes CSV resources under reasoning_benchmarks/src and accesses them via importlib.resources. These contain lists of names and statements used to generate the prompts and can be customized.