consol 0.3.0

Confident Solver enables you to solve problems confidently and efficiently by leveraging Large Language Models (LLMs) enhanced with robust statistical approaches, such as the SPRT

🎮 ConSol: Confident Solver

ConSol enables you to solve problems confidently and efficiently by leveraging Large Language Models (LLMs) enhanced with robust statistical approaches, such as the Sequential Probability Ratio Test (SPRT).

---

🚀 Quick Start

Installation

Install ConSol directly from PyPI:

pip install consol

Command Line Usage

Solve a problem using ConSol via the CLI:

$ consol --prompt "1 + 1 = ?"
2

SDK Usage

Integrate ConSol into your Python applications:

from consol import ConfidentSolver

# Initialize ConSol with custom parameters
consol = ConfidentSolver(
    llm_model="gpt-4o-mini",           # Supported models: "gpt-4o-mini", "o3-mini-low", etc.
    confidence_model="pvalue",        # Options: "msprt", "sprt", "pvalue", "bayesian_posterior", "vote40", "vote1"
    output_schema="float"             # Output formats: "abce", "float", etc.
)

# Invoke ConSol to solve a problem
answer = consol.invoke("1 + 1 = ?")
print(answer)
# Output: 2

---

🔍 About ConSol

ConSol is an innovative framework designed for precise and cost-effective problem-solving. It integrates statistical validation with LLMs to significantly enhance accuracy and efficiency, especially for mathematical computations and reasoning tasks.

Key Benefits

• 🚩 Higher Accuracy: ConSol improves OpenAI's o3-mini-low and o3-mini-medium performance on the AIME24 Benchmark by 10-17 percentage points—boosting accuracy from 60% to 70% (o3-mini-low) and 73% to 90% (o3-mini-medium).

• 💰 Cost Efficiency: ConSol reduces the number of output tokens generated by OpenAI's GPT-o3-mini models on the AIME24 Benchmark by 63.9%–84.8%, directly lowering computational and monetary costs.

For a comprehensive understanding and detailed methodology, please refer to our research publication.

Example

Accuracy

Without ConSol

$ consol --confidence_model vote1 --llm_model gpt-4o-mini --prompt "Jen enters a lottery by picking \$4\$ distinct numbers from \$S=\\{1,2,3,\\cdots,9,10\\}.\$ \$4\$ numbers are randomly chosen from \$S.\$ She wins a prize if at least two of her numbers were \$2\$ of the randomly chosen numbers, and wins the grand prize if all four of her numbers were the randomly chosen numbers. The probability of her winning the grand prize given that she won a prize is \$\\tfrac{m}{n}\$ where \$m\$ and \$n\$ are relatively prime positive integers. Find \$m+n\$."
# or simply consol --confidence_model vote1
# => 11.0

The result always changes whenever you re-run.

With ConSol

consol --confidence_model msprt --llm_model gpt-4o-mini --prompt "Jen enters a lottery by picking \$4\$ distinct numbers from \$S=\\{1,2,3,\\cdots,9,10\\}.\$ \$4\$ numbers are randomly chosen from \$S.\$ She wins a prize if at least two of her numbers were \$2\$ of the randomly chosen numbers, and wins the grand prize if all four of her numbers were the randomly chosen numbers. The probability of her winning the grand prize given that she won a prize is \$\\tfrac{m}{n}\$ where \$m\$ and \$n\$ are relatively prime positive integers. Find \$m+n\$."
# or simply consol
# => 116.0 (Always)

Typically it stops at 20-100 samples until finding consistency.

Efficiecny

With Prior Method (Ada-Cons)

$ consol --confidence_model bayesian_posterior --prompt "random integer between 0 to 20"
# => Random Number

Typically it stops at 40 trials, hitting the max trials

With Our Method (mixture SPRT)

consol --confidence_model msprt --prompt "random integer between 0 to 20"
# => Random Number

Typically it stops at 10 to 30 trials.

Verbose Output for Debugging

$ consol --debug
,answer,token_usage
0,116.0,484
1,47.0,81
2,11.0,69
3,116.0,483
4,37.0,337
5,7.0,221
6,3.0,168
7,37.0,116
8,116.0,585
9,21.0,114
10,9.0,156
11,49.0,67
12,19.0,165
13,20.0,81
14,22.0,222
15,116.0,514
16,145.0,117
17,49.0,97
18,15.0,135
19,116.0,427

The output shows the result as a table.