multilingual-gsm-symbolic

A Python package for generating synthetic multilingual math problems from symbolic templates. Allows you to create more than a thousand examples from just one problem and allows you to test if the LLMs actually understand the problem or whether it was just lucky pattern-matching.

⏳ Installation

pip install multilingual-gsm-symbolic

👩‍💻 Get started

from multilingual_gsm_symbolic import load_data, available_languages

# see possible languages
print(available_languages())
# {'eng': {'number of samples': 100}, 'dan': {'number of samples': 100}, ...}

# Load English templates
templates = load_data("eng")

# Generate concrete questions from a template
questions = templates[0].generate_questions(n=10)

for q in questions:
    print(q.question)
    print(q.answer)
    print()

Running experiments

You might often be interested in some sort of variation upon the dataset. E.g. does the performance degredation happens only due to the changes names:

# We can also control the synthetic generation: 
# fix numeric variables and only vary names/strings
defaults = templates[0].get_default_assignments()
number_vars = {var: val for var, val in defaults.items() if not isinstance(val, str)}
questions = templates[0].generate_questions(n=5, fixed=number_vars, verbose=False)

You can also inspect the available combinations directly:

# get up to 100 unique numeric assignments for a template
combinations = templates[0].get_combinations(limit=100, only_numeric=True)
print(len(combinations))
print(combinations[:3])

You could imagine similar ablations, but adding spelling errors, introducing irrelevant task information like "Hey just a small math question: {question}" or similar.

📋 Template format

Templates are TOML files with the following fields:

Computational validation is enforced by CI and is not stored in templates. Absent metadata fields are omitted from TOML and load as None in Python.

Annotated question syntax

{variable, default_value}   — placeholder in the question text
#init:
- $var = range(low, high)   — variable sampled from a range
- $var = sample([a, b, c])  — variable sampled from a list
#conditions:
- is_int(x / y)             — constraint that must hold for a combination to be valid
#answer: x * y + z          — Python expression evaluated to produce the numeric answer

question = "A fog bank rolls in over a city at 3 miles/hour. The city is 42 miles wide. How many hours will it take for the fog bank to cover the city?"

answer = "At 3 miles/hour, it will take 42/3=14 hours for the fog to cover the city."

id_orig = 0
id_shuffled = 0
creation = "example"
language = "eng"

question_annotated = """
A fog bank rolls in over a city at {speed,3} miles/hour. The city is {width,42} miles wide. How many hours will it take for the fog bank to cover the city?

#init:
- $speed = range(1, 20)
- $width = range(2, 100)

#conditions:
- is_int(width / speed)

#answer: width // speed
"""

answer_annotated = "At {speed} miles/hour, it will take {width}/{speed}={width//speed} hours for the fog to cover the city."

question = "A store sells apples for $2 each and oranges for $3 each. If you buy 4 apples and 5 oranges, how much do you spend?"

answer = "You spend 4*2 + 5*3 = 8 + 15 = $23."

id_orig = 0
id_shuffled = 0
creation = "example"
language = "eng"

question_annotated = """
A store sells apples for ${apple_price,2} each and oranges for ${orange_price,3} each. If you buy {n_apples,4} apples and {n_oranges,5} oranges, how much do you spend?

#init:
- $apple_price = range(1, 10)
- $orange_price = range(1, 10)
- $n_apples = range(1, 20)
- $n_oranges = range(1, 20)

#conditions:
- True

#answer: apple_price * n_apples + orange_price * n_oranges
"""

answer_annotated = "You spend {n_apples}*{apple_price} + {n_oranges}*{orange_price} = {n_apples*apple_price} + {n_oranges*orange_price} = ${apple_price*n_apples + orange_price*n_oranges}."

question = "A car travels at 60 mph for 3 hours. How far does it travel?"

answer = """
Distance = speed × time = 60 × 3 = 180 miles.
#### 180
"""

id_orig = 0
id_shuffled = 0
creation = "example"
language = "eng"

question_annotated = """
A car travels at {speed,60} mph for {hours,3} hours. How far does it travel?

#init:
- $speed = range(20, 100, 10)
- $hours = range(1, 9)

#conditions:
- is_int(speed * hours / 10)

#answer: speed * hours
"""

answer_annotated = """
Distance = speed × time = {speed} × {hours} = {speed * hours} miles.
#### {speed * hours}
"""

from multilingual_gsm_symbolic.templates import AnnotatedQuestion

template = AnnotatedQuestion.from_toml("my_template.toml")
questions = template.generate_questions(n=5)
for q in questions:
    print(q.question)
    print(q.answer)

🗃️ Data

The English templates are derived from Apple's GSM-Symbolic paper, from which the remainder is derived. For example, the Danish templates were initially translated using GPT-5.4, localized and reviewed by native speakers, and validated computationally and using Claude Opus. The original concrete problems are from GSM8k.

You can see the available languages as follows:

from multilingual_gsm_symbolic import available_languages, load_data

# see possible languages
print(available_languages())
# {'eng': {'number of samples': 100}, 'dan': {'number of samples': 100}, ...}

# Inspect template metadata:
template = load_data("dan")[0]
template.source_language  # 'eng'
template.initial_translation_model  # 'gpt-5.4'
template.human_validated  # 'by three native speakers'
template.error_analysis

The following table shows the list of validated languages:

Want to add a new language?

Want to add a new language or validate an existing one? Great to hear. The src/multilingual_gsm_symbolic/data/templates/ folder contains the templates for each language, and src/scripts/translate_templates.py can be used to translate the templates from one language to another. We have already pre-generated a few languages; see the templates folder for which ones. Each language folder also contains an instruction.toml with the prompt used to evaluate models (available through load_instruction), which should be validated along with the templates. Once you have validated the examples you can submit a PR with the changes.

📖 API reference

function load_data

load_data(language="eng", directory=None) → list[AnnotatedQuestion]

Load symbolic templates.

function load_replacements

load_replacements(language="eng") → dict

Load language-specific named values (e.g. lists of names, places) used inside templates.

function load_instruction

load_instruction(language="eng") → str

Load the language-specific instruction used to prompt a model with a question (solve step by step and put the final answer in \boxed{}). It is meant to be followed by a blank line and the question. Raises FileNotFoundError if the language has no instruction yet.

function load_gsm

load_gsm(language="eng", directory=None) → list[GSMProblem]

Load the bundled concrete problems for a given language.

class AnnotatedQuestion

Core class representing a symbolic template. Constructed from a TOML template file via AnnotatedQuestion.from_toml(path).

^{method AnnotatedQuestion.generate_questions}

Generate concrete Question instances from the template.

^{method AnnotatedQuestion.get_default_assignments}

Extract the default variable values from the question template placeholders.

^{method AnnotatedQuestion.get_combinations}

Enumerate unique valid assignments for a template.

^{method AnnotatedQuestion.format_question}

Render the question text for a given variable assignment.

^{method AnnotatedQuestion.format_answer}

Render the answer text for a given variable assignment.

class Question

Dataclass holding a single generated problem.

Acknowledgement

The symbolic template engine and the danish subset were originally developed as part of the m-gsm-symbolic project at the Centre for Humanities Computing by:

• Kenneth Enevoldsen
• Sofie Mosegaard
• Simon Enni

The initial template format was derived from Apple's GSM-Symbolic paper and the original concrete problems are from GSM8k.

The code was refactored for optimizations and usability by Kenneth Enevoldsen, who is also the current maintainer.