multilingual-gsm-symbolic

A Python package for generating synthetic multilingual math problems from symbolic templates. Allows you to create more than a thousand examples from just one problem and allows you to test if the LLMs actually understand the problem or whether it was just lucky pattern-matching.

⏳ Installation

pip install multilingual-gsm-symbolic

👩‍💻 Get started

from multilingual_gsm_symbolic import load_data, available_languages

# see possible languages
print(available_languages())
# {'eng': {'number of samples': 100}, 'dan': {'number of samples': 100}, ...}

# Load English templates
templates = load_data("eng")

# Generate concrete questions from a template
questions = templates[0].generate_questions(n=10)

for q in questions:
    print(q.question)
    print(q.answer)
    print()

Running experiments

You might often be interested in some sort of variation upon the dataset. E.g. does the performance degredation happens only due to the changes names:

# We can also control the synthetic generation: 
# fix numeric variables and only vary names/strings
defaults = templates[0].get_default_assignments()
number_vars = {var: val for var, val in defaults.items() if not isinstance(val, str)}
questions = templates[0].generate_questions(n=5, fixed=number_vars, verbose=False)

You can also inspect the available combinations directly:

# get up to 100 unique numeric assignments for a template
combinations = templates[0].get_combinations(limit=100, only_numeric=True)
print(len(combinations))
print(combinations[:3])

You could imagine similar ablations, but adding spelling errors, introducing irrelevant task information like "Hey just a small math question: {question}" or similar.

📋 Template format

Templates are TOML files with the following fields:

Computational validation is enforced by CI and is not stored in templates. Absent metadata fields are omitted from TOML and load as None in Python.

Annotated question syntax

{variable, default_value}   — placeholder in the question text
#init:
- $var = range(low, high)   — variable sampled from a range
- $var = sample([a, b, c])  — variable sampled from a list
#conditions:
- is_int(x / y)             — constraint that must hold for a combination to be valid
#answer: x * y + z          — Python expression evaluated to produce the numeric answer

question = "A fog bank rolls in over a city at 3 miles/hour. The city is 42 miles wide. How many hours will it take for the fog bank to cover the city?"

answer = "At 3 miles/hour, it will take 42/3=14 hours for the fog to cover the city."

id_orig = 0
id_shuffled = 0
creation = "example"
language = "eng"

question_annotated = """
A fog bank rolls in over a city at {speed,3} miles/hour. The city is {width,42} miles wide. How many hours will it take for the fog bank to cover the city?

#init:
- $speed = range(1, 20)
- $width = range(2, 100)

#conditions:
- is_int(width / speed)

#answer: width // speed
"""

answer_annotated = "At {speed} miles/hour, it will take {width}/{speed}={width//speed} hours for the fog to cover the city."

question = "A store sells apples for $2 each and oranges for $3 each. If you buy 4 apples and 5 oranges, how much do you spend?"

answer = "You spend 4*2 + 5*3 = 8 + 15 = $23."

id_orig = 0
id_shuffled = 0
creation = "example"
language = "eng"

question_annotated = """
A store sells apples for ${apple_price,2} each and oranges for ${orange_price,3} each. If you buy {n_apples,4} apples and {n_oranges,5} oranges, how much do you spend?

#init:
- $apple_price = range(1, 10)
- $orange_price = range(1, 10)
- $n_apples = range(1, 20)
- $n_oranges = range(1, 20)

#conditions:
- True

#answer: apple_price * n_apples + orange_price * n_oranges
"""

answer_annotated = "You spend {n_apples}*{apple_price} + {n_oranges}*{orange_price} = {n_apples*apple_price} + {n_oranges*orange_price} = ${apple_price*n_apples + orange_price*n_oranges}."

question = "A car travels at 60 mph for 3 hours. How far does it travel?"

answer = """
Distance = speed × time = 60 × 3 = 180 miles.
#### 180
"""

id_orig = 0
id_shuffled = 0
creation = "example"
language = "eng"

question_annotated = """
A car travels at {speed,60} mph for {hours,3} hours. How far does it travel?

#init:
- $speed = range(20, 100, 10)
- $hours = range(1, 9)

#conditions:
- is_int(speed * hours / 10)

#answer: speed * hours
"""

answer_annotated = """
Distance = speed × time = {speed} × {hours} = {speed * hours} miles.
#### {speed * hours}
"""

from multilingual_gsm_symbolic.templates import AnnotatedQuestion

template = AnnotatedQuestion.from_toml("my_template.toml")
questions = template.generate_questions(n=5)
for q in questions:
    print(q.question)
    print(q.answer)

🗃️ Data

The English templates are derived from Apple's GSM-Symbolic paper, from which the remainder is derived. For example, the Danish templates were initially translated using GPT-5.4, localized and reviewed by native speakers, and validated computationally and using Claude Opus. The original concrete problems are from GSM8k.

You can see the available languages as follows:

from multilingual_gsm_symbolic import available_languages, load_data

# see possible languages
print(available_languages())
# {'eng': {'number of samples': 100}, 'dan': {'number of samples': 100}, ...}

# Inspect template metadata:
template = load_data("dan")[0]
template.source_language  # 'eng'
template.initial_translation_model  # 'gpt-5.4'
template.human_validated  # 'by three native speakers'
template.error_analysis

The following table shows the list of validated languages:

Want to add a new language?

Want to add a new language or validate an existing one? Great to hear. The src/multilingual_gsm_symbolic/data/templates/ folder contains the templates for each language, and src/scripts/translate_templates.py can be used to translate the templates from one language to another. We have already pre-generated a few languages; see the templates folder for which ones. Each language folder also contains an instruction.toml with the prompt used to evaluate models (available through load_instruction), which should be validated along with the templates. Once you have validated the examples you can submit a PR with the changes.

📖 API reference

function load_data

load_data(language="eng", directory=None) → list[AnnotatedQuestion]

Load symbolic templates.

function load_replacements

load_replacements(language="eng") → dict

Load language-specific named values (e.g. lists of names, places) used inside templates.

function load_instruction

load_instruction(language="eng") → str

Load the language-specific instruction used to prompt a model with a question (solve step by step and put the final answer in \boxed{}). It is meant to be followed by a blank line and the question. Raises FileNotFoundError if the language has no instruction yet.

function load_gsm

load_gsm(language="eng", directory=None) → list[GSMProblem]

Load the bundled concrete problems for a given language.

class AnnotatedQuestion

Core class representing a symbolic template. Constructed from a TOML template file via AnnotatedQuestion.from_toml(path).

^{method AnnotatedQuestion.generate_questions}

Generate concrete Question instances from the template.

^{method AnnotatedQuestion.get_default_assignments}

Extract the default variable values from the question template placeholders.

^{method AnnotatedQuestion.get_combinations}

Enumerate unique valid assignments for a template.

^{method AnnotatedQuestion.format_question}

Render the question text for a given variable assignment.

^{method AnnotatedQuestion.format_answer}

Render the answer text for a given variable assignment.

class Question

Dataclass holding a single generated problem.

Acknowledgement

If you use this dataset please cite the paper:

@misc{enevoldsen2026multilingualgsmsymbolicdeterminescapability,
      title={Multilingual GSM-Symbolic: What determines capability transfer across languages?}, 
      author={Kenneth Enevoldsen and Riley Herchert and Sofie Mosegaard and Dan Saattrup Smart and Simon Enni and Isaac Chung and Sofie Bruun and Ayush Sunil Munot and Max Müller-Eberstein and Adnan El-Assadi and Elisa Bassignana and Gianluca Barmina and Hafsteinn Einarsson and Iben Nyholm Debess and Linda Freienthal and Lukas Galke Poech and Mike Zhang and Nicolas Legrand and Vladimir Salnikov and Yevhen Kostiuk and Zafar Hussain and Sagandeep Kaur and Agnes Toftgård and Marie Mattson and Kristoffer Nielbo},
      year={2026},
      eprint={2610.03367},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2610.03367}, 
}

The symbolic template engine and the danish subset were originally developed as part of the m-gsm-symbolic project at the Centre for Humanities Computing by:

• Kenneth Enevoldsen
• Sofie Mosegaard
• Simon Enni

The initial template format was derived from Apple's GSM-Symbolic paper and the original concrete problems are from GSM8k.