prompt_miner

Data-mining tools for the Cirilla project — gathering and shaping training data for a budget LLM.

Fandom Scraper

A lightweight Fandom wiki scraper built on the MediaWiki API.

No GPU required. No heavy ML dependencies. Works with any language supported by the target wiki.

How it works

1. The scraper enumerates every page on the wiki via the MediaWiki list=allpages API, across all content namespaces (auto-detected — e.g. articles and Maps). This includes subpages (e.g. Geralt of Rivia/Netflix series).
2. Each page is fetched through api.php and split into three artifacts:
   • the article body → plain .txt (tables are kept as readable rows, image captions and trailing sections like Trivia are preserved),
   • the infobox (race, profession, family, abilities, …) → structured .json,
   • the "Quick Answers" Q&A widget, when present → instruction .json.
3. The async crawl runs across many concurrent fetchers and stops on its own once every discovered page has been processed.

Discovery does not rely on an NER model, so it works regardless of language and requires no GPU. Beyond full-wiki enumeration you can also seed the crawl from specific categories, search queries, or JSON seed lists, and optionally follow in-article links (BFS crawl).

User-agent strings are rotated on every request (pool of 7 real browser UAs) with matching Accept-Language headers per language, so the scraper blends in with normal browser traffic.

Installation

# from the repo root
uv sync          # or: pip install -e .

Usage

Full-wiki dump (default)

from prompt_miner.fandom_scraping import scrape_fandom

scrape_fandom(
    out_path="./pages",          # article body .txt files
    instruct_path="./instruct",  # Quick-Answers Q&A .json files (when found)
    infobox_path="./infobox",    # infobox .json files (when present)
    n_workers=50,                # concurrent async fetchers (raise for speed)
    wiki="Witcher",              # fandom wiki subdomain
    lang="en",                   # language code
    namespaces="content",        # auto-detect content namespaces (e.g. 0 + Map)
    use_allpages=True,           # enumerate the whole wiki via list=allpages
    follow_links=False,          # redundant when use_allpages is on
)

This discovers and scrapes every article on the wiki and terminates when the frontier is drained.

Targeted scraping

Disable use_allpages and seed from any combination of sources instead:

# Specific categories (reusable across wikis) — names with or without "Category:"
scrape_fandom(out_path="./pages", instruct_path="./instruct", infobox_path="./infobox",
              wiki="Witcher", lang="en",
              use_allpages=False, categories=["Subpages", "Characters"])

# Search-bar queries, then follow links from the matches
scrape_fandom(out_path="./pages", instruct_path="./instruct", infobox_path="./infobox",
              wiki="Witcher", lang="en",
              use_allpages=False, search_queries=["witcher", "monster"], follow_links=True)

# JSON seed-title files (a folder of JSON arrays of page names)
scrape_fandom(out_path="./pages", instruct_path="./instruct", infobox_path="./infobox",
              wiki="Witcher", lang="en",
              use_allpages=False, in_path="./seeds", follow_links=True)

A seed file is just a JSON array of starting page names:

[
    "Geralt of Rivia", "Triss Merigold", "Vesemir", "Lambert",
    "Eskel", "Shani", "Zoltan Chivay", "Dandelion (Jaskier)"
]

Parameters

Parameter	Default	Description
out_path / instruct_path / infobox_path	—	Output folders for body text, Q&A JSON, infobox JSON.
in_path	None	Optional folder of JSON seed-title lists.
n_workers	10	Concurrent async fetchers.
wiki / lang	"Witcher" / "en"	Wiki subdomain and language code.
namespaces	"content"	"content" (auto-detect), an int, or a list of namespace ids.
use_allpages	True	Enumerate the entire wiki via list=allpages.
categories	None	List of category names to seed from.
search_queries	None	List of search-bar queries to seed from.
follow_links	False	Also crawl in-article links (BFS).
include_redirects	False	Include redirect pages (duplicate content).

Language support

The wiki URL is built from wiki + lang: https://<wiki>.fandom.com for English and https://<wiki>.fandom.com/<lang> otherwise. For the Polish Witcher wiki (witcher.fandom.com/pl):

scrape_fandom(
    out_path="./pages_pl",
    instruct_path="./instruct_pl",
    infobox_path="./infobox_pl",
    wiki="Witcher",
    lang="pl",
)

Languages with native Accept-Language headers: en, pl, de, fr, es, ru, pt, it, zh, ja, ko, cs, hu, tr, uk. Any other IETF language code is passed through with a generic fallback header.

Converting instructions to JSONL

If the scraper finds any Q&A pairs, convert them to the Cirilla .jsonl format:

from prompt_miner.fandom_scraping import instructions_into_conv

instructions_into_conv('./instruct', './fandom_instruct.jsonl')

Output format:

{"subject": "Shani", "text": [{"role": "user", "content": "What role did Shani play in the Battle of Brenna?"}, {"role": "assistant", "content": "Shani played a vital role..."}], "data type": "conv", "source": "fandom"}

Effectiveness

Full-wiki enumeration covers the entire site rather than only the pages reachable from a seed list. For the English Witcher wiki this is 12,307 articles (including 427 subpages) across the content namespaces.

Training Data Pipeline

After scraping, prompt_miner turns the raw articles into scored, structured instruct (Q&A) training data through a chain of LLM-prompted stages. This part needs a GPU and vLLM (already installed by uv sync) — every stage below loads a real quantized model (speakleash/Bielik-*, CYFRAGOVPL/Llama-PLLuM-*).

Stages

scraped .txt (+ infobox)
  → quality_check                    score 1-10, keep only pages above a threshold
  → extract_topics                   break the article into topics + supporting info
  → generate_subjects                group topics into broader subjects
  → create_syllabus                  design an ordered lesson outline per subject
  → extract_concepts                 break each syllabus section into atomic concept + key_info pairs
  → create_instruct_from_concept     generate a multi-hop Q&A conversation per concept
  → judge_instruct                   verify each conversation is grounded in its concept's key_info

Each stage is a script in scripts/, run in order (every script reads the previous stage's .jsonl and writes its own):

python scripts/assess_quality.py                 # -> prompt_miner/data/pl/async_fandom_scores.jsonl
python scripts/extract_topics.py                 # -> prompt_miner/data/pl/async_fandom_topics.jsonl
python scripts/generate_subjects.py              # -> prompt_miner/data/pl/async_fandom_subjects.jsonl
python scripts/create_syllabus.py                # -> prompt_miner/data/pl/async_fandom_syllabus.jsonl
python scripts/extract_concepts.py               # -> prompt_miner/data/pl/async_fandom_concepts.jsonl
python scripts/generate_instruct_from_concept.py # -> prompt_miner/data/pl/async_fandom_concept_instructs.jsonl
python scripts/judge_instruct.py                 # -> prompt_miner/data/pl/async_fandom_judged.jsonl

Edit the paths/models at the top of each script to point at a different scrape or language.

There's also an older, simpler path, scripts/generate_instruct.py, which generates instruct conversations directly from the raw article + infobox, skipping the topic/subject/syllabus/concept breakdown — useful for a quick pass, but without the per-concept grounding the judge stage checks for.

Prompts

All prompts live in prompt_miner/prompts/ as Jinja2 templates and are registered in prompt_manager.py. Every stage has a system prompt (task instructions + examples) and a user prompt (just the data), in both English and Polish (_pl suffix). Prompts are looked up by class name via AVAILABLE_PROMPTS:

from prompt_miner.prompts.prompt_manager import AVAILABLE_PROMPTS
AVAILABLE_PROMPTS["ExtractTopicsSystemPL"].render()

Language support

Like the scraper, the pipeline is duplicated per language — the scripts above point at prompt_miner/data/pl/... and use the *PL prompt classes; swap both for English (prompt_miner/data/en/..., drop the PL suffix).