theoryminer

Assemble theories from text. theoryminer reads a PDF, finds the sentences that make causal claims, extracts each cause → effect pair, and brings the many ways authors name the same construct together, with a thesaurus (ELSST) or without one. Then it draws a causal map: an interactive page where every arrow leads back to the sentences that claim it.

PDF ──harvest()──► sentences ──causenet()──► relations (cause span → effect span)
                                                  │
              ┌───────────────────────────────────┼───────────────────────────┐
              ▼                                   ▼                           ▼
standardize_constructs()            pairwise_grouping() / clusterer()      (spans)
span → top-k thesaurus concepts     spans → groups of similar spans
                                                  │
                                  ┌───────────────┴───────────────┐
                                  ▼                               ▼
                        standardize_groups()               label_groups()
                        group → thesaurus concept          group → label from its own words
                                  └───────────────┬───────────────┘
                                                  ▼
                        causal_map() ──► draw_map() (interactive page), save_map() (tables, GraphML)

Status: extraction, harmonisation and the causal map are written. The evaluation is planned.

Install

Install from PyPI with pip:

pip install theoryminer

The install includes UMAP for clusterer(..., umap={...}).

The newest code on GitHub, before the next release:

pip install "theoryminer @ git+https://github.com/rasoulnorouzi/theoryminer.git"

In a notebook, such as Google Colab, put %pip install at the start of the first cell. The tutorial notebook does this for you: click the Colab badge above.

To change the code, clone the repository and install it in editable mode:

git clone https://github.com/rasoulnorouzi/theoryminer.git
cd theoryminer
pip install -e ".[notebooks,dev,umap]"

GPU

You do not need to set anything. causenet(sentences) uses device="auto": an NVIDIA GPU when PyTorch can see one, else the GPU of an Apple Silicon Mac, else the CPU. device="cpu", "cuda" or "mps" choose one device.

• Google Colab: choose a GPU runtime (Runtime → Change runtime type → T4 GPU). Colab already has a GPU build of PyTorch, so pip installs nothing extra.
• Linux with an NVIDIA GPU: the normal PyTorch from pip includes GPU support.
• Windows with an NVIDIA GPU: pip gives a CPU-only PyTorch there. The package then runs on the CPU, without errors. For the GPU, install PyTorch from https://pytorch.org first, then install theoryminer.
• Apple Silicon Mac: "auto" uses the Apple GPU. If the model fails there, it moves to the CPU and the run goes on.

The first call to causenet() downloads the SocioCausaNet model (rasoultilburg/SocioCausaNet) from the Hugging Face Hub. The first call to a harmonizer function downloads the sentence-transformer (all-mpnet-base-v2 by default).

Quick start

from theoryminer import (harvest, extract_dois, extract_references, download_papers, causenet, clusterer,
                         label_groups, standardize_constructs, causal_map, draw_map, save_map)

data = harvest("raw_data/my_book.pdf", pages="12-540")   # clean sentences + a drop log
data = harvest("sample_files")                          # or every PDF in a folder; each sentence has its paper's DOI and title
dois = extract_dois("sample_files", save="outputs")     # optional: the DOI and title of each paper -> outputs/dois.csv
refs = extract_references("sample_files", save="outputs")   # the DOIs that the papers cite -> outputs/references.csv
report = download_papers(refs, "round_2", licences=["cc-by"])  # their open-access PDFs: the next snowball round
relations = causenet(data["sentences"])                 # one dict per cause → effect pair

groups = clusterer(relations)                           # group similar spans (HDBSCAN)
names = label_groups(groups)                            # name each group from its own words
matches = standardize_constructs(relations)             # or map each span to ELSST concepts

cmap = causal_map(relations, names)                     # nodes, edges, and the hidden edges
draw_map(cmap)                                          # the interactive page, with the DOI of each paper
save_map(cmap, "outputs/my_map")                        # nodes.csv, edges.csv, hidden.csv, map.graphml

harvest() takes one PDF or a folder of PDFs. In a folder, a file that is not a PDF, or a PDF that cannot be read, does not stop the run. The drop log gets one row for it, with the reason. Each sentence also gets the DOI and title of its paper. The DOI comes from the most certain source: the log downloads.csv that download_papers() writes into the folder, the file name of a downloaded PDF, the PDF metadata, then the first DOI on pages 1 and 2. causenet() copies the DOI and title to each relation, so draw_map(cmap) links each sentence to its paper without an extra step.

extract_references() finds the DOIs in the references of seed papers, for a snowball search. It repairs the DOIs that a PDF cuts at a line end. On the sample papers, 214 of its 215 DOIs exist at doi.org. download_papers() downloads their open-access PDFs. It checks the licence of each paper with OpenAlex first (free, no key), so licences=["cc-by"] downloads only CC BY papers. Then it tries free sources in turn until one gives a real PDF: OpenAlex, per-publisher plugins, PubMed Central, Europe PMC, Unpaywall, Semantic Scholar, Crossref, OSF preprints, and the doi.org page. The tutorial, section 11, runs one snowball round.

download_papers() gets open-access copies only. It does not log in to a publisher site and it does not use a browser, so a paper that is only behind a paywall stays not_found. Get such a paper through your library, and put the PDF into the folder with the others. Set respect_robots=False only when you have the right to do so. tmsr-doi-downloader stays as an opt-in source: sources=[..., "tmsr-doi-downloader"].

The causal map

A node is a construct: a group of spans, or a thesaurus concept. An arrow goes from a cause to an effect. Its width shows the number of papers that make the claim. On the page you can:

• hover over a node or an arrow, and click an arrow to read all its sentences, with the DOI of each paper (else its title, else its file name);
• search as a filter: "age" shows only the matching nodes and their neighbours (or 2 steps, or only the matches), and hides the rest;
• filter by paper, by role ("cause only", "cause and effect", "effect only"), by two-way pairs, and with min papers, min sentences and the self-loop switch; drag, zoom, and Reset;
• save a picture as PNG, JPG or SVG: the whole map, the current view, or one node with its neighbours.

The page is one HTML file that works offline. Nothing is deleted: a two-way claim keeps both arrows, and an edge that a filter hides stays in the data with its reason.

Sample papers

sample_files/ holds 7 open-access social-science papers with a CC BY licence. sample_files/SOURCES.md gives the title, authors, DOI and licence of each one. The tutorial and the tests use them.

Data you supply

Give harvest() your own PDFs. In a clone, raw_data/ is a good place for them: git ignores it.

The ELSST thesaurus (release 5) ships inside the package as theoryminer/data/ELSST_R5.rdf.gz. taxonomy="elsst" needs no extra file. A full path to another SKOS .rdf or .rdf.gz file also works as taxonomy=. The thesaurus vectors are computed once and cached in ~/.cache/theoryminer/taxonomy. Set the environment variable THEORYMINER_CACHE_DIR to use another folder. If the folder cannot be written, the functions still work; they only compute the vectors again in the next session. label_groups() needs no thesaurus at all.

Documentation

The documentation is at https://rasoulnorouzi.github.io/theoryminer/: a guide for each step, the reference of each function, and every setting with its default.

The learning material in the repository is in tutorials/:

The notebooks work from tutorials/ and from the repository root. The examples use the papers in sample_files/.

Tests

pip install -e ".[dev]"
pytest                 # the fast tests: no model download, a few seconds
pytest -m slow         # the tests that load the real models (about 1 GB on the first run)

GitHub Actions runs the fast tests on each push, on Linux, Windows and macOS. It also builds the package and tests the built wheel. The slow tests run on each push to main and every Monday.

Releases

A push to main never publishes. A release goes to PyPI in three steps:

1. Set the new version in pyproject.toml and in the first cell of tutorials/TUTORIAL.ipynb (%pip install -q "theoryminer==<version>"). Add a section to CHANGELOG.md, push, and wait for a green CI.
2. On GitHub: Releases → Draft a new release, tag v plus the version (for example v0.1.0), Generate release notes, Publish release.
3. In the Actions tab, open the "Publish to PyPI" run and approve it (Review deployments).

The workflow stops when the tag and the version differ. PyPI trusts the workflow through Trusted Publishing, so no token is stored in the repository.

Licence

The code has the MIT licence. See LICENSE.

These files in this repository keep their own licence:

• theoryminer/data/ELSST_R5.rdf.gz: ELSST, © CESSDA, CC BY-SA 4.0. See theoryminer/data/ELSST_LICENSE.md.
• sample_files/*.pdf: open-access papers, CC BY 4.0. See sample_files/SOURCES.md.
• theoryminer/data/cytoscape.min.js (Cytoscape.js) and the fcose layout files (layout-base.js, cose-base.js, cytoscape-fcose.js): MIT. Each page that draw_map() makes holds a copy, with the licence text.