theoryminer

Assemble theories from text. theoryminer reads a PDF, finds the sentences that make causal claims, extracts each cause → effect pair, and brings the many ways authors name the same construct together, with a thesaurus (ELSST) or without one.

PDF ──harvest()──► sentences ──causenet()──► relations (cause span → effect span)
                                                  │
              ┌───────────────────────────────────┼───────────────────────────┐
              ▼                                   ▼                           ▼
standardize_constructs()            pairwise_grouping() / clusterer()      (spans)
span → top-k thesaurus concepts     spans → groups of similar spans
                                                  │
                                  ┌───────────────┴───────────────┐
                                  ▼                               ▼
                        standardize_groups()               label_groups()
                        group → thesaurus concept          group → label from its own words

Status: extraction and harmonisation are written. The graph stage (theorize()) and the evaluation are planned.

Install

Install from GitHub with pip. You do not need to clone the repository:

pip install "theoryminer @ git+https://github.com/rasoulnorouzi/theoryminer.git"
pip install "theoryminer[umap] @ git+https://github.com/rasoulnorouzi/theoryminer.git"   # with UMAP

In a notebook, such as Google Colab, put %pip install at the start of the first cell. The tutorial notebook does this for you: click the Colab badge above.

To change the code, clone the repository and install it in editable mode:

git clone https://github.com/rasoulnorouzi/theoryminer.git
cd theoryminer
pip install -e ".[notebooks,dev,umap]"

GPU

You do not need to set anything. causenet(sentences) uses device="auto": an NVIDIA GPU when PyTorch can see one, else the GPU of an Apple Silicon Mac, else the CPU. device="cpu", "cuda" or "mps" choose one device.

• Google Colab: choose a GPU runtime (Runtime → Change runtime type → T4 GPU). Colab already has a GPU build of PyTorch, so pip installs nothing extra.
• Linux with an NVIDIA GPU: the normal PyTorch from pip includes GPU support.
• Windows with an NVIDIA GPU: pip gives a CPU-only PyTorch there. The package then runs on the CPU, without errors. For the GPU, install PyTorch from https://pytorch.org first, then install theoryminer.
• Apple Silicon Mac: "auto" uses the Apple GPU. If the model fails there, it moves to the CPU and the run goes on.

The first call to causenet() downloads the SocioCausaNet model (rasoultilburg/SocioCausaNet) from the Hugging Face Hub. The first call to a harmonizer function downloads the sentence-transformer (all-mpnet-base-v2 by default).

Quick start

from theoryminer import harvest, extract_dois, causenet, clusterer, label_groups, standardize_constructs

data = harvest("raw_data/my_book.pdf", pages="12-540")   # clean sentences + a drop log
data = harvest("sample_files")                          # or every PDF in a folder
dois = extract_dois("sample_files", save="outputs")     # the DOI of each paper -> outputs/dois.csv
relations = causenet(data["sentences"])                 # one dict per cause → effect pair

groups = clusterer(relations)                           # group similar spans (HDBSCAN)
names = label_groups(groups)                            # name each group from its own words
matches = standardize_constructs(relations)             # or map each span to ELSST concepts

harvest() takes one PDF or a folder of PDFs. In a folder, a file that is not a PDF, or a PDF that cannot be read, does not stop the run. The drop log gets one row for it, with the reason.

Sample papers

sample_files/ holds 7 open-access social-science papers with a CC BY licence. sample_files/SOURCES.md gives the title, authors, DOI and licence of each one. The tutorial and the tests use them.

Data you supply

Give harvest() your own PDFs. In a clone, raw_data/ is a good place for them: git ignores it.

The ELSST thesaurus (release 5) ships inside the package as theoryminer/data/ELSST_R5.rdf.gz. taxonomy="elsst" needs no extra file. A full path to another SKOS .rdf or .rdf.gz file also works as taxonomy=. The thesaurus vectors are computed once and cached in theoryminer/taxonomy_cache/. label_groups() needs no thesaurus at all.

Documentation

The notebooks work from tutorials/ and from the repository root. The examples use the papers in sample_files/.

Tests

pip install -e ".[dev]"
pytest                 # the fast tests: no model download, a few seconds
pytest -m slow         # the tests that load the real models (about 1 GB on the first run)

GitHub Actions runs the fast tests on each push, on Linux, Windows and macOS. It also builds the package and tests the built wheel. The slow tests run on each push to main and every Monday.

Releases

A push to main never publishes. A release goes to PyPI in three steps:

1. Set the new version in pyproject.toml, add a section to CHANGELOG.md, push, and wait for a green CI.
2. On GitHub: Releases → Draft a new release, tag v plus the version (for example v0.1.0), Generate release notes, Publish release.
3. In the Actions tab, open the "Publish to PyPI" run and approve it (Review deployments).

The workflow stops when the tag and the version differ. PyPI trusts the workflow through Trusted Publishing, so no token is stored in the repository.

Licence

The code has the MIT licence. See LICENSE.

Two kinds of data in this repository keep their own licence:

• theoryminer/data/ELSST_R5.rdf.gz: ELSST, © CESSDA, CC BY-SA 4.0. See theoryminer/data/ELSST_LICENSE.md.
• sample_files/*.pdf: open-access papers, CC BY 4.0. See sample_files/SOURCES.md.