Automated Intelligence Tests

This is a library to conduct automated tests for cognitive abilities. All tests should include instruction, parsing and scoring: instruct() builds the stimuli, parse() will do some simple parsing to fit the format of the scoring evaluate() function.

Install

pip install git+https://github.com/dtzx00/Automated-Intelligence-Tests.git

Once published to PyPI: pip install automated-intelligence-tests.

Dependencies (glove-word-embeddings, numpy) are declared in pyproject.toml and installed automatically — no separate requirements step. Python 3.9+.

For local development:

git clone https://github.com/dtzx00/Automated-Intelligence-Tests.git
cd Automated-Intelligence-Tests
pip install -e .

Quickstart

import automated_intelligence_tests as ait
ait.list_available_tests()
stim = ait.instruct("aut", cue="brick")
parsed = ait.parse("aut", raw, stim=stim)
result = ait.evaluate("aut", parsed)

instruct() returns a dict with test, instructions, response_format, and the test's own stimuli. evaluate() returns a dict with score (None if nothing scorable), n_valid, and per-test detail.

API

ait.list_available_tests()                   # short_name and long_name for every test
ait.instruct(test, **kwargs)                 # test in {"cat", "dat", "aut", "aut_1w", "cwt", "wat"}
ait.call_test_instruction("DAT", **kwargs)   # same as instruct(), case-insensitive name
ait.parse(test, raw, stim=None)              # some basic automated parsing
ait.evaluate(test, responses, **kwargs)      # evaluate the scores using parsed response        

Sub-packages can be used directly: from automated_intelligence_tests.cat import instruct, parse, evaluate.

CAT samples word pairs from a fixed list of 8,069 English word pairs at cosine distance 0.85–0.95. n_words controls how many pairs are returned at once. Providing cue=[(w1, w2), ...] uses those explicit pairs (and a single pair automatically embeds it in the instruction text).

DAT asks for n_words words that are as different from each other as possible. Optional cue seeds the first word; the participant then supplies the remaining words.

AUT samples (or accepts) a common object via cue. n_words is accepted for API consistency but currently unused.

AUT_1W is a standardised, cued AUT. One object, by default drawn uniformly from brick, paperclip, soap, wheel (any other can be passed as cue); up to ten optional lines a–j, each written as I will use the <object> as <usage> for <explanation>, returned as a JSON list of {"usage", "explanation"} pairs. instructions is the full rendered page, given unchanged to humans and LLMs. See aut_1w/readme.md.

CWT samples (or accepts via cue) 1–3 cue words controlled by n_words and asks for a short creative story that incorporates every cue word.

WAT is the Small World of Words continued word-association task. One cue, by default drawn uniformly from the 12,216 SWOW-EN2018 cues so every trial has a human reference distribution; up to three responses (the first words that come to mind), returned as {"unknown": bool, "responses": [r1, r2, r3]}. parse() applies the SWOW preprocessing. See wat/readme.md.

Scoring

• CAT — mean cosine similarity between the participant's word and each of the two cue words, averaged over items. Higher = stronger convergent association. Invalid or unembeddable words are marked valid: False and excluded rather than penalised.
• DAT — Olson's procedure exactly: mean pairwise cosine distance of the first 7 unique valid words, ×100. Returns score: None if fewer than 7 valid words are given.
• AUT — Beaty & Johnson SemDis: clean each use (how="heavy" stopwords, drop cue + plural), multiply leftover word vectors, take 1 − cosine to the cue in five spaces (cbow-subs-300d, cbow-ukwac-subs-300d, cbow-baroni-400d, tasa-lsa-300d, glove-6b-300d). score is the mean of those spaces, then the mean across uses. Higher = more original. First call downloads the five spaces into ~/.cache/glove-word-embeddings.
• AUT_1W — the first ten non-blank uses are scored by the AUT scorer above and its result returned unchanged. By default (noun_only=True) each use is the usage noun alone; noun_only=False scores usage + explanation joined. Replies that aren't a well-formed list of pairs, or that contain a usage that isn't one word after trimming stopwords, score None.
• CWT — Johnson et al. BERT DSI: split the story into sentences, keep BERT-large layers 6 and 7 for every real token, put those vectors in one list, take the mean pairwise 1 − cosine. Higher = more divergent integration. First call downloads BERT into ~/.cache/glove-word-embeddings/huggingface.
• WAT — mirrors the SWOW game's player feedback: typicality (mean p(response | cue) in the SWOW-EN norms; score), relevance (share of responses some human also gave), common_ground (first response is the modal human first response), fluency (responses per cue), imagery (mean concreteness), plus cue→response GloVe distance. evaluate.summarize() aggregates a session. The SWOW norms (CC BY-NC-ND) and concreteness norms are not bundled: download them and pass the paths as norms= and concreteness=.

All will use glove-word-embeddings, default model glove-840b-300d for CAT and DAT. The first evaluate() call downloads the embedding file and caches it under ~/.cache/glove-word-embeddings; later calls are offline. Other keys (glove-6b-300d, wiki-news-300d-1m, the flair-olson-* set, …) can be passed via model_key.

Layout

automated_intelligence_tests/
    __init__.py          dispatcher: list_available_tests() / instruct() / parse() / evaluate()
    cat/  dat/           per test: instruct.py, parse.py, evaluate.py, metadata.json,
    aut/  aut_1w/  cwt/  wat/      readme.md, example.ipynb
    cat/data/cat_word_pairs_en.txt CAT word-pair list
    wat/data/swow_en_cues.txt      WAT cue list (SWOW-EN2018 cues)

Each test folder carries a metadata.json holding its short name, long name and description. That file is what list_available_tests() reads, so dropping in a new test folder with one makes it discoverable automatically. Each also has its own readme.md and a runnable example.ipynb.

License

MIT