Automated Intelligence Tests

Minimal Python package providing automated tests of associative and creative ability, for both human and artificial intelligence. Each test exposes the same two calls: instruct() builds the stimuli and prompt, evaluate() scores the responses.

Install

pip install git+https://github.com/dtzx00/Automated-Intelligence-Tests.git

Once published to PyPI: pip install automated-intelligence-tests.

Dependencies (glove-word-embeddings, numpy) are declared in pyproject.toml and installed automatically — no separate requirements step. Python 3.9+.

For local development:

git clone https://github.com/dtzx00/Automated-Intelligence-Tests.git
cd Automated-Intelligence-Tests
pip install -e .

Quickstart

import automated_intelligence_tests as ait

ait.list_available_tests()        # {"CAT": "Convergent Association Task", ...}

# 1. Build the stimuli and prompt
task = ait.instruct("cat", n_words=10, seed=42)
print(task["instructions"])       # give this to a person or a model
print(task["response_format"])    # the shape the answer should come back in

# 2. Fill in "word_user" for each item, then score
responses = {
    "wordset_1": {"word_1": "television", "word_2": "lake", "word_user": "reflection"},
    # ...
}
result = ait.evaluate("cat", responses)
print(result["score"], result["n_valid"])

instruct() returns a dict with test, instructions, response_format, and the test's own stimuli. evaluate() returns a dict with score (None if nothing scorable), n_valid, and per-test detail.

API

ait.list_available_tests()               # -> {short_name: long_name} for every test
ait.instruct(test, **kwargs)             # test in {"cat", "dat", "aut", "cwt"}
ait.call_test_instruction("DAT", **kwargs)   # same as instruct(), case-insensitive name
ait.evaluate(test, responses, **kwargs)

Sub-packages can also be used directly: from automated_intelligence_tests.cat import instruct, evaluate.

CAT samples word pairs from a fixed list of 8,069 English word pairs at cosine distance 0.85–0.95. n_words controls how many pairs are returned at once. Providing cue=[(w1, w2), ...] uses those explicit pairs (and a single pair automatically embeds it in the instruction text).

DAT asks for n_words words that are as different from each other as possible. Optional cue seeds the first word; the participant then supplies the remaining words.

AUT samples (or accepts) a common object via cue. n_words is accepted for API consistency but currently unused.

CWT samples (or accepts via cue) 1–3 cue words controlled by n_words and asks for a short creative story that incorporates every cue word.

Scoring

• CAT — mean cosine similarity between the participant's word and each of the two cue words, averaged over items. Higher = stronger convergent association. Invalid or unembeddable words are marked valid: False and excluded rather than penalised.
• DAT — Olson's procedure exactly: mean pairwise cosine distance of the first 7 unique valid words, ×100. Returns score: None if fewer than 7 valid words are given.
• AUT — Beaty & Johnson SemDis: clean each use (how="heavy" stopwords, drop cue + plural), multiply leftover word vectors, take 1 − cosine to the cue in five spaces (cbow-subs-300d, cbow-ukwac-subs-300d, cbow-baroni-400d, tasa-lsa-300d, glove-6b-300d). score is the mean of those spaces, then the mean across uses. Higher = more original. First call downloads the five spaces into ~/.cache/glove-word-embeddings.
• CWT — Johnson et al. BERT DSI: split the story into sentences, keep BERT-large layers 6 and 7 for every real token, put those vectors in one list, take the mean pairwise 1 − cosine. Higher = more divergent integration. First call downloads BERT into ~/.cache/glove-word-embeddings/huggingface.

All will use glove-word-embeddings, default model glove-840b-300d for CAT and DAT. The first evaluate() call downloads the embedding file and caches it under ~/.cache/glove-word-embeddings; later calls are offline. Other keys (glove-6b-300d, wiki-news-300d-1m, the flair-olson-* set, …) can be passed via model_key.

Layout

automated_intelligence_tests/
    __init__.py          dispatcher: list_available_tests() / instruct() / evaluate()
    cat/  dat/           per test: instruct.py, evaluate.py, metadata.json,
    aut/  cwt/                     readme.md, example.ipynb
    cat/data/cat_word_pairs_en.txt CAT word-pair list

Each test folder carries a metadata.json holding its short name, long name and description. That file is what list_available_tests() reads, so dropping in a new test folder with one makes it discoverable automatically. Each also has its own readme.md and a runnable example.ipynb.

License

MIT