bnc-lookup 1.0.5

Static Hash-Based Lookup for BNC Terms

BNC Lookup

Is this token a word? O(1) answer. No setup. No dependencies.

A simple question deserves a simple answer. This library gives you instant yes/no validation against 669,000 word forms from the British National Corpus.

Quick Start

pip install bnc-lookup

from bnc_lookup import is_bnc_term

# That's it. Start validating.
is_bnc_term('the')          # True
is_bnc_term('however')      # True
is_bnc_term('nonetheless')  # True
is_bnc_term('xyzabc123')    # False

# Handles plurals automatically
is_bnc_term('computers')    # True

# Case insensitive
is_bnc_term('THE')          # True

Features

• Zero Dependencies - Pure Python, no external packages
• Zero I/O - No filesystem access, no database queries
• Zero Setup - No corpus downloads or configuration
• Microsecond Lookups - O(1) dictionary access
• Smart Plurals - Automatically checks singular forms
• Simple API - One function does it all

The Problem This Solves

In NLP, you frequently need to answer the question: "Is this token a real word?"

Not "what does it mean?" Not "give me synonyms." Just: is this a word?

is_bnc_term('computer')	is_bnc_term('asdfgh')
True	False

That's it. O(1) response. No ambiguity.

Why BNC?

The British National Corpus isn't an academic wordlist (too narrow). It's not a web scrape (too noisy). It's not slang (too ephemeral).

It's a 100-million-word corpus of real British English collected from written and spoken sources between 1991-1994. Books, newspapers, academic papers, conversations. The BNC frequency list captures ~669,000 unique word forms actually used by native speakers.

If a token passes the BNC test, you can be confident it's a word that real people actually use.

When to Use This

• Tokenization filtering: Keep real words, discard garbage
• Input validation: Reject nonsense in user input
• NLP preprocessing: Filter candidates before expensive operations
• Spell-check pre-filtering: Quick reject obvious non-words before fuzzy matching
• Data cleaning: Identify malformed or corrupted text

What This Doesn't Do

• No definitions, synonyms, or semantic relationships (use spaCy for that)
• No frequency counts or rankings (just yes/no)
• No spell-checking or suggestions (just existence check)

Documentation

For detailed usage, performance benchmarks, and advanced features, see the API Documentation.

How It Works

BNC terms are stored as MD5 hash suffixes in 256 frozenset buckets (by first two hex characters of the hash). Lookups hash the input, route to the correct bucket, and perform O(1) set membership. Modules are lazy-loaded on first access per bucket.

For the gory details, see Implementation Notes.

Development

git clone https://github.com/craigtrim/bnc-lookup.git
cd bnc-lookup
make install  # Install dependencies
make test     # Run tests
make all      # Full build pipeline

See API Documentation for detailed development information.

License

This package is dual-licensed:

• Software: MIT License
• BNC Data: BNC User Licence

See LICENSE for complete terms.

Attribution

This package contains data derived from the British National Corpus frequency lists:

BNC frequency lists compiled by Adam Kilgarriff. Source: https://www.kilgarriff.co.uk/BNClists/all.num.gz

The British National Corpus, version 3 (BNC XML Edition). 2007. Distributed by Bodleian Libraries, University of Oxford, on behalf of the BNC Consortium.

Note: This is a static snapshot of BNC frequency data. The data is not automatically updated.

Links

• Repository: github.com/craigtrim/bnc-lookup
• PyPI: pypi.org/project/bnc-lookup
• BNC: natcorp.ox.ac.uk
• Author: Craig Trim ([e-mail])