bnc-lookup 1.1.0

Static Hash-Based Lookup for BNC Terms

BNC Lookup

Is this token a word? O(1) answer. No setup. No dependencies.

A simple question deserves a simple answer. This library gives you instant yes/no validation against 669,000 word forms from the British National Corpus, plus frequency ranking.

Quick Start

pip install bnc-lookup

import bnc_lookup as bnc

# Check if a word exists
bnc.exists('the')          # True
bnc.exists('however')      # True
bnc.exists('xyzabc123')    # False

# Get frequency bucket (1=most common, 100=least common)
bnc.bucket('the')          # 1
bnc.bucket('python')       # 4
bnc.bucket('qwerty')       # 12
bnc.bucket('xyzabc123')    # None (not found)

# Handles plurals automatically
bnc.exists('computers')    # True
bnc.bucket('computers')    # 1

# Case insensitive
bnc.exists('THE')          # True

Features

• Zero Dependencies - Pure Python, no external packages
• Zero I/O - No filesystem access, no database queries
• Zero Setup - No corpus downloads or configuration
• Microsecond Lookups - O(1) dictionary access
• Smart Plurals - Automatically checks singular forms
• Frequency Ranking - 100 buckets from most to least common
• Simple API - Two functions: exists() and bucket()

The Problem This Solves

In NLP, you frequently need to answer the question: "Is this token a real word?"

Not "what does it mean?" Not "give me synonyms." Just: is this a word?

bnc.exists('computer')	bnc.exists('asdfgh')
True	False

That's it. O(1) response. No ambiguity.

Frequency Buckets

Words are ranked into 100 buckets based on their frequency in the BNC corpus:

Bucket	Description	Examples
1	Most frequent (~6,700 words)	the, of, and, is, computer
2-10	Very common	algorithm, python, beautiful
11-50	Common	qwerty, specialized terms
51-99	Less common	Rare but valid words
100	Least frequent	Obscure terms

import bnc_lookup as bnc

# Filter by frequency
def is_common_word(word):
    bucket = bnc.bucket(word)
    return bucket is not None and bucket <= 10

Why BNC?

The British National Corpus isn't an academic wordlist (too narrow). It's not a web scrape (too noisy). It's not slang (too ephemeral).

It's a 100-million-word corpus of real British English collected from written and spoken sources between 1991-1994. Books, newspapers, academic papers, conversations. The BNC frequency list captures ~669,000 unique word forms actually used by native speakers.

If a token passes the BNC test, you can be confident it's a word that real people actually use.

When to Use This

• Tokenization filtering: Keep real words, discard garbage
• Input validation: Reject nonsense in user input
• NLP preprocessing: Filter candidates before expensive operations
• Spell-check pre-filtering: Quick reject obvious non-words before fuzzy matching
• Data cleaning: Identify malformed or corrupted text
• Frequency-based filtering: Prefer common words over obscure ones

What This Doesn't Do

• No definitions, synonyms, or semantic relationships (use spaCy for that)
• No spell-checking or suggestions (just existence check)
• No irregular plural handling ("mice" → "mouse")

Documentation

For detailed usage, performance benchmarks, and advanced features, see the API Documentation.

Development

git clone https://github.com/craigtrim/bnc-lookup.git
cd bnc-lookup
make install  # Install dependencies
make test     # Run tests
make all      # Full build pipeline

See API Documentation for detailed development information.

License

This package is dual-licensed:

• Software: MIT License
• BNC Data: BNC User Licence

See LICENSE for complete terms.

Attribution

This package contains data derived from the British National Corpus frequency lists:

BNC frequency lists compiled by Adam Kilgarriff. Source: https://www.kilgarriff.co.uk/BNClists/all.num.gz

The British National Corpus, version 3 (BNC XML Edition). 2007. Distributed by Bodleian Libraries, University of Oxford, on behalf of the BNC Consortium.

Note: This is a static snapshot of BNC frequency data. The data is not automatically updated.

See Also

• wordnet-lookup - Similar O(1) lookup using the WordNet lexicon

Links

• Repository: github.com/craigtrim/bnc-lookup
• PyPI: pypi.org/project/bnc-lookup
• BNC: natcorp.ox.ac.uk
• Author: Craig Trim ([e-mail])