dictionary-crawler 0.1.0

A Python package for crawling Cambridge and OxfordLearners dictionaries

🦄 Dictionary Crawler

A Python package for crawling dictionary websites. Currently supports Cambridge Dictionary and Oxford Learners Dictionary.

Installation

pip install dictionary-crawler

Or install from source:

git clone https://github.com/arshinmq/dictionary-crawler.git
cd dictionary-crawler
pip install -e .

Features

• Async HTTP client for non-blocking requests
• Parse Cambridge and OxfordLearners Dictionary pages
• Extract:
  • Headwords and parts of speech
  • Pronunciations (IPA + audio URLs)
  • Definitions with CEFR levels
  • Examples and sources
  • Phrases, idioms, and phrasal verbs
  • Collocations
  • Verb forms
  • Topics and semantic domains (OxfordLearners)
• Structured Pydantic models for type-safe data access
• JSON serialization support

Quick Start

import asyncio
from dictionary_crawler import CambridgeCrawler, OxfordLearnersCrawler

async def main():
    # Cambridge Dictionary
    async with CambridgeCrawler() as crawler:
        page = await crawler.crawl("run")
        
        print(f"Search term: {page.search_term}")
        
        for section in page.sections:
            for entry in section.entries:
                print(f"\n{entry.headword} ({entry.part_of_speech})")
                
                if entry.pronunciations:
                    for p in entry.pronunciations:
                        print(f"  [{p.region}] {p.pronunciation}")
                
                for sense in entry.senses:
                    for defn in sense.definitions:
                        print(f"  - {defn.definition}")
                        if defn.examples:
                            for ex in defn.examples:
                                if ex.example:
                                    print(f"    e.g. {ex.example}")
    
    # OxfordLearners Dictionary
    async with OxfordLearnersCrawler() as crawler:
        page = await crawler.crawl("run")
        
        print(f"\nOxfordLearners - Search term: {page.search_term}")
        
        for definition in page.definitions:
            print(f"\n{definition.headword} ({definition.part_of_speech})")
            
            if definition.pronunciations:
                for p in definition.pronunciations:
                    print(f"  [{p.phonetic}]")
            
            for sense in definition.senses:
                print(f"  - {sense.definition}")
                if sense.examples:
                    for ex in sense.examples:
                        print(f"    e.g. {ex.example}")

asyncio.run(main())

API Reference

CambridgeCrawler

The main crawler class for Cambridge Dictionary.

from dictionary_crawler import CambridgeCrawler

async with CambridgeCrawler() as crawler:
    page = await crawler.crawl(word: str) -> CambridgePage

Parameters

• client (optional): Custom httpx.AsyncClient instance

OxfordLearnersCrawler

The main crawler class for OxfordLearners Dictionary.

from dictionary_crawler import OxfordLearnersCrawler

async with OxfordLearnersCrawler() as crawler:
    page = await crawler.crawl(word: str, direct_call: bool = False) -> OxfordLearnersPage

Parameters

• client (optional): Custom httpx.AsyncClient instance
• direct_call (optional): If using the exact endpoint, pass this as true so the package doesn't search the term through OxfordLearners API to find a match. Default is False.

Data Models

OxfordLearnersPage

The root model for OxfordLearners Dictionary data:

Field	Type	Description
search_term	str	The word that was searched
definitions	list[OxfordLearnersDefinition]	List of definitions

OxfordLearnersDefinition

Field	Type	Description
headword	str	The word/phrase
part_of_speech	str	noun, verb, adjective, etc.
pronunciations	list[OxfordLearnersPronunciation]	Pronunciations
cefr_level	CEFRLevel | None	CEFR level (A1-C2)
senses	list[OxfordLearnersSense]	Word senses with definitions
verb_forms	list[OxfordLearnersDefinitionVerbForm] | None	Verb conjugations
idioms	list[OxfordLearnersIdiom] | None	Related idioms
phrasal_verbs	list[str] | None	Related phrasal verbs

OxfordLearnersPronunciation

Field	Type	Description
phonetic	str	IPA transcription
pronunciation_url	str	Audio file URL

OxfordLearnersSense

Field	Type	Description
semantic_domain	str	Domain category
definition	str	The definition text
synonyms	list[str] | None	Synonyms
topics	list[OxfordLearnersSenseTopic] | None	Topics with CEFR levels
cefr_level	CEFRLevel | None	CEFR level (A1-C2)
grammar_labels	list[str] | None	Grammar labels
labels	list[str] | None	Usage labels
examples	list[OxfordLearnersExample] | None	Example sentences
collocations	list[OxfordLearnersSenseCollocation] | None	Collocations

OxfordLearnersIdiom

Field	Type	Description
idiom	str	The idiom text
senses	list[OxfordLearnersIdiomSense]	Idiom senses

CambridgePage

The root model containing all parsed data:

Field	Type	Description
search_term	str	The word that was searched
sections	list[CambridgeSection]	Dictionary sections (British, American, Business)
examples	list[CambridgeExample] | None	Example sentences
collocations	list[CambridgeCollocation] | None	Collocations

CambridgeSection

Field	Type	Description
type	CambridgeSectionType	British, American, or Business
entries	list[CambridgeEntry]	Dictionary entries

CambridgeEntry

Field	Type	Description
headword	str	The word/phrase
part_of_speech	str	noun, verb, adjective, etc.
pronunciations	list[CambridgePronunciation] | None	Pronunciations
senses	list[CambridgeSense]	Word senses with definitions
phrases	list[CambridgeSectionPhrase] | None	Related phrases
idioms	list[str] | None	Related idioms
phrasal_verbs	list[str] | None	Related phrasal verbs
verb_forms	list[CambridgeVerbForm] | None	Verb conjugations

CambridgePronunciation

Field	Type	Description
pronunciation	str	IPA transcription
audio_urls	list[str]	Audio file URLs
region	str	e.g., "UK", "US"

CambridgeSense

Field	Type	Description
cefr	CEFRLevel | None	CEFR level (A1-C2)
definitions	list[CambridgeDefinition]	Definitions

CambridgeDefinition

Field	Type	Description
definition	str	The definition text
examples	list[CambridgeDefinitionExample] | None	Example sentences
image_urls	list[str] | None	Illustration URLs

CEFRLevel

Enum with values: A1, A2, B1, B2, C1, C2

Exceptions

Exception	Description
DictionaryException	Base exception
DictionaryFetchException	HTTP request failed
DictionaryWordNotFoundException	Word not in dictionary
DictionaryParseException	Failed to parse page
CambridgeException	Cambridge-specific base
CambridgeFetchException	Cambridge fetch error
CambridgeWordNotFoundException	Word not in Cambridge
CambridgeParseException	Cambridge parse error
OxfordLearnersException	OxfordLearners-specific base
OxfordLearnersFetchException	OxfordLearners fetch error
OxfordLearnersWordNotFoundException	Word not in OxfordLearners
OxfordLearnersParseException	OxfordLearners parse error
OxfordLearnersUnexpectedContent	Unexpected OxfordLearners page content

JSON Export

Models can be serialized to JSON:

page = await crawler.crawl("example")
json_data = page.model_dump_json(indent=2, ensure_ascii=False)
print(json_data)

Development

Setup

python -m venv venv
source venv/bin/activate
pip install -e ".[dev]"

Running Tests

pytest

Project Structure

dictionary-crawler/
├── src/
│   └── dictionary_crawler/
│       ├── __init__.py
│       ├── base.py          # BaseCrawler ABC
│       ├── http.py          # HTTP client setup
│       ├── utils.py         # URL utilities
│       ├── exceptions.py    # Custom exceptions
│       ├── crawlers/
│       │   ├── cambridge.py # Cambridge parser
│       │   └── oxford_learners.py    # OxfordLearners parser
│       └── models/
│           ├── common.py    # Shared models (CEFRLevel)
│           ├── cambridge.py # Cambridge data models
│           └── oxford_learners.py    # OxfordLearners data models
└── tests/ # Will add soon

Roadmap (Future)

• Support for additional dictionaries (Merriam-Webster, Collins, etc.)
• Multi-language support (Spanish, French, German, etc.)
• Add more test coverage for OxfordLearners support

License

MIT