gd-core-web-sm 0.1.2

A spaCy-compatible Scottish Gaelic NLP pipeline with POS tagging and a POS-aware lemmatizer.

Scottish Gaelic Lemmatizer – CSRI Research Project

This project is part of an ongoing collaborative research initiative between the Cornell College Computer Science Department and Napier University in Edinburgh, Scotland. The goal is to build tools that enable deeper text analysis for the Scottish Gaelic language — a low-resource language with limited computational tools.

---

Project Context

• Year 1: A team of students created a cleaned corpus of Scottish Gaelic text and a tokenizer.
• Year 2 (Current): Focused on developing a rule-based lemmatizer, generating word frequency statistics, and implementing preprocessing techniques specific to Scottish Gaelic grammar and spelling.

---

What This Tool Does

This lemmatizer reduces inflected Scottish Gaelic word forms to their base forms (lemmas) using:

• A manually curated irregular dictionary for unpredictable wordforms (e.g., chunnaic → faic)
• A set of carefully tested suffix rules for regular morphological patterns (e.g., taighean → taigh)
• Preprocessing steps that handle accents, emphatic suffixes, prosthetic consonants, and lenition
• Frequency-guided refinement: rules and dictionary are informed by analysis of the most common words
• Output in a simple, editable format for future researchers to reuse or expand

---

Tree-like Folder Structure

lemmatizer_gaelic/
├── lemmatizer.py                       # Main lemmatization script using spaCy custom component
├── irregular_dict.json                 # Irregular word → lemma mappings
├── Latest_Corpus.txt                   # Cleaned corpus with "word source" format
├── lemmatized_output.txt               # Output from previous lemmatizer version / reference output
├── lemmatized_output_pos_aware_example.txt # Example output from the new POS-aware lemmatizer
├── compare_lemmas.py                   # Script to compare two lemmatizer output files
├── word_frequency.py                   # Word frequency analysis script (by source)
├── stopWords.txt                       # List of Gaelic stop words (excluded from stats)
├── CorpusBySource.py                   # Exports per-source texts from the raw corpus
└── README.md                           # Project documentation

---

How It Works

1. Corpus Preparation

• The input corpus (Latest_Corpus.txt) contains lines in the format:
• Only the first word of each line is used for lemmatization.

---

2. Preprocessing Steps

• Replace acute accents with grave accents
• Remove emphatic suffixes: -sa, -se, -san, -ne
• Strip prosthetic consonants: t-, h-, n- (at beginning)
• Remove lenition marker: h as second letter (e.g., bhean → bean)

---

3. Lemmatization Logic (lemmatizer.py)

• Apply preprocessing to each token
• If token exists in irregular dictionary → use its mapped lemma
• If not, apply suffix rules (-ean, -an, -achadh, etc.)
• If no rules match, return preprocessed token (unless it's too short)

---

Code Flowchart

Load input corpus file
  ↓
Process text with spaCy pipeline (Tokenizer -> POS Tagger -> Custom Lemmatizer)
  ↓                                  (This step assigns POS tags to tokens)
For each token in the spaCy `Doc` object:
  ↓
  Get lowercased raw text
  Initialize lemma with raw text
  ↓
  1. Check if raw token text exists in irregular dictionary
  ├─ Yes → Use lemma from irregulars (e.g., "deach" → "rach") and SKIP further steps for this token
  └─ No → Proceed to preprocessing
             ↓
      2. Preprocess token text (e.g., "chuala" → "cuala", "bhean" → "bean"):
        ├─ Replace acute accents with grave accents
        ├─ Remove emphatic suffixes
        ├─ Remove prosthetic consonants
        └─ Remove lenition marker
             ↓
      3. Check if preprocessed token text exists in irregular dictionary
      ├─ Yes → Use lemma from irregulars (e.g., "cuala" → "cluinn") and SKIP further steps
      └─ No → Proceed to POS-informed suffix rules
                  ↓
          4. Apply POS-informed suffix rules (based on `token.pos_`):
            ├─ If `token.pos_ == "NOUN"`:
            │  ├─ Apply longest noun suffixes first (e.g., "-aichean" -> "", "-annan" -> "", "-ean" -> "", "-an" -> "")
            │  └─ Examples: "eileanan" → "eilean", "stàitean" → "stàit", "lochan" → "loch"
            ├─ If `token.pos_ == "VERB"`:
            │  ├─ Apply longest verb suffixes first (e.g., "-eachadh" -> "-ich", "-eadh" -> "", "-te" -> "", "-ta" -> "")
            │  └─ Examples: "atharrachadh" → "atharraich", "stèidheachadh" → "stèidhich", "aonaichte" → "aonaich", "sònraichte" → "sònraich"
            ├─ If `token.pos_ == "ADJ"`:
            │  ├─ Apply adjective-specific rules
            │  └─ Examples: (comparatives, superlatives, ensuring base forms like "cumanta" remain unchanged)
            └─ If no POS-specific rule matched:
                   ↓
                  5. Fallback: If preprocessing changed the word and no rule applied, use preprocessed word as lemma.
                      If lemma length is 1 or less (and not an irregular), revert to original raw text.
  ↓
Assign final lemma to token in spaCy `Doc` object
  ↓
Repeat for all tokens
  ↓
Output results:
  ├─ Print token → lemma to console
  └─ Save results to `lemmatized_output_pos_aware_example.txt`
  ↓
Print summary:
  ├─ Changed by irregulars
  ├─ Changed by preprocessing
  ├─ Changed by suffix
  ├─ Total changed / unchanged words
  └─ Total operations applied

---

4. Word Frequency Analysis (word_frequency.py)

• Computes the top 100 most frequent words by source
• Excludes stopwords (stopWords.txt)
• Guides rule updates and dictionary development

---

Future Work

• Integrate a robust Gaelic POS tagger to improve lemmatizer accuracy
• Expand suffix rules and irregulars using linguistic input from Napier University
• Group frequent words by part of speech (noun, verb, adj) for better rule targeting
• Evaluate lemmatizer accuracy using a manually verified gold standard

---

Status

Work in Progress — June 2025