nltk-extratokenizers User Guide

Core NLTK model/data files packaged for pip installation

nltk-extratokenizers provides essential NLTK data files (tokenizers, taggers, corpora) as a pip-installable package, eliminating the need for nltk.download(). This package implements NLTK Issue #3413 by using Python entry points to make NLTK data available directly from pip.

---

Quick Start

1. Clone and Install Patched NLTK

Open PowerShell and run:

git clone https://github.com/Julien-ser/nltk_Group1_ENGG4450.git
cd nltk_Group1_ENGG4450
python -m pip install --upgrade pip setuptools
python -m pip install -e .

2. Install the Data Package

python -m pip install nltk-extratokenizers

---

Running Tests

To validate your installation and run all tests:

1. Install test dependencies:

   python -m pip install -r requirements-test.txt
   python -m pip install pytest pytest-mock
   

2. Run the test suite:

   pytest nltk/test
   

---

Troubleshooting & Validation

Common Issues & Solutions

1. Missing regex Module

If you see ModuleNotFoundError: No module named 'regex':

python -m pip install regex

Or rely on the fallback to Python's re module (less feature-complete).

2. Missing pytest-mock Plugin

If you see fixture 'mocker' not found when running tests:

python -m pip install pytest-mock

3. SyntaxError in Test File

If you see a SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes, check for stray shell commands in your Python files and remove them.

4. RecursionError in Zip Pointer Test

If you see RecursionError: maximum recursion depth exceeded in test_find_zip_root_returns_zip_pointer, ensure all zipfile mocks are complete and stub classes are present.

---

Usage Examples

Once installed, you can use NLTK tokenizers and other resources without calling nltk.download():

Sentence Tokenizer

from nltk.tokenize import PunktSentenceTokenizer
tok = PunktSentenceTokenizer()
sentences = tok.tokenize("Hello world. This is a test. Another sentence here.")
print(sentences)
# Output: ['Hello world.', 'This is a test.', 'Another sentence here.']

Word Tokenizer

from nltk.tokenize import word_tokenize
tokens = word_tokenize("Natural language processing is fun!")
print(tokens)
# Output: ['Natural', 'language', 'processing', 'is', 'fun', '!']

Stopwords

from nltk.corpus import stopwords
english_stops = set(stopwords.words('english'))
print('the' in english_stops)  # True

POS Tagger

from nltk.tag import pos_tag
from nltk.tokenize import word_tokenize
text = "The quick brown fox jumps over the lazy dog"
tokens = word_tokenize(text)
tags = pos_tag(tokens)
print(tags)
# Output: [('The', 'DT'), ('quick', 'JJ'), ('brown', 'JJ'), ...]

---

Verification Steps

1. Check Package Installation

python -m pip show nltk-extratokenizers

2. Verify Entry Points

import importlib.metadata as m
print("nltk_data entry points:")
for ep in m.entry_points(group="nltk_data"):
    print(f"  {ep.name} -> {ep.value}")

Expected output should include:

punkt -> nltk_punkt.data
averaged_perceptron_tagger -> nltk_punkt.data
stopwords -> nltk_punkt.data
wordnet -> nltk_punkt.data
...etc

3. Verify Data Loading from Pip Package

import nltk
path = nltk.data.find("tokenizers/punkt/english.pickle")
print(f"Punkt tokenizer location: {path}")
# Should show a path containing 'nltk_punkt' or 'site-packages'

4. Test Tokenizer Functionality

from nltk.tokenize import PunktSentenceTokenizer
try:
    tok = PunktSentenceTokenizer()
    result = tok.tokenize("Test sentence. Another one!")
    print("✓ Punkt tokenizer works correctly!")
    print(f"  Result: {result}")
except LookupError as e:
    print(f"✗ Error: {e}")
    print("  Make sure nltk-extratokenizers is installed and NLTK is patched.")

---

What's Included

This package provides the following NLTK data resources:

• punkt (Sentence tokenizer models)
• averaged_perceptron_tagger (POS tagger)
• stopwords (Stopword lists)
• wordnet (WordNet lexical database)
• omw-1.4 (Open Multilingual Wordnet)
• snowball_data (Snowball stemmer data)
• names (Name corpus)
• brown (Brown corpus)
• movie_reviews (Movie reviews corpus)

---

Requirements

• Python 3.8 or higher
• Patched NLTK from this repository
• setuptools >= 61.0

---

How It Works

This package uses Python's entry point system to register NLTK data resources. The patched NLTK's nltk.data.find() function:

1. Checks for pip-installed packages via nltk_data entry points
2. Falls back to the standard NLTK data search paths if not found

This allows NLTK data to be distributed and installed via pip, making it easier to manage dependencies and deploy applications.

---

Contributing

Contributions are welcome! Please visit the GitHub repository to report issues or submit pull requests.

---

License

This package is distributed under the Apache License 2.0, same as NLTK.

---

Related Projects

• NLTK - Natural Language Toolkit
• NLTK GitHub - Official NLTK repository

---

Citation

If you use this package in your research, please cite:

Bird, Steven, Edward Loper and Ewan Klein (2009).
Natural Language Processing with Python. O'Reilly Media Inc.

---

Package Name: nltk-extratokenizers
Version: 0.1.0
Repository: https://github.com/Julien-ser/nltk_Group1_ENGG4450

---

Publishing to PyPI

To publish this package to PyPI, follow these steps in PowerShell:

1. Build the Package

python -m pip install --upgrade build
python -m build
# This creates a 'dist/' folder with your package files

2. Upload to PyPI

python -m pip install --upgrade twine
python -m twine upload dist/*
# Enter your PyPI username and password when prompted

3. (Optional) Test Upload to TestPyPI

python -m twine upload --repository testpypi dist/*
# Visit https://test.pypi.org/project/nltk-extratokenizers/ to verify

Note:

• Make sure your pyproject.toml and/or setup.py are correctly filled out with package metadata.
• You must have a PyPI account. Register at https://pypi.org/account/register/
• For more details, see the official PyPI packaging guide.