nltk-extratokenizers 0.1.1

Core NLTK model/data files (punkt, tagger, stopwords, wordnet, etc.) packaged for pip installation

nltk-extratokenizers

Core NLTK model/data files packaged for pip installation

nltk-extratokenizers provides essential NLTK data files (tokenizers, taggers, corpora) as a pip-installable package, eliminating the need for nltk.download(). This package implements NLTK Issue #3413 by using Python entry points to make NLTK data available directly from pip.

Installation

pip install nltk-extratokenizers

Important: This package requires a patched version of NLTK that supports entry-point-based data loading. You'll need to use the modified NLTK from this repository:

git clone https://github.com/Julien-ser/nltk_Group1_ENGG4450.git
cd nltk_Group1_ENGG4450
pip install -e .

Then install the data package:

pip install nltk-extratokenizers

What's Included

This package provides the following NLTK data resources:

Tokenizers

• punkt - Sentence tokenizer models

Taggers

• averaged_perceptron_tagger - Part-of-speech tagger

Corpora & Lexical Resources

• stopwords - Stopword lists for multiple languages
• wordnet - WordNet lexical database
• omw-1.4 - Open Multilingual Wordnet
• snowball_data - Snowball stemmer data
• names - Name corpus
• brown - Brown corpus
• movie_reviews - Movie reviews corpus

Usage

Once installed, you can use NLTK tokenizers and other resources without calling nltk.download():

Example: Using Punkt Sentence Tokenizer

from nltk.tokenize import PunktSentenceTokenizer

# No nltk.download() needed!
tok = PunktSentenceTokenizer()
sentences = tok.tokenize("Hello world. This is a test. Another sentence here.")
print(sentences)
# Output: ['Hello world.', 'This is a test.', 'Another sentence here.']

Example: Using Word Tokenizer

from nltk.tokenize import word_tokenize

# Works without nltk.download()
tokens = word_tokenize("Natural language processing is fun!")
print(tokens)
# Output: ['Natural', 'language', 'processing', 'is', 'fun', '!']

Example: Using Stopwords

from nltk.corpus import stopwords

# No download required
english_stops = set(stopwords.words('english'))
print('the' in english_stops)  # True

Example: Using POS Tagger

from nltk.tag import pos_tag
from nltk.tokenize import word_tokenize

# Works without nltk.download()
text = "The quick brown fox jumps over the lazy dog"
tokens = word_tokenize(text)
tags = pos_tag(tokens)
print(tags)
# Output: [('The', 'DT'), ('quick', 'JJ'), ('brown', 'JJ'), ...]

Verification

To verify that the package is working correctly and data is being loaded from the pip package:

1. Check Package Installation

pip show nltk-extratokenizers

2. Verify Entry Points

import importlib.metadata as m

print("nltk_data entry points:")
for ep in m.entry_points(group="nltk_data"):
    print(f"  {ep.name} -> {ep.value}")

Expected output should include:

punkt -> nltk_punkt.data
averaged_perceptron_tagger -> nltk_punkt.data
stopwords -> nltk_punkt.data
wordnet -> nltk_punkt.data
...

3. Verify Data Loading from Pip Package

import nltk

# Check that punkt tokenizer is loaded from pip package
path = nltk.data.find("tokenizers/punkt/english.pickle")
print(f"Punkt tokenizer location: {path}")
# Should show a path containing 'nltk_punkt' or 'site-packages'

4. Test Tokenizer Functionality

from nltk.tokenize import PunktSentenceTokenizer

try:
    tok = PunktSentenceTokenizer()
    result = tok.tokenize("Test sentence. Another one!")
    print("✓ Punkt tokenizer works correctly!")
    print(f"  Result: {result}")
except LookupError as e:
    print(f"✗ Error: {e}")
    print("  Make sure nltk-extratokenizers is installed and NLTK is patched.")

Requirements

• Python 3.8 or higher
• NLTK (patched version from this repository)
• setuptools >= 61.0

How It Works

This package uses Python's entry point system to register NLTK data resources. The patched NLTK's nltk.data.find() function:

1. First checks for pip-installed packages via nltk_data entry points
2. Falls back to the standard NLTK data search paths if not found

This allows NLTK data to be distributed and installed via pip, making it easier to manage dependencies and deploy applications.

Troubleshooting

"LookupError: Resource not found"

If you get a LookupError when using NLTK resources:

1. Verify the package is installed:

   pip show nltk-extratokenizers
   

2. Check that you're using the patched NLTK:

   import nltk
   print(nltk.__file__)
   # Should point to your local NLTK clone, not site-packages
   

3. Verify entry points are registered:

   import importlib.metadata as m
   eps = list(m.entry_points(group="nltk_data"))
   print(f"Found {len(eps)} entry points")
   

"Module not found: nltk_punkt"

If you see import errors, make sure:

• The package is installed: pip install nltk-extratokenizers
• You're using the correct Python environment

Contributing

Contributions are welcome! Please visit the GitHub repository to report issues or submit pull requests.

License

This package is distributed under the Apache License 2.0, same as NLTK.

Related Projects

• NLTK - Natural Language Toolkit
• NLTK GitHub - Official NLTK repository

Citation

If you use this package in your research, please cite:

Bird, Steven, Edward Loper and Ewan Klein (2009).
Natural Language Processing with Python. O'Reilly Media Inc.

---

Package Name: nltk-extratokenizers
Version: 0.1.0
Repository: https://github.com/Julien-ser/nltk_Group1_ENGG4450