codexcollector 0.2.0

A wrapper for common Python packages to build a text corpus with a single line of code.

CodexCollector

A wrapper for common Python packages to build a text corpus with a single line of code.

Overview

CodexCollector provides a unified interface for document ingestion from both filesystem paths and web URLs. It automatically handles format detection, text extraction, metadata collection, and error logging, making it easy to build text corpora from diverse document sources.

Key Capabilities:

• Extract text from multiple document formats (Word, PowerPoint, PDF, plain text)
• Collect documents from local filesystems with recursive directory traversal
• Scrape and download documents from websites with configurable crawling depth
• Extract text directly from web pages (HTML content extraction)
• Automatic encoding detection for text files
• Metadata extraction (creation dates) where available
• Comprehensive error logging for failed ingestions
• Rate limiting and timeout controls for web requests
• File size validation and filtering

Installation

Requirements

• Python >=3.10
• Dependencies installed automatically via pip/uv

From PyPI (Recommended)

# Using uv
uv add codexcollector

# Using pip
pip install codexcollector

From Source

# Clone the repository
git clone https://github.com/NotAndroid37/codexcollector.git
cd codexcollector

# Using uv
uv sync

# Or using pip
pip install -e .

Quick Start

Basic Filesystem Collection

from codexcollector import CodexCollector

# Initialize collector with default settings
collector = CodexCollector()

# Collect documents from a directory
corpus = collector.collect('/path/to/documents')

# Access collected documents (returns a list)
for document in corpus:
    print(f"File: {document['filename']}")
    print(f"Source: {document['source']}")
    print(f"Text length: {len(document['text'])} characters")
    print(f"Date: {document['date']}")

Basic Web Collection (Document Files)

from codexcollector import CodexCollector

# Configure for web scraping - downloads linked document files
collector = CodexCollector(
    request_delay=2.0,  # 2 seconds between requests
    timeout=30,
    max_crawl_depth=2   # Crawl starting page + 2 levels deep
)

# Collect document files linked on the website
corpus = collector.collect('https://example.com/docs')

# Check for errors
if collector.error_log:
    print(f"Failed to process {len(collector.error_log)} files")
    for source, error in collector.error_log.items():
        print(f"  {source}: {error}")

Web Page Text Extraction

from codexcollector import CodexCollector

# Configure to extract text from web pages themselves
collector = CodexCollector(
    extract_page_text=True,  # Extract HTML page content instead of downloading files
    request_delay=2.0,
    max_crawl_depth=1
)

# Collect text from web pages
corpus = collector.collect('https://example.com/articles')

for page in corpus:
    print(f"Page: {page['filename']}")
    print(f"Content preview: {page['text'][:200]}...")

Note: Web page text extraction may encounter challenges such as 404 errors, permission denied responses, or rate limiting from target websites. These are server-side restrictions, not issues with CodexCollector. See the Web Page Text Extraction Considerations section for details.

Configuration

CodexCollector Parameters

Parameter	Type	Default	Description
supported_extensions	set[str]	{'.pdf', '.docx', ...}	File extensions to process (must include leading dot)
max_file_size_mb	int	100	Maximum file size in MB (0 = no limit)
request_delay	float	1.0	Delay between web requests in seconds
timeout	int	30	HTTP request timeout in seconds
excluded_dirs	set[str]	{'.git', '__pycache__', ...}	Directory names to skip during traversal
max_crawl_depth	int	2	Maximum depth for recursive web crawling (0 = starting page only)
default_encoding	str	'utf-8'	Default encoding when chardet fails or as fallback
max_collection_time	int	0	Maximum collection time in seconds (0 = no limit)
extract_page_text	bool	False	When True, extracts text from HTML pages instead of downloading linked files

Default supported extensions:

{'.pdf', '.docx', '.doc', '.pptx', '.ppt', '.txt', '.md', '.rtf'}

Default excluded directories:

{'.git', '__pycache__', '.pytest_cache', 'node_modules', '.vscode',
 '.idea', 'venv', '.env', 'logs', 'tmp', 'temp'}

Advanced Usage

Custom File Filtering

from codexcollector import CodexCollector

# Only collect PDFs and Word documents, limit file size
collector = CodexCollector(
    supported_extensions={'.pdf', '.docx'},
    max_file_size_mb=25,
    excluded_dirs={'archive', 'backup', 'temp'}
)

corpus = collector.collect('/data/documents')

Web Scraping with Depth Control

from codexcollector import CodexCollector

# Configure deep crawling with rate limiting
collector = CodexCollector(
    max_crawl_depth=3,       # Starting page + 3 levels
    request_delay=1.5,       # Respectful crawling
    timeout=60,
    max_collection_time=600  # 10 minute timeout
)

corpus = collector.collect('https://research-papers.example.com')

Progress Logging

from codexcollector import CodexCollector

# Log progress every 100 documents
collector = CodexCollector()
corpus = collector.collect(
    '/large/document/collection',
    progress_callback=100  # Logs at 100, 200, 300, etc. plus final count
)

# Output:
# INFO: Progress: 100 / 500 (20%)
# INFO: Progress: 200 / 500 (40%)
# ...

Custom Text Encoding

from codexcollector import CodexCollector

# Use Latin-1 as fallback for Western European documents
collector = CodexCollector(
    default_encoding='latin-1'
)

corpus = collector.collect('/path/to/european/docs')

Collection with Timeout

from codexcollector import CodexCollector

# Limit collection to 5 minutes
collector = CodexCollector(max_collection_time=300)

corpus = collector.collect('https://large-site.example.com')

print(f"Collected {len(corpus)} documents")
print(f"Errors: {len(collector.error_log)}")

Output Format

Corpus Structure

The collect() method returns a list of document dictionaries:

[
    {
        'filename': 'report.pdf',
        'source': '/path/to/report.pdf',
        'text': 'Document content...',
        'date': '2024-01-15T10:30:00'
    },
    {
        'filename': 'presentation.pptx',
        'source': 'https://example.com/presentation.pptx',
        'text': 'Slide text content...',
        'date': None
    }
]

Document Dictionary Fields

Field	Type	Description
filename	str	Name of the file extracted from path or URL
source	str | None	Original source path or URL
text	str	Extracted text content (empty string on failure)
date	str | None	ISO format creation date from metadata, or None if unavailable

Type Safety

CodexCollector provides a TypedDict definition for better type hints:

from codexcollector import Document

# Type-safe document creation
doc: Document = {
    'filename': 'example.pdf',
    'source': '/path/to/example.pdf',
    'text': 'Content here',
    'date': '2024-01-01T00:00:00'
}

# Type-safe corpus
corpus: list[Document] = [doc]

Error Handling

CodexCollector implements graceful degradation - individual document failures don't halt collection:

from codexcollector import CodexCollector

collector = CodexCollector()
corpus = collector.collect('/path/to/documents')

# Check error log for failures
if collector.error_log:
    print(f"\nFailed to process {len(collector.error_log)} files:")
    for source, error in collector.error_log.items():
        print(f"  {source}")
        print(f"    Error: {error}")

# Successful documents are still in the corpus
print(f"\nSuccessfully collected {len(corpus)} documents")

Custom Exceptions

from codexcollector import IngestionError, ConfigurationError

try:
    collector = CodexCollector(max_file_size_mb=-1)
except ConfigurationError as e:
    print(f"Invalid configuration: {e}")

# IngestionError is caught internally and logged to error_log

Web Page Text Extraction Considerations

The extract_page_text=True option enables extraction of text content directly from HTML pages. While powerful, this feature may encounter challenges depending on target website configurations:

Common Challenges

Issue	Cause	Mitigation
404 Not Found	Broken links, dynamic URLs, or removed content	Expected behavior; errors logged to error_log
403 Forbidden	Server blocks automated requests	Increase request_delay, respect robots.txt
Rate Limiting	Too many requests in short period	Use higher request_delay (2-5 seconds recommended)
Empty Text	JavaScript-rendered content	Not supported; use browser automation tools instead
Connection Timeouts	Slow servers or network issues	Increase timeout parameter

Best Practices for Web Page Extraction

from codexcollector import CodexCollector

# Conservative configuration for web page extraction
collector = CodexCollector(
    extract_page_text=True,
    request_delay=3.0,       # Generous delay between requests
    timeout=60,              # Allow slow responses
    max_crawl_depth=1,       # Limit scope initially
    max_collection_time=300  # Set overall timeout
)

corpus = collector.collect('https://example.com/articles')

# Always check error_log for issues
print(f"Collected {len(corpus)} pages")
print(f"Errors encountered: {len(collector.error_log)}")

# Review specific errors
for url, error in collector.error_log.items():
    if "403" in error or "Forbidden" in error:
        print(f"Access denied: {url}")
    elif "404" in error:
        print(f"Page not found: {url}")

Content Extraction

When extract_page_text=True, CodexCollector:

• Removes boilerplate elements (navigation, headers, footers, sidebars)
• Extracts main content from <main>, <article>, or similar containers
• Converts HTML to clean text using html2text
• Sets filename to the URL path
• Sets date to None (HTML pages rarely have reliable date metadata)

Supported File Formats

Format	Extensions	Library	Metadata
Word (modern)	.docx	python-docx	Creation date
Word (legacy)	.doc	docx2txt	None
PowerPoint	.pptx, .ppt	python-pptx	Creation date
PDF	.pdf	pypdf	Creation date
Plain text	.txt, .md, .rtf	Built-in	None

Note: For files without embedded metadata, the file system modification time is used as a fallback for the date field.

Architecture

Design Patterns

Protocol-Based Extension: Text extractors implement the TextExtractor protocol, enabling easy addition of new format support without modifying core code.

class TextExtractor(Protocol):
    def extract_text(self, file_path: Path) -> str: ...
    def extract_metadata(self, file_path: Path) -> dict[str, str | None]: ...

Composition Over Inheritance: Format-specific extractors are composed into the collector rather than inherited, allowing runtime configuration and testing.

Key Methods

Method	Description
collect(source, progress_callback=None)	Main public API for document collection
_is_url(source)	Detects whether source is URL or filesystem path
_discover_files_from_path(path)	Recursive filesystem traversal
_discover_files_from_url(url)	Web crawling to find document file links
_collect_page_text_from_web(url, ...)	Web crawling with HTML text extraction
_extract_text_from_html(html, url)	Extracts main content from HTML pages
_extract_from_file(file_path, source)	Format-specific text extraction
_download_file(url, temp_path)	Streaming download for web sources

Dependencies

Core Dependencies

• beautifulsoup4 - HTML parsing for web scraping
• chardet - Character encoding detection
• docx2txt - Legacy Word document support
• html2text - HTML to text conversion for web page extraction
• pandas - Data structure support
• pypdf - PDF text extraction
• python-docx - Modern Word document parsing
• python-pptx - PowerPoint parsing
• requests - HTTP requests for web scraping

Development Dependencies

• ipykernel - Jupyter notebook support
• pytest - Testing framework

Testing

Run the included test suite:

# Using pytest
uv run pytest tests_claude/ -v

# Or directly
pytest tests_claude/ -v

Performance Considerations

Large Collections

• Use max_file_size_mb to skip oversized files
• Set max_collection_time to prevent indefinite execution
• Use progress_callback with an interval (e.g., progress_callback=100) for monitoring

Web Scraping

• Adjust request_delay to respect server rate limits (recommended: >=1.0 seconds)
• Use max_crawl_depth to limit scope and prevent excessive crawling
• Set reasonable timeout values for slow servers

Text Encoding

• Use default_encoding to skip chardet for known-encoding collections
• For homogeneous sources, setting a specific encoding improves performance

Limitations

• PDF extraction: No OCR support - scanned documents without text layers return empty strings
• Legacy formats: .doc and .ppt files have limited metadata extraction
• Web scraping: Respects same-domain restriction; doesn't handle JavaScript-rendered content
• Web page extraction: May encounter 404s, 403s, and rate limiting from target servers (see Web Page Text Extraction Considerations)
• Encoding detection: May fail on very short files or files with mixed encodings
• File formats: No support for EPUB, ODT, or other formats (extensible via Protocol)

Use Cases

Building Text Corpora

from codexcollector import CodexCollector

# Collect academic papers
collector = CodexCollector(supported_extensions={'.pdf'})
corpus = collector.collect('/research/papers')

# Extract just the text for analysis
texts = [doc['text'] for doc in corpus]

Document Archive Mining

from codexcollector import CodexCollector

# Process large archive with progress logging every 100 documents
collector = CodexCollector(
    max_file_size_mb=50,
    excluded_dirs={'backup', 'archive_v1', 'duplicates'}
)

corpus = collector.collect(
    '/corporate/document/archive',
    progress_callback=100
)

Web Documentation Collection

from codexcollector import CodexCollector

# Scrape technical documentation (download linked files)
collector = CodexCollector(
    supported_extensions={'.pdf', '.docx'},
    max_crawl_depth=3,
    request_delay=2.0
)

corpus = collector.collect('https://docs.example.com/api')

Blog/Article Collection

from codexcollector import CodexCollector

# Extract text from blog posts (HTML content)
collector = CodexCollector(
    extract_page_text=True,
    request_delay=2.0,
    max_crawl_depth=2
)

corpus = collector.collect('https://blog.example.com')

# Filter out pages with minimal content
substantial_articles = [doc for doc in corpus if len(doc['text']) > 500]

Contributing

Contributions are welcome! Areas for improvement:

1. Additional format support: EPUB, ODT, etc.
2. OCR integration: For scanned PDFs
3. Parallel processing: For large collections
4. Incremental collection: Track processed files to avoid reprocessing
5. Custom stopword filtering: At collection time
6. Robots.txt compliance: Enhanced web scraping ethics

Development Process

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Add tests for new functionality
4. Ensure all tests pass (uv run pytest tests_claude/ -v)
5. Submit a pull request

Roadmap

• OCR integration for scanned documents (pytesseract)
• Parallel processing for large corpora (multiprocessing)
• Additional file format support (EPUB, ODT)
• Robots.txt compliance for web scraping
• Incremental collection with state persistence
• Content-type validation before download
• Language-specific encoding presets
• Custom extractor registration API
• JavaScript rendering support (Playwright/Selenium integration)

Known Issues

• Legacy Word (.doc) and PowerPoint (.ppt) files may not extract text correctly in all cases
• Web crawling doesn't handle JavaScript-rendered content
• Web page text extraction may fail on sites with strict access controls or rate limiting
• Temporary files from web downloads are created in current directory

License

This project is licensed under the MIT License.

Support

• Issues: GitHub Issues

---

Built for seamless document collection and text extraction.