pycewl 0.1.0

Python async re-implementation of CeWL (Custom Word List Generator)

pycewl

Python async re-implementation of CeWL (Custom Word List Generator)

A modern, high-performance Python implementation of CeWL with:

• Async spider using asyncio + httpx
• HTML parsing with beautifulsoup4
• Google Search discovery for seed URLs
• Smart Word Relevance Scoring using Google Natural Language API
• Production-grade packaging for PyPI

Installation

pip install pycewl

For development:

pip install pycewl[dev]

For PDF metadata extraction:

pip install pycewl[pdf]

Quick Start

Basic Usage

Spider a website and generate a wordlist:

pycewl https://example.com -w words.txt

With Options

# Set spider depth and show word counts
pycewl https://example.com -d 3 -c -w words.txt

# Extract emails too
pycewl https://example.com -e --email-file emails.txt -w words.txt

# Lowercase words, minimum 5 characters
pycewl https://example.com --lowercase -m 5 -w words.txt

Google Search Integration

Find seed URLs using Google Custom Search:

# Set environment variables
export GOOGLE_API_KEY="your-api-key"
export GOOGLE_SEARCH_ENGINE_ID="your-search-engine-id"

# Search and spider
pycewl --google-keyword "star trek fan site" -w words.txt

Smart Relevance Scoring

Group words by relevance to your search query using Google NLP:

# Set Google Cloud credentials
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account.json"

# Spider with relevance scoring
pycewl --google-keyword "star trek" --relevance-scoring \
    --related-file related.txt \
    --unrelated-file general.txt

Output structure:

=== Words Related to "star trek" ===
enterprise, 42
spock, 38
federation, 25
...

=== General Words (Not Query-Specific) ===
welcome, 15
contact, 12
page, 8
...

CLI Reference

Option	Description
-d, --depth INT	Spider depth (default: 2)
-m, --min-word-length INT	Minimum word length (default: 3)
-x, --max-word-length INT	Maximum word length
-w, --write PATH	Output file for words
-n, --no-words	Don't output wordlist
-c, --count	Show word counts
-g, --groups INT	Group words by count ranges
-o, --offsite	Allow spidering offsite URLs
-e, --email	Extract email addresses
--email-file PATH	Output file for emails
-a, --meta	Extract metadata from documents
--meta-file PATH	Output file for metadata
-k, --keep	Keep downloaded files
--lowercase	Convert words to lowercase
--with-numbers	Include words containing numbers
--convert-umlauts	Convert German umlauts to ASCII
-u, --user-agent TEXT	Custom user agent
--concurrency INT	Concurrent requests (default: 10)
--auth-type TEXT	Authentication type (basic/digest/bearer)
--auth-user TEXT	Authentication username
--auth-pass TEXT	Authentication password
--auth-token TEXT	Bearer/JWT token for authentication
--proxy-host TEXT	Proxy hostname
--proxy-port INT	Proxy port
-H, --header TEXT	HTTP header (Name: Value)
-v, --verbose	Verbose output
--google-keyword TEXT	Search Google for seed URLs
--google-max-results INT	Max Google results (default: 10)
--relevance-scoring	Enable word relevance scoring
--relevance-threshold FLOAT	Relevance threshold (default: 0.5)
--related-file PATH	Output for query-related words
--unrelated-file PATH	Output for general words
--version	Show version

Bearer Token Authentication

Authenticate with a bearer or JWT token:

# Using --auth-token (auto-detects bearer type)
pycewl https://api.example.com --auth-token "your-access-token" -w words.txt

# Explicit auth type
pycewl https://api.example.com --auth-type bearer --auth-token "eyJhbG..." -w words.txt

Python API

import asyncio
from pycewl import Crawler, CeWLConfig, SpiderConfig, WordConfig, WordExtractor

async def main():
    config = CeWLConfig(
        url="https://example.com",
        spider=SpiderConfig(depth=2, concurrency=5),
        word=WordConfig(min_length=4, lowercase=True),
    )

    crawler = Crawler(config)
    extractor = WordExtractor(config.word)

    async for result in crawler.crawl(["https://example.com"]):
        if result.html:
            extractor.process_html(result.html)

    for word, count in extractor.get_sorted_words()[:20]:
        print(f"{word}: {count}")

asyncio.run(main())

Google Cloud Setup

For Google Search

1. Go to Google Cloud Console
2. Create a project and enable the Custom Search API
3. Create an API key
4. Set up a Programmable Search Engine
5. Set environment variables:

   export GOOGLE_API_KEY="your-api-key"
   export GOOGLE_SEARCH_ENGINE_ID="your-cx-id"
   

For Relevance Scoring (NLP)

1. Enable the Natural Language API in Google Cloud Console
2. Create a service account with Natural Language API access
3. Download the JSON key file
4. Set environment variable:

   export GOOGLE_APPLICATION_CREDENTIALS="/path/to/key.json"
   

Development

# Clone repository
git clone https://github.com/digininja/CeWL.git
cd CeWL/pycewl

# Install dev dependencies
make install-dev

# Run tests
make test

# Run linting
make lint

# Format code
make format

# Build package
make build

License

MIT License - see LICENSE for details.

Credits

• Original CeWL by Robin Wood (digininja)
• Python implementation maintains feature parity with Ruby original