cewlio 1.0.1

Custom word list generator for web content

CeWLio 🕵️‍♂️✨

CeWLio is a powerful, Python-based Custom Word List Generator inspired by the original CeWL by Robin Wood. While CeWL is excellent for static HTML content, CeWLio brings modern web scraping capabilities to handle today's JavaScript-heavy websites. It crawls web pages, executes JavaScript, and extracts:

• 📚 Unique words (with advanced filtering)
• 📧 Email addresses
• 🏷️ Metadata (description, keywords, author)

Perfect for penetration testers, security researchers, and anyone needing high-quality, site-specific wordlists!

🤖 AI-Assisted Development: This project was created with the help of AI tools, but solves real-world problems in web scraping and word list generation. Every line of code has been carefully reviewed, tested, and optimized for production use.

---

🔄 CeWL vs CeWLio: What's Different?

Feature	Original CeWL	CeWLio
Language	Ruby	Python 3.12+
JavaScript Support	❌ Static HTML only	✅ Full JavaScript rendering
Browser Engine	Basic HTTP requests	🚀 Playwright (Chromium/Firefox/WebKit)
Modern Web Support	❌ Struggles with SPAs	✅ Handles React, Vue, Angular
Word Processing	Basic filtering	🎯 Advanced: length, case, umlauts, groups
Email Extraction	Basic regex	🔍 Smart: content + mailto links
API Access	❌ CLI only	✅ Python API + CLI
Testing	Limited	🧪 100% test coverage
Installation	Ruby gems	📦 pip install cewlio
Cross-Platform	Ruby dependencies	✅ Universal Python package
Active Development	❌ Limited updates	✅ Modern, actively maintained

---

🚀 Features

• JavaScript-Aware Extraction: Uses headless browser to render pages and extract content after JavaScript execution
• Advanced Word Processing:
  • Minimum/maximum word length filtering
  • Lowercase conversion
  • Alphanumeric or alpha-only words
  • Umlaut conversion (ä→ae, ö→oe, ü→ue, ß→ss)
  • Word frequency counting
• Word Grouping: Generate multi-word phrases (e.g., 2-grams, 3-grams)
• Email & Metadata Extraction: Find emails from content and mailto links, extract meta tags
• Flexible Output: Save words, emails, and metadata to separate files or stdout
• Professional CLI: All features accessible via command-line interface
• Comprehensive Testing: 100% test coverage

---

🛠️ Installation

From PyPI (Recommended)

pip install cewlio

From Source

git clone https://github.com/yourusername/cewlio
cd cewlio
pip install -e .

Dependencies

• Python 3.12+
• Playwright (for browser automation)
• BeautifulSoup4 (for HTML parsing)
• Requests (for HTTP handling)

---

⚡ Quick Start

Basic Usage

# Extract words from a website
cewlio https://example.com

# Save words to a file
cewlio https://example.com -o wordlist.txt

# Extract emails and metadata
cewlio https://example.com --email emails.txt --metadata meta.txt

Advanced Examples

Generate word groups with counts:

cewlio https://example.com --groups 3 --count -o phrases.txt

Custom word filtering:

cewlio https://example.com --min-length 4 --max-length 12 --lowercase --convert-umlauts

Handle JavaScript-heavy sites:

cewlio https://example.com -w 5 --visible

Extract only emails and metadata:

cewlio https://example.com --no-words --email emails.txt --metadata meta.txt

---

🎛️ Command-Line Options

Option	Description	Default
url	URL to process	Required
-o, --output	Output file for words	stdout
--email	Output file for email addresses	-
--metadata	Output file for metadata	-
--min-length	Minimum word length	3
--max-length	Maximum word length	No limit
--lowercase	Convert words to lowercase	False
--with-numbers	Include words with numbers	False
--convert-umlauts	Convert umlaut characters	False
--count	Show word counts	False
--groups	Generate word groups of specified size	-
-w, --wait	Wait time for JavaScript execution (seconds)	0
--visible	Show browser window	False
--timeout	Browser timeout (milliseconds)	30000
--no-words	Don't extract words (only emails/metadata)	False

---

📚 API Usage

Basic Python Usage

from cewlio import CeWLio

# Create instance with custom settings
cewlio = CeWLio(
    min_word_length=4,
    max_word_length=12,
    lowercase=True,
    convert_umlauts=True
)

# Process HTML content
html_content = "<p>Hello world! Contact us at [e-mail]</p>"
cewlio.process_html(html_content)

# Access results
print("Words:", list(cewlio.words.keys()))
print("Emails:", list(cewlio.emails))
print("Metadata:", list(cewlio.metadata))

Process URLs

import asyncio
from cewlio import CeWLio, process_url_with_cewlio

async def main():
    cewlio = CeWLio()
    success = await process_url_with_cewlio(
        url="https://example.com",
        cewlio_instance=cewlio,
        wait_time=5,
        headless=True
    )
    
    if success:
        print(f"Found {len(cewlio.words)} words")
        print(f"Found {len(cewlio.emails)} emails")

asyncio.run(main())

---

🧪 Testing

The project includes a comprehensive test suite with 33 tests covering all functionality:

• ✅ Core functionality tests (15 tests)
• ✅ HTML extraction tests (3 tests)
• ✅ URL processing tests (2 tests)
• ✅ Integration tests (3 tests)
• ✅ Edge case tests (10 tests)

Total: 33 tests with 100% success rate

For detailed testing information and development setup, see CONTRIBUTING.md.

---

🐛 Troubleshooting

Common Issues

"No module named 'playwright'"

pip install playwright
playwright install

JavaScript-heavy sites not loading properly

# Increase wait time for JavaScript execution
cewlio https://example.com -w 10

Browser timeout errors

# Increase timeout and wait time
cewlio https://example.com --timeout 60000 -w 5

---

🤝 Contributing

We welcome contributions! Please see CONTRIBUTING.md for detailed guidelines on:

• 🚀 Getting started with development
• 📝 Code style and formatting guidelines
• 🧪 Testing requirements and procedures
• 🔄 Submitting pull requests
• 🐛 Reporting issues
• 💡 Feature requests

Quick start:

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests for new functionality
5. Submit a pull request

For detailed development setup and guidelines, see CONTRIBUTING.md.

---

📝 Changelog

v1.0.0

• ✨ Initial release
• 🎯 Complete word extraction with filtering
• 📧 Email extraction from content and mailto links
• 🏷️ Metadata extraction from HTML meta tags
• 🔧 Professional CLI interface
• 🧪 Comprehensive test suite (33 tests)
• 📦 PyPI-ready packaging

---

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

---

🙏 Credits

• Inspired by CeWL by Robin Wood
• Built with Playwright for browser automation
• Uses BeautifulSoup4 for HTML parsing

---

📞 Support

• 🐛 Issues: GitHub Issues
• 📖 Documentation: GitHub Wiki
• 💬 Discussions: GitHub Discussions

---

Made with ❤️ for the security community