novel-cli 2.0.0

Web novel scraper and library manager with Rich UI

novel-cli

A CLI tool for scraping and managing web novels offline, with an optional web viewer.

Features

• Smart Scraping: Automatically skips completed novels with all chapters already downloaded, saving time and bandwidth
• Offline Reading: Download and store complete novels locally
• Web Viewer: Optional web-based reader for your library
• Multiple Sources: Support for various novel hosting sites ( Only tested on one site - RR)
• Library Management: Add, remove, export, and organize your collection
• Progress Tracking: Monitor download progress and completion status

Installation

From Source

git clone https://github.com/Azizgic/novel-cli.git
cd novel-cli
pip install -e .

From PyPI

pip install novel-cli

Quick Start

# Add a novel to your reading list
novel add https://www.royalroad.com/fiction/12345/some-novel

# Scrape all novels (smart mode - skips completed ones)
novel scrape

# Scrape a single novel
novel scrape "Novel Title"

# Update stats only (no chapter download)
novel update-stats

# List your library
novel list

# Show novel details
novel show "Novel Title"

# Remove a novel
novel remove "Novel Title"

# Export library
novel export json
novel export csv

# Show statistics
novel stats

# Clean up orphaned data
novel clean

Command Reference

Command	Description
novel add <url>	Add a novel URL to your reading list
novel scrape [title]	Download chapters (skips completed novels by default)
novel scrape --dry-run	Preview what would be scraped without downloading
novel update-stats [title]	Update metadata without downloading chapters
novel list	Display all novels in your library
novel show [title]	Show detailed information about a novel
novel remove [title]	Remove a novel from your library
novel export <json|csv>	Export your library to JSON or CSV
novel stats	Display library statistics
novel clean	Remove orphaned data and thumbnails
novel read	Start the web viewer

Web Viewer (Optional)

cd viewer && pnpm install
novel read
# Opens at http://localhost:4173

Data Storage

All your data is stored locally in ~/.novel-library/:

• library.db - SQLite database with chapters and metadata
• urls.txt - Your reading list
• thumbnails/ - Cover images
• library.json - Configuration file

Smart Scraping

The tool intelligently avoids rescraping novels that are already complete. A novel is skipped if:

• Its status is marked as "Completed"
• AND all chapters have already been downloaded

This saves time and bandwidth during regular scraping sessions. Other statuses (Ongoing, Hiatus, Dropped, etc.) are always scraped to catch updates.

Testing

The project includes comprehensive automated tests to ensure reliability.

Running Tests

# Install test dependencies
pip install pytest pytest-cov

# Run all tests
pytest tests/

# Run with verbose output
pytest tests/ -v

# Generate coverage report
pytest tests/ --cov=novel --cov-report=html

See tests/README.md for detailed testing documentation.

Recent Changes

Smart Skip Feature (v0.1.0)

• Added ScrapeChecker module to prevent rescraping completed novels
• Reduces unnecessary downloads and improves performance
• Fully automated testing with 30+ test cases
• See docs/ENHANCEMENT_SKIP_COMPLETED.md for details

Development

Project Structure

novel-cli/
├── novel/
│   ├── cli.py              # Command-line interface
│   ├── manage.py           # Library management
│   ├── paths.py            # Data storage paths
│   └── scraper/
│       ├── novel_scraper_indexer.py   # Main scraper
│       ├── scrape_checker.py          # Skip logic
│       ├── writers.py      # Database writer
│       └── base_scraper.py # Base scraper class
├── tests/
│   └── test_scrape_checker.py  # Test suite
├── examples/
│   └── integration_example.py   # Integration examples
└── viewer/                      # Web viewer (Svelte)

Contributing

Contributions are welcome! Please ensure:

1. All tests pass: pytest tests/
2. Code follows the existing style
3. New features include test coverage

Dependencies

• requests - HTTP requests
• beautifulsoup4 - HTML parsing
• cloudscraper - Handle CloudFlare protection
• selenium - Browser automation (if needed)
• markdownify - Convert HTML to Markdown
• lxml - XML/HTML processing

For development:

• pytest - Testing framework
• pytest-cov - Coverage reports

AI Use

Ai has been used to create a portion of this code. Specifically to write the tests, and other minor improvements. AI has also been used to improve readability and add helpful code comments.

License

MIT License - see LICENSE file for details

Support

Found a bug or have a suggestion? Open an issue on GitHub: https://github.com/Azizgic/novel-cli/issues

---

Made with ❤️ for novel readers