pageweaver 0.1.9

A web crawler to fetch web novel chapters and generate a PDF.

PageWeaver

This project is a CLI tool designed to crawl web novels from FreeWebNovel and generate a PDF document containing the chapters. The tool uses Python libraries such as requests, BeautifulSoup, and pylatex to fetch, process, and compile the novel content into a well-formatted PDF.

Features

• Fetches novel chapters from FreeWebNovel.
• Processes and cleans the text to remove non-UTF8 characters.
• Generates a PDF document with a title page, table of contents, and chapters.
• Supports multi-threaded crawling for faster processing.
• Option to allow non-English characters in the novel title and author name.

Requirements

• Python 3.9+
• requests
• beautifulsoup4
• pylatex
• argparse

Installation

Via pip

pip install pageweaver

Via source

git clone https://github.com/KTS-o7/pageweaver.git
cd pageweaver
pip install -r requirements.txt
python setup.py install

Usage

pageweaver <novel_url> <start_chapter_number> <end_chapter_number> [--output_dir <output_dir>] [--num-workers <num_workers>] [--allow-non-english]

Arguments

• novel_url: The FreeWebNovel URL of the novel to crawl.
• start_chapter: The starting chapter number.
• end_chapter: The ending chapter number.
• --output_dir: (Optional) The destination directory for the generated PDF. Defaults to the current working directory.
• --num-workers: (Optional) The number of workers to use for crawling. Defaults to 10.
• --allow-non-english: (Optional) Allow non-English characters in the novel title and author name.

Example Usage

pageweaver https://freewebnovel.com/global-fog-survival.html 1 15 --num-workers 5
pageweaver https://freewebnovel.com/global-fog-survival.html 1 30 --output_dir /path/to/output --allow-non-english

How It Works

• WebCrawler: Fetches the HTML content of the novel chapters and extracts the text.
• TextProcessor: Cleans the text by removing non-UTF8 characters and escaping LaTeX special characters.
• DocumentGenerator: Uses pylatex to create a PDF document with the novel content.
• NovelCrawlerService: Manages the crawling process, coordinates the fetching and processing of chapters, and generates the final PDF.

Example

To crawl the novel "Global Fog Survival" from chapters 1 to 2 and generate a PDF, run:

pageweaver https://freewebnovel.com/global-fog-survival.html 1 2 --num-workers 10

This will create a PDF document in the current working directory with the title and author extracted from the novel's metadata.

License

This project is licensed under the MIT License.

Contributing

Contributions are welcome! Please open an issue or submit a pull request for any improvements or bug fixes.

Contact

For any questions or support, please open an issue on the GitHub repository.

Disclaimer

This tool is not intended to promote piracy. It should be used for educational or personal reading purposes only. Please respect the copyrights of the original authors and publishers.

Authors

• Krishnatejaswi S
• Sridhar D Kedlaya

Star Graph