Skip to main content

A web crawler to fetch web novel chapters and generate a PDF.

Project description

PageWeaver

Python PyPI License ViewCount

This project is a CLI tool designed to crawl web novels from FreeWebNovel and generate a PDF document containing the chapters. The tool uses Python libraries such as requests, BeautifulSoup, and pylatex to fetch, process, and compile the novel content into a well-formatted PDF.

Features

  • Fetches novel chapters from FreeWebNovel.
  • Processes and cleans the text to remove non-UTF8 characters.
  • Generates a PDF document with a title page, table of contents, and chapters.
  • Supports multi-threaded crawling for faster processing.
  • Option to allow non-English characters in the novel title and author name.

Requirements

  • Python 3.9+
  • requests
  • beautifulsoup4
  • pylatex
  • argparse

Installation

Via pip

pip install pageweaver

Via source

git clone https://github.com/KTS-o7/pageweaver.git
cd pageweaver
pip install -r requirements.txt
python setup.py install

Usage

pageweaver <novel_url> <start_chapter_number> <end_chapter_number> [--output_dir <output_dir>] [--num-workers <num_workers>] [--allow-non-english]

Arguments

  • novel_url: The FreeWebNovel URL of the novel to crawl.
  • start_chapter: The starting chapter number.
  • end_chapter: The ending chapter number.
  • --output_dir: (Optional) The destination directory for the generated PDF. Defaults to the current working directory.
  • --num-workers: (Optional) The number of workers to use for crawling. Defaults to 10.
  • --allow-non-english: (Optional) Allow non-English characters in the novel title and author name.

Example Usage

pageweaver https://freewebnovel.com/global-fog-survival.html 1 15 --num-workers 5
pageweaver https://freewebnovel.com/global-fog-survival.html 1 30 --output_dir /path/to/output --allow-non-english

How It Works

  • WebCrawler: Fetches the HTML content of the novel chapters and extracts the text.
  • TextProcessor: Cleans the text by removing non-UTF8 characters and escaping LaTeX special characters.
  • DocumentGenerator: Uses pylatex to create a PDF document with the novel content.
  • NovelCrawlerService: Manages the crawling process, coordinates the fetching and processing of chapters, and generates the final PDF.

Example

To crawl the novel "Global Fog Survival" from chapters 1 to 2 and generate a PDF, run:

pageweaver https://freewebnovel.com/global-fog-survival.html 1 2 --num-workers 10

This will create a PDF document in the current working directory with the title and author extracted from the novel's metadata.

License

This project is licensed under the MIT License.

Contributing

Contributions are welcome! Please open an issue or submit a pull request for any improvements or bug fixes.

Contact

For any questions or support, please open an issue on the GitHub repository.

Disclaimer

This tool is not intended to promote piracy. It should be used for educational or personal reading purposes only. Please respect the copyrights of the original authors and publishers.

Authors

Star Graph

Star History Chart

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pageweaver-1.1.1.tar.gz (10.8 kB view details)

Uploaded Source

Built Distribution

pageweaver-1.1.1-py3-none-any.whl (10.5 kB view details)

Uploaded Python 3

File details

Details for the file pageweaver-1.1.1.tar.gz.

File metadata

  • Download URL: pageweaver-1.1.1.tar.gz
  • Upload date:
  • Size: 10.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.9.20

File hashes

Hashes for pageweaver-1.1.1.tar.gz
Algorithm Hash digest
SHA256 4d4ac707a5e89f2efa21d0324bea1a7a82b606281e0631cc0d92737aeca62065
MD5 810171ea932f4e7285519f2cb7959a0f
BLAKE2b-256 d63cc33667fbef514111de4ba05eaf2ddf7f670924e1a1475dceca73ca2fc25e

See more details on using hashes here.

File details

Details for the file pageweaver-1.1.1-py3-none-any.whl.

File metadata

  • Download URL: pageweaver-1.1.1-py3-none-any.whl
  • Upload date:
  • Size: 10.5 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.9.20

File hashes

Hashes for pageweaver-1.1.1-py3-none-any.whl
Algorithm Hash digest
SHA256 4385d2914c12828ca7c1cd806e36ee1764ee0b392251cde6363dc3fe2293924a
MD5 88d5d59b27ad999ea668ac22b7275f60
BLAKE2b-256 15b265ddc2a9a454cce9e9d15f8895434cd443d44dd3f48ca3f8d11562e0868f

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page