QChanCrawler

QChanCrawler retrieves posts from QQ channels and saves them to a SQLite database. It can crawl channels supplied explicitly or discover channels from the Explore recommendations.

Requirements

• Python 3.12 or later.

Installation

pip install qchancrawler

Usage

Crawl specified channels

qchancrawler -g ID1 ID2 ID3

The channel identifiers may be numeric guild IDs or QQ channel numbers. More than one identifier can be supplied; the channels are crawled concurrently.

Crawl recommended channels

qchancrawler -e

-e discovers channels from the explore section and crawls them. If no number is supplied, it requests 200 channels. You can also supply with inf or infinite to crawl infinitely from exploration.

Examples

qchancrawler -g 123456
qchancrawler -g 123456 Mine101 -o posts.db
qchancrawler -e -c cookies.txt -v
qchancrawler -e inf

Options:

• -g, --guild: QQ channel/guild IDs to crawl.
• -e, --explore: Discover and crawl recommended channels. The optional count defaults to 200.
• -c, --cookie: Path to an exported pd.qq.com cookie file. Optional for posts, but required to crawl comments and replies.
• -o, --outfile: SQLite database path. Defaults to ./qchan.db.
• -p, --page: Starting page number. Crawling continues until the API reports that there are no more pages. Defaults to 1.
• -v, --verbose: Enable verbose progress output.
• --version: Show the installed QChanCrawler version.

Database

The posts table contains these fields:

• id: Auto-incrementing SQLite row ID.
• guild_id: QQ guild/channel identifier.
• guild_name: QQ guild/channel name.
• item_id: Unique ID of the post, comment, or reply. Rows are deduplicated by item_id.
• item_type: One of post, comment, or reply.
• published_at: Item publication time as a Unix timestamp.
• author_name: Author nickname.
• author_id: Author identifier.
• author_location: Author province.
• icon_url: Author avatar URL.
• text: Item content.
• post_url: URL of the source post. Comments and replies use the URL of their parent post.

Item relationships

Post
|
|-- Comment
|   |
|   |-- Reply

Each item has its own item_id. Comments and replies are associated with the parent post through post_url.

Cookie file

The cookie file must use the Mozilla/Netscape cookies.txt format. Browser extensions and other cookie-export tools can create files in this format.

Cookies are optional when crawling posts, but are required for crawling comments and replies. It's also recommended for crawling exploration, because the exploration section will be duplicated in log-out status.

The cookie file must contain a pair of valid p_uin and p_skey.

Disclaimer

Use QChanCrawler only for channels and data you are authorized to access. Protect personal data and exported cookies, avoid excessive requests, respect service rate limits, and comply with QQ's terms and policies.