AioPTTCrawler 0.0.5

PTT crawler using asyncio

AioPTTCrawler (PTT 網路版爬蟲)

This is Python Package use to crawl PTT's article data by using asyncio.

Documentation

PyPi Page

pip install AioPTTCrawler

from AioPTTCrawler import AioPTTCrawler
ptt_crawler = AioPTTCrawler()

Usage

get data from PTT

ptt_crawler = AioPTTCrawler()

BOARD = "Gossiping"
ptt_data = ptt_crawler.get_board_articles(board=BOARD, page_count=10)

ptt_data is a PTTData object. To extract data you need to use get_article_dict(), get_article_dataframe(), etc

---

get dict from PTTData

article_dict = ptt_data.get_article_dict()
comment_dict = ptt_data.get_comment_dict()

article's dict format

[
    {
        "article" : "Article's ID. ex:M.1663144920.A.A6E",
        "article_title" : "Article's title. ex:[公告] 批踢踢27週年活動宣導公告更新",
        "user_id" : "Author's ID. ex: ubcs",
        "user_name" : "Author's name. ex:(覺★青年超冒險蓋)",
        "board" : "BBS Board ex: Gossiping",
        "datetime" : "Post time. ex: Wed Sep 14 16:41:58 2022.",
        "context" : "Context of article. ex: PTT 27 周年活動開始囉，本篇為置底宣導，詳情參閱下面資料...",
        "ip_address" : "IP address. ex: 59.120.192.119",
        "comment_list" : [
            {"comment_dict"},
            {"comment_dict"},
        ]
    }, {"..."}
]

comment's dict format

[
    {
        "article_id" : "Article's ID. ex:M.1663144920.A.A6E",
        "tag" : "comment's reaction. ex: 推 噓 →",
        "user_id" : "User's ID. ex: bill403777",
        "comment_order" : "order of comment. ex: 1",
        "context" : "Context of comment. ex: 錢",
        "datetime" : "Post time. ex: 09/14 16:42",
        "ip_address" : "27.53.96.42",
    }, {"..."}
]

use this article for example

Comparsion

Used time difference between normal method and async method

(unit: second)

Support

You may report bugs, ask for help and discuss various other issues on the issuse