Weibo Scraper

Simple weibo tweet scraper . Crawl weibo tweets without authorization. There are many limitations in official API . In general , we can inspect mobile site which has it's own API by Chrome.

Why

1. Crawl weibo data in order to research big data .

2. Back up data for weibo's shameful blockade .

Installation

pip

$ pip install weibo-scraper

Or Upgrade it.

$ pip install --upgrade weibo-scraper

pipenv

$ pipenv install weibo-scraper

Or Upgrade it.

$ pipenv update --outdated # show packages which are outdated

$ pipenv update weibo-scraper # just update weibo-scraper

Only Python 3.6+ is supported

Usage

1. Firstly , you can get weibo profile by name or uid .

>>> from weibo_scraper import get_weibo_profile
>>> weibo_profile = get_weibo_profile(name='来去之间',)
>>> ....

You will get weibo profile response which is type of weibo_base.UserMeta, and this response include fields as below

2. Secondly , via tweet_container_id to get weibo tweets is a rare way to use but it also works well .

>>> from weibo_scraper import  get_weibo_tweets
>>> for tweet in get_weibo_tweets(tweet_container_id='1076033637346297',pages=1):
>>>     print(tweet)
>>> ....

3. Of Course , you can also get raw weibo tweets by nick name which is exist . And the param of pages is optional .

>>> from weibo_scraper import  get_weibo_tweets_by_name
>>> for tweet in get_weibo_tweets_by_name(name='嘻红豆', pages=1):
>>>     print(tweet)
>>> ....

3. If you want to get all tweets , you can set the param of pages as None

>>> from weibo_scraper import  get_weibo_tweets_by_name
>>> for tweet in get_weibo_tweets_by_name(name='嘻红豆', pages=None):
>>>     print(tweet)
>>> ....

4. There is a giant update since 1.0.5 🍰!

You can also get formatted tweets via api of weibo_scrapy.get_formatted_weibo_tweets_by_name,

>>> from weibo_scraper import  get_formatted_weibo_tweets_by_name
>>> result_iterator = get_formatted_weibo_tweets_by_name(name='嘻红豆', pages=None)
>>> for user_meta in result_iterator:
>>>     for tweetMeta in user_meta.cards_node:
>>>         print(tweetMeta.mblog.text)
>>> ....

Weibo Flasgger

Weibo Flasgger is a web api document for weibo scraper , and powered by flasgger .

P.S

1. Inspiration from Twitter-Scraper .

2. For 'XIHONGDOU' .

3. Welcome To Fork Me .

LICENSE

MIT