tweetscrape 0.1.5

Scrape the Twitter frontend API without any authentication and restriction.

Twitter’s API is annoying to work with, and has lots of limitations — luckily their frontend (JavaScript) has it’s own API, which I reverse–engineered. No API rate limits. No restrictions. Extremely fast.

You can use this library to get the text of any user’s Tweets trivially. Follow the creator’s blog at shirishkadam.com for updates on progress.

This project is inspired from Kenneth Reitz’s similar project kennethreitz/twitter-scraper which is limited to python 3.6 anc above.

Getting Started

$ git clone https://github.com/5hirish/tweet_scrapper.git
$ cd tweet_scrapper
$ pip install -r requirements.txt
$ python -m tweetscrape.twitter_scrape -u "@5hirish" -p 3
$ python -m tweetscrape.twitter_scrape -s "#Python" -p 4
$ python -m tweetscrape.twitter_scrape -s "Avengers Infinity War" -p 2

Usage

from tweetscrape.profile_tweets import TweetScrapperProfile

tweet_scrapper = TweetScrapperProfile("@5hirish", 1)
tweets = tweet_scrapper.get_profile_tweets()
for tweet in tweets:
    print(str(tweet))

Read more on tweetscrape usage here.

Id:973027095411437568   Type:tweet  Time:1520822712000
Author:Reza_Zadeh   AuthorId:92839676
Text:  There's a lot of computational power that goes into mining bitcoin, in particular to find little bits of data with certain SHA256 hashes. Instead, would've been great if that compute power were used to solve challenging NP-hard problems. Human progress becomes side-effect of hype
Links:[]
Hastags:[]
Mentions:[]
Replies:12  Favorites:187   Retweets:51

Id:972778151796510721   Type:tweet  Time:1520763359000
Author:gensim_py    AuthorId:3110758625
Text:   Calling all #Gensim users! Please help improve Gensim by giving us your feedback in this short survey.https://radimrehurek.com/gensim/survey.html …
Links:['https://t.co/sJsDTXv6QF']
Hastags:['#Gensim']
Mentions:[]
Replies:8   Favorites:57    Retweets:46

Id:953706491881680896   Type:tweet  Time:1516216321000
Author:QCon AuthorId:14100646
Text:  “ML for Question and Answer Understanding @Quora” #machinelearning @nikhilbd presentation is now live on @infoqhttp://bit.ly/2Da8WuX
Links:['https://t.co/buseQxF9mS']
Hastags:['#machinelearning']
Mentions:['@Quora', '@nikhilbd', '@InfoQ']
Replies:0   Favorites:15    Retweets:5

Id:970542118350462976   Type:tweet  Time:1520230247000
Author:GabbbarSingh AuthorId:108391251
Text:  Gary Oldman, playing Winston Churchill, wins the best actor for Darkest hour. Nothing wrong with awarding the craft of acting even though you play a murderer, but showing the cold blooded tyrant Churchill, in a positive light, deserves condemnation from Indians. #FuckChurchill
Links:[]
Hastags:['#FuckChurchill']
Mentions:[]
Replies:30  Favorites:581   Retweets:253

Id:970008694783176704   Type:tweet  Time:1520103069000
Author:maxmunnecke  AuthorId:201907594
Text:  New Jupyter notebook on topic modelling with SpaCy, Gensim and Textacy. Combining 'Termite Plot' and 'pyLDAvis' visualizations makes sense when evaluating topic models. Try out the notebook: https://nbviewer.jupyter.org/github/repmax/topic-model/blob/master/topic-modelling.ipynb … #dataviz #nlp #digitalhumanities @gensim_py @stanfordnlp @uwdatapic.twitter.com/ngyGZopw7g
Links:['https://t.co/0o0FEOAl20', 'https://t.co/ngyGZopw7g']
Hastags:['#dataviz', '#nlp', '#digitalhumanities']
Mentions:['@gensim_py', '@stanfordnlp', '@uwdata']
Replies:0   Favorites:170   Retweets:61

....

Requirements

• Python 3.X

Python Package dependencies listed in requirements.txt

Features

• Extract user tweets with all meta-data

• Extracts external links, hashtags and mentions from a tweet

• Extracts reply, favorite and retweet counts of a tweet

TODO

• [x] Extract tweets from a twitter user’s profile

• [x] Extract tweets from twitter search

• [ ] Extract tweets from a twitter thread, given the thread link

• [ ] Extract the quoted tweet along with a tweet

Contributions

Please see the contributing documentation for some tips on getting started.

Maintainers

• @5hirish - Shirish Kadam