Scrape the Twitter frontend API without any authentication and restriction.
Project description
Twitter’s API is annoying to work with, and has lots of limitations — luckily their frontend (JavaScript) has it’s own API, which I reverse–engineered. No API rate limits. No restrictions. Extremely fast.
You can use this library to get the text of any user’s Tweets trivially. Follow the creator’s blog at shirishkadam.com for updates on progress.
This project is inspired from Kenneth Reitz’s similar project kennethreitz/twitter-scraper which is limited to python 3.6 anc above.
Getting Started
$ git clone https://github.com/5hirish/tweet_scrapper.git
$ cd tweet_scrapper
$ pip install -r requirements.txt
$ python -m tweetscrape.twitter_scrape -u "@5hirish" -p 3
$ python -m tweetscrape.twitter_scrape -s "#Python" -p 4
$ python -m tweetscrape.twitter_scrape -s "Avengers Infinity War" -p 2
Usage
from tweetscrape.profile_tweets import TweetScrapperProfile
tweet_scrapper = TweetScrapperProfile("@5hirish", 1)
tweets = tweet_scrapper.get_profile_tweets()
for tweet in tweets:
print(str(tweet))
Read more on tweetscrape usage here.
Id:973027095411437568 Type:tweet Time:1520822712000 Author:Reza_Zadeh AuthorId:92839676 Text: There's a lot of computational power that goes into mining bitcoin, in particular to find little bits of data with certain SHA256 hashes. Instead, would've been great if that compute power were used to solve challenging NP-hard problems. Human progress becomes side-effect of hype Links:[] Hastags:[] Mentions:[] Replies:12 Favorites:187 Retweets:51 Id:972778151796510721 Type:tweet Time:1520763359000 Author:gensim_py AuthorId:3110758625 Text: Calling all #Gensim users! Please help improve Gensim by giving us your feedback in this short survey.https://radimrehurek.com/gensim/survey.html … Links:['https://t.co/sJsDTXv6QF'] Hastags:['#Gensim'] Mentions:[] Replies:8 Favorites:57 Retweets:46 Id:953706491881680896 Type:tweet Time:1516216321000 Author:QCon AuthorId:14100646 Text: “ML for Question and Answer Understanding @Quora” #machinelearning @nikhilbd presentation is now live on @infoqhttp://bit.ly/2Da8WuX Links:['https://t.co/buseQxF9mS'] Hastags:['#machinelearning'] Mentions:['@Quora', '@nikhilbd', '@InfoQ'] Replies:0 Favorites:15 Retweets:5 Id:970542118350462976 Type:tweet Time:1520230247000 Author:GabbbarSingh AuthorId:108391251 Text: Gary Oldman, playing Winston Churchill, wins the best actor for Darkest hour. Nothing wrong with awarding the craft of acting even though you play a murderer, but showing the cold blooded tyrant Churchill, in a positive light, deserves condemnation from Indians. #FuckChurchill Links:[] Hastags:['#FuckChurchill'] Mentions:[] Replies:30 Favorites:581 Retweets:253 Id:970008694783176704 Type:tweet Time:1520103069000 Author:maxmunnecke AuthorId:201907594 Text: New Jupyter notebook on topic modelling with SpaCy, Gensim and Textacy. Combining 'Termite Plot' and 'pyLDAvis' visualizations makes sense when evaluating topic models. Try out the notebook: https://nbviewer.jupyter.org/github/repmax/topic-model/blob/master/topic-modelling.ipynb … #dataviz #nlp #digitalhumanities @gensim_py @stanfordnlp @uwdatapic.twitter.com/ngyGZopw7g Links:['https://t.co/0o0FEOAl20', 'https://t.co/ngyGZopw7g'] Hastags:['#dataviz', '#nlp', '#digitalhumanities'] Mentions:['@gensim_py', '@stanfordnlp', '@uwdata'] Replies:0 Favorites:170 Retweets:61 ....
Requirements
Python Package dependencies listed in requirements.txt
Features
Extract user tweets with all meta-data
Extracts external links, hashtags and mentions from a tweet
Extracts reply, favorite and retweet counts of a tweet
TODO
[x] Extract tweets from a twitter user’s profile
[x] Extract tweets from twitter search
[ ] Extract tweets from a twitter thread, given the thread link
[ ] Extract the quoted tweet along with a tweet
Contributions
Please see the contributing documentation for some tips on getting started.
Maintainers
@5hirish - Shirish Kadam
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for tweetscrape-0.1.5.2-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 332d0317a70705180b6b8a78e988a4a713c3f92d6f0dd102b2f3cfb9105e3b94 |
|
MD5 | df2c83567b3728622867caec61506875 |
|
BLAKE2b-256 | efcbd52a18d72141c5b37f24e74b4c3d180c0e71a9738ad304ada6317fc3a2e2 |