Skip to main content

Twitter stream & search API grabber

Project description


Twitter stream + search API grabber handling various config options such as collecting only during specific time periods, or limiting the collection to some locations. Automatically goes back to "fill in the gaps" when there are cuts in the tweet collection.

Python >= 3.7 compatible.


  • Install gazouilloire

    pip install gazouilloire
  • Install Elasticsearch, version 7.X (you can also use Docker for this)

  • Init gazouilloire collection in a specific directory...

    gazouilloire init path/to/collection/directory
  • ...or in the current directory

    gazouilloire init

a config.json file is created. Open it to configure the collection parameters.

  • Set your Twitter API key and generate the related Access Token

    "twitter": {
       "key": "<Consumer Key (API Key)>xxxxxxxxxxxxxxxxxxxxx",
       "secret": "<Consumer Secret (API Secret)>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
       "oauth_token": "<Access Token>xxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
       "oauth_secret": "<Access Token Secret>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
  • Write down the list of desired keywords and @users and/or the list of desired url_pieces as json arrays:

    "keywords": [
        "\"mots successifs\"",
    "url_pieces": [

    Some advanced filters can be used in combination with the keywords, such as -undesiredkeyword, filter:links, -filter:media, -filter:retweets, etc. See Twitter API's documentation for more details.

    Avoid using accented characters (Twitter will automatically return both tweets with and without accents, for instance searching "heros" will find both tweets with "heros" and "héros").

    Note that there are three possibilities to filter further:

    • language: in order to collect only tweets written in a specific language : just add "language": "fr" to the config (the language should be written in ISO 639-1 code)

    • geolocation: just add "geolocation": "Paris, France" field to the config with the desired geographical boundaries or give in coordinates of the desired box (for instance [48.70908786918211, 2.1533203125, 49.00274483644453, 2.610626220703125])

    • time_limited_keywords: in order to filter on specific keywords during planned time periods, for instance:

    "time_limited_keywords": {
          "#fosdem": [
              ["2021-01-27 04:30", "2021-01-28 23:30"]
  • Setup extra options:

    • resolve_redirected_links: set to true or false to enable or disable automatic resolution of all links found in tweets ( links are always handled, but this allows resolving also all other shorteners like

    • grab_conversations: set to true to activate automatic iterative collection of all tweets to which collected tweets are answering (warning: one should account for the presence of these when processing data, it often results in collecting tweets way out of the collection time period).

    • catchup_past_week: Twitter's free API allows to collect tweet up to 7 days in the past only which gazouilloire does by default, set this option to false to disable this and only collect tweets posted after the collection was started.

    • download_media: set "download_media": {"photo": true, "video": false, "animated_gif": false} to activate automatic downloading of photos posted by users, without videos or gifs (this does not include images from social cards). All fields can also be set to true to download everything. Setup the media_directory field in complement to setup the absolute path where Gazouilloire should store the images and videos on the machine.

    • timezone: adjust the timezone within which tweets timestamps should be computed. Allowed values are proposed on Gazouilloire's startup when setting up an invalid one.

  • Run with:

    gazouilloire run path/to/collection/directory

    or, to run the script in the current directory:

    gazouilloire run
  • The tool can also run as daemon with:

    gazouilloire start
  • Stop the daemon with :

    gazouilloire stop
  • Access the current collection status (running/not running, nomber of collected docs, disk usage, etc.) with

    gazouilloire status
  • Gazouilloire stores its current search state in the collection directory. This means that if you restart Gazouilloire, it will not search again for tweets that were already found. If you want a fresh start (e.g. if you modify the query terms in config.json), you can reset the search state with:

    gazouilloire reset -i none

    The --es-index/-i option allows you to also remove the links or tweets Elasticsearch indices. To remove only links and search state:

    gazouilloire reset -i links

    To remove only tweets and search state:

    gazouilloire reset -i tweets

    To remove links, tweets and search state:

    gazouilloire reset
  • Data is stored in your ElasticSearch, which you can direcly query. But you can also export it easily in csv format:

    # Export all fields from all tweets:
    gazouilloire export
    # or
    gazou export
  • By default, the export command writes in stdout. You can also use the -o option to write into a file:

    gazou export > my_tweets_file.csv
    # is equivalent to
    gazou export -o my_tweets_file.csv
  • Other available options:

    # Export a csv of all tweets having a specific word in their text:
    gazou export medialab
    # Export a csv of all tweets between 2 dates (the last date is excluded):
    gazou export --since "2021-03-24T12:00" --until "2021-03-24T13:00"
    # or
    gazou export --since "2021-03-24" --until "2021-03-25"
    # Export a csv of all tweets having one of many specific words in their text:
    gazou export medialab digitalhumanities datajournalism '#python'
    # Export only a selection of columns:
    gazouilloire export --columns/-c id,user_screen_name,local_time,links
    # or
    gazou export --select/-s id,user_screen_name,local_time,links
    # Other example: export only the text of the tweets:
    gazou export -s text
    # Exclude tweets from conversations or from quotes (i.e. that do not match the keywords defined in config.json)
    gazou export --exclude-threads
    # Exclude retweets from the export
    gazou export --exclude-retweets
    # Export all tweets matching a specific Elasticsearch term query, for instance by user name:
    gazou export "{'user_screen_name': 'medialab_ScPo'}"
    # Take a csv file with an "id" column and return all tweets matching these ids:
    gazou export --export-tweets-from-file yourfile.csv


  • Elasticsearch

    • Remember to set the heap size (at 1GB by default) when moving to production. 1GB is fine for indices under 15-20 million tweets, but be sure to set a higher value for heavier corpora.

      Set these values here /etc/elasticsearch/jvm.options (if you use Elasticsearch as a service) or here your_installation_folder/config/jvm.options (if you have a custom installation folder):


      Here the heap size is set at 2GB (set the values at -Xms5g -Xmx5g if you need 5GB, etc).

    • If you encounter this Elasticsearch error message: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]:

      :arrow_right: Increase the max_map_count value:

      sudo sysctl -w vm.max_map_count=262144


    • If you get a ClusterBlockException [SERVICE_UNAVAILABLE/1/state not recovered / initialized] when starting Elasticsearch:

      :arrow_right: Check the value of gateway.recover_after_nodes in /etc/elasticsearch/elasticsearch.yml:

      sudo [YOUR TEXT EDITOR] /etc/elasticsearch/elasticsearch.yml

      Edit the value of gateway.recover_after_nodes to match your number of nodes (usually 1 - easily checked here : http://host:port/_nodes).

Publications using Gazouilloire

Publications talking about Gazouilloire

Credits & License

Benjamin Ooghe-Tabanou, Jules Farjas, Béatrice Mazoyer & al @ Sciences Po médialab

Read more about Gazouilloire's migration from Python2 & Mongo to Python3 & ElasticSearch in Jules' report.

Discover more of our projects at médialab tools.

This work is supported by DIME-Web, part of DIME-SHS research equipment financed by the EQUIPEX program (ANR-10-EQPX-19-01).

Gazouilloire is a free open source software released under GPL 3.0 license.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

gazouilloire-1.0.0a11.tar.gz (53.7 kB view hashes)

Uploaded Source

Built Distribution

gazouilloire-1.0.0a11-py3-none-any.whl (67.6 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page