Skip to main content

[Author] PyPi [Python 3.7] license

Python Twitter data collector built on Tweepy that allow users to dynamically collect accounts and statuses from Twitter during streaming, and automatically generate Datasets from collected data that you can as CSV.

This library provides a framework that you can use to build your own data collector, specifying which are your features that have to be extracted from Twitter accounts/statuses.

Creating your Twitter dataset: 1. Instantiate an AccountCollector and/or StatusCollector class in according to what you want collect, accounts, statuses or both. At this step you can re-defined your own features that have to be extracted from twitter data, you have to pass dict-like parameters in the following form: <feature_name, function> where the function has to be applied to the user or status object. Please refer to documentation for more details about Twitter objects 2. Instantiate the OnlineStreamer passing the collector as parameter 3. Start streaming on some topics 4. Save the created dataset at specified location

NB: It is not mandatory to use both collectors and streamer but you can also use Collectors alone, for instance if you already have the users and/or statuses to collect and you don’t need to stream anything.

NEW FEATURES: * Offline collection by name, allow user to make a query by name and collect some name-similar users extracting features defined in the collector constructor

INSTALLATION

The package is available on PyPi here

$ pip install ptdc

EXAMPLE USAGE

Import modules

from ptdc import authenticate, AccountCollector, OnlineStreamer, StatusCollector

Define tokens

Replace the following tokens with yours, see Twitter developers authentication for more details about how obtain them.

consumer_key = "xxxxxxxxxxx"
consumer_key_secret = "xxxxxxxxxxxxx"
access_token = "xxxxxxxxxxxxxxxxxxxxxx"
access_token_secret = "xxxxxxxxxxxxxxxxxx"

Create the default Tweepy API object of tweepy

api = authenticate(consumer_key=consumer_key, consumer_key_secret=consumer_key_secret, access_token=access_token, access_token_secret=access_token_secret)

Create your own Collectors for collecting data

Create your own StatusCollector object

s_collector = StatusCollector(api=api)

Create your own AccountCollector object, which will collect also statuses

collector = AccountCollector(api=api, statuses_collector=s_collector)

Create the Streamer

Create Online Streamer that will collect data (in this case will collect only 5 accounts)

streamer = OnlineStreamer(api=api, collector=collector, data_limit=5, n_statuses=400)

Start streaming

You can start streaming in all ways defined by Tweepy, see its doc for more details

streamer.stream(track=['famous', 'web', 'vip', 'holiday', 'pic', 'photo'], is_async=False)

Save dataset/s

After streaming ended (in according to your defined limits), save DataFrame/s generated into csv file/s. You just need to access the collector object and call the save_dataset method providing the path.

streamer.collector.save_dataset(path="../data/accounts.csv")

Questions and Contributing

Feel free to post questions and problems on the issue tracker. Pull requests are welcome!

Feel free to fork and modify or add new features and functionality to the library

Metadata

Release files for ptdc 1.3.6

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for ptdc 1.3.6
File Size Uploaded
ptdc-1.3.6.tar.gz 12.6 kB Details

Built distribution (wheel)

Table of built distributions (wheels) for ptdc 1.3.6
File Interpreter ABI Platform
ptdc-1.3.6-py3-none-any.whl Python 3 none any Details

Total release size: 25.2 kB

Release files / ptdc-1.3.6.tar.gz

Download URL ptdc-1.3.6.tar.gz
Size 12.6 kB
Tags Source
SHA-256 checksum
How to use checksums
8b260fa671a22cb6da9926c22bf8a534f55ca9c49554b545f448e7d19606ff29
BLAKE2b-256 checksum
How to use checksums
facdfbaca49ce420fbde077676f3622e07a1523a28fd522aaf91af2e9455c88f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.7.3

Release files / ptdc-1.3.6-py3-none-any.whl

Download URL ptdc-1.3.6-py3-none-any.whl
Size 12.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
3989fb71f47f010d3666a45e82f1b61188fa6e7a776ab41f4be41ac2810c58e6
BLAKE2b-256 checksum
How to use checksums
9b1197f701a573e265ea369aca35b9d3645e84ebb778f3b225439c81e76b247a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.7.3

Release history Release notifications | RSS feed

This release

1.3.6 This release

2 release files

1.3.5

2 release files

1.2.9

2 release files

1.2.8

2 release files

1.2.6

2 release files

1.2.5

2 release files

1.2.2

2 release files

1.2.1

2 release files

1.2.0

2 release files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page