Tools to pull text data from github, slack, website, stackoverflow, etc.
Project description
spider-nest
Tools to pull text data from github, slack, website, stackoverflow, etc.
Components
gitpump
Fetch github data, including repos, issues, discussions, trendings. Requires github account tokens.
slackpump
Fetch slack messages of a workspace. Requires slack bot token.
stackpump
Fetch questions/answers from stackoverflow by web parser
web2markdown
Get web pages and parse text content to markdown content.
From source code
git clone git@github.com:yhmo/spider-nest.git
cd spider-nest
pip3 install -r ./requirements.txt
Installation
pip3 install spider-nest
Requirement
python >= 3.7
How to run examples
To run the examples of gitpump, follow the steps:
- create a github personal token by following this guide
- create a text file
token_github.txt
under the examples folder - put some github tokens into the
token_github.txt
, no delimiter, one line for each token - now you can run examples
example_github_xxx.py
To run the example of example_slack_xxx.py
- create a slack bot, bind the bot to a workspace, get the token
- create a text file
token_slack.txt
under the examples folder - put the bot token into the
token_slack.txt
- now you can run examples
example_slack_xxx.py
You can run other examples without specific setting
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
spider-nest-0.2.0.tar.gz
(67.3 kB
view hashes)