# robot_detection
robot_detection is a python module to detect if a given HTTP User Agent is a web crawler. It uses the list of registered robots from http://www.robotstxt.org: (Robots Database)[http://www.robotstxt.org/db.html)
## Usage
There is only one, function, is_robot that takes a string (unicode or not) and returns True iff that string matches a known robot in the robotstxt.org robot database
### Example
>>> import robot_detection >>> robot_detection.is_robot(user_agent_string)
## Updating
You can download a new version of the Robot Database from (this link)[http://www.robotstxt.org/dbexport.html].
Download the database dump, and run the file robot_detection.py with the file as first argument.
$ wget http://www.robotstxt.org/db/all.txt $ python robot_detection.py all.txt
If the database has changed, it’ll print out the new version of robot_useragents variable that you need to put into the source code.
## Tests
Some simple unittests are included. Running the tests.py file will run the tests.
1 maintainer
Release files for robot-detection 0.2.8
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| robot-detection-0.2.8.tar.gz | 77.3 kB | Details |
Release files / robot-detection-0.2.8.tar.gz
| Download URL | robot-detection-0.2.8.tar.gz |
|---|---|
| Size | 77.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
90793053beefdf9d8177abed6632acdb2089ec3bb3938f055f00689f1ac5c0c7
|
|
BLAKE2b-256 checksum How to use checksums |
1e42049b4338c370161568b4f3c76a2be81691a1b0c46b6569b33dcd08958ebb
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
