A python library (written in rust) for large scale batch analysis of robots.txt files
Project description
rbdt
🚨🚨🚨🚨
rbdt is a work in progress, currently being extracted out of another (private) project for the purpose of open sourcing and better software engineering.
🚨🚨🚨🚨
rbdt is a python library written in rust that parses robots.txt files.
rbdt features:
- MIT license, have fun.
- Written in Rust, so it is fast.
- Callable from Python, so it is useful.
- Has been and continues to be run against millions of unique robots.txt files.
- Forgiving, corrects some typical mistakes in files written by hand, like recognizing
dissallows
probably meant to bedisallow
. - Intentionally provides direct access to the parsed robots.txt representation (unlike Reppy or Google's parser).
- Ability to compare which user agent has more privilege given to it by the website owner, both heuristically and logically.
rbdt anti-features:
- rbdt isn't meant to be used as part of a web crawler, but as part of a large scale analysis of robots.txt files. If ends up being useful for web crawlers eventually, that's great and only incidental.
Development
maturin develop
python py_tests/tests.py
Contributions
File a ticket or send a PR if you'd like.
To Do
- Real Open Sourcing Hours
- Changelog
- Write documentation and put them somewhere
- branch protection for main, no direct writes only PR's
- automated tests
- Crawl-delay parsing and restructuring of the data representation.
- Be able to detect whether a crawler can access a specific page.
- More tests of all the various edge cases.
- Benchmarks, (maybe someday never).
- Publish it as a Rust library as well (maybe).
- Get Rust tests working (maybe).
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
No source distribution files available for this release.See tutorial on generating distribution archives.
Built Distributions
Close
Hashes for rbdt-0.0.3_alpha2-cp38-cp38-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | aa7c014b4e5b1695647f940841fe18766e0554483bd0b607284fbc8b82f0de6d |
|
MD5 | 51e3dd407d8a06d25ab1173efbff0c39 |
|
BLAKE2b-256 | 39238258845ecdf4c7f43211d001fac85a2b2b5d7cd987d621a79df363a2c127 |
Close
Hashes for rbdt-0.0.3_alpha2-cp36-cp36m-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dbb6f5de815a5d9fc36578f13c9114c0dfb1cb1c1c5b1c0836ea0d5697db63c7 |
|
MD5 | b8f8630a3863345eef87b241aa99e592 |
|
BLAKE2b-256 | d616987ac9327291d846fb3865afe3a6f66189c21bffa77f34d9d69f67ff6a85 |