DeepCrawl Robots.txt live checker
> cat urls.txt
https://www.ebay.at/
https://www.ebay.at/adchoice
https://www.ebay.at/sl/sell
https://www.ebay.at/mye/myebay/watchlist
https://www.ebay.at/sch/ebayadvsearch
https://www.ebay.at/sch/Kleidung-Accessoires-/11450/i.html
https://www.ebay.at/sch/Auto-Tuning-Styling-/107059/i.html
https://www.ebay.at/sch/Modeschmuck-/10968/i.html
https://www.ebay.at/sch/Damenschuhe-/3034/i.html
> cat robots.txt
User-agent: *
Disallow: /sch/
pip install deepcrawl_robots
from deepcrawl_robots import Processor
urls_path = "Path to urls file"
robots_txt_path = "Path to robots.txt file"
processor = Processor(
user_agent="User agent",
urls_file_path=urls_path,
robots_file_path=robots_txt_path
)
> cat result.txt
https://www.ebay.at/mye/myebay/watchlist,true
https://www.ebay.at/adchoice,true
https://www.ebay.at/,true
https://www.ebay.at/sl/sell,true
https://www.ebay.at/sch/Kleidung-Accessoires-/11450/i.html,false
https://www.ebay.at/sch/ebayadvsearch,true
https://www.ebay.at/sch/Modeschmuck-/10968/i.html,false
https://www.ebay.at/sch/Auto-Tuning-Styling-/107059/i.html,false
https://www.ebay.at/sch/Damenschuhe-/3034/i.html,false
2 maintainers
Release files for deepcrawl-robots 0.0.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| deepcrawl_robots-0.0.5.tar.gz | 1.3 MB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| deepcrawl_robots-0.0.5-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 2.7 MB
Release files / deepcrawl_robots-0.0.5.tar.gz
| Download URL | deepcrawl_robots-0.0.5.tar.gz |
|---|---|
| Size | 1.3 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
82480e1c42d0dca6aa5557dceb443ff32a521b5c07d41604edaaf48cda24f228
|
|
BLAKE2b-256 checksum How to use checksums |
0471fd97748d4cbfdfa749747fd2024e62b1e6ee82cc92c50204ced9986e616b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.2.0 pkginfo/1.6.1 requests/2.25.1 setuptools/51.0.0 requests-toolbelt/0.9.1 tqdm/4.54.1 CPython/3.7.5
|
Release files / deepcrawl_robots-0.0.5-py3-none-any.whl
| Download URL | deepcrawl_robots-0.0.5-py3-none-any.whl |
|---|---|
| Size | 1.3 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
a40dfb7d24230d5fd8121284d41f6b8d8f625613e348a55274e32895eaadbbe0
|
|
BLAKE2b-256 checksum How to use checksums |
a1d2596eac60e310048a99b4ef1fc9f3a23abf0bc969df7f2d011749dadc54ac
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.2.0 pkginfo/1.6.1 requests/2.25.1 setuptools/51.0.0 requests-toolbelt/0.9.1 tqdm/4.54.1 CPython/3.7.5
|
