scrapy-mosquitera - tools for filtered scraping
How can I scrape items off a site from the last five days?
—Scrapy User
That question started the development of scrapy-mosquitera, a tool to help you restrict crawling and scraping scope using matchers.
Matchers are simple Python functions that return the validity of an element under certain restrictions.
The first goal in the project was date matching, but you can create your own matcher for your own crawling and scraping needs.
How it works
In the case where the dates are available in the URLs, you will just use the matcher function directly in your code:
from scrapy_mosquitera.matchers import date_matches
date = scrape_date_from_url(url)
if date_matches(data=date, after='5 days ago'):
yield Request(url=url, callback=self.parse_item)
To handle the case when the date is only available at the time when you scrape the items, scrapy-mosquitera provides a PaginationMixin to control the crawl according to the dates scraped.
Head on to the remaining of the documentation for more details.
Installation
The quick way:
pip install scrapy-mosquitera
Metadata
Release files for scrapy-mosquitera 0.1.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| scrapy-mosquitera-0.1.1.tar.gz | 18.4 kB | Without provenance | Details for scrapy-mosquitera-0.1.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| scrapy_mosquitera-0.1.1-py2.py3-none-any.whl | Python 2, Python 3 | none | any | Details for scrapy_mosquitera-0.1.1-py2.py3-none-any.whl |
Total release size: 27.1 kB
Release files / scrapy-mosquitera-0.1.1.tar.gz
| Download URL | scrapy-mosquitera-0.1.1.tar.gz |
|---|---|
| Size | 18.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
2ba3752240999a9111851b0cd0e4d31e3f073cbd241bd7afcc64db420d0b62b7
|
|
BLAKE2b-256 checksum How to use checksums |
0b6d4edc4532bc7181299cbee894b460d44b0b26d57ce09fce637077683735ad
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
Release files / scrapy_mosquitera-0.1.1-py2.py3-none-any.whl
| Download URL | scrapy_mosquitera-0.1.1-py2.py3-none-any.whl |
|---|---|
| Size | 8.7 kB |
| Tags | Python 2 Python 3 |
|
SHA-256 checksum How to use checksums |
92472f527dfb33efcc6733641de622c0537b71ee89a14111fb651c8f6c4d2a70
|
|
BLAKE2b-256 checksum How to use checksums |
b079d188e5de92c8699480fa464867982c50d0728408e54be275c14524a1aec3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
Security for scrapy-mosquitera 0.1.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.