Pyscalpel
Your easy-to-use, fast and powerful web scraping library.
Why?
I already knew scrapy which is the reference in python for web scraping. But two things bothered me.
- I feel like scrapy cannot integrate into an existing project, you need to treat your web scraping stuff like a project on its own.
- Usage of Twisted who is a veteran in asynchronous programming, but I think that there are better asynchronous frameworks today. Note that this second point is not true anymore as I'm writing the document since scrapy adds support for asyncio
After having made this observation I decided to create pyscalpel. And let's be honest, I also want to have my own web scraping library, and it is fun to write one ;)
Installation
pip install pyscalpel # to only use the asyncio backend
pip install pyscalpel[gevent] # to install the gevent backend
pip install pyscalpel[trio] # to installl the trio backend
pip install pyscalpel[full] # to install all the backends
If you know about poetry you can use it instead of pip.
poetry add pyscalpel # to only use the asyncio backend
poetry add pyscalpel[gevent] # to install the gevent backend
poetry add pyscalpel[trio] # to install the trio backend
poetry add pyscalpel[full] # to install all the backends
pyscalpel works starting from python 3.7, it relies on robust packages:
- configuror: A configuration toolkit.
- httpx: A modern http client.
- selenium: A library for controlling a browser.
- gevent: An asynchronous framework using the synchronous way. (optional)
- trio: A modern asynchronous framework using
async/awaitsyntax. (optional) - anyio: An asynchronous networking and concurrency library that works on top of either asyncio or trio.
- parsel: A library elements in HTML/XML documents.
- attrs: A library helping to write classes without pain.
- fake-useragent: A simple library to fake a user agent.
- rfc3986: A library for url parsing and validation.
- msgpack: A library allowing for fast serialization/deserialization of data structures.
Documentation
The documentation is available at https://scalpel.readthedocs.io/en/latest/.
Usage
To give you an overview of what can be done, this is a simple example of quote scraping. Don't hesitate to look at the examples folder for more snippets to look at.
with gevent
from pathlib import Path
from scalpel import Configuration
from scalpel.green import StaticSpider, StaticResponse, read_mp
def parse(spider: StaticSpider, response: StaticResponse) -> None:
for quote in response.xpath('//div[@class="quote"]'):
data = {
'message': quote.xpath('./span[@class="text"]/text()').get(),
'author': quote.xpath('./span/small/text()').get(),
'tags': quote.xpath('./div/a/text()').getall()
}
spider.save_item(data)
next_link = response.xpath('//nav/ul/li[@class="next"]/a').xpath('@href').get()
if next_link is not None:
response.follow(next_link)
if __name__ == '__main__':
backup = Path(__file__).parent / 'backup.mp'
config = Configuration(backup_filename=f'{backup}')
spider = StaticSpider(urls=['http://quotes.toscrape.com'], parse=parse, config=config)
spider.run()
print(spider.statistics())
# you can do whatever you want with the results
for quote_data in read_mp(filename=backup, decoder=spider.config.msgpack_decoder):
print(quote_data)
with anyio
from pathlib import Path
import anyio
from scalpel import Configuration
from scalpel.any_io import StaticResponse, StaticSpider, read_mp
async def parse(spider: StaticSpider, response: StaticResponse) -> None:
for quote in response.xpath('//div[@class="quote"]'):
data = {
'message': quote.xpath('./span[@class="text"]/text()').get(),
'author': quote.xpath('./span/small/text()').get(),
'tags': quote.xpath('./div/a/text()').getall()
}
await spider.save_item(data)
next_link = response.xpath('//nav/ul/li[@class="next"]/a').xpath('@href').get()
if next_link is not None:
await response.follow(next_link)
async def main():
backup = Path(__file__).parent / 'backup.mp'
config = Configuration(backup_filename=f'{backup}')
spider = StaticSpider(urls=['http://quotes.toscrape.com'], parse=parse, config=config)
await spider.run()
print(spider.statistics())
# you can do whatever you want with the results
async for item in read_mp(backup, decoder=spider.config.msgpack_decoder):
print(item)
if __name__ == '__main__':
# by default, this will run the asyncio backend, if you want the trio backend, you must first install the trio
# package and replace the following line with: anyio.run(main, backend='trio').
anyio.run(main)
Known limitations
pyscalpel aims to handle SPA (single page application) through the use of selenium. However, due to the synchronous nature of selenium, it is hard to leverage anyio and gevent asynchronous feature. You will notice that the selenium spider is slower than the static spider. For more information look at the documentation.
Warning
pyscalpel is a young project, so it is expected to have breaking changes in the api without respecting the semver principle. It is recommended to pin the version you are using for now.
Metadata
Release files for pyscalpel 0.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| pyscalpel-0.2.0.tar.gz | 32.6 kB | Without provenance | Details for pyscalpel-0.2.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| pyscalpel-0.2.0-py3-none-any.whl | Python 3 | none | any | Details for pyscalpel-0.2.0-py3-none-any.whl |
Total release size: 73.2 kB
Release files / pyscalpel-0.2.0.tar.gz
| Download URL | pyscalpel-0.2.0.tar.gz |
|---|---|
| Size | 32.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
2c8a555f8ed1d4bc0553c9a74cf1643761c4ac145ad2c9762fd8fbd2a4bf501d
|
|
BLAKE2b-256 checksum How to use checksums |
8c9e8a3d54296332cb034cdfb0e555e3b23319ada0f491299390eeff997ae36b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.1.13 CPython/3.7.13 Linux/5.13.0-1025-azure
|
Release files / pyscalpel-0.2.0-py3-none-any.whl
| Download URL | pyscalpel-0.2.0-py3-none-any.whl |
|---|---|
| Size | 40.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
4d7f858748b03f3267041b17b7d129b933648f6e71b1f6fcd1201571b1661fce
|
|
BLAKE2b-256 checksum How to use checksums |
88c60d3e09dcc2d3fffe9cd3bd3500392992a4074edb661847ff4787c7aa05ad
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.1.13 CPython/3.7.13 Linux/5.13.0-1025-azure
|
Security for pyscalpel 0.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.