Scrapfly SDK
Installation
pip install scrapfly-sdk
You can also install extra dependencies
pip install "scrapfly-sdk[seepdup]"for performance improvementpip install "scrapfly-sdk[concurrency]"for concurrency out of the box (asyncio / thread)pip install "scrapfly-sdk[scrapy]"for scrapy integrationpip install "scrapfly-sdk[all]"Everything!
For use of built-in HTML parser (via ScrapeApiResponse.selector property) additional requirement of either parsel or scrapy is required.
For reference of usage or examples, please checkout the folder /examples in this repository.
Get Your API Key
You can create a free account on Scrapfly to get your API Key.
Migration
Migrate from 0.7.x to 0.8
asyncio-pool dependency has been dropped
scrapfly.concurrent_scrape is now an async generator. If the concurrency is None or not defined, the max concurrency allowed by
your current subscription is used.
async for result in scrapfly.concurrent_scrape(concurrency=10, scrape_configs=[ScrapConfig(...), ...]):
print(result)
brotli args is deprecated and will be removed in the next minor. There is not benefit in most of case versus gzip regarding and size and use more CPU.
What's new
0.8.x
- Better error log
- Async/Improvement for concurrent scrape with asyncio
- Scrapy media pipeline are now supported out of the box
1 maintainer
Release files for scrapfly-sdk 0.8.17
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| scrapfly-sdk-0.8.17.tar.gz | 27.8 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| scrapfly_sdk-0.8.17-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 60.3 kB
Release files / scrapfly-sdk-0.8.17.tar.gz
| Download URL | scrapfly-sdk-0.8.17.tar.gz |
|---|---|
| Size | 27.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
442437a301a68b76755ff1bdc3cb7156b481ce54b9575dcfa2c090322ac7905f
|
|
BLAKE2b-256 checksum How to use checksums |
f4c29563531094bd0a66a00192115bfd625304761e2e7245c1637a3351001157
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.9.6
|
Release files / scrapfly_sdk-0.8.17-py3-none-any.whl
| Download URL | scrapfly_sdk-0.8.17-py3-none-any.whl |
|---|---|
| Size | 32.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
97d07f62806c240f24f347d09ea83f287a06ebad1e55178d524b3123bfc42224
|
|
BLAKE2b-256 checksum How to use checksums |
0465ad3858cb59b05400c144e75469ff23007ef1e249525e2e86766413cbecc4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.9.6
|
