Skip to main content

Scrapfly SDK for Scrapfly

Project description

Scrapfly SDK

Installation

pip install scrapfly-sdk

You can also install extra dependencies

  • pip install "scrapfly-sdk[seepdup]" for performance improvement
  • pip install "scrapfly-sdk[concurrency]" for concurrency out of the box (asyncio / thread)
  • pip install "scrapfly-sdk[scrapy]" for scrapy integration
  • pip install "scrapfly-sdk[scrapy]" Everything!

Get Your API Key

You can create a free account on Scrapfly to get your API Key.

Migration

Migrate from 0.7.x to 0.8

asyncio-pool dependency has been dropped

scrapfly.concurrent_scrape is now an async generator. If the concurrency is None or not defined, the max concurrency allowed by your current subscription is used.

    async for result in scrapfly.concurrent_scrape(concurrency=10, scrape_configs=[ScrapConfig(...), ...]):
        print(result)

brotli args is deprecated and will be removed in the next minor. There is not benefit in most of case versus gzip regarding and size and use more CPU.

What's new

0.8.x

  • Better error log
  • Async/Improvement for concurrent scrape with asyncio
  • Scrapy media pipeline are now supported out of the box

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

scrapfly-sdk-0.8.0.tar.gz (23.9 kB view hashes)

Uploaded Source

Built Distribution

scrapfly_sdk-0.8.0-py3-none-any.whl (28.2 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page