Apify SDK for Python
Verified details
These details have been verified by PyPIProject links
GitHub Statistics
Maintainers
Project description
Apify SDK for Python
The Apify SDK for Python is the official library to create Apify Actors in Python. It provides useful features like Actor lifecycle management, local storage emulation, and Actor event handling.
If you just need to access the Apify API from your Python applications, check out the Apify Client for Python instead.
Installation
The Apify SDK for Python is available on PyPI as the apify package.
For default installation, using Pip, run the following:
pip install apify
For users interested in integrating Apify with Scrapy, we provide a package extra called scrapy.
To install Apify with the scrapy extra, use the following command:
pip install apify[scrapy]
Documentation
For usage instructions, check the documentation on Apify Docs.
Examples
Below are few examples demonstrating how to use the Apify SDK with some web scraping-related libraries.
Apify SDK with HTTPX and BeautifulSoup
This example illustrates how to integrate the Apify SDK with HTTPX and BeautifulSoup to scrape data from web pages.
from bs4 import BeautifulSoup
from httpx import AsyncClient
from apify import Actor
async def main() -> None:
async with Actor:
# Retrieve the Actor input, and use default values if not provided.
actor_input = await Actor.get_input() or {}
start_urls = actor_input.get('start_urls', [{'url': 'https://apify.com'}])
# Open the default request queue for handling URLs to be processed.
request_queue = await Actor.open_request_queue()
# Enqueue the start URLs.
for start_url in start_urls:
url = start_url.get('url')
await request_queue.add_request(url)
# Process the URLs from the request queue.
while request := await request_queue.fetch_next_request():
Actor.log.info(f'Scraping {request.url} ...')
# Fetch the HTTP response from the specified URL using HTTPX.
async with AsyncClient() as client:
response = await client.get(request.url)
# Parse the HTML content using Beautiful Soup.
soup = BeautifulSoup(response.content, 'html.parser')
# Extract the desired data.
data = {
'url': actor_input['url'],
'title': soup.title.string,
'h1s': [h1.text for h1 in soup.find_all('h1')],
'h2s': [h2.text for h2 in soup.find_all('h2')],
'h3s': [h3.text for h3 in soup.find_all('h3')],
}
# Store the extracted data to the default dataset.
await Actor.push_data(data)
Apify SDK with PlaywrightCrawler from Crawlee
This example demonstrates how to use the Apify SDK alongside PlaywrightCrawler from Crawlee to perform web scraping.
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext
from apify import Actor
async def main() -> None:
async with Actor:
# Retrieve the Actor input, and use default values if not provided.
actor_input = await Actor.get_input() or {}
start_urls = [url.get('url') for url in actor_input.get('start_urls', [{'url': 'https://apify.com'}])]
# Exit if no start URLs are provided.
if not start_urls:
Actor.log.info('No start URLs specified in Actor input, exiting...')
await Actor.exit()
# Create a crawler.
crawler = PlaywrightCrawler(
# Limit the crawl to max requests. Remove or increase it for crawling all links.
max_requests_per_crawl=50,
headless=True,
)
# Define a request handler, which will be called for every request.
@crawler.router.default_handler
async def request_handler(context: PlaywrightCrawlingContext) -> None:
url = context.request.url
Actor.log.info(f'Scraping {url}...')
# Extract the desired data.
data = {
'url': context.request.url,
'title': await context.page.title(),
'h1s': [await h1.text_content() for h1 in await context.page.locator('h1').all()],
'h2s': [await h2.text_content() for h2 in await context.page.locator('h2').all()],
'h3s': [await h3.text_content() for h3 in await context.page.locator('h3').all()],
}
# Store the extracted data to the default dataset.
await context.push_data(data)
# Enqueue additional links found on the current page.
await context.enqueue_links()
# Run the crawler with the starting URLs.
await crawler.run(start_urls)
What are Actors?
Actors are serverless cloud programs that can do almost anything a human can do in a web browser. They can do anything from small tasks such as filling in forms or unsubscribing from online services, all the way up to scraping and processing vast numbers of web pages.
They can be run either locally, or on the Apify platform, where you can run them at scale, monitor them, schedule them, or publish and monetize them.
If you're new to Apify, learn what is Apify in the Apify platform documentation.
Creating Actors
To create and run Actors through Apify Console, see the Console documentation.
To create and run Python Actors locally, check the documentation for how to create and run Python Actors locally.
Guides
To see how you can use the Apify SDK with other popular libraries used for web scraping, check out our guides for using Requests and HTTPX, Beautiful Soup, Playwright, Selenium, or Scrapy.
Usage concepts
To learn more about the features of the Apify SDK and how to use them, check out the Usage Concepts section in the sidebar, particularly the guides for the Actor lifecycle, working with storages, handling Actor events or how to use proxies.
Project details
Verified details
These details have been verified by PyPIProject links
GitHub Statistics
Maintainers
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file apify-2.6.0.tar.gz.
File metadata
- Download URL: apify-2.6.0.tar.gz
- Upload date:
- Size: 635.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.12.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
f51e5fb639662f3cff89100055f97657e5ec64656cb5c0a6ed8d50c0200122cd
|
|
| MD5 |
eafd0f7558b39d21d34a1f4e51cc20c4
|
|
| BLAKE2b-256 |
c651bf25c0051b86b3bf14856c00f5cd9ee4c581a6421b1837172ac70ad8abfe
|
Provenance
The following attestation bundles were made for apify-2.6.0.tar.gz:
Publisher:
release.yaml on apify/apify-sdk-python
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
apify-2.6.0.tar.gz -
Subject digest:
f51e5fb639662f3cff89100055f97657e5ec64656cb5c0a6ed8d50c0200122cd - Sigstore transparency entry: 232960397
- Sigstore integration time:
-
Permalink:
apify/apify-sdk-python@d852b4b895e7a771596cb4302bb3fd2ae6d00a6c -
Branch / Tag:
refs/heads/master - Owner: https://github.com/apify
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yaml@d852b4b895e7a771596cb4302bb3fd2ae6d00a6c -
Trigger Event:
workflow_dispatch
-
Statement type:
File details
Details for the file apify-2.6.0-py3-none-any.whl.
File metadata
- Download URL: apify-2.6.0-py3-none-any.whl
- Upload date:
- Size: 67.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.12.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
84cf7e8766b01020d9b45e076386a6faebd118d24c59ee517692e1d213881784
|
|
| MD5 |
5fe174c53d3da18745850b75af1c87f9
|
|
| BLAKE2b-256 |
65096f2cc05387734fb87276e0031f87f89f1b4098fd400845c1b22bd48c70a1
|
Provenance
The following attestation bundles were made for apify-2.6.0-py3-none-any.whl:
Publisher:
release.yaml on apify/apify-sdk-python
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
apify-2.6.0-py3-none-any.whl -
Subject digest:
84cf7e8766b01020d9b45e076386a6faebd118d24c59ee517692e1d213881784 - Sigstore transparency entry: 232960408
- Sigstore integration time:
-
Permalink:
apify/apify-sdk-python@d852b4b895e7a771596cb4302bb3fd2ae6d00a6c -
Branch / Tag:
refs/heads/master - Owner: https://github.com/apify
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yaml@d852b4b895e7a771596cb4302bb3fd2ae6d00a6c -
Trigger Event:
workflow_dispatch
-
Statement type:
