Skip to main content

Lightweight async data gathering for Python

Project description

Python Versions

DataService

Lightweight - async - data gathering for Python.

DataService is a lightweight web scraping and general purpose data gathering library for Python.

Designed for simplicity, it’s built upon common web scraping and data gathering patterns.

No complex API to learn, just standard Python idioms.

Dual synchronous and asynchronous support.

Installation

Please note that DataService requires Python 3.11 or higher.

You can install DataService via pip:

pip install python-dataservice

You can also install the optional playwright dependency to use the PlaywrightClient:

pip install python-dataservice[playwright]

To install Playwright, run:

python -m playwright install

or simply:

playwright install

How to use DataService

To start, create a DataService instance with an Iterable of Request objects. This setup provides you with an Iterator of data objects that you can then iterate over or convert to a list, tuple, a pd.DataFrame or any data structure of choice.

start_requests = [Request(url="https://books.toscrape.com/index.html", callback=parse_books_page, client=HttpXClient())]
data_service = DataService(start_requests)
data = tuple(data_service)

A Request is a Pydantic model that includes the URL to fetch, a reference to the client callable, and a callback function for parsing the Response object.

The client can be any async Python callable that accepts a Request object and returns a Response object. DataService provides an HttpXClient class by default, which is based on the httpx library, but you are free to use your own custom async client.

The callback function processes a Response object and returns either data or additional Request objects.

In this trivial example we are requesting the Books to Scrape homepage and parsing the number of books on the page.

Example parse_books_page function:

def parse_books_page(response: Response):
    articles = response.html.find_all("article", {"class": "product_pod"})
    return {
        "url": response.url,
        "title": response.html.title.get_text(strip=True),
        "articles": len(articles),
    }

This function takes a Response object, which has a html attribute (a BeautifulSoup object of the HTML content). The function parses the HTML content and returns data.

The callback function can return or yield either data (dict or pydantic.BaseModel) or more Request objects.

If you have used Scrapy before, you will find this pattern familiar.

For more examples and advanced usage, check out the examples section.

For a detailed API reference, check out the API section.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

python_dataservice-0.12.0.tar.gz (22.7 kB view details)

Uploaded Source

Built Distribution

python_dataservice-0.12.0-py3-none-any.whl (26.2 kB view details)

Uploaded Python 3

File details

Details for the file python_dataservice-0.12.0.tar.gz.

File metadata

  • Download URL: python_dataservice-0.12.0.tar.gz
  • Upload date:
  • Size: 22.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.8.4 CPython/3.12.7 Linux/6.5.0-1025-azure

File hashes

Hashes for python_dataservice-0.12.0.tar.gz
Algorithm Hash digest
SHA256 f3dcf62a01d0a769e3b265fab5a45398a7c006e26042a46326c889244fdc52d9
MD5 e75730053614f101d3cc7e90a9887585
BLAKE2b-256 c54051176b910543226106ab88c05132f49630e172411dac8bd6c50a447e4b92

See more details on using hashes here.

File details

Details for the file python_dataservice-0.12.0-py3-none-any.whl.

File metadata

File hashes

Hashes for python_dataservice-0.12.0-py3-none-any.whl
Algorithm Hash digest
SHA256 292fd73882eae647d4a89a963068efdad3723a6a3ee9d4ac3d51374c450c3a28
MD5 df79a70bef86e5430a860395357cce8c
BLAKE2b-256 69a7544f9b4fc168b5c71093df68beec39768ba5b1b1c270b1890e8f3fb31c47

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page