Skip to main content

Lightweight async data gathering for Python

Project description

Python Versions

DataService

Lightweight - async - data gathering for Python.

DataService is a lightweight web scraping and general purpose data gathering library for Python.

Designed for simplicity, it’s built upon common web scraping and data gathering patterns.

No complex API to learn, just standard Python idioms.

Dual synchronous and asynchronous support.

Installation

Please note that DataService requires Python 3.11 or higher.

You can install DataService via pip:

pip install python-dataservice

You can also install the optional playwright dependency to use the PlaywrightClient:

pip install python-dataservice[playwright]

To install Playwright, run:

python -m playwright install

or simply:

playwright install

How to use DataService

To start, create a DataService instance with an Iterable of Request objects. This setup provides you with an Iterator of data objects that you can then iterate over or convert to a list, tuple, a pd.DataFrame or any data structure of choice.

start_requests = [Request(url="https://books.toscrape.com/index.html", callback=parse_books_page, client=HttpXClient())]
data_service = DataService(start_requests)
data = tuple(data_service)

A Request is a Pydantic model that includes the URL to fetch, a reference to the client callable, and a callback function for parsing the Response object.

The client can be any async Python callable that accepts a Request object and returns a Response object. DataService provides an HttpXClient class by default, which is based on the httpx library, but you are free to use your own custom async client.

The callback function processes a Response object and returns either data or additional Request objects.

In this trivial example we are requesting the Books to Scrape homepage and parsing the number of books on the page.

Example parse_books_page function:

def parse_books_page(response: Response):
    articles = response.html.find_all("article", {"class": "product_pod"})
    return {
        "url": response.url,
        "title": response.html.title.get_text(strip=True),
        "articles": len(articles),
    }

This function takes a Response object, which has a html attribute (a BeautifulSoup object of the HTML content). The function parses the HTML content and returns data.

The callback function can return or yield either data (dict or pydantic.BaseModel) or more Request objects.

If you have used Scrapy before, you will find this pattern familiar.

For more examples and advanced usage, check out the examples section.

For a detailed API reference, check out the API section.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

python_dataservice-0.8.0.tar.gz (21.1 kB view details)

Uploaded Source

Built Distribution

python_dataservice-0.8.0-py3-none-any.whl (24.2 kB view details)

Uploaded Python 3

File details

Details for the file python_dataservice-0.8.0.tar.gz.

File metadata

  • Download URL: python_dataservice-0.8.0.tar.gz
  • Upload date:
  • Size: 21.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.8.3 CPython/3.11.9 Darwin/23.6.0

File hashes

Hashes for python_dataservice-0.8.0.tar.gz
Algorithm Hash digest
SHA256 c4867033d3bed881b971b98a74ec4e589d3a5d4b40d7c96880e7a5533a4eacd2
MD5 4b35601358806fdb35b483bc70a9863c
BLAKE2b-256 737357f7efdca8299027530ee7c68df488e120922a52b2f5b6a943657c45fde5

See more details on using hashes here.

Provenance

File details

Details for the file python_dataservice-0.8.0-py3-none-any.whl.

File metadata

File hashes

Hashes for python_dataservice-0.8.0-py3-none-any.whl
Algorithm Hash digest
SHA256 41ae53a3912154e569d01733c517e19fa2e40ae345a49f3331d7ba460103702c
MD5 95bbe14b4ee541c394fe56e9ac29a317
BLAKE2b-256 affaa447ddda1d5ba3e2012ae754a1ce5c7b67705ef1571180e51e0913bfc798

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page