web-poet 0.3.0

Zyte's Page Object pattern for web scraping

web-poet implements Page Object pattern for web scraping. It defines a standard for writing web data extraction code, which allows the code to be portable & reusable.

License is BSD 3-clause.

Installation

pip install web-poet

It requires Python 3.7+.

Overview

web-poet is a library which defines a standard on how to write and organize web data extraction code.

If web scraping code is written as web-poet Page Objects, it can be reused in different contexts. For example, such code can be developed in an IPython notebook, then tested in isolation, and then plugged into a Scrapy spider, or used as a part of some custom aiohttp-based web scraping framework.

Currently, the following integrations are available:

• Scrapy, via scrapy-poet

See Documentation for more.

Developing

Setup your local Python environment via:

1. pip install -r requirements-dev.txt

2. pre-commit install

Now everytime you perform a git commit, these tools will run against the staged files:

• black

• isort

• flake8

You can also directly invoke pre-commit run –all-files or tox -e linters to run them without performing a commit.