data-extractor

Combine XPath, CSS Selectors and JSONPath for Web data extracting.

Quickstarts

Installation

Install the stable version from PYPI.

pip install "data-extractor[jsonpath-extractor]"  # for extracting JSON data
pip install "data-extractor[lxml]"  # for extracting HTML data

Or install the latest version from Github.

pip install "data-extractor[jsonpath-extractor] @ git+https://github.com/linw1995/data_extractor.git@master"

Extract JSON data

Currently supports to extract JSON data with below optional dependencies

install one dependency of them to extract JSON data.

Extract HTML(XML) data

Currently supports to extract HTML(XML) data with below optional dependencies

lxml for using XPath
cssselect for using CSS-Selectors

Usage

from data_extractor import Field, Item, JSONExtractor


class Count(Item):
    followings = Field(JSONExtractor("countFollowings"))
    fans = Field(JSONExtractor("countFans"))


class User(Item):
    name_ = Field(JSONExtractor("name"), name="name")
    age = Field(JSONExtractor("age"), default=17)
    count = Count()


assert User(JSONExtractor("data.users[*]"), is_many=True).extract(
    {
        "data": {
            "users": [
                {
                    "name": "john",
                    "age": 19,
                    "countFollowings": 14,
                    "countFans": 212,
                },
                {
                    "name": "jack",
                    "description": "",
                    "countFollowings": 54,
                    "countFans": 312,
                },
            ]
        }
    }
) == [
    {"name": "john", "age": 19, "count": {"followings": 14, "fans": 212}},
    {"name": "jack", "age": 17, "count": {"followings": 54, "fans": 312}},
]

Changelog

v1.0.1

Build

Supports Python 3.13

Contributing

Environment Setup

Clone the source codes from Github.

git clone https://github.com/linw1995/data_extractor.git
cd data_extractor

Setup the development environment. Please make sure you install the pdm, pre-commit and nox CLIs in your environment.

make init
make PYTHON=3.7 init  # for specific python version

Linting

Use pre-commit for installing linters to ensure a good code style.

make pre-commit

Run linters. Some linters run via CLI nox, so make sure you install it.

make check-all

Testing

Run quick tests.

make

Run quick tests with verbose.

make vtest

Run tests with coverage. Testing in multiple Python environments is powered by CLI nox.

make cov

Release files for data-extractor 1.0.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for data-extractor 1.0.1
File	Size	Uploaded
data-extractor-1.0.1.tar.gz	28.2 kB	Oct 13, 2024	Details

Built distribution (wheel)

Table of built distributions (wheels) for data-extractor 1.0.1
File	Interpreter	ABI	Platform	Reset
data_extractor-1.0.1-py3-none-any.whl 18.3 kB Oct 13, 2024	Python 3	none	any	Details

Total release size: 46.5 kB

Release files / data-extractor-1.0.1.tar.gz

Download URL	data-extractor-1.0.1.tar.gz
Size	28.2 kB
Tags	Source
SHA-256 checksum How to use checksums	`3ff9424d4859ecd1a4f3b0f9f5b614117a0efa2747493365169b54c6af51aa90`
BLAKE2b-256 checksum How to use checksums	`19ab4a9fff19fe0fcb15eb83083fa51288bd1f21aa1acbc12fba8dd93c8c6597`
Upload date	Oct 13, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.9.20`

Release files / data_extractor-1.0.1-py3-none-any.whl

Download URL	data_extractor-1.0.1-py3-none-any.whl
Size	18.3 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`2126dc68207b650ae884cac6caf8dec0388c6334ad379069b166d6336e27b1e7`
BLAKE2b-256 checksum How to use checksums	`1f9e4c7f72bd7e7a0879eb751c8599dda74a8542d736e97c8f673eaff8226b68`
Upload date	Oct 13, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.9.20`