scrapeghost
scrapeghost is an experimental library for scraping websites using OpenAI's GPT.
Source: https://github.com/jamesturk/scrapeghost
Documentation: https://jamesturk.github.io/scrapeghost/
Issues: https://github.com/jamesturk/scrapeghost/issues
Use at your own risk. This library makes considerably expensive calls ($0.36 for a GPT-4 call on a moderately sized page.) Cost estimates are based on the OpenAI pricing page and not guaranteed to be accurate.
Features
The purpose of this library is to provide a convenient interface for exploring web scraping with GPT.
While the bulk of the work is done by the GPT model, scrapeghost provides a number of features to make it easier to use.
Python-based schema definition - Define the shape of the data you want to extract as any Python object, with as much or little detail as you want.
Preprocessing
- HTML cleaning - Remove unnecessary HTML to reduce the size and cost of API requests.
- CSS and XPath selectors - Pre-filter HTML by writing a single CSS or XPath selector.
- Auto-splitting - Optionally split the HTML into multiple calls to the model, allowing for larger pages to be scraped.
Postprocessing
- JSON validation - Ensure that the response is valid JSON. (With the option to kick it back to GPT for fixes if it's not.)
- Schema validation - Go a step further, use a
pydanticschema to validate the response. - Hallucination check - Does the data in the response truly exist on the page?
Cost Controls
- Scrapers keep running totals of how many tokens have been sent and received, so costs can be tracked.
- Support for automatic fallbacks (e.g. use cost-saving GPT-3.5-Turbo by default, fall back to GPT-4 if needed.)
- Allows setting a budget and stops the scraper if the budget is exceeded.
Metadata
Release files for scrapeghost 0.6.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| scrapeghost-0.6.0.tar.gz | 17.3 kB | Without provenance | Details for scrapeghost-0.6.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| scrapeghost-0.6.0-py3-none-any.whl | Python 3 | none | any | Details for scrapeghost-0.6.0-py3-none-any.whl |
Total release size: 37.0 kB
Release files / scrapeghost-0.6.0.tar.gz
| Download URL | scrapeghost-0.6.0.tar.gz |
|---|---|
| Size | 17.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
78d49016c59d907f659b6bd3f470555b374fc7a9b085a8feeac31a4f0df59404
|
|
BLAKE2b-256 checksum How to use checksums |
0660fd7bc7b7f3bac6dff42d019de01372120bd91a14055e7ae3db1a3d825f55
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.5.1 CPython/3.11.4 Darwin/22.6.0
|
Release files / scrapeghost-0.6.0-py3-none-any.whl
| Download URL | scrapeghost-0.6.0-py3-none-any.whl |
|---|---|
| Size | 19.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
3afa8d6e48cfcc37704c500930763fd5022062e6447f5748ce4771992fac87e1
|
|
BLAKE2b-256 checksum How to use checksums |
d9d1d779b4e7ca8195d0514814bc2432e282aaca72be122801a1467000dc17f2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.5.1 CPython/3.11.4 Darwin/22.6.0
|
Security for scrapeghost 0.6.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.