Scraipe
Scraping and analysis framework. Under development.
Features
- Versatile Scraping: Leverage custom scrapers that handle Telegram messages, news articles, and links that require multiple ingress rules.
- LLM Analysis: Process text using OpenAI models with built-in Pydantic validation.
- Workflow Management: Combine scraping and analysis in a single fault-tolerant workflow--ideal for Jupyter notebooks.
- High Performance: Asynchronous IO-bound tasks are seamlessly integrated in the synchronous API.
- Modular: Extend the framework with new scrapers or analyzers as your data sources evolve.
- Customizable Ingress: Easily define rules to dynamically route different links to their appropriate scrapers.
- Detailed Logging: Monitor scraping and analysis operations through robust errors for improved debugging and transparency.
Check out the demo.
Help
See documentation for details.
Installation
Ensure you are using Python>=3.10. Install Scraipe and all built-in scrapers/analyzers:
pip install scraipe[extended]
Alternatively, install the core library with:
pip install scraipe
Example
# Import components from scraipe
from scraipe.defaults import TextScraper
from scraipe.defaults import TextStatsAnalyzer
from scraipe import Workflow
# Initialize the scraper and analyzer
scraper = TextScraper()
analyzer = TextStatsAnalyzer()
# Create the workflow instance
workflow = Workflow(scraper, analyzer)
# List urls to scrape
urls = [
"https://example.com",
"https://rickandmortyapi.com/api/character/1",
"https://ckaestne.github.io/seai/"
]
# Run the workflow
workflow.scrape(urls)
workflow.analyze()
# Print the results
results = workflow.export()
print(results)
Contributing
Contributions are welcome. Please open an issue or submit a pull request for improvements.
Run poetry install --with dev,docs --extras extended to install all dependences for the project.
Maintainer
This project is maintained by nibs.
1 maintainer
Metadata
1 maintainer
Release files for scraipe 0.1.73
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| scraipe-0.1.73.tar.gz | 32.5 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| scraipe-0.1.73-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 76.1 kB
Release files / scraipe-0.1.73.tar.gz
| Download URL | scraipe-0.1.73.tar.gz |
|---|---|
| Size | 32.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
77b233c81d26abf07baba3f95baca5be1d8ce2e89071a6a81a8ad5d9f95287bb
|
|
BLAKE2b-256 checksum How to use checksums |
097a2421a53d1ad19cd35d48ef583a76bf3afe82a2270caea5262ecc7b04d96b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.1.2 CPython/3.12.9 Linux/5.15.167.4-microsoft-standard-WSL2
|
Release files / scraipe-0.1.73-py3-none-any.whl
| Download URL | scraipe-0.1.73-py3-none-any.whl |
|---|---|
| Size | 43.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
01d440e535c22e30afe4daa57bd4bbc5cfbd6c7fa934153df2a431ed6f278612
|
|
BLAKE2b-256 checksum How to use checksums |
4c96f216a2fe101f3765dd63293521dbe49f9d10988218a7551e5f977fa9d4ce
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.1.2 CPython/3.12.9 Linux/5.15.167.4-microsoft-standard-WSL2
|
Security for scraipe 0.1.73
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
