WebSense
"Making sense of the web."
WebSense is a Python library that transforms raw websites into structured, meaningful data. It leverages AI through the ask2api library to semantically understand page content, allowing you to extract complex data structures without writing brittle CSS selectors or XPath expressions.
Features
- Semantic Understanding: Uses LLMs to interpret content meaning, not just match patterns
- Resilient: Adapts to layout changes—if the meaning is there, WebSense finds it
- Minimalist API: Extract data in 3 lines of code
- Auto-Cleaning: Intelligent noise removal filters focus on meaningful content
- Flexible Schemas: Use JSON schemas or provide examples for schema inference
- Web Search Integration: Search the web and scrape top results in one go
- Multi-Source Consolidation: Aggregate information from multiple websites into one structured result
- Modular Design: Fetch, search, clean, and parse stages can be customized independently
Installation
pip install websense
For development:
git clone https://github.com/atasoglu/websense.git
cd websense
pip install -e ".[dev]"
Quick Start
Extract data with just an example:
from websense import Scraper
scraper = Scraper()
data = scraper.scrape(
"https://github.com/atasoglu/ask2api",
example={
"project_name": "string",
"description": "string",
"stars": 0,
"is_active": True
}
)
print(data)
You can provide a strict JSON schema for validation:
schema = {
"type": "object",
"properties": {
"title": {"type": "string"},
"price": {"type": "number"},
"in_stock": {"type": "boolean"}
},
"required": ["title", "price"]
}
data = scraper.scrape("https://example.com/product", schema=schema)
Specify a different language model for extraction:
scraper = Scraper(model="gpt-4")
Web Search & Consolidation
Search the web and consolidate information from the top 3 results:
data = scraper.search_and_scrape(
"latest news about SpaceX Starship",
max_results=3,
example={
"status": "string",
"last_launch": "string",
"summary": "brief overview"
}
)
WebSense intelligently crawls multiple sources and uses an LLM-based "judge" to synthesize the most accurate data from all sources.
CLI Usage
WebSense provides a command-line interface for quick data extraction:
# Extract structured data from a webpage
websense scrape https://example.com --example schema.json --verbose
# Search the web and consolidate top 3 results
websense search-scrape "Nvidia stock performance 2024" --top-k 3 --example '{"price": "str"}'
# Search search only (returns titles and URLs)
websense search "query" --verbose
# Get cleaned content only
websense content https://example.com --output content.md
Available options for scrape command:
| Option | Description |
|---|---|
--model, -m |
LLM model name |
--schema, -s |
JSON schema (file path or raw JSON string) |
--example, -e |
JSON example (file path or raw JSON string) |
--output, -o |
Output file path |
--timeout, -t |
Request timeout (default: 10) |
--retries, -r |
Retry attempts (default: 3) |
--verbose, -v |
Enable verbose output |
Pro Tip: You can pass raw JSON strings directly to the CLI:
websense scrape https://example.com -e '{"title": "string"}'
How It Works
WebSense follows a three-stage pipeline:
- Fetch (
fetcher.py): Downloads and retrieves the webpage - Clean (
cleaner.py): Removes noise and extracts meaningful text - Parse (
parser.py): Uses AI to extract structured data based on your schema/example
Contributing
Contributions are welcome! Please:
- Fork the repository
- Create a feature branch (
git checkout -b feature/my-feature) - Commit changes (
git commit -m 'Add my feature') - Push to the branch (
git push origin feature/my-feature) - Open a Pull Request
License
MIT
Metadata
Release files for websense 0.4.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| websense-0.4.1.tar.gz | 20.0 kB | Without provenance | Details for websense-0.4.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| websense-0.4.1-py3-none-any.whl | Python 3 | none | any | Details for websense-0.4.1-py3-none-any.whl |
Total release size: 33.6 kB
Release files / websense-0.4.1.tar.gz
| Download URL | websense-0.4.1.tar.gz |
|---|---|
| Size | 20.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
45144cb7c4d4a45a6f41f147ce6d24e0e38fb61706196b6d99b526c78c82c34e
|
|
BLAKE2b-256 checksum How to use checksums |
b9b866cb937d9fbcfa069950d257da5efbb9f0999891232228dc40c3d7665990
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Release files / websense-0.4.1-py3-none-any.whl
| Download URL | websense-0.4.1-py3-none-any.whl |
|---|---|
| Size | 13.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
938a9e6f2aa1f795e314847ee440f41bbaa5006509e0cf3eb967b907a43f7c8b
|
|
BLAKE2b-256 checksum How to use checksums |
1a7e4395c60b9ad62950dd8404e59b6f61dec18c235a4ce5ae779c446cfe9477
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Security for websense 0.4.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.