Python client for the WebCache REST API
Project description
webcache-client
Python client for the WebCache REST API.
Installation
Install directly from the local package:
pip install ./client
Or add it to a requirements.txt alongside the webcache service:
dwilson-webcache-client>=0.2.0
Requires: Python 3.11+, httpx>=0.27.0
Quick start
from webcache_client import WebCacheClient
client = WebCacheClient("http://localhost:8000")
# Store a page
client.store(url="https://example.com/page", content="<html>…</html>", client_name="my_scraper")
# Retrieve by exact URL
entry = client.get("https://example.com/page")
print(entry["content"])
# Search by URL substring (returns metadata only, no content)
results = client.search(url_contains="example.com")
# Delete by content hash
client.delete(content_hash=entry["content_hash"])
client.close()
Use as a context manager to close the HTTP connection automatically:
with WebCacheClient("http://localhost:8000") as client:
client.store(url=url, content=html, client_name="my_scraper")
API reference
WebCacheClient(base_url, timeout=30.0)
| Parameter | Type | Description |
|---|---|---|
base_url |
str |
Base URL of the WebCache service, e.g. http://localhost:8000 |
timeout |
float |
Request timeout in seconds (default 30.0) |
store(url, content, client_name, lookup_time=None) → dict
Cache a web page.
| Parameter | Type | Description |
|---|---|---|
url |
str |
Full URL of the page, including any query parameters |
content |
str |
Raw page content (HTML, XML, etc.) |
client_name |
str |
Identifier for the scraper that fetched the page |
lookup_time |
datetime (optional) |
When the page was fetched from origin; defaults to utcnow() |
Returns the stored entry's metadata dict. HTTP 201 means the page was newly stored; HTTP 200 means identical content already existed (no write performed).
get(url) → dict | None
Retrieve the most recent cached entry for an exact URL. Returns the full entry dict including content, or None if not found.
get_by_hash(content_hash) → dict | None
Retrieve a cached entry by its BLAKE2b content hash. Returns None if not found.
search(url_contains) → list[dict]
Return metadata for all cached entries whose URL contains the given substring. Content is not included — call get or get_by_hash to fetch the full page.
# Matches https://example.com/products?page=1, ?page=2, etc.
results = client.search(url_contains="example.com/products")
for entry in results:
print(entry["url"], entry["content_hash"])
delete(content_hash) → None
Delete a cached entry and its associated storage file by content hash.
health() → dict
Check that the service is reachable. Returns {"status": "ok"} when healthy.
close() → None
Close the underlying HTTP connection. Called automatically when used as a context manager.
Entry schema
Fields returned by store, get, and get_by_hash:
| Field | Type | Description |
|---|---|---|
url |
str |
The cached page URL |
content_hash |
str |
BLAKE2b hash of the content (also the storage filename) |
client_name |
str |
Scraper that submitted the entry |
lookup_time |
str (ISO 8601) |
When the page was fetched from origin |
created_at |
str (ISO 8601) |
When the entry was stored in the cache |
content |
str |
Decompressed page content (full-entry endpoints only) |
search results omit the content field.
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file dwilson_webcache_client-1.0.1.tar.gz.
File metadata
- Download URL: dwilson_webcache_client-1.0.1.tar.gz
- Upload date:
- Size: 3.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.13.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
68b8843bc45267d8a8966717a8a067bb37a68c225155afa2d0c71b9e4d9cf1e1
|
|
| MD5 |
1f1df7eea274728018bc247475ff6c48
|
|
| BLAKE2b-256 |
3eca5d1443c5b134b1325d3f6cbe7ce77790c112859dad26ad8acf953bd7b3d8
|
File details
Details for the file dwilson_webcache_client-1.0.1-py3-none-any.whl.
File metadata
- Download URL: dwilson_webcache_client-1.0.1-py3-none-any.whl
- Upload date:
- Size: 4.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.13.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
b23c3dc9a951af24007732e1d685dc45420aeae88b0113fa375e5bc42f0c389a
|
|
| MD5 |
dd716a547f6e2111ed9d54d002bce601
|
|
| BLAKE2b-256 |
4996066efbf73da582e3782b39d9379d31cb3d5062b86c8d753c03b63221e161
|
