Crawler
Distributed web crawler for schema.org structured data.
Architecture
Master/worker pattern running as separate pods in Kubernetes:
- Master: Flask API + job scheduler
- Worker: Queue processor (embedding + upload to Azure AI Search)
Flow: Parse schema.org sitemaps → queue JSON files → embed → upload
Endpoints
GET /- Web UIGET /api/status- System statusPOST /api/sites- Add site to crawlGET /api/queue/status- Queue statistics
Commands
Run make help for the full list. Key targets:
make dev # Run master + worker via Docker Compose
make test # Run pytest
make build # Build image to ACR
make deploy # Deploy to AKS via Helm
1 maintainer
Metadata
1 maintainer
Release files for nlweb-crawler 0.7.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| nlweb_crawler-0.7.1.tar.gz | 85.6 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| nlweb_crawler-0.7.1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 180.1 kB
Release files / nlweb_crawler-0.7.1.tar.gz
| Download URL | nlweb_crawler-0.7.1.tar.gz |
|---|---|
| Size | 85.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
8b94995fcead145721f3f29d8becaa50f901d983df6fea9452c5368e944fbe32
|
|
BLAKE2b-256 checksum How to use checksums |
a1dae5f7c5dd199f55a6467c6d7d29b350d5cab332aac5aa4c6e2fd93f08674d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.10.4 {"installer":{"name":"uv","version":"0.10.4","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / nlweb_crawler-0.7.1-py3-none-any.whl
| Download URL | nlweb_crawler-0.7.1-py3-none-any.whl |
|---|---|
| Size | 94.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e813dee6adfd4a293bd1a635550b7e72ea9aa466c19d3871ba55d23417e23658
|
|
BLAKE2b-256 checksum How to use checksums |
d69103a4c6693bfce71f9ff882cc8564ffd3f297074ae21586189362d6ced289
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.10.4 {"installer":{"name":"uv","version":"0.10.4","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
