aegis-stack-crawl4ai
Web crawling for Aegis Stack projects. Point it at a site, or at a site's history in the Wayback Machine, and every page lands in your database as markdown and the HTML it came from, with its images, stylesheets and downloads kept beside it. Wraps crawl4ai (Apache 2.0). Listed in the plugin directory.
It is what NWVault Revival was recovered with: the original Neverwinter Vault, 2002 to 2014, rebuilt from the archive.
What it does
- Crawls live sites. Seeds, link depth, a page cap, a CSS selector for the part that matters, include and exclude globs.
- Crawls the past. One domain from the Wayback Machine, between two dates, each page at its nearest capture, paced so the archive does not throttle it.
- Keeps the assets. The files a page references are fetched once per site and stored by content, so the same GIF on three hundred pages is one object.
- Exports a site as plain files: pages, assets and a
manifest.json, as a zip or into the object store.
How it connects
aegis add crawl4ai renders an ordinary service into the project, wired into the parts the
stack already has:
- Database. Two tables,
crawled_pageandcrawled_asset, in their owncrawlerPostgres schema (unqualified on SQLite), created by an Alembic migration. - Worker. Crawls run on the plugin's own
crawlerqueue (arq, TaskIQ or Dramatiq, whichever the stack uses), so a webserver restart leaves them running. Without a worker they run in the webserver. - Jobs. Each crawl is a framework job: progress streams over
/api/v1/jobs/{job_id}/events, and pages are stored one at a time as they land. - Object store. Assets and stored exports live in the project's
app.core.storage. - Backend. Routes under
/api/v1/crawler, and acrawlcommand in the project CLI. - Agents. Three read tools (
crawler_sites,crawler_pages,crawler_page) in the app's tool registry, so MCP clients and the chat can read the store. - Health and frontends. A
crawlerhealth check, a card and modal on the dashboard, and a Crawler page in the htmx Overseer: Overview, Sites, Pages, and Crawl to start and follow a job.
Inside, the routes, jobs and tools go through one facade (CrawlerService) to four domains:
archive, assets, export and wayback. The Overseer map above is that structure, read from the
code.
Install
Inside an Aegis Stack project (aegis-stack 0.19.1 or later):
uv pip install aegis-stack-crawl4ai
aegis plugins list # crawl4ai shows as ready
aegis add crawl4ai
The plugin needs the database component; aegis add adds it first when the project lacks
it. The worker component is recommended. The fetch is HTTP only, with no browser, so
client-rendered content is out of scope.
First crawl
<project> crawl site https://example.com --depth 1 --max-pages 20
<project> crawl stats
Or from the API, which answers with a job to follow:
curl -X POST localhost:8000/api/v1/crawler/crawl-batch \
-H 'content-type: application/json' \
-d '{"urls": ["https://example.com"], "options": {"depth": 1, "max_pages": 20}}'
Docs
- Using it: the API, the CLI, Python inside the app, agent tools, progress, and a worked bulk fetch.
- Crawling: crawl options, archive snapshots, assets, export.
- Reference:
settings, the tables, why
crawl4ai>=0.8.6, the files it adds.
Develop
make install # uv sync
make test # pytest
make check # lint + test
make fix # ruff format + autofix
tests/test_plugin.py checks the spec contract here. The tests under
templates/{{ project_slug }}/tests/ ship into the generated project and run
against the real service there, including a layout test that holds the plugin to
the host project's structure.
License
Apache 2.0, same as upstream crawl4ai.
Metadata
Release files for aegis-stack-crawl4ai 0.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| aegis_stack_crawl4ai-0.2.0.tar.gz | 421.3 kB | With provenance | Details for aegis_stack_crawl4ai-0.2.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| aegis_stack_crawl4ai-0.2.0-py3-none-any.whl | Python 3 | none | any | Details for aegis_stack_crawl4ai-0.2.0-py3-none-any.whl |
Total release size: 537.4 kB
Release files / aegis_stack_crawl4ai-0.2.0.tar.gz
| Download URL | aegis_stack_crawl4ai-0.2.0.tar.gz |
|---|---|
| Size | 421.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
9e41797fc21cd168615246e0f188be59d804667c92a79133dd2d1eb73d5f7a17
|
|
BLAKE2b-256 checksum How to use checksums |
b63309a81a1c8c3259c0d8cdff3041ca32e0fca296246681c30ec327c45a09a9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 8, 2026.
Transparency logRelease files / aegis_stack_crawl4ai-0.2.0-py3-none-any.whl
| Download URL | aegis_stack_crawl4ai-0.2.0-py3-none-any.whl |
|---|---|
| Size | 116.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
8039d31e7abd36399ce255e401d587551fde84e6112dad46f70d1607ae943f3b
|
|
BLAKE2b-256 checksum How to use checksums |
3224922d84e4cd44b5587e1a737007202eabef67eb127a5736529fdfb8c1de87
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 8, 2026.
Transparency logSecurity for aegis-stack-crawl4ai 0.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for aegis_stack_crawl4ai-0.2.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 8, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.