Skip to main content

aegis-stack-crawl4ai

Web crawling for Aegis Stack projects. Point it at a site, or at a site's history in the Wayback Machine, and every page lands in your database as markdown and the HTML it came from, with its images, stylesheets and downloads kept beside it. Wraps crawl4ai (Apache 2.0). Listed in the plugin directory.

It is what NWVault Revival was recovered with: the original Neverwinter Vault, 2002 to 2014, rebuilt from the archive.

What it does

  • Crawls live sites. Seeds, link depth, a page cap, a CSS selector for the part that matters, include and exclude globs.
  • Crawls the past. One domain from the Wayback Machine, between two dates, each page at its nearest capture, paced so the archive does not throttle it.
  • Keeps the assets. The files a page references are fetched once per site and stored by content, so the same GIF on three hundred pages is one object.
  • Exports a site as plain files: pages, assets and a manifest.json, as a zip or into the object store.

How it connects

The crawler's architecture in Overseer

aegis add crawl4ai renders an ordinary service into the project, wired into the parts the stack already has:

  • Database. Two tables, crawled_page and crawled_asset, in their own crawler Postgres schema (unqualified on SQLite), created by an Alembic migration.
  • Worker. Crawls run on the plugin's own crawler queue (arq, TaskIQ or Dramatiq, whichever the stack uses), so a webserver restart leaves them running. Without a worker they run in the webserver.
  • Jobs. Each crawl is a framework job: progress streams over /api/v1/jobs/{job_id}/events, and pages are stored one at a time as they land.
  • Object store. Assets and stored exports live in the project's app.core.storage.
  • Backend. Routes under /api/v1/crawler, and a crawl command in the project CLI.
  • Agents. Three read tools (crawler_sites, crawler_pages, crawler_page) in the app's tool registry, so MCP clients and the chat can read the store.
  • Health and frontends. A crawler health check, a card and modal on the dashboard, and a Crawler page in the htmx Overseer: Overview, Sites, Pages, and Crawl to start and follow a job.

The Crawler page in Overseer

Inside, the routes, jobs and tools go through one facade (CrawlerService) to four domains: archive, assets, export and wayback. The Overseer map above is that structure, read from the code.

Install

Inside an Aegis Stack project (aegis-stack 0.16.1 or later):

uv pip install aegis-stack-crawl4ai
aegis plugins list    # crawl4ai shows as ready
aegis add crawl4ai

The plugin needs the database component; aegis add adds it first when the project lacks it. The worker component is recommended. The fetch is HTTP only, with no browser, so client-rendered content is out of scope.

First crawl

<project> crawl site https://example.com --depth 1 --max-pages 20
<project> crawl stats

Or from the API, which answers with a job to follow:

curl -X POST localhost:8000/api/v1/crawler/crawl-batch \
  -H 'content-type: application/json' \
  -d '{"urls": ["https://example.com"], "options": {"depth": 1, "max_pages": 20}}'

Docs

  • Using it: the API, the CLI, Python inside the app, agent tools, progress, and a worked bulk fetch.
  • Crawling: crawl options, archive snapshots, assets, export.
  • Reference: settings, the tables, why crawl4ai>=0.8.6, the files it adds.

Develop

make install     # uv sync
make test        # pytest
make check       # lint + test
make fix         # ruff format + autofix

tests/test_plugin.py checks the spec contract here. The tests under templates/{{ project_slug }}/tests/ ship into the generated project and run against the real service there, including a layout test that holds the plugin to the host project's structure.

License

Apache 2.0, same as upstream crawl4ai.

Metadata

Release files for aegis-stack-crawl4ai 0.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for aegis-stack-crawl4ai 0.1.1
File Size Uploaded Provenance Actions
aegis_stack_crawl4ai-0.1.1.tar.gz 420.0 kB With provenance Details for aegis_stack_crawl4ai-0.1.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for aegis-stack-crawl4ai 0.1.1
File Interpreter ABI Platform Actions
aegis_stack_crawl4ai-0.1.1-py3-none-any.whl Python 3 none any Details for aegis_stack_crawl4ai-0.1.1-py3-none-any.whl

Total release size: 534.5 kB

Release files / aegis_stack_crawl4ai-0.1.1.tar.gz

Download URL aegis_stack_crawl4ai-0.1.1.tar.gz
Size 420.0 kB
Tags Source
SHA-256 checksum
How to use checksums
11fa39aa8bf54acdd3a2a453f002e1dfafeaee42cafe061f63afca4beb0a8606
BLAKE2b-256 checksum
How to use checksums
e5b97c240ccd822fac50e5b6cf611e7393c2678f0518d7fecc229abbb1c50a1e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 6, 2026.

Transparency log

Release files / aegis_stack_crawl4ai-0.1.1-py3-none-any.whl

Download URL aegis_stack_crawl4ai-0.1.1-py3-none-any.whl
Size 114.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
09f473d2cc0deac8123b874403db2a84b745c8733980d94b7123b249df449120
BLAKE2b-256 checksum
How to use checksums
94ddb1346a49318cb50fe8ac437b092c7aa55e12114b1c5062574f8ae3760691
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 6, 2026.

Transparency log

Security for aegis-stack-crawl4ai 0.1.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for aegis_stack_crawl4ai-0.1.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 6, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page