klyrek-crawler

Breadth-first crawler that maps a target's pages, forms, and script references into a klyrek_core.models.ScanResult. Every fetch goes through a KlyrekHTTPClient (klyrek-http), so scope enforcement and rate limiting are inherited automatically — the crawler additionally skips enqueuing out-of-scope links so it doesn't spend its page budget chasing hosts it isn't authorized to touch.

from klyrek_core.config import KlyrekConfig
from klyrek_core.models import Target
from klyrek_core.scope import AuthorizationScope
from klyrek_http.client import KlyrekHTTPClient
from klyrek_crawler.crawler import Crawler

scope = AuthorizationScope(authorized_hosts=["target.com", "*.target.com"])
target = Target(base_url="https://target.com/", hosts=["target.com"])

with KlyrekHTTPClient(scope, config=KlyrekConfig(rate_limit_per_host=2)) as client:
    crawler = Crawler(client, max_pages=200, max_depth=3)
    result = crawler.crawl(target)

print(len(result.endpoints), "endpoints discovered")

Endpoints are tagged by how they were found: source="crawler" for pages, "crawler-form" for form actions (with field names captured as params), and "crawler-script" for referenced JavaScript files (the raw material klyrek-js consumes).