ApiCrawl

Crawl API documentation into structured, searchable markdown.

Point it at any API docs URL — an OpenAPI/Swagger spec, a Swagger UI / Redoc / Stoplight / Scalar page, ReadMe / Mintlify / Fern hosted docs, a Postman collection, a Google Discovery document, an llms.txt index, or plain HTML docs — and it discovers the underlying spec where one exists, crawls the pages where one doesn't, classifies the content with an LLM, extracts authentication instructions, and writes everything as a local markdown tree you can grep, read, or feed to any tool.

pip install apicrawl
playwright install chromium   # used to render JS-heavy docs sites

export GOOGLE_API_KEY=...     # LLM access is required (Gemini primary)
export GROQ_API_KEY=...       # optional fallback provider

apicrawl https://petstore3.swagger.io --output ./api-docs

Output layout:

api-docs/<catalog_id>/
  index.md            # API name, metadata, description, auth instructions
  manifest.json       # listing of everything ingested (also the completion marker)
  sections/<slug>.md  # docs pages / spec tag groups (markdown + frontmatter)
  endpoints/<slug>.md # one file per endpoint: parameters, examples, TypeScript types

Library usage:

import asyncio
from apicrawl import ingest_to_dir

result = asyncio.run(ingest_to_dir("https://docs.example.com/api", "./api-docs"))
print(result.entry.name, result.pages_ingested)

Custom storage — implement IngestionSink and receive the parsed catalog entry, sections, and endpoints as plain pydantic models, streamed in batches:

from apicrawl import IngestionSink, ingest

class MySink(IngestionSink):
    async def emit_sections(self, sections): ...
    async def emit_endpoints(self, endpoints): ...

asyncio.run(ingest("https://docs.example.com/api", MySink()))

Notes:

• LLM keys are required — page classification and auth extraction are LLM-powered. Set GOOGLE_API_KEY (and optionally GROQ_API_KEY) in the environment or a .env file.
• Node.js is optional — if a node binary is on PATH, endpoint pages include generated TypeScript request/response types (via a bundled openapi-typescript). Without Node, ingestion still works; the TS sections are simply omitted.

License: Apache-2.0