apicrawl 0.1.0

Crawl API documentation (OpenAPI, Swagger, ReadMe, Mintlify, Fern, llms.txt, plain HTML) into structured, searchable markdown

ApiCrawl

Crawl API documentation into structured, searchable markdown.

Point it at any API docs URL — an OpenAPI/Swagger spec, a Swagger UI / Redoc / Stoplight / Scalar page, ReadMe / Mintlify / Fern hosted docs, a Postman collection, a Google Discovery document, an llms.txt index, or plain HTML docs — and it discovers the underlying spec where one exists, crawls the pages where one doesn't, classifies the content with an LLM, extracts authentication instructions, and writes everything as a local markdown tree you can grep, read, or feed to any tool.

pip install apicrawl
playwright install chromium   # used to render JS-heavy docs sites

export GOOGLE_API_KEY=...     # LLM access is required (Gemini primary)
export GROQ_API_KEY=...       # optional fallback provider

apicrawl https://petstore3.swagger.io --output ./api-docs

Output layout:

api-docs/<catalog_id>/
  index.md            # API name, metadata, description, auth instructions
  manifest.json       # listing of everything ingested (also the completion marker)
  sections/<slug>.md  # docs pages / spec tag groups (markdown + frontmatter)
  endpoints/<slug>.md # one file per endpoint: parameters, examples, TypeScript types

Library usage:

import asyncio
from apicrawl import ingest_to_dir

result = asyncio.run(ingest_to_dir("https://docs.example.com/api", "./api-docs"))
print(result.entry.name, result.pages_ingested)

Custom storage — implement IngestionSink and receive the parsed catalog entry, sections, and endpoints as plain pydantic models, streamed in batches:

from apicrawl import IngestionSink, ingest

class MySink(IngestionSink):
    async def emit_sections(self, sections): ...
    async def emit_endpoints(self, endpoints): ...

asyncio.run(ingest("https://docs.example.com/api", MySink()))

Notes:

• LLM keys are required — page classification and auth extraction are LLM-powered. Set GOOGLE_API_KEY (and optionally GROQ_API_KEY) in the environment or a .env file.
• Node.js is optional — if a node binary is on PATH, endpoint pages include generated TypeScript request/response types (via a bundled openapi-typescript). Without Node, ingestion still works; the TS sections are simply omitted.

License: Apache-2.0