llms-ink

Turn any public link — a site, docs section, page, GitHub repo, PDF or video — into llms.txt or llms-full.txt, from Python. A client for llms.ink with LangChain and LlamaIndex loaders. No required dependencies.

pip install llms-ink

Use

from llms_ink import Client

ink = Client()
index = ink.llms("llmstxt.org")                         # llms.txt: an index of links
docs = ink.llms("docs.astro.build/en/guides/", mode="full")  # llms-full.txt: the content

print(docs.title, docs.kind, docs.tokens)
print(docs.text)

llms() returns a Result: url, kind (site, section, page, repo, document, video), source (native = published by the site, generated = made by llms.ink), mode, title, text, tokens, pages, model, cached, generated_at, and media for videos (duration, downloads, transcript segments). fresh=True skips the cache and rebuilds.

Check a site's published llms.txt:

report = ink.check("llmstxt.org")
print(report.found, report.summary.errors, [i.message for i in report.issues])

API key (optional)

Without a key you share the free anonymous limits (10 requests/minute, 60 new links a day per IP; cached links are free). A key gets its own daily quota:

export LLMS_INK_KEY=lk_...

or Client(key="lk_..."). It is sent as Authorization: Bearer. A bad key raises InvalidKey; it never falls back to anonymous. Ask for a key.

Errors

from llms_ink import Client, LlmsInkError, QuotaExceeded, RateLimited

try:
    Client().llms("example.com")
except (RateLimited, QuotaExceeded) as err:
    print(f"wait {err.retry_after}s")
except LlmsInkError as err:
    print(err.status, err.code, err.detail)

LlmsInkError carries the API's status, code (see the error table), detail and errors. InvalidKey, RateLimited and QuotaExceeded are subclasses.

LangChain

pip install 'llms-ink[langchain]'

from llms_ink.langchain import LlmsInkLoader

docs = LlmsInkLoader(["llmstxt.org", "github.com/AnswerDotAI/llms-txt"], mode="full").load()
docs[0].metadata  # {'url', 'title', 'kind', 'source', 'mode', 'tokens', 'generated_at'}

LlamaIndex

pip install 'llms-ink[llama-index]'

from llama_index.core import VectorStoreIndex
from llms_ink.llama_index import LlmsInkReader

docs = LlmsInkReader(mode="full").load_data(["docs.astro.build/en/guides/"])
index = VectorStoreIndex.from_documents(docs)

Command line

python -m llms_ink llmstxt.org            # llms.txt
python -m llms_ink llmstxt.org --full     # llms-full.txt
python -m llms_ink llmstxt.org --json     # the whole result

Development

python3 -m venv .venv && .venv/bin/pip install -e '.[langchain,llama-index]' pytest
.venv/bin/pytest                # against a local stub server
.venv/bin/pytest -m live        # one real call to https://llms.ink

LLMS_INK_API points the client at another API base URL.

MIT licensed.