llm-fragments-site-text

A fragment loader for LLM that converts websites into markdown plaintext using Trafilatura.

Installation

Install this plugin in the same environment as LLM:

llm install llm-fragments-site-text

Usage

Use -f 'site:URL' to fetch and convert a webpage to plaintext markdown with metadata. This plugin uses Trafilatura to extract clean text content and metadata from websites, and formats it as markdown.

Example:

llm -f 'site:https://example.com/article' "What is this article about?"

The output includes:

• Site name (if available)
• Title (if available)
• Author (if available)
• Date (if available)
• Description (if available)
• Main content (as markdown with links preserved)

Development

To set up this plugin locally, first checkout the code. Then create a new virtual environment:

cd llm-fragments-site-text
python -m venv venv
source venv/bin/activate

Install the dependencies and test dependencies:

pip install -e '.[test]'

To run the tests:

pytest

Dependencies

• httpx: For making HTTP requests
• trafilatura: For extracting clean text and metadata from websites
• llm: The LLM CLI tool this plugin extends