md-summarizer 0.1.66

AI-powered Markdown summarizer that preserves structure and code blocks

Markdown Summarizer

An AI-powered tool that reduces the token size of Markdown documents while preserving their essential structure, code blocks, and meaning. Designed to help fit documentation into AI context windows without losing critical information.

Features

• Token Reduction: Intelligently reduces document size
• Preserves markdown heading hierarchy
• Protects code blocks and technical details
• Bottom-up concurrent processing
• Real-time progress updates
• Streaming API
• Multiple AI provider support
• Token usage tracking and reporting

Usage

from md_summarizer import (
    MarkdownSummarizer, 
    ProgressStatus
)

# Basic usage
summarizer = MarkdownSummarizer()
result = await summarizer.summarize(content)

# Check token usage
usage = summarizer.agent.usage
print(f"Input tokens: {usage.request_tokens}")
print(f"Output tokens: {usage.response_tokens}")
print(f"Reduction: {(1 - usage.response_tokens/usage.request_tokens):.1%}")

# Streaming updates
async for update in summarizer.stream(content):
    if update.status == ProgressStatus.STARTING:
        print(f"Processing {update.total_sections} sections...")
    elif update.status == ProgressStatus.SECTION_COMPLETE:
        print(f"Completed section: {update.section_title}")
    elif update.status == ProgressStatus.COMPLETE:
        print("Done!")
        print(update.content)
        # Show token usage after completion
        usage = summarizer.agent.usage
        print("\nToken Usage:")
        print(f"Input tokens: {usage.request_tokens}")
        print(f"Output tokens: {usage.response_tokens}")
        print(f"Reduction: {(1 - usage.response_tokens/usage.request_tokens):.1%}")

# Progress update types:
# - ProgressStatus.STARTING: total_sections count
# - ProgressStatus.SECTION_COMPLETE: section_title of completed section
# - ProgressStatus.COMPLETE: final content
# - ProgressStatus.ERROR: error details if something fails

# Customize prompts
summarizer.system_prompt = "Your custom system prompt"
summarizer.user_prompt = "Your custom user prompt"

Example output:

Status: ProgressStatus.STARTING, Total Sections: 4
Status: ProgressStatus.SECTION_COMPLETE, Section: Subsection 2.1
Status: ProgressStatus.SECTION_COMPLETE, Section: Section 1
Status: ProgressStatus.SECTION_COMPLETE, Section: Section 2
Status: ProgressStatus.SECTION_COMPLETE, Section: Test Document
Status: ProgressStatus.COMPLETE

Token Usage:
Input tokens: 1250
Output tokens: 450
Reduction: 64.0%

Installation

Install from PyPI:

pip install md-summarizer

Or install from source:

git clone https://github.com/celtiberi/md-summarizer.git
cd md-summarizer
pip install -e .

Configuration

Set environment variables or use .env file:

OPENAI_API_KEY=your-key
MODEL=gpt-3.5-turbo
PROVIDER=openai
LOG_LEVEL=INFO

How it Works

1. Parses markdown into hierarchical sections
2. Processes sections bottom-up (children before parents)
3. Preserves heading levels and structure
4. Provides real-time progress updates
5. Combines processed sections into final document

Development

# Install development dependencies
pip install -e ".[test]"

# Run tests
make test

Contributing

Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first.

License

This project is licensed under the MIT License - see the LICENSE file for details.