sitejuicer 0.2.6

Convert URLs to markdown using Jina Reader

SiteJuicer 🧃

Extract clean, readable content from any website with one command

Installation • Quick Start • Features • Documentation • Examples • Contributing

📖 Overview

SiteJuicer is a powerful command-line utility that converts web content to clean, readable formats like Markdown, HTML, and JSON using Jina Reader. It helps extract valuable content from websites while removing clutter like ads, navigation menus, and other distractions.

SiteJuicer is fully cross-platform, working seamlessly on Windows, macOS, and Linux systems.

Perfect for: Content extraction, web scraping, research, offline reading, and data collection.

✨ Features

• 📝 Clean content extraction - Remove ads, navigation menus, and other distractions
• 🔄 Multiple output formats - Convert to Markdown, HTML, or JSON
• 🖼️ Image extraction - Download and save images from the page (optional)
• 🔗 Link extraction - Identify and list all links in the content (optional)
• 📚 Table of contents - Generate a TOC based on headings (optional)
• 📊 Batch processing - Process multiple URLs at once from a CSV or text file
• 🌐 API server - Run as a local API server with FastAPI
• 🖥️ Web UI - Use the graphical interface with Streamlit
• 🔑 API Key Support - Use your own Jina Reader API key for enhanced features

🚀 Quick Start

# Install
pip install sitejuicer

# Convert a URL to markdown (saved as {title}.md)
sitejuicer https://example.com

# Specify an output format
sitejuicer https://example.com --format html

📦 Installation

# Install from PyPI
pip install sitejuicer

# Optional UI components
pip install "sitejuicer[ui]"

Installation with Optional Components

# For HTML conversion
pip install sitejuicer[html]

# For API server
pip install sitejuicer[server]

# For web UI
pip install sitejuicer[ui]

# For all features
pip install sitejuicer[all]

📋 Usage

Basic Usage

# Convert a URL to markdown (output will be saved to {title}.md)
sitejuicer https://example.com

# Specify an output filename
sitejuicer https://example.com my-output

# Specify output format
sitejuicer https://example.com --format html
sitejuicer https://example.com --format json

Using API Keys

# Save a Jina Reader API key for enhanced features
sitejuicer --api YOUR_API_KEY_HERE

# Clear a previously saved API key
sitejuicer --clear-api

When an API key is configured, SiteJuicer will automatically use it to access the Jina Reader API with enhanced features. If there's an issue with the API key, SiteJuicer will gracefully fall back to the public service.

API keys must be in the format jina_XXXXXXXXXXXXXXXXXXXXXXXX_XXXXXXXXXXXXXXXX and are stored securely at ~/.sitejuicer/config.ini with restricted permissions.

Managing PyPI Tokens

# Save your PyPI token for package publishing
sitejuicer --pypi-token pypi-YOUR_TOKEN_HERE

# Clear a previously saved PyPI token
sitejuicer --clear-pypi-token

# Generate a .pypirc file using your saved token
sitejuicer --generate-pypirc

This feature allows you to securely store your PyPI token and automatically generate a valid .pypirc file for publishing packages. The token is stored securely at ~/.sitejuicer/config.ini with restricted permissions, and the generated .pypirc file will have appropriate permissions to protect your credentials.

Output Format Options

# Apply a template to the output
sitejuicer https://example.com --template blog

# Download and include images locally
sitejuicer https://example.com --include-images

# Specify a directory for downloaded images
sitejuicer https://example.com --include-images --image-dir ./my-images

Content Filtering Options

# Extract only the main content
sitejuicer https://example.com --main-content-only

# Include only specific HTML elements
sitejuicer https://example.com --include-elements p,h1,h2,h3,img

# Exclude specific HTML elements
sitejuicer https://example.com --exclude-elements aside,nav,footer

Batch Processing

# Process URLs from a text file (one URL per line)
sitejuicer --batch urls.txt --output-dir ./converted

# Process URLs from a CSV file
sitejuicer --batch data.csv --output-dir ./converted

🌐 API Server Mode

# Start the API server on the default port (8000)
sitejuicer --server

# Start on a custom port
sitejuicer --server --port 5000

Once started, you can access the API documentation at http://localhost:8000/docs

🖥️ Web UI Mode

# Start the web UI
sitejuicer --ui

This will launch a Streamlit-based web interface at http://localhost:8501

🔄 How It Works

SiteJuicer uses the Jina Reader API to extract clean content from web pages. The processing pipeline includes:

1. Fetching the web content using the Jina Reader API
2. Optional content filtering to extract only the main content
3. Optional image downloading and reference updating
4. Optional link extraction and listing
5. Optional table of contents generation
6. Formatting and saving the content in the desired format (Markdown, HTML, or JSON)

🧩 Programmatic Usage

SiteJuicer can also be used programmatically in your Python scripts:

from sitejuicer.core import fetch_content, format_markdown, save_markdown

# Fetch content from a URL with options
result = fetch_content(
    "https://example.com", 
    options={
        "main_content_only": True,
        "include_images": True,
        "image_dir": "images/",
        "api_key": "jina_2caa021406de46c7837bc05cce50e14d_qcv8AqU_xS_fCtKz3nJ2qS5IcDk"  # Optional: use your Jina Reader API key
    }
)

# Extract content and title from the result
content = result["content"]
title = result["title"]
url = result["url"]

# Format as markdown with options
markdown = format_markdown(
    content, 
    url=url, 
    title=title
)

# Save to a file with specific format
save_markdown(
    markdown, 
    "example.md", 
    url, 
    title, 
    output_format="markdown"
)

See the examples/usage_example.py file for a complete example.

🔌 API Integration

SiteJuicer integrates with the Jina Reader API to process web content. Key details:

• Endpoint: https://r.jina.ai/{url}
• Authentication: Bearer token with format Bearer jina_XXXXXXXXXXXXXXXXXXXXXXXX_XXXXXXXXXXXXXXXX
• Response: JSON object with processed content in requested format
• Error Handling: Graceful fallback to public service when API is unavailable or authentication fails

For detailed information about the API integration and authentication, see:

• Jina API Integration
• Jina API Authentication

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🤝 Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines.

---

Created with ❤️ by Jake Rains

If you find SiteJuicer useful, consider starring the repository or buying me a coffee.