sitejuicer 0.2.1

Convert URLs to markdown using Jina Reader

SiteJuicer 🧃

Extract clean, readable content from any website with one command

Installation • Quick Start • Features • Documentation • Examples • Contributing

📖 Overview

SiteJuicer is a powerful command-line utility that converts web content to clean, readable formats like Markdown, HTML, and JSON using Jina Reader. It helps extract valuable content from websites while removing clutter like ads, navigation menus, and other distractions.

Perfect for: Content extraction, web scraping, research, offline reading, and data collection.

✨ Features

• 📝 Clean content extraction - Remove ads, navigation menus, and other distractions
• 🔄 Multiple output formats - Convert to Markdown, HTML, or JSON
• 🖼️ Image extraction - Download and save images from the page (optional)
• 🔗 Link extraction - Identify and list all links in the content (optional)
• 📚 Table of contents - Generate a TOC based on headings (optional)
• 📊 Batch processing - Process multiple URLs at once from a CSV or text file
• 🌐 API server - Run as a local API server with FastAPI
• 🖥️ Web UI - Use the graphical interface with Streamlit
• 🔑 API Key Support - Use your own Jina Reader API key for enhanced features

🚀 Quick Start

# Install
pip install sitejuicer

# Convert a URL to markdown (saved as {title}.md)
sitejuicer https://example.com

# Specify an output format
sitejuicer https://example.com --format html

📦 Installation

# From PyPI (recommended)
pip install sitejuicer

# Optional UI components
pip install "sitejuicer[ui]"

# For developers
pip install "sitejuicer[dev]"

Installation with Optional Components

# For HTML conversion
pip install sitejuicer[html]

# For API server
pip install sitejuicer[server]

# For web UI
pip install sitejuicer[ui]

# For all features
pip install sitejuicer[all]

From Source

git clone https://github.com/jakerains/sitejuicer.git
cd sitejuicer
pip install -e .

📋 Usage

Basic Usage

# Convert a URL to markdown (output will be saved to {title}.md)
sitejuicer https://example.com

# Specify an output filename
sitejuicer https://example.com my-output

# Specify output format
sitejuicer https://example.com --format html
sitejuicer https://example.com --format json

Using API Keys

# Save a Jina Reader API key for enhanced features
sitejuicer --api YOUR_API_KEY_HERE

# Clear a previously saved API key
sitejuicer --clear-api

When an API key is configured, SiteJuicer will automatically use it to access the Jina Reader API with enhanced features. If there's an issue with the API key, SiteJuicer will gracefully fall back to the public service.

API keys must be in the format jina_XXXXXXXXXXXXXXXXXXXXXXXX_XXXXXXXXXXXXXXXX and are stored securely at ~/.sitejuicer/config.ini with restricted permissions.

Output Format Options

# Apply a template to the output
sitejuicer https://example.com --template blog

# Download and include images locally
sitejuicer https://example.com --include-images

# Specify a directory for downloaded images
sitejuicer https://example.com --include-images --image-dir ./my-images

Content Filtering Options

# Extract only the main content
sitejuicer https://example.com --main-content-only

# Include only specific HTML elements
sitejuicer https://example.com --include-elements p,h1,h2,h3,img

# Exclude specific HTML elements
sitejuicer https://example.com --exclude-elements aside,nav,footer

Batch Processing

# Process URLs from a text file (one URL per line)
sitejuicer --batch urls.txt --output-dir ./converted

# Process URLs from a CSV file
sitejuicer --batch data.csv --output-dir ./converted

🌐 API Server Mode

# Start the API server on the default port (8000)
sitejuicer --server

# Start on a custom port
sitejuicer --server --port 5000

Once started, you can access the API documentation at http://localhost:8000/docs

🖥️ Web UI Mode

# Start the web UI
sitejuicer --ui

This will launch a Streamlit-based web interface at http://localhost:8501

🔄 How It Works

SiteJuicer uses the Jina Reader API to extract clean content from web pages. The processing pipeline includes:

1. Fetching the web content using the Jina Reader API
2. Optional content filtering to extract only the main content
3. Optional image downloading and reference updating
4. Optional link extraction and listing
5. Optional table of contents generation
6. Formatting and saving the content in the desired format (Markdown, HTML, or JSON)

🧩 Programmatic Usage

SiteJuicer can also be used programmatically in your Python scripts:

from sitejuicer.core import fetch_content, format_markdown, save_markdown

# Fetch content from a URL with options
result = fetch_content(
    "https://example.com", 
    options={
        "main_content_only": True,
        "include_images": True,
        "image_dir": "images/",
        "api_key": "jina_2caa021406de46c7837bc05cce50e14d_qcv8AqU_xS_fCtKz3nJ2qS5IcDk"  # Optional: use your Jina Reader API key
    }
)

# Extract content and title from the result
content = result["content"]
title = result["title"]
url = result["url"]

# Format as markdown with options
markdown = format_markdown(
    content, 
    url=url, 
    title=title
)

# Save to a file with specific format
save_markdown(
    markdown, 
    "example.md", 
    url, 
    title, 
    output_format="markdown"
)

See the examples/usage_example.py file for a complete example.

🔌 API Integration

SiteJuicer integrates with the Jina Reader API to process web content. Key details:

• Endpoint: https://r.jina.ai/{url}
• Authentication: Bearer token with format Bearer jina_XXXXXXXXXXXXXXXXXXXXXXXX_XXXXXXXXXXXXXXXX
• Response: JSON object with processed content in requested format
• Error Handling: Graceful fallback to public service when API is unavailable or authentication fails

For detailed information about the API integration and authentication, see:

• Jina API Integration
• Jina API Authentication

🐳 Docker Support

SiteJuicer can be run in Docker containers for isolated deployment. See Docker Usage for details.

🛠️ For Developers

Building and Publishing to PyPI

SiteJuicer is available on PyPI, which means users can install it with pip install sitejuicer. If you're a developer wanting to build and publish updates to PyPI, follow these steps:

1. Update version number in sitejuicer/__init__.py

2. Build the distribution packages:

   python -m pip install --upgrade build twine
   python -m build
   

3. Check your distribution packages:

   twine check dist/*
   

4. Upload to TestPyPI (optional but recommended):

   twine upload --repository-url https://test.pypi.org/legacy/ dist/*
   

5. Install from TestPyPI to verify (in a new environment):

   pip install --index-url https://test.pypi.org/simple/ sitejuicer
   

6. Upload to PyPI:

   twine upload dist/*
   

For more detailed instructions on packaging Python projects, see the Python Packaging User Guide.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

---

Created with ❤️ by Jake Rains

If you find SiteJuicer useful, consider starring the repository or buying me a coffee.