codename-hunter 0.1.3

A powerful tool for extracting and enhancing markdown content from web pages

Hunter

Hunter (package name: codename_hunter) makes it easy to convert any web page content into clean, well-formatted Markdown. Built primarily for passing web page content to AI Code Editing tools, but useful for any web content to Markdown conversion needs.

Table of Contents

• Features
• Installation
• Usage
• Configuration
• Development
• Testing
• Contributing
• License

Features

• 🔍 Smart Content Extraction: Seamlessly extract structured content (headings, paragraphs, lists, code blocks, links, images) from any web page.
• 🤖 AI-Powered Enhancement: Optional integration with Together.ai to automatically refine and enhance Markdown formatting
• 📋 Clipboard Integration: Instantly copy the processed Markdown content to your clipboard
• 💾 File Saving: Save extracted content to disk with automatic URL-based filenames and timestamps - helpful when working with AI Code Editors that support file tagging for context.

Installation

Prerequisites

• Python 3.8+
• pip (Python package installer)

Install from PyPI

pip install codename_hunter  # Installs as 'hunter' command-line tool

Install from Source

git clone https://github.com/joenandez/codename_hunter.git
cd codename_hunter
pip install -e .

Package Name Note

While the package is named codename_hunter on PyPI, you'll use it simply as hunter in your terminal:

# Install the package
pip install codename_hunter

# Use the tool
hunter https://example.com/article

Usage

Hunter provides a simple command-line interface to extract and enhance Markdown content from web pages.

Basic Usage

# Extract and enhance content from a URL (copies to clipboard)
hunter https://example.com/article

# Save output to disk (defaults to "hunter_docs" folder)
hunter https://example.com/article -d

# Save to a custom folder
hunter https://example.com/article -d custom_folder

# Save to disk and force directory creation
hunter https://example.com/article -d custom_folder --force-dir

# Extract without AI enhancement
hunter https://example.com/article --no-enhance

# Extract without copying to clipboard
hunter https://example.com/article --no-copy

Command Options

• -d/--save-to-disk [folder]: Save output to disk (defaults to "hunter_docs")
• --force-dir: Create output directory without prompting
• --no-enhance: Disable AI-powered content enhancement
• --no-copy: Disable automatic copying to clipboard

Configuration

Hunter uses environment variables and an optional .env file for configuration.

Together AI Configuration

To enable AI-powered enhancements, you need a Together.ai API key.

Method 1: Environment Variable (Recommended)

export TOGETHER_API_KEY='your_api_key_here'  # On Windows: set TOGETHER_API_KEY=your_api_key_here

To unset the API key:

unset TOGETHER_API_KEY  # On Unix/macOS
set TOGETHER_API_KEY=   # On Windows

Method 2: .env File

Create a .env file in your working directory:

TOGETHER_API_KEY=your_api_key_here

Additional Settings

# Model Selection
TOGETHER_MODEL=mistralai/Mistral-7B-Instruct-v0.2

# Token Limits
TOGETHER_MAX_TOKENS=4000

# Temperature Setting
TOGETHER_TEMPERATURE=0.1

# Output Format
OUTPUT_FORMAT=markdown

# Console Style (dark/light)
CONSOLE_STYLE=dark

Development

Setup Development Environment

1. Clone the repository

git clone https://github.com/joenandez/codename_hunter.git
cd codename_hunter

2. Create a virtual environment

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

3. Install development dependencies

pip install -e ".[dev]"

Project Structure

codename_hunter/
├── hunter/
│   ├── __init__.py
│   ├── __main__.py
│   ├── main.py
│   ├── constants.py
│   ├── formatters.py
│   ├── parsers.py
│   └── utils/
│       ├── ai.py
│       ├── errors.py
│       ├── fetcher.py
│       └── progress.py
├── tests/
│   ├── test_parsers.py
│   ├── test_formatters.py
│   └── test_utils.py
├── project_docs/      # Project documentation
├── hunter_docs/       # Generated documentation
├── assets/           # Project assets
├── .github/          # GitHub configuration
├── README.md
├── CHANGELOG.md
├── CONTRIBUTING.md
├── LICENSE
└── pyproject.toml

Testing

Run the test suite:

pytest

Contributing

This project is currently in a read-only state and is not accepting pull requests. However, we welcome:

• Bug reports and feature requests through GitHub Issues
• Questions and discussions in the Issues section
• Using and forking the project for your own needs

See CONTRIBUTING.md for more details about this policy and how to effectively report issues.

License

This project is licensed under the MIT License - see the LICENSE file for details.