josty 0.4.0

What It Is

Josty (from Persian جستن / Jostan — to seek) queries keyless public search backends in parallel, fuses rankings with Reciprocal Rank Fusion (RRF), canonicalizes URLs, strips tracking telemetry, and extracts bounded Markdown from target pages.

It provides a dependable, structured search subprocess and async Python API without requiring search API keys, background daemons, or heavy browser dependencies.

Installation

# Recommended: Instant cached execution (zero persistent virtualenv overhead)
uvx josty "Python 3.13 changes" --limit 5

# Global CLI installation via uv:
uv tool install josty

# Alternative installation via pipx or standard pip:
pipx install josty
pip install josty

Quickstart

1. CLI Usage

# Basic web search (top 5 results)
josty "Python 3.13 features" --limit 5

# Developer profile (boosts GitHub, PyPI, crates.io, MDN, StackOverflow)
josty "FastAPI dependency injection" --profile dev --limit 5

# Academic profile (boosts arXiv, PubMed, IEEE, Nature, OpenAlex)
josty "retrieval augmented generation" --profile academic --limit 5

# Domain filtering (up to 5 domains)
josty "httpx connection reset" --site github.com --site stackoverflow.com

# Open Source discovery mode
josty "document indexing" --mode oss --github

# Extract clean, bounded Markdown from top result pages
josty "RRF rank fusion algorithm" --limit 3 --fetch

2. Versioned JSON Output

stdout emits pure, parseable JSON conforming to a strict schema contract (schema_version: "1.0"):

{
  "schema_version": "1.0",
  "query": "Python 3.13 features",
  "status": "complete",
  "count": 3,
  "partial": false,
  "cached": false,
  "providers": [
    { "provider": "bing,brave,duckduckgo", "ok": true, "result_count": 5 },
    { "provider": "google,mojeek,startpage", "ok": true, "result_count": 5 }
  ],
  "results": [
    {
      "title": "What's New In Python 3.13 — Python 3.13.0 documentation",
      "url": "https://docs.python.org/3/whatsnew/3.13.html",
      "snippet": "Python 3.13 includes an experimental free-threaded build mode...",
      "sources": ["bing,brave,duckduckgo", "google,mojeek,startpage"],
      "score": 0.032787,
      "content": "## What's New In Python 3.13\n\nThis article explains the new features...",
      "extraction_method": "trafilatura"
    }
  ]
}

Python API & Integrations

Direct Async Python API

import asyncio
from josty import Josty

async def main():
    engine = Josty(profile="dev")
    run = await engine.research_run("Linux kernel initial release year", limit=3)
    
    if run.status != "failed":
        for result in run.results:
            print(f"[{result.title}]({result.url})\n{result.snippet}\n")

asyncio.run(main())

Function Calling Tool Schema

search_tool_definition = {
    "type": "function",
    "function": {
        "name": "web_search",
        "description": "Search the web for up-to-date documentation and technical resources. Returns ranked results.",
        "parameters": {
            "type": "object",
            "properties": {
                "query": {
                    "type": "string",
                    "description": "The search query."
                },
                "fetch": {
                    "type": "boolean",
                    "description": "Set to true to fetch and extract clean Markdown page content.",
                    "default": False
                },
                "profile": {
                    "type": "string",
                    "enum": ["general", "dev", "academic"],
                    "description": "Ranking profile boosting authoritative technical or academic domains.",
                    "default": "general"
                },
                "mode": {
                    "type": "string",
                    "enum": ["plain", "exact", "oss"],
                    "description": "Search mode ('oss' filters for open-source repositories).",
                    "default": "plain"
                }
            },
            "required": ["query"]
        }
    }
}

Technical Specifications & Architecture

graph TD
    Query["Search Query"] --> Cache{"SQLite WAL Cache<br/>Tiered TTL (d:30m/news:1h/w:2h, else 6h)<br/>5k rows / 50 MB, SERP-only"}
    
    Cache -- Cache Hit --> Out["<b>Pure JSON Output</b><br/>(schema_version: 1.0)"]
    
    Cache -- Cache Miss --> Fanout["<b>Async Parallel Fanout</b><br/>(DDGS Engine Groups)"]
    
    Fanout --> B1["Backend Group 1<br/>(Bing, Brave, DDG)"]
    Fanout --> B2["Backend Group 2<br/>(Google, Mojeek, Startpage)"]
    Fanout --> GH["GitHub Search<br/>(Optional --github)"]
    
    B1 --> Circuit["<b>Circuit Breakers</b><br/>(Sliding Window)"]
    B2 --> Circuit
    GH --> Circuit
    
    Circuit --> RRF["<b>Domain-Weighted RRF Fusion</b><br/>(k=60 + Dev/Academic Profiles)"]
    
    RRF --> Canon["<b>URL Canonicalization</b><br/>(RFC 3986 + Tracking Stripper)"]
    
    Canon --> Fetch{"<b>--fetch Active?</b>"}
    
    Fetch -- Yes --> Traf["Trafilatura Extractor<br/>Bounded Markdown"]
    Fetch -- No --> Out
    Traf --> Out

    style Query fill:#dbeafe,stroke:#1e40af,stroke-width:2px;
    style Out fill:#dcfce7,stroke:#15803d,stroke-width:2px;
    style RRF fill:#fef3c7,stroke:#b45309,stroke-width:2px;

Roadmap

See ROADMAP.md for v0.4.0: error_kind=empty, diagnose challenged, no hidden query rewrite, and a bounded cache. Query relaxation, news engine filters, and hard host floors are out of scope.

Development

# Clone repository
git clone https://github.com/Alih-b/josty.git
cd josty

# Install in editable mode with dev dependencies
python -m pip install -e ".[dev]"

# Run test suite
pytest -q

# Lint and check code style
ruff check .

License

MIT © Ali Bayest. See LICENSE for details.