ai-url-aggregator

Note: This is a small experimental library, provided as-is.

ai-url-aggregator is a Python tool that leverages Perplexity and OpenAI to search the internet for relevant URLs: it expands your query into several searches, runs them, normalizes, deduplicates and checks the results, and then classifies and ranks them with GPT.

Features

1. Query Expansion
   An OpenAI model turns your query into several search queries, varying the wording, the language and the angle (for example, directories), to find more of the websites you’re after.
2. Web Search
   Perplexity’s Search API runs those queries (up to 5 per request, each request billed once) and returns ranked pages with their title and an extract. Every URL comes from the search index, so none are made up.
3. Clean & Filter
   • Normalizes URLs: adds https:// to bare domains such as www.example.com, lowercases the host, and drops trailing slashes and fragments.
   • Removes duplicates, preferring https:// over http:// and treating www.example.com and example.com as the same site.
4. Online Check
   • Verifies each URL’s availability (final status 200 or 403) concurrently with requests, following redirects and falling back from HEAD to GET.
   • Skips URLs, including redirect targets, that resolve to private, loopback or other non-public addresses. This is a best-effort guard against SSRF: the host is resolved again when connecting, so it doesn't stop DNS rebinding.
5. Relevance Ranking
   • Uses an OpenAI model to pick the pages relevant to your query, judging each by its title and extract, classifying it (official site, directory, article or other) and ordering them from most to least relevant. Only URLs from the search results are kept.

Installation

1. Install via PyPI

Requires Python 3.11 or later.

uv add ai-url-aggregator

The package is imported as ai_url_aggregator, with underscores.

2. Set Environment Variables

You must provide your Perplexity and OpenAI API keys:

export PERPLEXITY_API_KEY="PERPLEXITY_API_KEY"
export OPENAI_API_KEY="OPENAI_API_KEY"

Replace "PERPLEXITY_API_KEY" and "OPENAI_API_KEY" with your actual API keys. They are read the first time an API is called, so importing the package works without them. You can also pass your own clients instead (see Configuration).

3. (Optional) Install from Source

1. Clone or Download this repository.
2. Install Dependencies:

   uv sync
   

   This ensures all required libraries (like openai, perplexityai, requests, etc.) are installed, along with pytest.
3. Run the Tests:

   uv run pytest
   

   The tests use fake API clients and a local HTTP server, so they need neither API keys nor internet access.

How It Works

1. expand_query(query: str, num_queries: int = 5, ...) -> list[str]

   • Returns query followed by up to num_queries - 1 related queries written by an OpenAI model (gpt-6-luna with low reasoning effort by default).
   • If the model refuses or its output is cut off, it logs a warning and returns just query.

2. search_web(queries: Sequence[str], ...) -> list[SearchResult]

   • Runs the queries through Perplexity’s Search API, 5 per request.
   • Returns a SearchResult with the url, title and snippet of each page found.

3. keep_https(urls: list[str]) -> list[str]

   • Normalizes the URLs (see normalize_url) and removes duplicates, preferring https:// and ignoring www..
   • Drops anything that isn’t an http(s) URL.

4. check_urls_online(urls: list[str], ...) -> list[str]

   • Checks the URLs concurrently and keeps the reachable ones (final status 200 or 403).
   • Follows redirects, and tries GET when HEAD fails or returns another status, without downloading the body.
   • Treats URLs that resolve to non-public addresses as offline unless allow_private_addresses=True.

5. search_for_web_urls(query: str, num_queries: int = 5, ...) -> list[SearchResult]

   • Brings all the above together:
     1. Expands the query.
     2. Searches the web.
     3. Normalizes and deduplicates the results, preferring HTTPS.
     4. Returns the results whose URLs are online.

6. get_top_relevant_websites(results: Sequence[SearchResult | str], query: str | None = None, ...) -> list[Website]

   • Uses an OpenAI model to select the results relevant to query, judging each by its title and snippet, and returns them as Website(name, url, kind), most relevant first. kind is "official_site", "directory", "article" or "other".
   • Accepts plain URLs too, although the model then has only the URL to go on.
   • Discards any URL the model returns that isn’t in the list.
   • Returns an empty list, and logs a warning, if the model refuses or its output is cut off.

The OpenAI calls use the Responses API with store=False, so the responses aren’t stored for later retrieval.

Configuration

search_for_web_urls accepts these keyword arguments:

expand_query and get_top_relevant_websites accept the openai_* arguments, search_web accepts max_results, country and perplexity_client, and check_urls_online accepts timeout, max_workers and allow_private_addresses.

For a model without reasoning, such as gpt-4o, leave the reasoning effort out and set a temperature instead:

results = search_for_web_urls(
    query="...",
    openai_model="gpt-4o",
    openai_reasoning_effort=None,
    openai_temperature=0,
)

You can pass your own clients; any client you leave out is created from the environment variables:

from openai import OpenAI
from perplexity import Perplexity
from ai_url_aggregator import search_for_web_urls

results = search_for_web_urls(
    query="...",
    perplexity_client=Perplexity(api_key="..."),
    openai_client=OpenAI(api_key="..."),
)

Progress is reported through the standard logging module, under the ai_url_aggregator logger. To see it:

import logging

logging.basicConfig(level=logging.INFO)

Usage Example

Once installed and your environment variables are set, you can do:

from pprint import pprint

from ai_url_aggregator import (
    search_for_web_urls,
    get_top_relevant_websites
)

# Example query:
query = "Give me a list of all the real estate agencies in Uruguay."

# Step 1: Search the web and keep the unique results that are online
results = search_for_web_urls(query=query, country="UY")

print("--- Online URLs ---")
pprint([result.url for result in results])

# Step 2: Get the websites relevant to the query, classified and ranked
most_important_websites = get_top_relevant_websites(results, query=query)

print("--- Most Important Websites ---")
pprint(most_important_websites)

Example output of step 2 (illustrative; results vary between runs):

[Website(name='InfoCasas', url='https://www.infocasas.com.uy', kind='directory'),
 Website(name='CasasWeb', url='https://www.casasweb.com.uy', kind='directory'),
 Website(name='Mercado Libre Inmuebles', url='https://www.mercadolibre.com.uy/inmuebles', kind='directory'),
 Website(name='Uruguay Inmobiliarias', url='https://www.uruguayinmobiliarias.com', kind='directory')]

License

This project is distributed under the MIT License. See LICENSE for more information.

All suggestions and improvements are welcome!