fossick

fossick gets text off the web and into a program. You give it a search query, a URL, an arXiv id, a YouTube link, a GitHub repository or a local file, and it gives back markdown that a person or a language model can read. It was written for AI agents that need to look things up, so its functions return plain data (strings, lists and dictionaries) that can be printed or passed to a model as they are.

The library works in layers. search sends a query to several search engines at once and merges their answers into one ranked list. read takes a single target and returns its text. Under read, fetch downloads a page and to_md turns its HTML into markdown. Many sites refuse plain downloads from scripts, so fetch can also load a page in a headless browser, in a stealth browser built to pass bot checks, or in a copy of Chrome you have logged in to. For pages that need clicking and typing, cdp_connect drives that Chrome directly. Everything here is also available from the shell as the fossick command and to agents as an MCP server.

Install

uv add fossick

Searching and plain downloads need nothing else. Rendering JavaScript, the stealth fetcher and google run a headless browser through scrapling. If those calls raise BrowserUnavailable, run scrapling install to download its browsers. The browser automation in fossick.cdp drives your installed Google Chrome.

Quick start

Three calls cover most uses. search returns a list of results, each a dictionary with the page’s title, its URL in href and a snippet in body. score is the merged ranking score, and engines names the search engines that returned that page. method='flashrank' reorders the results with a small relevance model, which needs pip install flashrank; without a method the merged order is kept.

results = search('koshas github fts semantic code graph', method='flashrank', n=5)
for r in results: print(r['score'], r['engines'], r['title'], r['href'])

0.030536 ['brave', 'yahoo'] github.com › codegraph-ai › CodeGraphGitHub - codegraph-ai/CodeGraph: CodeGraph builds a semantic... https://github.com/codegraph-ai/CodeGraph
0.016129 ['yahoo'] vedicreader.github.io › kosha › graphgraph – koshas - vedicreader.github.io https://vedicreader.github.io/kosha/graph.html
0.015873 ['brave'] fts · GitHub Topics · GitHub https://github.com/topics/fts?o=desc&s=updated
0.016393 ['yahoo'] pypi.org › project › koshaskoshas · PyPI https://pypi.org/project/koshas/
0.015625 ['brave'] GitHub - colbymchenry/codegraph: Pre-indexed code knowledge graph, auto syncs on code changes, for Claude Code, Codex, Gemini, Cursor, OpenCode, AntiGravity, Kiro, and Hermes Agent — fewer tokens, fewer tool calls, 100% local https://github.com/colbymchenry/codegraph

res = research('sqlite WAL mode vs journal mode', n=5)
print(res['digest'][:500])
print([s['href'] for s in res['sources']])

## Write-Ahead Logging
https://www.sqlite.org/wal.html

Write-Ahead Logging

Table Of Contents

# 1\. Overview

The default method by which SQLite implements atomic commit and rollback is a rollback journal. Beginning with version 3.7.0 (2010-07-21), a new "Write-Ahead Log" option (hereafter referred to as "WAL") is available.

There are advantages and disadvantages to using WAL instead of a rollback journal. Advantages include:

1. WAL is significantly faster in most scenarios. 
  2. WAL provid
['https://www.sqlite.org/wal.html', 'https://blog.sqlite.ai/journal-modes-in-sqlite', 'https://mohit-bhalla.medium.com/understanding-wal-mode-in-sqlite-boosting-performance-in-sql-crud-operations-for-ios-5a8bd8be93d2', 'https://til.simonwillison.net/sqlite/enabling-wal-mode', 'https://fly.io/blog/wal-mode-in-litefs/']

page = read('https://en.wikipedia.org/wiki/Web_scraping')
print(page.text[:400])

The legality of web scraping varies across the world. In general, web scraping may be against the terms of service of some websites, but the enforceability of these terms is unclear.[11]

In the United States, website owners can use three major legal claims to prevent undesired web scraping: (1) copyright infringement (compilation), (2) violation of the Computer Fraud and Abuse Act ("CFAA"), and (

One function for every source

read is the function to reach for when you have a target and want its text. It first calls what_is to decide what the target is, and the answer names the reader it will use: dir for a local folder, file for a local file, arxiv for an arXiv URL or id, youtube for a YouTube link, github for a repository, ghfile for a file on GitHub, pdf for a URL ending in .pdf or /pdf, and web for any other http or https URL. A path that exists on disk wins over every URL rule.

Whatever the reader, the result has the same seven fields:

Here read opens this repository’s own README:

target = str(repo_root()/'README.md')
assert what_is(target) == 'file'
result = read(target)
assert sorted(result) == ['kind', 'meta', 'ok', 'skipped', 'source', 'text', 'title']
assert (result.ok, result.kind, result.title) == (True, 'file', 'fossick')
assert '# fossick' in result.text and result.skipped is None
result.kind, result.title, result.ok

A folder or a GitHub repository is a tree of files rather than one document, so read does not turn it into text. It returns ok=True with an empty text and puts the local path in meta['path'], and the caller decides which files to open. For a GitHub repository that path is a fresh clone in fossick’s cache. With pages=True, a PDF comes back as a list of (page_index, text) pairs, counted from 0, so that a citation can point at a page.

Sometimes a target is recognised but yields nothing: a site serves a bot check instead of the page, a YouTube video has no English captions, or a URL that looked like a PDF returns something else. read reports these with ok=False and a reason in skipped, and does not raise. It raises ValueError only when the target is none of the kinds above.

Modules

Common calls

search('query', n=10, region='auto', timelimit='y')   # results from the past year
research('query', n=5)                    # search, read the top pages, return a digest that cites them
fetch(url, auto=True)                     # try stronger download methods until one is not blocked
crawl(url, max_pages=5, same_domain=True) # follow links from url, staying on its site
google('query', n=10)                     # Google's own ranking, through the stealth browser
cdp = await cdp_connect(); pg = await cdp.open_page(url); print(await pg.snapshot())

From the shell: fossick search "q", fossick fetch URL --auto, fossick research "q". For agents: uvx fossick-mcp, or fossick-mcp --http to serve over HTTP instead of standard input and output.

A logged-in Chrome

Some pages are only shown to a logged-in user. fossick reaches them through a separate copy of Chrome that runs with remote debugging switched on, on port 9223, with its own profile in ~/.cache/fastcdp/cdp-chrome. cdp_connect starts it the first time it is needed. Log in to a site once in that window and the cookies stay in the profile, so later calls such as fetch(url, session=True) arrive logged in. fetch(url, auto=True) and read fall back to this Chrome when every other method is blocked.

To start it by hand in a visible window, for logging in, run await cdp_setup(headless=False). The fastcdp-setup command, installed with fossick’s fastcdp dependency, adds a launcher for the same Chrome and profile to your applications.