sourcesQL

Web research you can query, inspect and keep working with.

sourcesQL is an open-source research engine and MCP App. A connected assistant turns your request into SQL, searches the web, builds a structured dataset and presents the finished table in chat. You can inspect its sources, expand a row into a follow-up investigation, amend a field, review a later refresh and move the saved research between compatible installations.

Use it when the answer needs rows: a vendor comparison, a project pipeline, an organization inventory or a collection of claims with supporting sources. The result stays available for further SQL instead of becoming another table trapped in a conversation.

Keep working with saved research

Saved research includes the result, its supporting sources and the investigations that follow from it.

• Expand. Choose a row or a field within it and ask a follow-up question. The findings open in a linked child table, with the research tree keeping the path back to the parent. For example, expand one company into a table of its announced facilities.
• Amend. Correct a value and record why. Amendments have a separate revision history, so original observations stay available. Follow-up SQL can use the original result or a pinned amended view.
• Refresh. Check a field, row or table again, then review the proposed changes. Explicitly accept supported updates or keep the current values; earlier observations and amendments remain available.
• Check for updates. Revisit selected findings, look for new entities, or do both as separate research scopes. Review dated evidence for additions, changed claims and contradictions. Publish your accepted choices as a new table; missing search hits do not remove existing findings. The first version works with complete tables up to 100 rows and 200 KB.
• Compact checkpoints. Save the question, selected rows and fields, exact revisions, query details and pending job IDs in a small continuation record. Resume in another compatible client without copying the conversation or restarting research. Missing data and changed revisions are reported explicitly.
• Portable snapshots. Export a research tree as a bundle and restore it in another compatible sourcesQL store. Bundles carry original results, sources, amendments, linked investigations, refresh decisions, checkpoints and manual update review history. Reopen saved findings without repeating web discovery.

See the research lifecycle and snapshot/export contracts for supported scopes, review behavior and bundle limits.

How research works

The assistant plans the search terms. sourcesQL executes the query and keeps the data, evidence and execution receipt.

1. Discover broadly. Use complementary search terms across terminology, source types and angles. Search results are candidate pages, not verified entities.
2. Reduce with SQL. Merge overlapping results, deduplicate sources, filter available fields and join saved datasets before spending on semantic processing.
3. Extract selectively. Apply semantic matching, extraction, scoring or normalization to candidates that still need it, using explicitly configured models.
4. Present the finished table. Keep intermediate work behind the scenes. Open the final selection with readable labels, useful source links and an intentional row order.
5. Continue from saved research. Reopen, query, correct, expand or refresh a result without repeating discovery unnecessarily.

SQL filters run before semantic work where the admitted query plan permits it. A final LIMIT bounds returned rows; it does not make earlier search or model work cheaper.

This query runs against the included fictional fixtures, with no API keys or network calls:

SELECT
  SEM_EXTRACT(content, 'project name explicitly stated') AS project,
  SEM_EXTRACT(
    content,
    'planned capacity in MW, number only',
    type := 'DOUBLE'
  ) AS planned_capacity_mw,
  url AS source_url
FROM WEB_SEARCH('fictional Aurora Birch campuses')
WHERE SEM_MATCH(content, 'describes a specific data-center campus')
ORDER BY project, source_url;

The engine supports WEB_SEARCH, WEB_FETCH, SEM_MATCH, SEM_EXTRACT, SEM_EXTRACT_ALL, SEM_SCORE and SEM_NORM. It accepts an explicit subset of DuckDB-style SELECT and nonrecursive WITH queries. See the SQL reference for signatures, supported staging and refusals. Operator availability also depends on the selected provider profile.

The interactive workspace

The MCP App adds an interactive table to compatible clients, including ChatGPT, Claude and Cursor. Use contextual table actions or ask the connected assistant to continue from a saved result. Linked tables are accessible through the research tree. Other MCP clients can use the same tools without rendering the app.

The client guide covers connection steps and supported interactions. Host support varies: Cursor can render the table while leaving actions that require agent messaging disabled.

The app decorates column labels and source addresses without rewriting data. Missing values stay missing. Search excerpts remain identified as excerpts, and a supporting quote establishes what a source says rather than proving the claim true.

Choose an installation

Every path starts in fixture mode. Live search and model calls require credentials, an enabled trusted profile and explicit execution budgets.

Local Docker

Clone this repository and run from its root with Docker running:

git clone https://github.com/JohnnyFiv3r/sourcesQL.git
cd sourcesQL
docker compose up --build -d sourcesql-app

Open localhost:3117. The initial workspace uses synthetic data. Its named volume preserves results and corrections across restarts.

To connect a local MCP client over stdio, first build the app image:

docker compose build sourcesql-mcp

Then configure the client to run docker compose run --rm -T --no-deps sourcesql-mcp from this checkout. The Docker guide includes a complete MCP configuration, storage details and the CLI container. The default stdio container has no network; live providers require deliberate network and credential configuration.

A local endpoint alone is not reachable by hosted ChatGPT. Use the hosted path below for a remotely accessible, authenticated connection.

Python and CLI

Install from PyPI with Python 3.12–3.14 on macOS or Linux:

python -m pip install sourcesql
sourcesql --store ./research-data demo --output ./research-demo
sourcesql --store ./research-data capabilities --json

The installed demo includes fictional sources and needs no credentials or checkout. Each successful query returns an immutable result_id, typed columns, a bounded row preview and an execution receipt. Use the ID printed by demo to reopen or export it:

sourcesql --store ./research-data read RESULT_ID --offset 0 --limit 100
sourcesql --store ./research-data export --result RESULT_ID --output result-bundle.json
sourcesql --store ./restored import result-bundle.json --no-network

For source development, clone this repository, install uv and run uv sync --frozen. Then use uv run sourcesql in place of sourcesql. See package installation and release checks.

For Python integration, import Engine from sourcesql.engine. The examples cover discovery, saved-result refinement, row expansion and snapshot comparison.

To run the app from source, install Node.js 24, prepare the root Python environment with uv sync --frozen, then run:

cd apps/research
npm ci
npm run build
npm start

Open localhost:3117, or use npm run stdio with a local MCP client. See the example client configuration. CLI and app stores are separate by default; use the MCP query/result-reference workflow for app research, or research bundles to move it between stores.

Actual SQL execution requires the supported OS protections: macOS sandbox-exec, or Linux Landlock and seccomp. Missing protection causes a refusal. Use Docker on Windows. See runtime qualification for the tested environments.

Personal cloud: Vercel + MotherDuck

This path uses your own accounts and one owner workspace:

ChatGPT / MCP client
        │
        ▼
Vercel: MCP tools, app UI, authentication
        │
        ▼
MotherDuck: saved research, evidence, corrections, jobs
        │
        └── on-demand Flight: Python research execution

Ordinary reads, tree navigation and corrections use stored records without starting Python. Research runs as a bounded asynchronous job. One job runs at a time per owner namespace; queued jobs resume explicitly. A lost response is recovered using the same request and job IDs.

The installed package includes guided setup:

sourcesql setup personal-cloud --help
sourcesql setup personal-cloud plan --config personal.json --workspace ./my-cloud
sourcesql setup personal-cloud apply --workspace ./my-cloud --approve-plan PLAN_SHA256

Follow the personal-cloud setup guide to prepare your accounts, owner identity and token-free configuration. The plan records the exact destination, public source hashes, execution policy and proposed changes before applying them. Repeating an approved plan recovers its saved resources; uncertain creation stops for inspection.

Setup provisions a Vercel Preview gateway and a bounded MotherDuck Flight. It keeps credentials server-side, defaults to fixture mode and leaves account creation, OAuth registration and the exact-domain protection exception as visible account steps. Verify one fixture job and reopen its stored result before enabling live research. The client guide covers ChatGPT, Claude and Cursor, including which host interactions have actually been qualified.

Connect ChatGPT or Claude to the stable HTTPS /mcp URL using OAuth. Both have passed sign-in and saved-table rendering on the Hobby path. That one connection provides the research tools and interactive app; no plugin ZIP upload is needed. After a deployment, refresh the existing connection's tool definitions and reopen saved research.

Eligible personal, non-commercial adopters can choose Vercel Hobby; no Pro-only feature is required by this configuration. MotherDuck, search and model consumption are separate. Included allowances may cover light use, but application budgets do not guarantee that a platform account with billing enabled stops before overage.

Configure live search and models

Start with the provider guide and profile template. Keep real configuration and secrets outside version control.

For search-only work, omit model and embedding settings. An Exa API key does not enable semantic operators or authorize spending. Request budgets must fit the trusted profile; source fetching also requires approved destination hosts.

Hosted setup binds one search provider to each reviewed Flight. The configured profile determines the available search modes, candidate limits, fetching and semantic operators. Start with a small search, inspect its receipt, and configure wider coverage deliberately. A search-only profile returns source excerpts; it does not provide semantic extraction or a complete entity census.

See the hosted provider guide, hosted policy and Exa research notes for configuration and current limits.

Status and development

The current status records package, Docker, provider and host qualification separately. Fresh-account Hobby onboarding, broad live research quality and the complete Claude/Cursor lifecycle still need qualification. Hosted semantic models are unavailable in the reviewed profiles.

Run local checks from the repository root:

uv run pytest -q
npm ci --prefix apps/research
npm --prefix apps/research run check

Positive SQL worker tests need permission to launch the OS sandbox. A sandbox refusal is not a passing execution test. See qualification and app integration for the execution, evidence and storage contracts.

Read the checkpoint and manual update guide for continuation, evidence and review contracts. The roadmap covers the remaining adoption checks, larger engine-backed updates and later opt-in monitoring.

Licensed under Apache 2.0. Attribution and source reuse are recorded in NOTICE and provenance.