solace-researcher 0.1.0

Autonomous multi-agent research system with persistent per-agent state and cross-agent corroboration.

Solace Researcher

A continuous, tick-driven research agent. No chat turns — it runs as a daemon, persists its own state to disk, works through a task queue, proposes its own next tasks when the queue empties, and periodically reviews its own recent self-directed decisions to steer future ones.

Quickstart

pip install .
export GROQ_API_KEY=your_groq_key      # console.groq.com
export TAVILY_API_KEY=your_tavily_key  # tavily.com
solace-dashboard

pip install . puts solace-dashboard (and solace-researcher, the terminal-only version) on your PATH — run either from anywhere afterward, no need to stay in this folder. solace-dashboard opens a local web page at http://127.0.0.1:5055: pick agent count (1–5), topics, and duration in the form, hit Start, and watch tick/claim/verification counts update live per agent. Ctrl+C in the terminal stops the server; running it again against the same output directory resumes from saved state instead of repeating research.

(First install creates a throwaway build/ folder and *.egg-info folder next to the source — that's normal pip/setuptools output, not part of the project. Safe to delete; already git-ignored.)

That's everything needed to run it. Everything below this line is internals — useful for extending the code, not required to use it.

---

Other ways to run it

Terminal-only version (no dashboard)

solace-researcher

Same underlying run, input() prompts instead of a web form.

As a library

RunConfig/run/run_single are exported at the package's top level — nothing in agent.py/hub.py/llm_functions.py needs to be read or touched:

from solace_researcher import RunConfig, run

config = RunConfig(
    groq_api_key="gsk_...",
    tavily_api_key="tvly-...",
    topics=["stateful agent architectures", "vector database benchmarks"],
    num_agents=4,             # defaults to len(topics) if omitted -- one shard per topic
    duration_minutes=10,      # wall-clock budget per shard
)
hub, results = run(config)

print(hub.summary())
for claim, shards in hub.corroborated_claims(min_shards=2):
    print(f"[{claim.confidence:.2f}] ({', '.join(shards)}) {claim.text}")

run_single(config) is the same idea for one agent instead of a shard pool. See RunConfig in sdk.py for every field (feature toggles included — vector memory, code repair, iteration limits).

This is a config layer over fully open code, not a way of hiding it — everything sdk.py calls is the same public code the rest of this README documents. If you want the daemon (runs forever, resumable, no time budget) instead of a bounded run, see below.

As a daemon (continuous, resumable, no time budget)

export GROQ_API_KEY=your_groq_key
export TAVILY_API_KEY=your_tavily_key
python3 -m solace_researcher.main

Security: never commit real keys to this repo. Both are read from environment variables only. If a key was ever pasted anywhere outside your own terminal/environment (a chat, a screenshot, a doc), rotate it.

Runs forever. Ctrl+C shuts down gracefully and force-saves state first.

State persists in agent_state.json. Notes accumulate in research_notes/research_log.md (human-readable) and research_notes/research_log.jsonl (structured).

Offline demo (no API keys, no network, mocked calls)

python3 -m solace_researcher.agent

Self-direction, claim extraction, and self-reflection are all off by default in the demo (their _fn params default to None → old, simpler behavior, documented per-feature below). Useful for checking the plumbing before spending API credits.

How it works (internals)

What's actually autonomous

• Search grounding: real_reasoning and real_self_direct search via Tavily, then have Groq synthesize an answer from the results — plain search-API + inference, not a separate browsing-agent product. Every sentence in the synthesized answer is tagged inline as it's generated: [SOURCE: <url>] if it's a fact taken directly from a retrieved result, [ESTIMATE] if it's the model's own inference. Those tags are what real_extract_claims (below) reads.

• Structured claim extraction: task results used to collapse into one claim at a flat 0.7/0.3 confidence. real_extract_claims now splits a result into multiple individual claims, each carrying verified: bool and source_url: str | None — a claim backed by a real [SOURCE: url] tag is stored differently from one that's an [ESTIMATE], instead of both landing in the same bucket. No extractor wired → falls back cleanly to the old single-claim behavior.

• Self-direction: when the task queue empties, the agent calls real_self_direct, which proposes its own next research task from its open hypotheses' unknowns and recent findings — no human has to queue the next thing.

• Self-reflection, first order: every few ticks, _self_reflect reviews reflection_log — a record of the agent's own recent self-proposed tasks and how they turned out (success, verified-claim ratio) — and produces a reflection_bias via real_self_reflect. That bias is fed back into the context for the next self-directed proposal. So the loop is: agent proposes a task → task outcome gets logged → agent reviews its own log → that review shapes the next proposal. Verified end-to-end in testing (a later self-direction call demonstrably receives the prior reflection's assessment as context).

• Self-reflection, second order (calibration): first-order reflection only judges raw task outcomes. _meta_reflect goes one level further — it reviews reflection_history (a trace of the agent's own past reflections, not raw tasks) against what actually happened afterward, and judges whether those past self-assessments were well-calibrated. The result, meta_confidence, then scales how much a new first-order reflection is allowed to move reflection_bias — effective_delta = raw_delta * meta_confidence. A track record of badly-calibrated self-judgment measurably damps the agent's own future self-judgment instead of letting it compound unchecked. Tested end to end: a run with deliberately miscalibrated mock reflections drove meta_confidence down and visibly shrank later reflection_bias moves compared to earlier ones.

  Precision on both layers, because it matters if you're reading the code and not just this file: real_self_reflect and real_meta_reflect are structurally identical to real_judge or real_support_judge — format some text, send it to Groq, parse the JSON that comes back. The feedback loops around them are real and tested (output of past decisions measurably changes future decisions, and a calibration check measurably changes how much a new judgment is trusted). What this is not is introspective self-awareness or reasoning about its own reasoning process in the phenomenal sense — there's no representation of why the agent believes something that it can inspect independent of these scored logs. This is the same family of technique as Reflexion-style agents (verbal self-critique with memory feeding into future decisions) — a real, literature-grounded pattern, not a claim about subjective experience. Call it a two-level self-directed feedback loop, not self-recursive thinking.

• Everything already covered: task → oracle review → code-gate → (genesis+evaluator | reasoning) → claim(s) recorded in knowledge store → evidence applied to hypotheses per-claim → staggered contradiction sweep → staggered self-reflection → notes written — all still true, all still tested.

State vector (state.py, 13 dimensions)

Two derived scalars come off this vector each tick: binding (coherence — is the vector's norm stable or extreme) and self_model (surprise — distance between current state and the running memory mean, decayed). All 13 raw dimensions are wired to something real; none are placeholders.

#	Field	What it actually measures
0	backlog_pressure	pending tasks / capacity
1	recent_failure_rate	failures in last N tasks / N
2	vitality	1 - recent_failure_rate
3	idle_ticks_norm	ticks since last task, normalized
4	novelty	embedding distance from recent task history
5	cumulative_success	lifetime success rate
6	consecutive_failures_norm	current failure streak, normalized
7	avg_duration_norm	execution time trend
8	self_direction_ratio	of recent finished tasks, fraction the agent proposed itself vs. was assigned
9	verified_claim_ratio	of recent claims, fraction that came back verified=True (real source_url) vs. estimate
10	contradiction_pressure	of concepts touched in the last sweep, fraction with an active conflict
11	hypothesis_revision_rate	how often a hypothesis's status actually flipped (not just accumulated evidence) in the recent window, per hypothesis tracked
12	meta_reflection_delta	size of the last change to reflection_bias from _self_reflect

Slots 8–12 depend on extract_claims_fn and self_reflect_fn being wired (they are, in main.py; they aren't in the offline demo) — without them those slots stay at 0, same fallback pattern as everything else here.

Token/cost tracking and benchmark graphs

Every LLM and search call is logged to research_notes/token_usage.jsonl as it happens (crash-safe, same append-immediately pattern as the research log itself) — tagged with which function made the call (oracle, genesis, reasoning, judge, support_judge, self_direct, extract_claims, self_reflect, meta_reflect), tick number, prompt/completion tokens, latency, and cost in USD.

pip install matplotlib   # not in requirements.txt -- benchmarking tool, not a runtime dep
python3 analyze_metrics.py                        # defaults to ./research_notes/token_usage.jsonl
python3 analyze_metrics.py path/to/token_usage.jsonl

Produces a per-call-site summary table, a CSV (token_usage_summary.csv), and three PNGs: cumulative cost over the run, tokens by call site, and latency distribution by call site — the actual artifacts for a benchmark writeup, not just numbers in a terminal.

Cost math uses GROQ_PRICING in metrics.py, currently openai/gpt-oss-120b at $0.15/M input, $0.60/M output tokens. Verify against groq.com/pricing before trusting the dollar figures for anything investor- or reviewer-facing — pricing pages change, this was current as of when the file was written, not guaranteed current when you read it. Tavily search cost is a rough paid-tier estimate for the same reason; verify at tavily.com/pricing.

Vector memory (semantic search over claims)

knowledge.py's SemanticStore buckets claims by concept for exact lookups — that's cheap and still how contradiction checking works. On top of that, vector_memory.py adds real cross-concept semantic retrieval: store.semantic_search("some query", k=5) returns claims ranked by similarity to the query regardless of which concept they got filed under, backed by an actual similarity index instead of a dict scan.

Backend, auto-selected in this order:

1. FAISS + GPU — if faiss is installed and a GPU is visible (faiss.get_num_gpus() > 0). IndexFlatIP moved onto the GPU via index_cpu_to_gpu — real CUDA-accelerated search.
2. FAISS on CPU — if faiss is installed but no GPU.
3. Pure NumPy — if faiss isn't installed at all. Brute-force cosine similarity. Mathematically identical results to the FAISS backends at this scale (thousands of claims), just not index-accelerated.

Nothing else in the codebase needs to know or care which backend is active — same search() call, same return shape. Check agent.vector_memory.backend at runtime to see which one you're actually running on.

Wired into _self_seed: when the agent proposes its own next task, it now pulls claims that are semantically relevant to its open unknowns, not just the most recent claims within the same concept bucket — a claim filed under a different concept can still surface if it's actually relevant.

On embedding quality, honestly: embeddings.py's embed() is a hashed bag-of-words (the "hashing trick") — a lexical/token-overlap proxy for similarity, not a trained semantic embedding model. It catches shared vocabulary, not shared meaning; two sentences about the same thing using different words won't necessarily land close together. Good enough to keep near-duplicate tasks from colliding and to give semantic_search real signal over exact-bucket lookup — not a substitute for a real sentence-embedding model if you need genuine paraphrase-level recall. Swapping in something like sentence-transformers locally is a drop-in replacement for just the embed() function body; nothing downstream needs to change.

A real bug found and fixed while building this: embed() originally used Python's built-in hash() on tokens, which is randomized per-process by default (PYTHONHASHSEED, a security feature — hash("foo") returns a different number in two separate python3 runs). That's invisible within a single run, but this system persists embeddings to disk and reloads them in a new process on every restart. With the built-in hash(), a freshly-embedded query after a restart would land in effectively random buckets relative to vectors indexed in the previous process — silently corrupting every similarity comparison across restarts, including the pre-existing novelty-scoring embedding history in state.py. Fixed by switching to hashlib.md5 (stable across processes/machines/Python versions — used for stability, not any cryptographic property) and verified with a same-process-vs-fresh-process test before and after.

Multi-agent (hub.py)

Runs N independent SolaceResearcher shards, each with its own full state (own agent_state.json, own tick loop, own reflection history), pooling their claims into one SharedKnowledgeHub. Threading, not multiprocessing — shard work is I/O-bound (waiting on Groq/Tavily), so Python releases the GIL during the actual network wait and threads can share the hub's memory directly.

from hub import SharedKnowledgeHub, run_shards

def make_factory(shard_id, topic):
    def factory():
        agent = build_agent()  # from main.py, or your own construction
        agent.notes_dir = f"./shards/{shard_id}"
        agent.queue.add_task(f"Investigate {topic}", source="assigned", concept=topic)
        return agent
    return factory

hub = SharedKnowledgeHub(path="./hub_state")
factories = {"a": make_factory("a", "topic X"), "b": make_factory("b", "topic Y")}
results = run_shards(factories, hub, max_ticks=20)
print(hub.summary())

Corroboration is the actual point, not just pooling: when two different shards independently produce a claim that's semantically close (via vector_memory.py's semantic_search, not exact-string matching), that's real signal — independent convergence from separate research paths is stronger evidence than one shard saying something once. hub.corroborated_claims(min_shards=2) returns exactly that subset, and a corroborated claim's confidence gets a small bump (capped at 0.95) each time another shard independently reaches it. Tested end to end: three shards, two converging on the same finding from different task text, one unrelated — the hub correctly pooled to 2 distinct claims, flagged exactly one as corroborated, and credited the right two shard IDs.

A real threading bug found and fixed while building this: metrics.py's token tracker is a module-level singleton — harmless with one agent, but once multiple shard threads run in the same process, each shard's constructor sets tracker.path to its own notes_dir. As a plain instance attribute, every shard's write would race to set the same shared value — whichever shard finished constructing last would silently steal every other shard's token logging into its own file, and current_tick would reflect whatever thread last set it regardless of which shard actually made the call. Fixed with threading.local() for both path and current_tick, verified with a direct concurrent-write test (5 threads, each asserting it kept its own path/tick throughout).

On "multiverse" (the sim-file concept this is loosely inspired by): there's no actual parallel-reality mechanic here, and there wasn't one worth porting — it maps cleanly onto an ensemble of run configs instead: different shards given different topic focus, different starting reflection_bias, different search depth, then compared via hub.corroborated_claims() and hub.summary(). That's the honest, useful version of the idea.

Tests

tests/ holds standalone regression checks (plain assert + print, not pytest) for issues that were found and fixed by running this against real usage:

python3 tests/run_all.py

Test	What it guards against
test_fix1_hypothesis_linkage.py	Self-directed tasks used to accumulate claims under self_directed_research with no hypothesis attached — their findings never counted as evidence for anything. Now linked to the first open tracked hypothesis.
test_fix2_source_verification.py	verify_source_fn independently checks a claim's source_url once, at creation time — downgrades a claim to estimate-tier if the page doesn't actually support it, leaves it alone if verification itself is inconclusive (a fetch failure isn't evidence the claim is wrong).
test_fix3_retry_backoff.py	judge_fn calls in the contradiction sweep now retry with exponential backoff on transient errors, and a pair is only cached as "checked" after a real verdict — not on failure, which used to mean a pair that hit a rate limit was silently skipped forever.
test_fix4_persistence_atomic.py	Persistence.save() writes to a .tmp file and atomically replaces the real state file — a crash mid-write can no longer corrupt agent_state.json in place.
test_fix5_task_injection.py	external_tasks_path lets another process drop JSON-line tasks into a running daemon (echo '{"objective": "..."}' >> external_tasks.jsonl) — consumed and truncated each tick, malformed lines skipped without blocking the rest of the batch.
test_fix6_genesis_live_data.py	needs_live_data() detects objectives needing current information and routes a reasoning_fn fetch into genesis_fn's context first, instead of genesis_fn writing code that tries to hit the network from evaluator.py's bare subprocess (which has none).
test_fix7_code_repair.py	evaluator.run_with_repair(): a missing dependency gets installed and the same code retried; any other error gets handed to genesis_fn for a real fix; an unfixable error stops exactly at max_iterations instead of looping forever.

On fix 2 specifically, since it caused a real problem worth naming: verification must run exactly ONCE per claim, at the moment the claim is created — never as a recurring sweep over already-verified claims. A sweep-style design re-checks every claim ever produced on every pass, with no per-claim memory of "already handled" — call volume only grows, tick over tick, for the life of the run. verify_source_fn is deliberately wired into _record_outcome (creation time) rather than alongside the staggered contradiction sweep for exactly this reason.

Self-healing code execution

evaluator.py's run_with_repair() (used by default — enable_code_repair=True) iterates on a failed genesis run instead of just logging the failure:

• Missing dependency → the module name is extracted from the real ModuleNotFoundError, regex-validated to rule out anything that isn't a plain package name, installed via a separate pip install subprocess, and the same code is retried — no wasted genesis_fn call on an error that's already fixed.
• Any other failure → the error is handed back to genesis_fn along with the original objective, and the revised code is tried instead.
• Bounded by code_repair_max_iterations (default 3) either way — won't loop forever on something unfixable.

Trust boundary, stated plainly: the generated code itself still runs in a bare subprocess with no network access — that's unchanged. The only new network-capable action is pip install <name>, run as its own separate subprocess, and only for a name Python's own import machinery already asked for. Sandboxed execution and unsandboxed dependency resolution are different guarantees; this keeps the first and deliberately relaxes the second. Set enable_code_repair=False if you want the original single-attempt behavior back.

Files

• sdk.py — config-driven entry point (RunConfig, run(), run_single()) — set keys/topics/agent count/duration, no internals reading required.
• quickstart.py — runnable demo of sdk.py, works out of the box once API keys are set.
• main.py — the daemon entrypoint (python3 main.py, runs forever, resumable). Continuous loop, graceful shutdown, self-direction, claim extraction, first- and second-order self-reflection, source verification, and external task injection all wired in.
• llm_functions.py — real Groq + Tavily functions: oracle, genesis, reasoning, judge, support_judge, self_direct, extract_claims, self_reflect, meta_reflect, verify_source.
• metrics.py — token/cost/latency tracking, one JSONL row per LLM/search call, tagged by call site and tick. Thread-safe (thread-local path/tick) for multi-agent runs.
• analyze_metrics.py — reads token_usage.jsonl, produces a summary table, CSV, and benchmark graphs (cumulative cost, tokens by call site, latency by call site).
• vector_memory.py — cross-concept semantic search over claims, GPU via FAISS if available, correct numpy fallback if not.
• hub.py — SharedKnowledgeHub + run_shards: multi-agent pooling with semantic dedup and cross-shard corroboration.
• embeddings.py — content-based vectorizer (hashed bag-of-words) for novelty scoring and vector memory. Stable across process restarts (hashlib.md5, not Python's randomized built-in hash).
• agent.py — the tick loop itself (SolaceResearcher class) plus the mocked demo under __main__.
• state.py — telemetry → binding (coherence) → self_model (surprise/confidence); 13-dimensional, all slots wired, see table above.
• knowledge.py — semantic store: concept → claims, each with confidence/verified/source_url.
• hypotheses.py — persistent research questions, updated in place across ticks.
• contradictions.py — scoped, staggered contradiction detection, with retry/backoff.
• tasks.py — priority queue, concept/hypothesis linkage.
• evaluator.py — sandboxed execution + real pass/fail verification for generated code, with self-healing iteration (missing-package install, genesis-driven repair) — see below.
• persistence.py — atomic save/load of everything above.
• notes.py — the only output surface. No chat replies, no return-to-caller.
• tests/ — regression checks for the fixes in the table above; tests/run_all.py runs them all.

What still needs your input

• real_genesis doesn't currently have code-execution-time internet access (it writes code, but that code runs in a bare subprocess with no network) — needs_live_data() (see tests table above) covers the common case by fetching first and handing genesis the data as context, but it's a keyword gate, not exhaustive. Widen LIVE_DATA_KEYWORDS in agent.py if you hit a phrasing it misses.
• Tavily's free tier is 1,000 searches/month — real_reasoning and real_self_direct both consume it on every grounded call. Fine for testing/demo use; budget for it before running this unattended for long stretches.
• _self_seed's hypothesis linkage picks the first open hypothesis found, not the most relevant one — fine with one active research question at a time, worth revisiting if you're tracking several unrelated hypotheses concurrently.