Quant_LLM_Wiki

English | 简体中文

A Karpathy-shaped wiki-first knowledge base for quantitative investment research.

Quant_LLM_Wiki turns WeChat articles, web pages, and research PDFs into an LLM-built Markdown knowledge base for quantitative research. It follows Andrej Karpathy's LLM-built KB method: a raw/ ingest layer, an LLM-compiled wiki/ of concept articles, and a schema/ that the LLM and tools both follow. Vector RAG is preserved as a fallback substrate, not the primary retrieval path. Three durable verbs — ingest, query, lint — drive everything. A built-in Rethink Layer scores novelty and quality of brainstormed ideas before output.

The goal is research inspiration and cross-document idea combination, not producing trade-ready strategies.

Features

• Multi-source ingestion — single URLs, batch URL lists, local HTML, or PDFs; warns on re-ingesting rejected sources
• LLM enrichment — extract idea blocks, transfer targets, combination hooks, failure modes; concurrent with configurable parallelism
• Wiki-first retrieval — both ask and brainstorm query stable concepts first; vector RAG runs only as fallback
• Rethink Layer — post-generation novelty (vector similarity) + quality (LLM-as-judge) scoring on brainstormed ideas
• Schema-enforced wiki — wiki_lint checks required frontmatter, sections, and source anchors on every run; --fix auto-repairs via LLM
• Query → wiki feedback — every ask/brainstorm logs back into the wiki; lint --maintain distills logs into gap-filling suggestions
• Interactive agent — LangGraph ReAct agent with 15 tools (plus 7 memory tools) and real-time progress streaming
• Agent skills — multi-step workflows (full ingest, concept review, KB health check, wiki explanation, strategy brainstorm) codified as on-disk SOPs the agent matches by trigger and follows, pausing wherever a step needs your decision
• Workflow memory — the agent resumes prior context across sessions: handoff notes, tasks, decisions, and per-thread research notes, kept strictly out of the wiki
• Strategy conversations — bring a fuzzy strategy direction; the agent clarifies, maps wiki coverage, proposes ideas with sources and failure modes, and converges into a strategy brief on disk
• Provider-agnostic — any OpenAI-compatible LLM (Zhipu GLM, DeepSeek, Moonshot, Qwen, OpenAI, Ollama, etc.)
• Local-first — all data as Markdown + ChromaDB on disk

For the full architecture, three-verb pipeline, retrieval invariants, and design principles, see docs/architecture.md.

Quick start

Pick one install flow and stay in that column.

1. Install

# A. pipx (recommended)
pipx install quant-llm-wiki

# B. git clone + editable install
git clone https://github.com/jackwu321/Quant_LLM_Wiki.git
cd Quant_LLM_Wiki && python3 -m venv .venv && source .venv/bin/activate
pip install -e .

pipx ≥ 1.5 required (older pipx ships pip 24.0 which mis-parses langgraph's newer wheel metadata). If install fails with ResolutionImpossible, upgrade pipx first: python3 -m pip install --user --upgrade --break-system-packages pipx && hash -r.

2. Pick a workspace

qlw writes data under whichever directory it considers your KB root, resolved in this order: explicit --kb-root arg → $QLW_KB_ROOT → current working directory.

# pipx users — bootstrap a workspace and fetch schema/
mkdir -p ~/my-kb && cd ~/my-kb
export QLW_KB_ROOT="$PWD"
curl -fsSL https://github.com/jackwu321/Quant_LLM_Wiki/archive/refs/heads/main.tar.gz \
  | tar xz --strip=1 --wildcards "*/schema/*" "*/llm_config.example.env"

# clone users — the clone IS the workspace
cd Quant_LLM_Wiki

3. Configure the LLM

cp llm_config.example.env .env
# Edit .env with your API key and provider settings

.env is auto-loaded from $QLW_KB_ROOT/.env → $(pwd)/.env → the package directory. Or export directly in your shell. See llm_config.example.env for provider examples.

4. Try the worked example (no real research data needed)

cd examples/tiny_kb
export QLW_KB_ROOT="$PWD"

qlw enrich            # LLM-enrich the pre-seeded sample articles
qlw embed             # build the vector index
qlw compile           # compile the wiki
qlw ask --query "What signals do these articles describe?"
qlw brainstorm --query "Combine momentum and sector ETF rotation"

See examples/tiny_kb/README.md for what gets produced and where to look.

5. Run on your own articles

qlw ingest --url "https://mp.weixin.qq.com/s/..."   # WeChat / web URL
qlw ingest --html-file saved.html                    # saved page
qlw ingest --pdf-file paper.pdf                      # research PDF
qlw ingest --url-list urls.txt                       # batch from a list

qlw enrich --limit 10
qlw embed
qlw ask --query "What momentum factors are discussed?"
qlw brainstorm --query "Combine momentum and volatility timing for ETF rotation"

Ingestion auto-runs enrich → compile → embed after success. Use --no-enrich to skip enrichment (compile still runs); --no-compile writes raw only (skips enrich, compile, and embed). If LLM_API_KEY is unset, raw is written but enrich/compile/embed are skipped with a message. Each URL has a 120 s ceiling; each LLM enrichment has 360 s. Override with INGEST_URL_TIMEOUT / LLM_ARTICLE_TIMEOUT.

Wiki maintenance

qlw lint                       # schema + health audit
qlw lint --fix                 # LLM auto-repair of non-compliant concepts
qlw lint --maintain            # gap analysis: unmapped sources, under-supported, stale
qlw lint --maintain --apply    # apply query-derived state updates (idempotent)

Agent mode

qlw agent                                       # interactive REPL
qlw agent --query "list all articles"           # one-shot
qlw agent --query "brainstorm: factor timing + risk parity"
qlw agent --thread futures                      # resume a named memory thread
qlw agent --no-memory                           # fully stateless run
qlw memory show                                 # inspect workflow memory

The agent dispatches the 15 tools listed in docs/architecture.md#agent-layer, plus 7 workflow-memory tools when memory is enabled (the default). Multi-step workflows — full ingest, concept review, KB health check, wiki explanation, strategy brainstorm — run as skills: on-disk SOPs the agent matches by trigger and follows, pausing wherever a step needs your decision. qlw memory promote-procedure <id> turns your own recurring flows into KB-level skills.

Workflow memory (<kb_root>/.qlw/memory/) gives sessions continuity: a human-editable workflow.md plus a SQLite store of sessions, tasks, decisions, and per-thread research notes, all inspectable via qlw memory. Open a session with a fuzzy strategy direction ("想看看宏观周期和商品期限结构有没有结合点") and the agent clarifies constraints, maps wiki coverage, proposes candidates with sources and failure modes, and — only on your explicit go-ahead — converges the conversation into a strategy brief under outputs/brainstorms/.

Configuration

Legacy ZHIPU_* variables are also accepted as fallbacks. On HTTP 429, later requests in the same process honor a shared cooldown (Retry-After header when present).

Article status lifecycle and content_type classification are documented in docs/metadata-schema.md.

Documentation

• docs/architecture.md — system architecture, three-verb pipeline, retrieval invariants, design principles
• docs/metadata-schema.md — article frontmatter fields and enums
• docs/brainstorm-output-spec.md — output contract for brainstorm runs
• docs/releasing.md — release process for maintainers
• schema/ — the contract enforced by qlw lint
• For CLI reference, qlw <subcommand> --help is the source of truth.

Running tests

Tests live in the repo, not the wheel — run from a git clone checkout (install flow B).

python3 -m unittest discover -s tests -p 'test_*.py' -v
python3 -m unittest discover -s tests/robustness -p 'test_*.py' -v

The tests/robustness/ suite covers Layer 1 (tool inputs), Layer 2 (workflow integration), Layer 3 (agent routing), and Layer 4 (LLM API timeouts/retries).

Contributing

1. Fork and create a feature branch
2. Write tests for new functionality
3. Ensure python3 -m unittest discover -s tests -p 'test_*.py' passes
4. Open a Pull Request

License

MIT — see LICENSE.

Disclaimer

Quant_LLM_Wiki is a research tool for generating investment strategy ideas. It does not produce trade-ready strategies or financial advice. All generated ideas require independent validation, backtesting, and risk assessment before any real-world application. Use at your own risk.