shouchao 0.2.1

Web Information Retrieval Assistant - aggregate, analyze, and brief on web information

ShouChao (手抄) - Web Information Retrieval Assistant

Aggregates news from 100+ major media sources across 10 languages, converts articles to structured markdown, indexes them into a ChromaDB knowledge base, and provides AI-powered briefings and analysis for investment, immigration, and study abroad scenarios.

Features

• 10-Language Coverage: Chinese, English, Japanese, French, Russian, German, Italian, Spanish, Portuguese, Korean
• 100+ News Sources: Reuters, BBC, NHK, Le Monde, TASS, DW, ANSA, El Pais, Folha, Yonhap, and many more
• Multiple Reader Backends: requests, curl_cffi, DrissionPage, Playwright with human-like browsing behavior
• RSS + Web Reading: RSS feeds for efficient discovery, web reading for full articles
• Markdown Storage: Articles saved as {lang}/{site}/{date}/{title}.md with YAML front matter
• ChromaDB Knowledge Base: GangDan-compatible vector database for semantic search
• AI Analysis: Investment, immigration, study abroad, and general news analysis via Ollama
• News Briefings: Daily, weekly, and domain-specific briefings with LLM summarization
• Three Interfaces: CLI, GUI (tkinter), and Web (Flask) dashboard
• i18n: Full 10-language UI support
• Stock Market Treemap: Real-time global market heatmap (A-Share, HK, US) with sector visualization

Requirements

• Python >= 3.10
• Ollama (for AI features: analysis, briefings, semantic search)

Installation

pip install shouchao

Or install from source:

git clone https://github.com/cycleuser/ShouChao.git
cd ShouChao
pip install -e .

Optional dependencies

pip install shouchao[all]        # All optional modules + readability
pip install shouchao[curl]       # curl_cffi for better browser simulation
pip install shouchao[browser]    # DrissionPage (system Chrome)
pip install shouchao[readability] # Better content extraction

Quick Start

# List available news sources
shouchao sources --language en

# Fetch news
shouchao fetch --language en --max-articles 50

# Generate daily briefing
shouchao briefing --language zh --output daily_briefing.md

# Start web dashboard
shouchao web

Stock Market Heatmap

# Open stock market treemap visualization
# Access at http://localhost:5000/market
shouchao web
# Then navigate to /market or click "股市热力图" in sidebar

The heatmap shows:

• A-Share (Shanghai/Shenzhen): 30+ sectors, real-time data from East Money API
• HK Stocks: Major sectors and companies
• US Stocks: NASDAQ/NYSE tech giants and major indices
• Global View: Combined view of all markets

Features:

• Color-coded performance (Red=Up, Green=Down - China convention)
• Market-cap weighted sizing
• Sector grouping
• Click for stock details
• Real-time refresh

Usage

CLI Options

Command	Description
shouchao fetch	Fetch news from sources
shouchao search "query"	Search indexed news
shouchao briefing	Generate news briefings
shouchao analyze "query"	Analyze news for scenarios
shouchao index	Index articles into ChromaDB
shouchao sources	List/manage news sources
shouchao config	View/update configuration
shouchao web	Start Flask web server
shouchao gui	Launch tkinter GUI

Global Flags

Flag	Description
-V, --version	Show version
-v, --verbose	Verbose output
--json	JSON output
-q, --quiet	Suppress non-essential output
--data-dir PATH	Custom data directory

Fetch Examples

shouchao fetch --language zh --max 20              # Chinese news
shouchao fetch --language en --source "Reuters"     # Specific source
shouchao fetch --fetcher curl                       # Use curl_cffi backend
shouchao fetch --language ja,ko --max 5             # Multiple languages

Analysis Scenarios

shouchao analyze "Impact of new EU AI Act" --scenario investment
shouchao analyze "Canada immigration policy 2026" --scenario immigration
shouchao analyze "UK university tuition changes" --scenario study_abroad
shouchao analyze "Global semiconductor trends" --scenario general

Python API

from shouchao import fetch_news, search_news, analyze_news, list_sources

# List sources
result = list_sources(language="en")
print(result.data["count"])  # Number of English sources

# Fetch news
result = fetch_news(language="en", max_articles=10)
print(result.data["fetched"])  # Articles fetched

# Search
result = search_news(query="climate change", top_k=5)
for r in result.data["results"]:
    print(r["metadata"]["title"])

# Analyze
result = analyze_news(query="market trends", scenario="investment")
print(result.data["content"])

Agent Integration (OpenAI Function Calling)

ShouChao exposes OpenAI-compatible tools for LLM agents:

from shouchao.tools import TOOLS, dispatch

response = client.chat.completions.create(
    model="gpt-4o",
    messages=messages,
    tools=TOOLS,
)

result = dispatch(
    tool_call.function.name,
    tool_call.function.arguments,
)

CLI Help

Project Structure

shouchao/
├── core/
│   ├── config.py        # Configuration management
│   ├── sources.py       # 100+ news source registry
│   ├── fetcher.py       # HTTP fetcher backends
│   ├── rss.py           # RSS/Atom feed parser
│   ├── converter.py     # HTML-to-Markdown pipeline
│   ├── storage.py       # Article file storage
│   ├── indexer.py       # ChromaDB indexer
│   ├── ollama_client.py # Ollama API client
│   ├── analyzer.py      # LLM analysis engine
│   ├── briefing.py      # Briefing generator
│   └── market_map.py    # Stock market treemap data
├── cli.py               # CLI interface
├── gui.py               # Tkinter GUI
├── app.py               # Flask web server
├── api.py               # Python API
├── tools.py             # OpenAI tools
└── i18n.py              # 10-language translations

Development

git clone https://github.com/cycleuser/ShouChao.git
cd ShouChao
pip install -e ".[dev]"
python -m pytest tests/test_unified_api.py -v

License

GPL-3.0-or-later