ShouChao (手抄) - Web Information Retrieval Assistant

Aggregates news from 100+ major media sources across 10 languages, converts articles to structured markdown, indexes them into a ChromaDB knowledge base, and provides AI-powered briefings and analysis for investment, immigration, and study abroad scenarios.

Features

• 10-Language Coverage: Chinese, English, Japanese, French, Russian, German, Italian, Spanish, Portuguese, Korean
• 100+ News Sources: Reuters, BBC, NHK, Le Monde, TASS, DW, ANSA, El Pais, Folha, Yonhap, and many more
• Multiple Reader Backends: requests, curl_cffi, DrissionPage, Playwright with human-like browsing behavior
• RSS + Web Reading: RSS feeds for efficient discovery, web reading for full articles
• Markdown Storage: Articles saved as {lang}/{site}/{date}/{title}.md with YAML front matter
• ChromaDB Knowledge Base: GangDan-compatible vector database for semantic search
• AI Analysis: Investment, immigration, study abroad, and general news analysis via Ollama
• News Briefings: Daily, weekly, and domain-specific briefings with LLM summarization
• Three Interfaces: CLI, GUI (tkinter), and Web (Flask) dashboard
• i18n: Full 10-language UI support
• Stock Market Treemap: Real-time global market heatmap (A-Share, HK, US) with sector visualization

Requirements

• Python >= 3.10
• Ollama (for AI features: analysis, briefings, semantic search)

Installation

pip install shouchao

Or install from source:

git clone https://github.com/cycleuser/ShouChao.git
cd ShouChao
pip install -e .

Optional dependencies

pip install shouchao[all]        # All optional modules + readability
pip install shouchao[curl]       # curl_cffi for better browser simulation
pip install shouchao[browser]    # DrissionPage (system Chrome)
pip install shouchao[readability] # Better content extraction

Quick Start

# List available news sources
shouchao sources --language en

# Fetch news
shouchao fetch --language en --max-articles 50

# Generate daily briefing
shouchao briefing --language zh --output daily_briefing.md

# Start web dashboard
shouchao web

Stock Market Heatmap

# Open stock market treemap visualization
# Access at http://localhost:5000/market
shouchao web
# Then navigate to /market or click "股市热力图" in sidebar

The heatmap shows:

• A-Share (Shanghai/Shenzhen): 30+ sectors, real-time data from East Money API
• HK Stocks: Major sectors and companies
• US Stocks: NASDAQ/NYSE tech giants and major indices
• Global View: Combined view of all markets

Features:

• Color-coded performance (Red=Up, Green=Down - China convention)
• Market-cap weighted sizing
• Sector grouping
• Click for stock details
• Real-time refresh

Usage

CLI Options

Global Flags

Fetch Examples

shouchao fetch --language zh --max 20              # Chinese news
shouchao fetch --language en --source "Reuters"     # Specific source
shouchao fetch --fetcher curl                       # Use curl_cffi backend
shouchao fetch --language ja,ko --max 5             # Multiple languages

Analysis Scenarios

shouchao analyze "Impact of new EU AI Act" --scenario investment
shouchao analyze "Canada immigration policy 2026" --scenario immigration
shouchao analyze "UK university tuition changes" --scenario study_abroad
shouchao analyze "Global semiconductor trends" --scenario general

Python API

from shouchao import fetch_news, search_news, analyze_news, list_sources

# List sources
result = list_sources(language="en")
print(result.data["count"])  # Number of English sources

# Fetch news
result = fetch_news(language="en", max_articles=10)
print(result.data["fetched"])  # Articles fetched

# Search
result = search_news(query="climate change", top_k=5)
for r in result.data["results"]:
    print(r["metadata"]["title"])

# Analyze
result = analyze_news(query="market trends", scenario="investment")
print(result.data["content"])

Agent Integration (OpenAI Function Calling)

ShouChao exposes OpenAI-compatible tools for LLM agents:

from shouchao.tools import TOOLS, dispatch

response = client.chat.completions.create(
    model="gpt-4o",
    messages=messages,
    tools=TOOLS,
)

result = dispatch(
    tool_call.function.name,
    tool_call.function.arguments,
)

CLI Help

Project Structure

shouchao/
├── core/
│   ├── config.py        # Configuration management
│   ├── sources.py       # 100+ news source registry
│   ├── fetcher.py       # HTTP fetcher backends
│   ├── rss.py           # RSS/Atom feed parser
│   ├── converter.py     # HTML-to-Markdown pipeline
│   ├── storage.py       # Article file storage
│   ├── indexer.py       # ChromaDB indexer
│   ├── ollama_client.py # Ollama API client
│   ├── analyzer.py      # LLM analysis engine
│   ├── briefing.py      # Briefing generator
│   └── market_map.py    # Stock market treemap data
├── cli.py               # CLI interface
├── gui.py               # Tkinter GUI
├── app.py               # Flask web server
├── api.py               # Python API
├── tools.py             # OpenAI tools
└── i18n.py              # 10-language translations

Development

git clone https://github.com/cycleuser/ShouChao.git
cd ShouChao
pip install -e ".[dev]"
python -m pytest tests/test_unified_api.py -v

License

GPL-3.0-or-later