English | 简体中文

hfpapers-crawler

Naming philosophy: claw (sharp grasp) ≠ crawl (creep). hfpclawer = HuggingFace Papers + claw + er = "A sharp tool that claws HF papers with precision" 🦞

Not a crawler — faster, sharper, more precise. Same series: OpenClaw, Hermes Agent ecosystem.

A multi-source academic paper clawler for PDE / neural operator / physics-informed ML. Built with SQLite paper_store, Crossref cross-validation, anti-crawl Scrapy pipelines, and MCP server.

✨ Features

• Research-engineered paper discovery — arXiv / OpenReview / Semantic Scholar multi-source clawling with relevance scoring, citation-graph analysis, and SimClusters-style community-guided 2-hop expansion (graph expand-hub --community: seed → its Louvain communities → community hub papers, faithfully mapped from x-algorithm SimClusters) for topic-focused "papers like this one" exploration.
• Auditable scholarly-network recommendations — every expansion run can emit a JSONL audit trail (which papers were adopted into the frontier, hub scores, community labels), and the optional learned layer (hfpclawer rank train, lightgbm → native model) trains on that trail with tree feature importance as the explanation. Heuristic layer stays 0-token default; learning is opt-in (hfpclawer[rank]).
• Verification status machine (v0.16+) — every paper carries an explicit, derived state: pending → verified / stale / suspect. Metadata conflicts (e.g. a DOI resolving to a different arXiv ID than recorded) are flagged suspect by symbolic 0-LLM checks and require human adjudication — no silent corruption, no LLM-judged verdicts.
• First-party recommendation signals — search history × text similarity + relevance scoring + verification-state gating, all computed from your local store. No external dependency: works fully offline, no Zotero required. Zotero (when present) is an optional enhancement adapter — only DOI/arXiv-bearing scholarly items sync back as interest signals (pydantic contract hfpapers/contracts.py).
• Repo-scoped profiles → recommendations (v0.16.5+) — every repo consuming hfpclawer is its own virtual user: hfpclawer init scaffolds a REPO_USER.md interest profile (real personal interests live in ~/.hfpclawer/profile.yaml, never in public repos), and hfpclawer recommend fuses config + repo + machine layers with per-hit provenance, through the verification gate.
• Declarations as explicit feedback (v0.16.9+) — REPO_USER.md v2 accepts/rejects blocks (SKILL.state: active/superseded/revoked; scope: topic-exclusion filters candidates, scope: self-constraint documents repo choices without filtering papers about them); paper-level accepts fold into the positive pool as strongest-layer examples.
• Zotero sync-back (v0.16.7) — hfpclawer zotero sync-back pulls Favor-tagged items through the scholarly contract (web pages/programs/reports are dropped before any lookup) and marks matching local papers favorited — an interest signal the learned ranker consumes.
• Zero-config positive-example pool (v0.16.8) — hfpclawer pool accumulates training signal from fully local, label-free sources: hub audit trails, store verification status and Zotero favorites (layered weak labels: verified/manual/favorited/adopted vs truncated). Suspect papers never enter; live gates keep the pool clean; append-only, gitignored, zero telemetry — open-source safe.
• Zero-token operation ready — deterministic change detection + cron/monitor layering keeps routine monitoring at 0 LLM cost (see Hermes Agent integration skills).

---

Quick Install

pip install hfpclawer

Dependencies

• Core (auto-installed): pyyaml, requests, beautifulsoup4, typer, etc.
• QUIC transport (optional): pip install hfpclawer[quic] — HTTP/3 download fallback for arXiv (TCP to arXiv is reset on CN networks; QUIC/UDP is not). hfpclawer fetch falls back tcp → quic automatically.
• LLM features (optional): pip install hfpclawer[llm] — for sniff / analyze commands
• PDF conversion (optional): pip install hfpclawer[pdf]
• Scrapy spiders (optional): pip install hfpclawer[scrapy]
• Dev (testing): pip install hfpclawer[dev]
• arXiv local search (optional): pip install hfpclawer[arxiv] documents the metadata dependency only (PyPI doesn't support git+https). See docs/kaggle-metadata.md for manual git clone + OAI-PMH or Kaggle setup.
• Citation audit (optional): pip install hfpclawer[audit] declares namespace only. See hfpclawer/citation_audit.py for manual setup.

Local Development

git clone <your-repo>
cd hfpapers-clawler

# Create virtual environment
python3 -m venv venv
source venv/bin/activate

# Install in editable mode with dev dependencies
pip install -e ".[dev]"

# Verify
hfpclawer --help

Configuration

First run hfpclawer init to generate config and env template:

hfpclawer init --quick          # Quick mode (defaults)
# or
hfpclawer init                  # Interactive wizard
cp .env.template .env           # Fill in API keys
# Edit config.yaml to customize search queries

Or manually create files (see docs/USAGE.md for full reference):

---

CLI Commands

# Search for new papers
hfpclawer search                    # Default 3 pages, threshold 30
hfpclawer search --max-pages 5      # More pages
hfpclawer search --dry-run          # Show only, don't save

# Full pipeline: search → download → convert
hfpclawer full

# SQLite Paper Store operations
hfpclawer store stats               # Storage statistics
hfpclawer store search              # List all papers
hfpclawer store search --keyword "FNO"
hfpclawer store verify --aid 2301.11167

# Download & convert
hfpclawer download                  # Download top-20 PDFs
hfpclawer fetch 2502.05171          # Single paper via tcp→quic→hint chain
hfpclawer fetch 2502.05171 -k source  # tex source bundle (tar.gz)
hfpclawer convert                   # PDF → Markdown

# MCP Server (for Hermes Agent / OpenCode)
hfpclawer mcp                       # Default port :8765

---

Python API

from hfpapers.paper_store import PaperStore, PaperRecord, ensure_paper

# Create a store
store = PaperStore(db_path="/tmp/papers.db")

# Add a paper
rec = PaperRecord(
    title="Fourier Neural Operator",
    abstract="Learning PDE solution operators with Fourier transforms",
    year=2023,
    source="my_app",
    relevance=90,
)
sf_id = store.upsert_paper(rec)
store.add_identifier(sf_id, "arxiv", "2010.08895")

# Search
papers = store.search_papers("neural operator")
for p in papers:
    print(f"[{p.relevance}] {p.title}")

# Hardware probe
from hfpapers.hardware import HardwareProbe
hw = HardwareProbe()
print(f"Hardware: {hw.summary()}")

---

MCP Server

hfpapers-clawler ships with a built-in MCP server for AI agent integration:

hfpclawer mcp

Register in Hermes Agent ~/.hermes/config.yaml:

mcp:
  servers:
    hfpapers:
      command: "hfpclawer"
      args: ["mcp", "--port", "8765"]

Available MCP tools (4 core always active; heavy ops use CLI):

Tool	MCP (auto)	CLI preferred
hfpclawer_search	✅	—
hfpclawer_info	✅	—
hfpclawer_list	✅	—
hfpclawer_stats	✅	—
hfpclawer_download	⚠️ available	hfpclawer download --limit N
hfpclawer_convert	⚠️ available	hfpclawer convert
hfpclawer_full	⚠️ available	hfpclawer full

Heavy operations (download/convert/full) are hidden from tools/list by default to save tokens. They remain callable via direct tools/call — or better, use the CLI for progress feedback.

---

Architecture

┌─ CLI (Typer) ─┐  ┌─ MCP Server ─┐
└──────┬────────┘  └──────┬───────┘
       └────────┬──────────┘
                ▼
┌─ Scrapy Layer (Multi-source) ───────────┐
│  ArxivSearchSpider | OpenReviewSpider    │
│  HFPapersSpider | MultiSourceSpider      │
│  Middleware: UA random, delay, proxy...  │
│  Pipeline: Store→Classify→Export→DL     │
└──────────────────┬──────────────────────┘
                   ▼
┌─ Paper Store (SQLite) ──────────────────┐
│  papers (Snowflake ID) | identifiers    │
│  crossref_cache | CrossrefClient        │
└─────────────────────────────────────────┘

---

Tests

pip install -e ".[dev]"
pytest tests/ -v           # Run all tests
pytest tests/ --cov=hfpapers  # With coverage

---

License

MIT

Hermes Agent Skills

These skills automate common hfpclawer workflows inside Hermes Agent (or any AI coding assistant that supports the Hermes skill format):

Skill	Purpose	Install
hfpclawer-paper-search	Daily paper discovery → download → wiki	hermes skills install https://raw.githubusercontent.com/diamond2nv/hfpapers-crawler/main/skills/hfpclawer-paper-search/SKILL.md
hfpclawer-citation-audit	Verify citations via S2 + OpenAlex	hermes skills install https://raw.githubusercontent.com/diamond2nv/hfpapers-crawler/main/skills/hfpclawer-citation-audit/SKILL.md
hfpclawer-academic-integrity	Paper draft integrity: extract → verify → flag FABRICATED	hermes skills install https://raw.githubusercontent.com/diamond2nv/hfpapers-crawler/main/skills/hfpclawer-academic-integrity/SKILL.md
hfpclawer-formula-verify	LaTeX formula cross-validation (SymPy ↔ Wolfram, dimensional)	hermes skills install https://raw.githubusercontent.com/diamond2nv/hfpapers-crawler/main/skills/hfpclawer-formula-verify/SKILL.md

After installing, load with skill_view(name='hfpclawer-paper-search') in any Hermes conversation.

Acknowledgments

This project incorporates code adapted from:

• academic-research-skills by Cheng-I Wu (https://github.com/Imbad0202/academic-research-skills)

  • hfpclawer/_text_similarity.py — title normalization and similarity scoring
  • hfpclawer/citation_audit_s2.py — Semantic Scholar API client (architecture reference)
  • hfpclawer/citation_audit_oa.py — OpenAlex API client (architecture reference) Licensed under CC BY-NC 4.0 (https://creativecommons.org/licenses/by-nc/4.0/)

• mirobody by thetahealth (https://github.com/thetahealth/mirobody)

  • Design inspiration only (no code reuse): the controlled-vocabulary alignment spectrum (symbolic decision over learned ranking in high-error-cost domains), abstain-as-first-class-state, and COVERAGE_FLOOR ratchet discipline shaped the v0.16 metadata verification & self-assessment roadmap. Licensed under its own terms (https://github.com/thetahealth/mirobody)

Links

• Full Usage Guide
• Architecture
• Developer Guide
• Paper Store Reference