crawly: the literature assistant

crawly (crawly: the literature assistant) is an end-to-end academic research assistant for discovering scientific papers across 8+ official Open Access databases, downloading full-text PDFs within a strict Legal Open-Access framework by default, exporting BibTeX citations, and generating AI-powered literature reviews using either local self-hosted LLMs (Ollama, LM Studio, vLLM, llama.cpp) or commercial/OpenAI-compatible APIs (OpenAI, Gemini, Anthropic, OpenRouter, Groq, DeepSeek).

• Repository: https://github.com/lokeshsk/crawly
• Documentation & Command Builder Website: https://lokeshsk.github.io/crawly/
• Maintainer: Lokesh Kumar (crawly.theliteratureassistant@gmail.com)

---

1. Quick Start for Users (No pip install or Dependencies Required!)

⚡ One-Line Instant Install (Zero Setup)

• Linux / macOS / WSL:

  curl -fsSL https://lokeshsk.github.io/crawly/install.sh | bash
  

• Windows (PowerShell):

  irm https://lokeshsk.github.io/crawly/install.ps1 | iex
  

• Python / PyPI (pip):

  pip install crawly-ai
  

---

Manual Download & Repository Clone

Option A: Universal Cross-Platform Bundle (Works on BOTH Windows & Linux / WSL)

Unzip dist/crawly-universal-windows-linux.zip (or run directly from dist/):

• On Windows (Command Prompt / PowerShell / Double-Click):

  dist\crawly.bat
  

• On Linux / WSL / macOS:

  ./dist/crawly
  

• Direct Single-File Archive (crawly.pyz):

  python dist/crawly.pyz
  

Option B: Native Standalone Binary (Zero Python Setup Needed)

• Linux / WSL:

  ./dist/crawly-linux-x86_64
  

• Windows:

  .\dist\crawly-windows-x86_64.exe
  

---

2. Setting Up & Customizing config.json

All configuration values—including AI providers, model names, base URLs, API keys, temperature, PDF parsing limits, default search databases, timeouts, user-agents, and API endpoints—are stored in config.json in the project root (or current working directory).

Nothing is hardcoded: if config.json is deleted, crawly automatically regenerates it with clean default values and built-in _documentation labels.

Three Ways to Change Settings in config.json

1. Edit config.json directly in any text editor (VS Code, Notepad, nano, vim).
2. Interactive Terminal UI: Launch ./dist/crawly and choose Option 5 (Configure AI Models & Settings).
3. CLI Command:

   # View current config.json status
   ./dist/crawly config --show
   
   # Update values in config.json via CLI flags
   ./dist/crawly config --provider ollama --model llama3.2 --temperature 0.2
   ./dist/crawly config --provider lmstudio --base-url http://localhost:1234/v1 --model local-model
   ./dist/crawly config --provider openai_compatible --base-url https://api.openai.com/v1 --api-key sk-... --model gpt-4o-mini
   ./dist/crawly config --provider gemini --api-key YOUR_GEMINI_KEY --model gemini-2.5-flash
   ./dist/crawly config --provider anthropic --api-key YOUR_ANTHROPIC_KEY --model claude-3-5-sonnet-latest
   

---

Complete config.json Field Reference

Variable Name	Default Value	What to Put / Allowed Values
ai_provider	"ollama"	Which AI engine to use: • "ollama" — Local self-hosted Ollama server • "lmstudio" — Local self-hosted LM Studio server • "openai_compatible" — Any OpenAI-compatible server (OpenAI, OpenRouter, Groq, DeepSeek, Together, vLLM, llama.cpp) • "gemini" — Google Gemini API • "anthropic" — Anthropic Claude API • "extractive" — Offline NLP & BM25 extractive engine (no LLM server or API key needed)
ai_model	"llama3.2"	Model identifier for your chosen provider: • Ollama: "llama3.2", "qwen2.5:7b", "mistral", "deepseek-r1:8b" • LM Studio: "local-model" or the loaded GGUF model ID • OpenAI / OpenRouter: "gpt-4o-mini", "deepseek-chat", "meta-llama/llama-3.1-8b-instruct" • Gemini: "gemini-2.5-flash", "gemini-2.5-pro" • Anthropic: "claude-3-5-sonnet-latest"
ai_base_url	"http://localhost:11434"	Server base URL for local or OpenAI-compatible providers: • Ollama: "http://localhost:11434" • LM Studio: "http://localhost:1234/v1" • vLLM / llama.cpp: "http://localhost:8000/v1" • OpenAI: "https://api.openai.com/v1" • OpenRouter: "https://openrouter.ai/api/v1" • Groq: "https://api.groq.com/openai/v1"
ai_api_key	""	API key for commercial/cloud providers (openai_compatible, gemini, anthropic). Leave "" for local Ollama or LM Studio. Can also be set via env vars (CRAWLY_AI_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY, ANTHROPIC_API_KEY).
temperature	0.2	Float between 0.0 and 1.0. Lower values (0.1–0.3) produce more factual, deterministic literature summaries.
max_pdf_pages	25	Maximum number of PDF pages to extract per paper with pypdf for AI summarization and RAG indexing.
max_text_chars	14000	Maximum characters of extracted full-text PDF content passed to the LLM per paper.
default_sources	["openalex", "semantic_scholar", "arxiv", "europe_pmc", "crossref", "doaj", "hal", "openaire"]	List of open academic databases searched by default when --sources is not specified.
default_limit_per_source	10	Default maximum number of papers to fetch from each database per search query.
default_sort	"relevance"	Default sorting order for search results: "relevance", "desc" (newest year first), "asc" (oldest year first), or "citations" (most cited first).
request_timeout	12	HTTP timeout in seconds for database queries and metadata resolution.
unpaywall_email	"crawly.theliteratureassistant@gmail.com"	Polite contact email sent to the Unpaywall Legal Open-Access API when resolving OA PDF links by DOI.
semantic_scholar_api_key	""	Optional Semantic Scholar API key for higher rate limits (leave "" for public tier).
polite_user_agent	"crawly/2.0 (crawly: the literature assistant; https://github.com/lokeshsk/crawly; mailto:crawly.theliteratureassistant@gmail.com)"	User-Agent header used for polite academic API requests (OpenAlex, Crossref, Unpaywall, Europe PMC).
browser_user_agent	"Mozilla/5.0 ..."	Browser User-Agent header used when downloading PDFs from publisher repositories.
use_scihub_fallback	false	Keep false for strict Legal Open-Access compliance. Even if set to true, crawly displays the Legal Warning & Non-Liability Disclaimer requiring explicit risk acceptance.
scihub_cdn_url	"https://sci.bban.top/pdf/{doi}.pdf"	Template URL ({doi} placeholder) used only if the user explicitly accepts legal risk for third-party fallback.
scihub_mirrors	["https://sci-hub.ee", ...]	List of mirror URLs tried only if the user explicitly accepts legal risk for third-party fallback.
api_endpoints	{...}	Dictionary of base URLs for OpenAlex, Semantic Scholar, arXiv, Europe PMC, Crossref, DOAJ, HAL, OpenAIRE, Unpaywall, Gemini, and Anthropic.

---

Example config.json Setups for Popular AI Providers

A. Local Ollama (Default — Free & Private)

1. Install Ollama (https://ollama.com) and pull a model: ollama pull llama3.2
2. Set in config.json:

{
  "ai_provider": "ollama",
  "ai_model": "llama3.2",
  "ai_base_url": "http://localhost:11434",
  "ai_api_key": "",
  "temperature": 0.2
}

B. Local LM Studio (Free & Private GUI Server)

1. Open LM Studio, load any GGUF model, and start the Local Server on port 1234.
2. Set in config.json:

{
  "ai_provider": "lmstudio",
  "ai_model": "local-model",
  "ai_base_url": "http://localhost:1234/v1",
  "ai_api_key": "",
  "temperature": 0.2
}

C. OpenAI / OpenRouter / Groq / DeepSeek (openai_compatible)

{
  "ai_provider": "openai_compatible",
  "ai_model": "gpt-4o-mini",
  "ai_base_url": "https://api.openai.com/v1",
  "ai_api_key": "sk-proj-YOUR_API_KEY",
  "temperature": 0.2
}

D. Google Gemini API (gemini)

{
  "ai_provider": "gemini",
  "ai_model": "gemini-2.5-flash",
  "ai_api_key": "AIzaSyYOUR_GEMINI_API_KEY",
  "temperature": 0.2
}

E. Offline Extractive Engine (extractive — No LLM or Internet Needed)

{
  "ai_provider": "extractive",
  "ai_model": "extractive-bm25"
}

---

3. CLI Subcommands & Usage

All commands display modern multi-column progress bars (spinner, progress bar, percentage, M/N counter, elapsed time, and ETA) and work identically whether you invoke ./dist/crawly, dist\crawly.bat, ./dist/crawly-linux-x86_64, or python crawly.py:

# 1. Launch Interactive Terminal UI Dashboard
./dist/crawly

# 2. Search across all legal open databases (saves CSV + BibTeX)
./dist/crawly search "synthetic aperture radar" -o sar_papers.csv --limit 10

# 3. Citation Graph Snowballing (trace forward citing & backward referenced works)
./dist/crawly snowball "10.1109/TGRS.2023.123456" -l 10
./dist/crawly snowball "synthetic aperture radar imaging" -o snowball_radar.csv

# 4. Download PDFs using strictly Legal Open Access repositories (Default)
./dist/crawly download sar_papers.csv

# 5. Run AI Literature Review Modes
#    Mode 1: Per-Paper Structured Summaries (Objective, Methodology, Key Findings, Limitations)
./dist/crawly ai sar_papers.csv --mode 1
#    Mode 2: Synthesized Cross-Paper Literature Review & Markdown Comparison Matrix
./dist/crawly ai sar_papers.csv --mode 2
#    Mode 3: Citation-Grounded RAG Q&A over Downloaded PDFs
./dist/crawly ai sar_papers.csv --mode 3 -q "What are the main compressed sensing methods for SAR?"
#    Mode gaps: Automated Research Gaps & Open Challenges Roadmap
./dist/crawly ai sar_papers.csv --mode gaps

# 6. Generate an Interactive Single-File HTML Review Dashboard
./dist/crawly report sar_papers.csv

# 7. Sync Papers & Full-Text PDFs to Local Zotero Desktop
./dist/crawly zotero sar_papers.csv -c "SAR_Survey"

# 8. Literature Watchdog (Scan latest preprints and generate alerts digest)
./dist/crawly watch "SAR despeckling" -o watchdog_radar.md

# 9. Run the complete end-to-end pipeline (Search -> Download Legal OA PDFs -> AI Review)
./dist/crawly pipeline "deep learning SAR despeckling" -o sar_dl --limit 8 --mode all

# 10. Export any CSV dataset to BibTeX (.bib) for Zotero / Mendeley / Overleaf
./dist/crawly export-bibtex sar_papers.csv

# 11. Run System & API Health Doctor
./dist/crawly doctor

---

4. For Contributors & Developers

Every Pull Request submitted to crawly: the literature assistant is verified by the automated GitHub Actions quality gate (.github/workflows/ci-tests-coverage.yml).

# 1. Install developer, test, and build dependencies
pip install -r requirements.txt coverage pytest pytest-cov pyinstaller

# 2. Run the full test suite and enforce 100% code coverage
python -m coverage run -m unittest discover -s tests -v
python -m coverage report -m --fail-under=100

# 3. Run from source
python crawly.py
python downloader.py

# 4. Rebuild dist/crawly.pyz, launchers, universal zip, and native binary
python build_release.py --native

Contact & Credits

• Author & Maintainer: Lokesh Kumar (crawly.theliteratureassistant@gmail.com)
• Credits: Original v1 Google Scholar script inspired by https://jovian.ai/saini-9.

Support / Donation

If you like crawly: the literature assistant, you can buy me a cup of tea :)