Grabber

A flexible and extensible template-based web crawler with automatic site detection. Easily crawl news articles, stock data, and more from Korean websites.

✨ Features

• 🎯 Automatic Template Detection: Automatically selects the optimal template based on URL
• 🚀 Simple Interface: One-line crawling with the Grabber class
• ⚡ High-Performance Async: Optimized async crawling up to 10,000+ RPS locally
• 📰 30+ Supported Sites: Major Korean news sites and financial platforms
• 🔧 Extensible: Easy to add custom templates for new sites
• 💾 Multiple Export Formats: Save as JSON, CSV, or TXT
• 🔄 Batch Processing: Crawl multiple URLs efficiently with connection pooling
• 🛡️ Smart Bot Detection Bypass: Advanced header management and session persistence
• 🎛️ Adaptive Performance: Dynamic concurrency and delay adjustment based on site response
• 📊 Site Profiling: Pre-configured optimal settings for each news site
• 🖥️ Mode-based CLI: One grabber command for HTML, news, audio and site-exploration modes
• 🤖 Agent-friendly: JSON/NDJSON output, meaningful exit codes, and a self-describing capabilities command

📦 Installation

git clone https://github.com/unohee/grabber.git
cd grabber
pip install -e .

# With JS rendering support (SPA/SSR news sites)
pip install -e ".[js]" && playwright install chromium

🖥️ Command Line

Installing the package puts a grabber command on your PATH. Modes are subcommand groups:

grabber doctor                 # check binaries, optional extras and credentials
grabber grab <URL>             # HTML / article scraping
grabber news search "반도체"    # keyword search via the Naver News API
grabber audio get <URL>        # YouTube / SoundCloud audio download
grabber explore run suno       # Suno / Udio media discovery
grabber probe <URL>            # which template and backend would handle this URL?
grabber capabilities           # dump the whole command surface as JSON

HTML mode

# Single URL, human output
grabber grab "https://www.newsis.com/view/NISX20240101_0002"

# Batch from a file, 8 workers, machine-readable stream
grabber grab -f urls.txt -w 8 --ndjson > out.jsonl

# From stdin, trimmed for an LLM's context budget
cat urls.txt | grabber grab -f - --json --max-chars 2000 --fields url,title,content

# Export to CSV / Markdown (format inferred from the extension)
grabber grab "$URL" -o article.md

News mode

Needs NAVER_CLIENT_ID / NAVER_CLIENT_SECRET in .env (or --env-file).

grabber news search "삼성전자" --days 3 --max 50 --json
grabber news collect "환율" --max 10 --max-chars 1500 -o out.md   # search + scrape

Audio mode

Requires yt-dlp on PATH. Downloads go to $GRABBER_DATA_DIR/audio/<source>/ (default ~/.grabber), or -d <dir>.

grabber audio info "https://youtu.be/VIDEOID" --json      # metadata only
grabber audio search "lofi piano" -n 10 --urls-only       # SoundCloud search
grabber audio get "https://youtu.be/VIDEOID" -d ./downloads

# Long corpus runs: queue once, drain repeatedly (resume-safe)
grabber audio queue -f songs.csv --source youtube
grabber audio batch --source youtube -w 4 --limit 500

audio queue accepts a plain URL list or a CSV carrying a youtube_id, video_id, track_id or url column.

Connections are direct by default; pass --proxy URL or --proxy-pool to route through a proxy. Recent yt-dlp versions need a JavaScript runtime (e.g. deno) for YouTube downloads — grabber doctor flags this. SoundCloud is unaffected.

Explore mode

Requires the js extra (pip install -e ".[js]" && playwright install chromium).

grabber explore run suno --pages 20 --batches 3
grabber explore stats suno --json
grabber explore export suno --type audio -o suno_audio.txt
grabber explore download suno --type audio --limit 100

For agents

grabber capabilities --json     # every command, option, type, default and choice

stdout carries only the payload; progress and logs go to stderr. Exit codes: 0 success · 1 partial · 2 usage error · 3 all failed · 4 missing dependency or credentials · 5 no results. Full contract — record schema, error codes, recipes — in docs/AGENT_CLI.md.

🚀 Quick Start

Simple Usage

from grabber import Grabber

# One-line crawling
data = Grabber.quick_grab("https://www.ajunews.com/view/20240101000000000")

if data:
    print(f"Title: {data.data.get('title')}")
    print(f"Content: {data.data.get('content')}")

Basic Usage

from grabber import Grabber

# Create a Grabber instance
grabber = Grabber()

# Crawl a news article (automatic template detection)
data = grabber.grab("https://n.news.naver.com/article/001/0014000000")

if data:
    print(f"Source: {data.source}")
    print(f"Title: {data.data.get('title')}")
    print(f"Content: {data.data.get('content')}")

Batch Crawling

# Crawl multiple URLs
urls = [
    "https://www.ajunews.com/view/20240101000000000",
    "https://www.businesspost.co.kr/BP?command=article_view&num=123456",
    "https://www.thebell.co.kr/free/content/ArticleView.asp?key=202401010000000000"
]

results = grabber.grab_batch(urls)

for result in results:
    if result:
        print(f"{result.source}: {result.data.get('title')}")

Stock Data Crawling

# Crawl stock information
data = grabber.grab_stock("005930", source="naver")  # Samsung Electronics

if data:
    print(f"Company: {data.data.get('company_name')}")
    print(f"Price: {data.data.get('current_price')}")

🆕 Asynchronous Crawling (High Performance)

import asyncio
from grabber.core import AsyncTemplateCrawler
from templates.async_naver_news_template import AsyncNaverNewsTemplate

async def async_crawl():
    # Create async template with optimization
    template = AsyncNaverNewsTemplate()
    
    # Create async crawler with performance tuning
    async with AsyncTemplateCrawler(
        template=template, 
        max_concurrent=32,  # Optimized for Ryzen 5800X
        delay=0.3,          # Optimal delay for news sites
        timeout=30
    ) as crawler:
        # Single crawl
        result = await crawler.crawl("인공지능")
        
        # Multiple crawls concurrently  
        keywords = ["AI", "머신러닝", "딥러닝"]
        results = await crawler.crawl_multiple(keywords)
        
        # Performance: 10x faster than synchronous
        print(f"Crawled {len(results)} keywords")
        
# Run async crawling
asyncio.run(async_crawl())

📋 Supported Sites

News Sites

• 아주뉴스 (ajunews.com)
• 비즈니스포스트 (businesspost.co.kr)
• 이데일리 (edaily.co.kr)
• 이투데이 (etoday.co.kr)
• 한국경제 (hankyung.com)
• 매일경제 (mk.co.kr)
• 네이버 뉴스 (news.naver.com)
• 뉴시스 (newsis.com)
• 뉴스핌 (newspim.com)
• 더벨 (thebell.co.kr)
• And 20+ more...

Financial Sites

• 네이버 금융 (finance.naver.com)
• FnGuide (comp.fnguide.com)

🔧 Advanced Usage

Custom Template

from grabber import Grabber, SiteTemplate

class MyCustomTemplate(SiteTemplate):
    def get_site_name(self):
        return "MyCustomSite"
    
    def build_url(self, target):
        return f"https://mycustomsite.com/{target}"
    
    def extract_data(self, soup):
        return {
            "title": soup.find("h1").text,
            "content": soup.find("article").text
        }

# Use custom template
grabber = Grabber(template=MyCustomTemplate)
data = grabber.grab("article/123")

Save Results

# Save crawled data
data = grabber.grab(url)

if data:
    # Save as JSON
    grabber.save_to_file(data, "output.json", format="json")
    
    # Save as CSV
    grabber.save_to_file(data, "output.csv", format="csv")
    
    # Save as TXT
    grabber.save_to_file(data, "output.txt", format="txt")

List Available Templates

# Get all available templates
templates = grabber.list_templates()
for name, site_name in templates.items():
    print(f"{name}: {site_name}")

# Get supported domains
domains = grabber.get_supported_domains()
print(f"Supported domains: {domains}")

🏗️ Architecture

grabber/
├── grabber/     # Core package
│   ├── core/            # Core components
│   ├── grabber.py       # Main interface
│   └── tools/           # Utilities
├── templates/           # Site-specific templates
│   ├── *_template.py    # Individual site templates
│   ├── extractors/      # Data extractors
│   └── validators/      # Data validators
└── examples/            # Usage examples

🧪 Testing

# Install development dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Run with coverage
pytest --cov=grabber

🤝 Contributing

Contributions are welcome! To add support for a new site:

1. Create a new template in templates/ directory
2. Inherit from SiteTemplate class
3. Implement required methods:
   • get_site_name()
   • build_url(target)
   • extract_data(soup)
   • clean_data(raw_data)
   • validate_data(data)

Example:

from grabber.core.site_template import SiteTemplate

class NewSiteTemplate(SiteTemplate):
    def get_site_name(self):
        return "NewSite"
    
    def build_url(self, target):
        return f"https://newsite.com/{target}"
    
    def extract_data(self, soup):
        # Implementation here
        pass

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Built with BeautifulSoup4 and Selenium
• Uses undetected-chromedriver for anti-detection
• Inspired by the need for flexible web scraping solutions

📧 Contact

• Issues: GitHub Issues

🚀 Performance Optimization

System Requirements

• CPU: Multi-core processor recommended (tested on AMD Ryzen 7 5800X)
• RAM: 4GB+ recommended for large-scale crawling
• Network: Stable internet connection with low latency

Performance Benchmarks

Site	RPS	Concurrency	Success Rate
Naver News	11.9	32	100%
HTTPBin (local)	10,544	128	100%
General News	7-15	16-32	95%+

Optimization Tips

1. Use async crawling for I/O-bound operations
2. Adjust concurrency based on target site's rate limits
3. Enable connection pooling for batch operations
4. Use site profiling for automatic optimization

📈 Roadmap

• Add more news sites
• Support for international sites
• Async crawling support (v1.1.0)
• Performance optimization for news sites (v1.2.0)
• Smart bot detection bypass (v1.2.0)
• REST API interface
• Docker support
• Cloud deployment guides
• Distributed crawling support

---

Made with ❤️ by unohee