ankur-scraper 0.1.6

A robust website scraper that supports static and dynamic pages with intelligent content extraction.

🕷️ Ankur Scraper

Ankur Scraper is a modular, production-ready website scraping tool built with Python. It crawls and extracts structured content from websites — including dynamic pages rendered with JavaScript — and saves the results in a clean JSON format.

---

🚀 Features

• ✅ Crawl internal links (with max depth)
• ✅ Extract visible, structured text (section-wise)
• ✅ Supports static and dynamic (JS-rendered) pages
• ✅ Respects robots.txt
• ✅ CLI interface with arguments
• ✅ Logs everything to file + rich-colored terminal
• ✅ Testable, extensible, and publishable as a Python package

---

📦 Project Structure

ankur_scraper/
├── ankur_scraper
│   ├── core/              # Crawling and extraction logic
│   ├── logs/              # Package-level logs
│   ├── cli.py             # Command-line interface
│   ├── dispatcher.py      # Orchestrates scraper execution
│   └── _version.py
├── tests/                 # Unit & integration tests
├── docs/                  # 📖 Documentation (per-module)
├── logging_config.py
├── setup.py
├── requirements.txt
└── README.md

📖 Module Documentation

• Core Scrapers (core/)
• Command Line Interface (cli.py)
• Dispatcher (dispatcher.py)
• Logging System
• Testing Guide

---

🔧 Installation

Clone the project and install dependencies:

git clone https://github.com/your-org/ankur_scraper.git
cd ankur_scraper
python -m venv venv
source venv/bin/activate  # or venv\Scripts\activate on Windows

pip install -r requirements.txt

# Install Playwright drivers (for dynamic scraping)
playwright install

---

🕹️ Usage (CLI)

python -m ankur_scraper.cli \
  --url "https://example.com" \
  --depth 1 \
  --dynamic \
  --timeout 10

Command Line Options

Option	Description
--url	Starting URL to scrape (required)
--depth	How deep to crawl within the domain
--dynamic	Use dynamic scraping (Playwright)
--timeout	Timeout for each page (seconds)

---

📝 Usage Examples

# Basic usage
ankur-scraper --url https://example.com

# With depth and output
ankur-scraper --url https://example.com --depth 2

# With dynamic scraping and timeout
ankur-scraper --url https://example.com --dynamic --timeout 30

from ankur_scraper.dispatcher import run_scraper

data = run_scraper("https://example.com/", depth=1)

{
  "data": [
    {
      "content": "...",
      "metadata": {...}

    }
  ],
  "summary": {
    "successful_pages": success_count,
    "failed_pages": fail_count,
    "total_sections": total_sections
  }
}

---

🧪 Running Tests

Unit tests:

pytest tests/ --tb=short

Integration tests (live web):

pytest tests/test_integration.py -m integration

Add pytest.ini for markers:

[pytest]
markers =
    integration: mark tests as integration

---

📄 Output Format

Every page section is saved as a structured object:

{
  "content": "Text content here...",
  "metadata": {
    "section": "About Us",
    "source_url": "https://example.com/about",
    "extraction_time": "2025-07-14 12:34:56"
  }
}

---

📚 Logging

Logs are written to both terminal and file:

• logs/info.log → general operations
• logs/error.log → failed links and errors
• logs/general.log → warnings, summaries

Terminal output is rich-colored with emojis and timestamps. (See Logging Docs for details.)

---

📦 Packaging & Publishing

This scraper is structured as a pip-installable package.

Install locally:

pip install .

Run from anywhere:

ankur-scraper --url "https://example.com"

Publish to PyPI

pip install build twine
python -m build
twine upload dist/*

⚠️ Update version in setup.py before publishing!

---

📌 Dependencies

• httpx – Fast, async-capable HTTP requests
• beautifulsoup4 + lxml – HTML parsing
• tldextract – Domain filtering
• playwright – JS rendering (headless)
• rich – Beautiful terminal output
• pytest – Testing

---

🤝 Contributing

1. Fork the repo
2. Make changes in a branch
3. Run tests: pytest
4. Submit a PR

---

🧠 Future Roadmap

• Save to Markdown or plaintext
• URL exclusion filters
• Config file / ENV mode
• Docker support
• CI/CD pipeline

---

🧑‍💻 Maintainer

Made with ❤️ by Ankur Global Solutions