Agastya

High-performance web scraping and MongoDB utilities for Python.

Features

• CustomLogger: Fast, color-coded logging with minimal overhead
• Record: Lightweight MongoDB document wrapper with state management
• fetch(): Smart HTTP client with caching, retries, and validation
• clean_dict(): Comprehensive text/HTML cleaning utilities
• run_engine(): Adaptive concurrent task executor with auto-scaling threads

Installation

pip install agastya

Quick Start

Basic Usage

from agastya import fetch, CustomLogger, Record

# Initialize logger
logger = CustomLogger(name=__file__)

# Fetch with caching
response = fetch(
    url="https://example.com",
    save_dir="cache",
    filename="page.html",
    max_retries=3
)

# Work with MongoDB records
from pymongo import MongoClient
db = MongoClient()['mydb']
collection = db['items']

doc = collection.find_one({"_id": "123"})
record = Record(doc, collection, logger)

# Mark states
record.mark_done("scraped", meta={"items": 42})
record.mark_fail("parsed", meta={"error": "timeout"})

Configuration

Create a config.py in your project root:

from pymongo import MongoClient

client = MongoClient("mongodb://localhost:27017")
db = client['your_database']

Advanced Features

Smart caching with validation:

response = fetch(
    url="https://example.com/data",
    save_dir="cache",
    filename="data.html",
    valid_rules={
        "required": ["expected text"],
        "forbidden": ["error", "404"],
        "remove": True  # delete invalid cache
    }
)

Concurrent processing:

from agastya import run_engine
from pymongo import MongoClient

db = MongoClient()['mydb']
collection = db['tasks']

def worker(doc):
    # Process document
    print(f"Processing {doc['_id']}")

query = {"status": "pending"}
run_engine(collection, query, worker, THREADS_START=50)

Requirements

• Python 3.8+
• requests
• lxml
• pymongo
• urllib3

Development

# Clone repository
git clone https://github.com/yourusername/agastya.git
cd agastya

# Install in editable mode with dev dependencies
pip install -e ".[dev]"

# Run tests
pytest

License

MIT License - see LICENSE file for details