myrmex

The compact web crawling toolkit.

Unlike full-featured frameworks, myrmex does not implement an entire scraping pipeline. Instead, it focuses exclusively on core crawling functionality. Higher-level scraping logic is left to the specific implementation of your scraper.

If you're looking for a complete scraping framework, consider Scrapy.

myrmex provides a minimal interface through two crawler classes — Crawler and TorCrawler — for regular HTTP crawling and Tor-based anonymous crawling, respectively.

Key Capabilities

• Asynchronous context management for automatic resource handling
• Built on aiohttp for HTTP requests
• Executes synchronous operations using the native asyncio thread pool (non-blocking)
• Functional-style error handling via Result
• Configurable per-operation timeouts for robust request management

Installation

Install via pip:

pip install myrmex

Or using uv:

uv add myrmex

Please note that the following libraries will be installed alongside myrmex:

• aiohttp – for HTTP requests
• aiohttp-socks – for SOCKS5 proxy support
• stem – for Tor control port integration
• result – for functional-style error handling

Configuration

Crawler accepts the following options:

…and TorCrawler accepts the following options during initialization:

Usage Example

The example below demonstrates how to fetch your current IP address over the Tor network:

import asyncio
from myrmex import TorCrawler

async def main():
    async with TorCrawler("socks5h://127.0.0.1:9050", password="password") as crawler:
        await crawler.rotate_ip()  # optional: rotates IP before request
        result = await crawler.fetch("http://httpbin.org/ip")
        if result.is_ok():
            print("Current IP:", result.unwrap())

asyncio.run(main())

Tor Setup

Since TorCrawler is strictly associated with Tor network usage, ensure that you have a configured and running Tor instance before using it.

Update your torrc configuration file with the following:

SocksPort 0.0.0.0:9050
ControlPort 0.0.0.0:9051
HashedControlPassword ***

To generate a hashed password:

tor --hash-password your_password

Start Tor manually in the background:

tor &