myrmex 0.1.1

The compact web crawling toolkit

myrmex

The compact web crawling toolkit.

Unlike full-featured frameworks, myrmex does not implement an entire scraping pipeline. Instead, it focuses exclusively on core crawling functionality. Higher-level scraping logic is left to the specific implementation of your scraper.

If you're looking for a complete scraping framework, consider Scrapy.

myrmex provides a minimal interface through two crawler classes — Crawler and TorCrawler — for regular HTTP crawling and Tor-based anonymous crawling, respectively.

Key Capabilities

• Asynchronous context management for automatic resource handling
• Built on aiohttp for HTTP requests
• Executes synchronous operations using the native asyncio thread pool (non-blocking)
• Functional-style error handling via Result
• Configurable per-operation timeouts for robust request management

Installation

Install via pip:

pip install myrmex

Or using uv:

uv add myrmex

Please note that the following libraries will be installed alongside myrmex:

• aiohttp – for HTTP requests
• aiohttp-socks – for SOCKS5 proxy support
• stem – for Tor control port integration
• result – for functional-style error handling

Configuration

Crawler accepts the following options:

Parameter	Type	Default	Description
timeout	int	10	Timeout (in seconds) for HTTP requests.
headers	dict	None	HTTP headers to include with each request.

…and TorCrawler accepts the following options during initialization:

Parameter	Type	Default	Description
address	str	None	SOCKS5 proxy address for routing traffic through Tor.
password	str	None	Control port password for authenticating with the Tor proxy.
timeout	int	10	Timeout (in seconds) for HTTP requests.
headers	dict	None	HTTP headers to include with each request.

Usage Example

The example below demonstrates how to fetch your current IP address over the Tor network:

import asyncio
from myrmex import TorCrawler

async def main():
    async with TorCrawler("socks5h://127.0.0.1:9050", password="password") as crawler:
        await crawler.rotate_ip()  # optional: rotates IP before request
        result = await crawler.fetch("http://httpbin.org/ip")
        if result.is_ok():
            print("Current IP:", result.unwrap())

asyncio.run(main())

Tor Setup

Since TorCrawler is strictly associated with Tor network usage, ensure that you have a configured and running Tor instance before using it.

Update your torrc configuration file with the following:

SocksPort 0.0.0.0:9050
ControlPort 0.0.0.0:9051
HashedControlPassword ***

To generate a hashed password:

tor --hash-password your_password

Start Tor manually in the background:

tor &