crawlier 0.0.0.1

A compact, multi-threaded web crawler for desktop (PansiluBot) and mobile (MethmiBot) modes

Crawlier — Alpha v0.0.0.1

A compact, friendly README for the Crawlier project (PansiluBot / MethmiBot).

---

What is Crawlier?

Crawlier is a Python web crawler that supports both a desktop mode (PansiluBot) and a mobile mode (MethmiBot). It focuses on practical, configurable crawling with useful data extraction features such as metadata, links, forms, images, files, technology detection, and optional CAPTCHA integration.

Features

• Multi-threaded crawling
• Configurable crawl depth and per-request delay
• Optional respect for robots.txt (toggleable)
• Keyword, metadata, images, videos, forms, files and social link extraction
• Technology detection and basic SEO analysis
• Outputs to JSON, CSV and an SQLite database
• Optional CAPTCHA solver integration (requires external service)
• Live logging support (suitable for UIs like Gradio)

---

Quick Install

Clone the repo and install dependencies:

git clone https://github.com/yourusername/crawlier.git
cd crawlier
pip install -r requirements.txt

Run tests (optional):

python -m unittest discover tests

Note: Python 3.10+ recommended.

---

CLI Usage

Run a basic crawl:

python -m crawlier -d example.com -m pc

Deep crawl (mobile UA, many threads):

python -m crawlier -d example.com -m mobile -t 20 --delay 0.5 --depth 5

Ignore robots.txt (only if you have permission):

python -m crawlier -d example.com -m pc --no-robots

CLI options summary:

• -d, --domain : Target domain to crawl (required)
• -m, --mode : Crawler mode (mobile / pc), default: pc
• -t, --threads : Max concurrent threads (default: 10)
• --delay : Delay between requests in seconds (default: 1.0)
• --depth : Maximum crawl depth (default: 3)
• --no-robots : Ignore robots.txt rules
• --captcha-key : API key for CAPTCHA solving service
• -o, --output : Output file (default: crawl_results.json)
• --db : SQLite database file (default: crawl_data.db)

---

Python API (programmatic)

Use run_crawl() for live logs (generator style):

from crawlier import run_crawl

for log in run_crawl("https://example.com", mode="pc", max_threads=5, delay=1, max_depth=2):
	print(log)

Use the Crawlier class directly for fine-grained control:

from crawlier import Crawlier

crawler = Crawlier(
	target_domain="example.com",
	mode="pc",
	max_threads=5,
	delay=1,
	max_depth=2,
	respect_robots=True,
	captcha_solver=None,
	db_file="crawl_data.db"
)

crawler.start_crawl()
crawler.save_results("crawl_results.json")
crawler.close()

---

Output

Crawlier writes:

• JSON output (default crawl_results.json)
• CSV export of URL details
• An SQLite DB (default crawl_data.db)
• A human-friendly text report alongside JSON (if enabled)

Make sure the output/ directory exists before running, or provide a path that does.

---

Example: run_crawl() with mobile UA

for log in run_crawl(
	url="https://example.com",
	mode="mobile",
	max_threads=10,
	delay=0.5,
	max_depth=3,
	ignore_robots=False
):
	print(log)

---

Changelog — Alpha v0.0.0.1

• Initial release
• Supports PC (PansiluBot) and Mobile (MethmiBot) modes
• Multi-threaded crawling with configurable depth/delay
• Robots.txt respect toggle
• Optional CAPTCHA solving (requires external service)
• Live logging for UI integration (Gradio)

---

Notes & Tips

• Use conservative max_threads / delay values on large sites to avoid rate-limiting.
• Respect site terms of service and robots.txt unless you have permission to ignore it.
• For CAPTCHA handling or browser-level JS challenges, integrate a headless browser or external solver.

---

If you'd like, I can also:

• add badges (build / python version)
• create a minimal setup.py / pyproject.toml
• extract requirements.txt and validate imports

Enjoy!