SmartSpider 0.0.3

A modular adaptive web crawler framework with incremental, continuous, and scope-limited crawling support.

Summary — The 9 Foundational Crawler Categories

Category	Core Question It Answers	Example Type
Purpose	Why is it crawling?	Scraping crawler
Crawl Strategy	How does it traverse pages?	BFS, DFS, Focused
Scheduling Behavior	When does it crawl?	Incremental, Continuous
Architecture	How is it organized?	Distributed crawler
Scope	How wide does it crawl?	Site-wide crawler
Data Access Method	Where does it get data from?	API crawler
Ethical Behavior	How does it treat site rules?	Polite crawler
Intelligence Level	How does it make decisions?	LLM-guided crawler
Integration Role	How does it fit into a pipeline?	Modular crawler

By Purpose (Goal or Intent)

Defines why the crawler exists — what it aims to achieve.

Type Description

• Discovery Crawler - Finds and collects URLs or metadata; builds an index or frontier.
• Scraping Crawler - Extracts structured data from pages (e.g., tables, text, entities).
• Monitoring Crawler - Tracks updates or changes in content over time.
• Archival Crawler - Saves full page copies for preservation or offline analysis.
• Testing / Auditing Crawler - Used for SEO, broken-link checking, or site compliance validation.

By Crawl Strategy (Traversal Logic)

Defines how pages are selected or ordered during crawling.

Type Description

• Breadth-First (BFS) - Crawl all pages at one depth before moving deeper.
• Depth-First (DFS) - Crawl one path as deep as possible before backtracking.
• Priority-Based - Assign numerical priority scores to URLs.
• Adaptive Adjust - strategy dynamically based on feedback or results.
• Context-Aware - Use HTML structure and semantics to guide crawling decisions.

By Scheduling Behavior (Temporal Logic)

Defines when and how often the crawler operates.

Type Description

• One-Shot - Runs once and stops after completion.
• Incremental - Re-crawls known pages periodically to detect updates.
• Continuous - Never stops constantly cycles through crawl/revisit loops.
• Event-Driven - Triggered by signals such as webhooks or detected changes.

By Scope (Coverage Target)

Defines how broadly the crawler explores the web.

Type Description

• Focused - only relevant pages based on topic or keyword.
• Site-Wide - Restricted to one domain or subdomain.
• Multi-Domain - Crawls a fixed list of domains.
• Vertical / Domain-Specific - Focused on one industry (e.g., sports, jobs, e-commerce).
• Web-Scale - Crawls the entire public web; search-engine level.

By Architecture (System Design)

Defines how the crawler is built and organized internally.

Type-Description

• Centralized Single control node managing all crawl tasks.
• Distributed Multiple coordinated nodes sharing workload and URL queues.
• Peer-to-Peer Decentralized — nodes share discovered URLs without a central coordinator.
• Cloud / Serverless Uses scalable, ephemeral functions to perform crawl tasks.

By Data Access Method

Defines where and how data is retrieved.

Type Description

• HTML / Page Crawler Fetches and parses HTML pages.
• API Crawler Pulls structured data through APIs.
• Headless / Rendered Crawler Uses browsers or headless engines to handle JavaScript.
• Hybrid Crawler Combines HTML, API, and headless methods adaptively.

By Ethical or Policy Behavior

Defines how the crawler interacts with site policies.

Type Description

• Polite Crawler Obeys robots.txt, rate limits, and crawl delays.
• Aggressive Crawler Ignores some restrictions (not recommended).
• Authenticated Crawler Operates within login-required environments.

By Intelligence Level

Defines how much decision-making or AI the crawler uses.

Type Description

• Rule-Based Follows fixed rules or regex filters.
• Heuristic-Based Uses handcrafted scoring or relevance functions.
• ML-Guided Uses machine learning for link scoring or prioritization.
• LLM-Guided Uses large language models to interpret pages and guide navigation.

By Integration Role (System Position)

Defines how the crawler fits into the larger data pipeline.

Type Description

• Standalone Crawler - Operates independently, outputs raw pages or URLs.
• Coupled Crawler - Integrates scraping logic directly inside the crawl loop.
• Modular Crawler - Works with separate scraper, parser, and storage components.
• Streaming Crawler - Feeds data continuously into real-time pipelines (Kafka, etc.).