Configurable crawler for web-scraping
Project description
arc-crawler
arc-crawler is a flexible Python module designed to simplify complex web scraping tasks.
It focuses on efficient, resumable data acquisition, structured output management,
and customizable data processing.
✨ Features
- Flexible Fetching Strategies: Supports both high-speed parallel and precise sequential URL fetching with customizable delays.
- Resumable Progress Tracking: Automatically tracks progress, allowing seamless resumption from interruptions and incremental data appending.
- Efficient Output Management: Writes metadata alongside fetched data,
making very large output files easily accessible and queryable via built-in
IndexReader. - Customizable Data Processing: Offers flexible callback functions for pre-request actions, post-response processing, and custom metadata indexing.
🚀 Quick Start
Installation
You can easily install arc-crawler using pip:
pip install arc-crawler
Basic Usage
This example shows you how to quickly crawl a few Wikipedia pages
and save only their <body> contents using the built-in html_body_processor.
from arc_crawler import Crawler, html_body_processor
def crawl_wikipedia():
urls_to_fetch = [
"https://en.wikipedia.org/wiki/JavaScript",
"https://en.wikipedia.org/wiki/Go_(programming_language)",
"https://en.wikipedia.org/wiki/Python_(programming_language)",
"https://en.wikipedia.org/wiki/Node.js",
]
# Initialize the crawler, outputting to a new './output' directory
crawler = Crawler(out_file_path="./output")
# Fetch URLs, process responses to save only the <body> tag, and save to 'wiki-programming' files
reader = crawler.get(
urls_to_fetch,
out_file_name="wiki-programming",
response_processor=html_body_processor, # Use the built-in processor
)
print(f"Successfully gathered {len(reader)} Wikipedia entries.")
print(f"Dataset file is located at: {reader.path}")
if __name__ == "__main__":
crawl_wikipedia()
📚 More Examples & Advanced Usage
For more detailed examples demonstrating advanced features like custom response/request processors, metadata indexing, and handling json content, please refer to the basic.py and advanced.py file within the repository.
Additionally, for comprehensive API documentation, including all available parameters, return types,
and internal workings, explore the detailed docstrings within the arc-crawler module's source code
(e.g., use help(Crawler) or help(Crawler.get) in your Python interpreter or access docs via IDE Tools).
💡 Best Practice: Separate Scraping from Parsing
For optimal efficiency and maintainability in web scraping, it is strongly recommended
to prioritize saving raw, unaltered response data first within your response_processor callbacks.
Avoid implementing complex parsing logic directly during the fetching phase.
This approach offers significant advantages:
- Iterative Improvement: You can refine and improve your parsing logic on local data without needing to re-fetch everything each time.
- Resilience & Speed: The fetching stage focuses purely on data acquisition, improving resilience to network issues and maximizing download speed.
- Resource Management: Offload CPU and memory-intensive parsing to a separate process or script, preventing it from slowing down I/O-bound fetching operations.
In essence, with arc-crawler, focus on saving the raw source data. Parsing is a distinct, subsequent step best performed after fetching.
Changelog
0.1.1
Minor performance optimizations and structural changes
- Performance improvements for cases when comparing large list of target URLs against those already fetched.
- Moved
IndexReaderrelated imports toarc_crawler.readerfor clarity.
0.1.0
Initial release
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file arc_crawler-0.1.1.tar.gz.
File metadata
- Download URL: arc_crawler-0.1.1.tar.gz
- Upload date:
- Size: 19.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/2.1.3 CPython/3.12.6 Windows/11
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
a017ffb0e3353a5c7dd8fef1c615049060ab55f7766643c603582618cba51cae
|
|
| MD5 |
5734e2e3bb0c1ac84ceaf88b33efc1bf
|
|
| BLAKE2b-256 |
6f8d0729a6b7611f1cc7fed01b04271be58b9a8969d12d04a47fc2b4b847526f
|
File details
Details for the file arc_crawler-0.1.1-py3-none-any.whl.
File metadata
- Download URL: arc_crawler-0.1.1-py3-none-any.whl
- Upload date:
- Size: 24.6 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/2.1.3 CPython/3.12.6 Windows/11
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
a70d302a10370218d0b70d6b8957cc23a6ccce0b2bfe31c16c2cc6f136ff8d3d
|
|
| MD5 |
e2523a77c4316baab50bf0951faf2cae
|
|
| BLAKE2b-256 |
a666709c66a020fb7d53fcec15e818b67e81a5e6818521e66bad591b2f9e070c
|
