Collection of different tools for async web scraping, crawling and parsing
Verified details
These details have been verified by PyPIProject links
GitHub Statistics
Maintainers
Project description
aiofetch
A Python toolkit for asynchronous web scraping with built-in error tracking and metadata management.
Features
Web Processing
- Asynchronous file downloading with progress tracking
- Rate limiting with configurable delays
- Smart retry logic with timeout handling
- Domain-aware crawling with URL validation
Content Processing
- Flexible HTML content parsing
- Custom selector-based metadata extraction
- Automated link and image extraction
- URL normalization and path handling
File & Data Management
- Asynchronous file operations
- Concurrent chunk-based downloads
- Smart path handling and file naming
- JSON data management with validation
Error Handling & Progress Tracking
- Comprehensive error tracking and reporting
- Progress monitoring for long operations
- Detailed logging with configurable outputs
- Operation statistics and summaries
Metadata Management
- Efficient in-memory caching
- Field-based search functionality
- Automatic metadata indexing
- Structured data validation
Installation
git clone https://github.com/spike1236/aiofetch.git
pip install -r requirements.txt
Key Components
- AsyncDownloader: Parallel file downloading with progress tracking
- BatchProcessor: Process items in configurable batches
- RateLimiter: Control request frequency
- MetadataExtractor: HTML metadata extraction with custom selectors
- PathHandler: Path and filename utilities
- FileIO: Async/sync file operations
- BaseCrawler: Extensible crawler base class with domain validation
- LoggerFactory: Enhanced logging with file and console outputs
Requirements
- Python 3.9+
- aiofiles
- aiohttp
- BeautifulSoup4
Quick start
import asyncio
from aiofetch import (
AsyncDownloader,
MetadataExtractor,
ContentParser,
FileIO
)
async def main():
# Initialize components
downloader = AsyncDownloader(concurrent_limit=20)
parser = ContentParser()
file_io = FileIO()
# Download files
urls = [
("https://example.com/file1.pdf", "downloads/file1.pdf"),
("https://example.com/file2.pdf", "downloads/file2.pdf")
]
await downloader.download_batch(urls)
# Parse HTML content
html = """<html><body>
<h1>Title</h1>
<img src="image.jpg" alt="Test">
</body></html>"""
# Extract metadata
extractor = MetadataExtractor()
metadata = extractor.extract_from_html(html, {
'title': 'h1',
'images': ('img', 'src')
})
# Save results
await file_io.save_json(metadata, 'output/metadata.json')
if __name__ == "__main__":
asyncio.run(main())
License
MIT License - see LICENSE file for details
Author
Akram Rakhmetulla (akram042006@gmail.com)
Project details
Verified details
These details have been verified by PyPIProject links
GitHub Statistics
Maintainers
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
aiofetch-0.0.1.tar.gz
(13.4 kB
view details)
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
aiofetch-0.0.1-py3-none-any.whl
(12.7 kB
view details)
File details
Details for the file aiofetch-0.0.1.tar.gz.
File metadata
- Download URL: aiofetch-0.0.1.tar.gz
- Upload date:
- Size: 13.4 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.12.8
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
9b9b58661b2b616a248f5529c6d092410fb3f9d8d53b5fce7e4ad56e04a449cf
|
|
| MD5 |
eb497beca3ad5ad67b3fc6beeab0d7cb
|
|
| BLAKE2b-256 |
5338d341266d07db143f87a282e794d4975d893105ba88aced079e9f6d824e79
|
Provenance
The following attestation bundles were made for aiofetch-0.0.1.tar.gz:
Publisher:
workflow.yml on spike1236/aiofetch
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
aiofetch-0.0.1.tar.gz -
Subject digest:
9b9b58661b2b616a248f5529c6d092410fb3f9d8d53b5fce7e4ad56e04a449cf - Sigstore transparency entry: 168957916
- Sigstore integration time:
-
Permalink:
spike1236/aiofetch@44133ad3214f897cc1ba1515b50e00d07eb29c88 -
Branch / Tag:
refs/tags/v0.0.1 - Owner: https://github.com/spike1236
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
workflow.yml@44133ad3214f897cc1ba1515b50e00d07eb29c88 -
Trigger Event:
push
-
Statement type:
File details
Details for the file aiofetch-0.0.1-py3-none-any.whl.
File metadata
- Download URL: aiofetch-0.0.1-py3-none-any.whl
- Upload date:
- Size: 12.7 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.12.8
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
2087a2f306f9c9259687ea12bd6ff7cb8117819e22143fcbeaa8d5b421314767
|
|
| MD5 |
d9ef75d515eaefbe091be96c695f54cf
|
|
| BLAKE2b-256 |
3ff8f22bb7d33ee758fd77c20fcf3a7d9521f829e43e3ddfc41a7544dbd041df
|
Provenance
The following attestation bundles were made for aiofetch-0.0.1-py3-none-any.whl:
Publisher:
workflow.yml on spike1236/aiofetch
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
aiofetch-0.0.1-py3-none-any.whl -
Subject digest:
2087a2f306f9c9259687ea12bd6ff7cb8117819e22143fcbeaa8d5b421314767 - Sigstore transparency entry: 168957918
- Sigstore integration time:
-
Permalink:
spike1236/aiofetch@44133ad3214f897cc1ba1515b50e00d07eb29c88 -
Branch / Tag:
refs/tags/v0.0.1 - Owner: https://github.com/spike1236
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
workflow.yml@44133ad3214f897cc1ba1515b50e00d07eb29c88 -
Trigger Event:
push
-
Statement type:
