A package to fetch web resources and cache HTML content.
Project description
fetch_web_resource
fetch_web_resource 是一个用于抓取网页资源并缓存 HTML 内容的 Python 包。
功能
- 抓取网页内容
- 提取网页中的图片和文件链接
- 缓存抓取的 HTML 内容
- 使用随机 User-Agent 模拟请求
安装
你可以使用以下命令安装该包：
pip install .
使用方法
初始化
首先，你需要初始化 HTMLFetcher 类：
from web_fetch.web_fetch import HTMLFetcher
import diskcache
cache = diskcache.Cache('./html_cache')
fetcher = HTMLFetcher(cache=cache, max_concurrent_per_domain=5)
抓取网页内容
你可以使用 fetch_html_batch 方法抓取一批网页内容：
results = [
SearchResult(url="https://example.com/page1"),
SearchResult(url="https://example.com/page2"),
]
async for result in fetcher.fetch_html_batch(results, timeout=5):
print(result)
提取 URL 资源
你可以使用 _extract_urls 方法从 HTML 内容中提取图片和文件链接：
html_content = "<html>...</html>"
base_url = "https://example.com"
url_resource = fetcher._extract_urls(html_content, base_url)
print(url_resource)
环境变量
你需要在项目根目录下创建一个 .env 文件，并添加你的 API 密钥：
API_KEY=your_api_key_here
依赖
该项目依赖以下 Python 包：
- requests
- beautifulsoup4
- diskcache
- python-dotenv
- htmldate
- pydantic
贡献
欢迎贡献代码！请 fork 本仓库并提交 pull request。
许可证
该项目使用 MIT 许可证。
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
web_fetch-0.1.1.1.tar.gz
(5.4 kB
view details)
Built Distribution
File details
Details for the file web_fetch-0.1.1.1.tar.gz.
File metadata
- Download URL: web_fetch-0.1.1.1.tar.gz
- Upload date:
- Size: 5.4 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.10.15
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | e334a76299c54e5b1a289e917a720fde216256d25131c7af3217d57ac10fd7f1 |
|
| MD5 | 51630ff6194194e0859e421b91cec46f |
|
| BLAKE2b-256 | 745d72728d9b764c49658d00fb1a4449600395970290f42b2ea840baba0b65f8 |
File details
Details for the file web_fetch-0.1.1.1-py3-none-any.whl.
File metadata
- Download URL: web_fetch-0.1.1.1-py3-none-any.whl
- Upload date:
- Size: 5.8 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.10.15
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 61ef9c66abd4da0437f4f2bcbbb144ef273f9db4b01f8e458f3afadd3097fe77 |
|
| MD5 | 985f03ec67b332ec025e0070c50d78ff |
|
| BLAKE2b-256 | 80061880d2ddbab8499efd94359b8aa6170722e3d95c96e0e8ec4445bfd557cb |
