fetch_web_resource
fetch_web_resource 是一个用于抓取网页资源并缓存 HTML 内容的 Python 包。
功能
- 抓取网页内容
- 提取网页中的图片和文件链接
- 缓存抓取的 HTML 内容
- 使用随机 User-Agent 模拟请求
安装
你可以使用以下命令安装该包：
pip install .
使用方法
初始化
首先，你需要初始化 HTMLFetcher 类：
from web_fetch.web_fetch import HTMLFetcher
import diskcache
cache = diskcache.Cache('./html_cache')
fetcher = HTMLFetcher(cache=cache, max_concurrent_per_domain=5)
抓取网页内容
你可以使用 fetch_html_batch 方法抓取一批网页内容：
results = [
SearchResult(url="https://example.com/page1"),
SearchResult(url="https://example.com/page2"),
]
async for result in fetcher.fetch_html_batch(results, timeout=5):
print(result)
提取 URL 资源
你可以使用 _extract_urls 方法从 HTML 内容中提取图片和文件链接：
html_content = "<html>...</html>"
base_url = "https://example.com"
url_resource = fetcher._extract_urls(html_content, base_url)
print(url_resource)
环境变量
你需要在项目根目录下创建一个 .env 文件，并添加你的 API 密钥：
API_KEY=your_api_key_here
依赖
该项目依赖以下 Python 包：
- requests
- beautifulsoup4
- diskcache
- python-dotenv
- htmldate
- pydantic
贡献
欢迎贡献代码！请 fork 本仓库并提交 pull request。
许可证
该项目使用 MIT 许可证。
1 maintainer
Release files for web-fetch 0.1.1.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| web_fetch-0.1.1.3.tar.gz | 5.7 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| web_fetch-0.1.1.3-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 11.5 kB
Release files / web_fetch-0.1.1.3.tar.gz
| Download URL | web_fetch-0.1.1.3.tar.gz |
|---|---|
| Size | 5.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
0b24325ef84d067665b6697a60a3e758e42c5b7c6a6d822345debdaf02b35893
|
|
BLAKE2b-256 checksum How to use checksums |
a61aedb0918de8e5c2429fbd3ac252f73c899ce2be1ae1a913bf1165ae529557
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.1 CPython/3.10.15
|
Release files / web_fetch-0.1.1.3-py3-none-any.whl
| Download URL | web_fetch-0.1.1.3-py3-none-any.whl |
|---|---|
| Size | 5.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
03db5100301b327b6c0dbd589cf22363b6e1e24a869dc97bb409c727c91d49d8
|
|
BLAKE2b-256 checksum How to use checksums |
6eca48eca3d682193c5e5e699967ef7c6c4348dafaef44c062ffe7ac967b4fad
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.1 CPython/3.10.15
|
