Web crawler for datatist.com
Project description
MCP Aim Crawler
这是一个基于MCP协议的网站爬虫项目。
项目结构
mcp_aim_crawler/
├── tests/ # 测试用例目录
└── mcp_aim_crawler/ # 主要源代码目录
├── crawler.py # 爬虫核心逻辑
└── logger.py # 日志配置
环境要求
- Python 3.12.9
- Poetry
- pyenv
安装方法
- 使用pyenv安装Python 3.12.9:
pyenv install 3.12.9
pyenv local 3.12.9
- 安装项目依赖:
poetry install
运行方法
poetry run python -m mcp_aim_crawler
环境变量
在项目根目录创建 .env 文件，包含以下配置:
WEBSITE_URL: 目标网站URLUSERNAME: 登录用户名PASSWORD: 登录密码MCP_TOKEN: MCP令牌
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
mcp_aim_crawler-0.1.0.tar.gz
(27.1 kB
view details)
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file mcp_aim_crawler-0.1.0.tar.gz.
File metadata
- Download URL: mcp_aim_crawler-0.1.0.tar.gz
- Upload date:
- Size: 27.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.1.0 CPython/3.12.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
ddddb2445a492ef8de7a4c2a23fb979525dbeef0f9e63a54f65fbfcb8c0950ef
|
|
| MD5 |
8a596b2b114797dd0ff95d2b44f40f59
|
|
| BLAKE2b-256 |
d00903c9571c1f6a1158856f17c335bd04cd4e9ae685a15cdbb6a54366e52bb9
|
File details
Details for the file mcp_aim_crawler-0.1.0-py3-none-any.whl.
File metadata
- Download URL: mcp_aim_crawler-0.1.0-py3-none-any.whl
- Upload date:
- Size: 7.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.1.0 CPython/3.12.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
96e5c8fd684c9b160e988e4ab5f8c903813c51aa1ef6b5e09b980951b67189d1
|
|
| MD5 |
131f11c84b6e4fb97674e899f936cad0
|
|
| BLAKE2b-256 |
ed2dde799a235757f393fa7f0ff2139d2e302b60160b49b88652d3bc948ba012
|
