A Spider FrameWork
Project description
openget
A Spider FrameWork.
Installation
You can install openget by simply doing:
pip install openget
pip install git+https://github.com/dytttf/openget.git@main
pip install git+https://github.com/dytttf/openget.git@dev
Usage
Very Important Thing This line must be the first line in code
from openget.spiders import *
命令行支持
- 生成配置文件样例
# 环境变量类型的配置文件
python -m openget --gen_env_example
About Environment Variable
优先级
- $(pwd)/.env
- ~/.openget/.env
docker build
cd docekr
bash build.sh
TODO
- ftp download
- auto add task
- statis crawl speed
- browser support: zhipin.com tmall.com
- sqlite.db rename
- use cmd to create spider template
- 金融许可证 卡死问题处理
- 环境变量定义优化
- 新项目创建优化
- 极简模式、完整模式 减少包的依赖
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
openget-0.3.0.tar.gz
(69.8 kB
view hashes)
Built Distribution
openget-0.3.0-py3-none-any.whl
(80.8 kB
view hashes)