A Spider FrameWork
Project description
openget
A Spider FrameWork.
Installation
You can install openget by simply doing:
pip install openget
pip install git+https://github.com/dytttf/openget.git@main
pip install git+https://github.com/dytttf/openget.git@dev
Usage
Very Important Thing This line must be the first line in code
from openget.spiders import *
命令行支持
- 生成配置文件样例
# 环境变量类型的配置文件
python -m openget --gen_env_example
About Environment Variable
优先级
- $(pwd)/.env
- ~/.openget/.env
docker build
cd docekr
bash build.sh
TODO
- ftp download
- auto add task
- statis crawl speed
- browser support: zhipin.com tmall.com
- sqlite.db rename
- use cmd to create spider template
- 金融许可证 卡死问题处理
- 环境变量定义优化
- 新项目创建优化
- 极简模式、完整模式 减少包的依赖
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
openget-0.3.3.tar.gz
(72.4 kB
view details)
Built Distribution
openget-0.3.3-py3-none-any.whl
(81.7 kB
view details)
File details
Details for the file openget-0.3.3.tar.gz.
File metadata
- Download URL: openget-0.3.3.tar.gz
- Upload date:
- Size: 72.4 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.11.4
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 1bcf9f090d4d7d007e96ac1108332b931ceba10b4a0897b6a4c5b3638810b34e |
|
| MD5 | a4f532375e6f799a84bbd69374866bcb |
|
| BLAKE2b-256 | 09b48572890a12c818969b13932368452126f51f613b86ba4493d48028a6cee9 |
File details
Details for the file openget-0.3.3-py3-none-any.whl.
File metadata
- Download URL: openget-0.3.3-py3-none-any.whl
- Upload date:
- Size: 81.7 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.11.4
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | cb3c80a9590367d6a2f3135756bc9990d696f148ea16a9e3c2c7a0b8c341ae75 |
|
| MD5 | 53415ba177830f5bab6893e6e2076a84 |
|
| BLAKE2b-256 | a4742766c34d4065bf412cce8ae8354d3b02b3b1f5169f2f70c1ceeea2767170 |
