A Spider FrameWork
Project description
openget
A Spider FrameWork.
Installation
You can install openget by simply doing:
pip install openget
pip install git+https://github.com/dytttf/openget.git@main
pip install git+https://github.com/dytttf/openget.git@dev
Usage
Very Important Thing This line must be the first line in code
from openget.spiders import *
命令行支持
- 生成配置文件样例
# 环境变量类型的配置文件
python -m openget --gen_env_example
- 查看配置文件内容
# 环境变量类型的配置文件
python -m openget --show_env_config
About Environment Variable
优先级
- $(pwd)/.env
- ~/.openget/.env
docker build
cd docekr
bash build.sh
TODO
- ftp download
- auto add task
- statis crawl speed
- browser support: zhipin.com tmall.com
- sqlite.db rename
- use cmd to create spider template
- 金融许可证 卡死问题处理
- 环境变量定义优化
- 新项目创建优化
- 极简模式、完整模式 减少包的依赖
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
openget-0.3.5.tar.gz
(70.3 kB
view details)
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
openget-0.3.5-py3-none-any.whl
(81.5 kB
view details)
File details
Details for the file openget-0.3.5.tar.gz.
File metadata
- Download URL: openget-0.3.5.tar.gz
- Upload date:
- Size: 70.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.0.0 CPython/3.11.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
46d31a5ff744b1f3add60f059defc835b890230725b10d7cc4d177b4143c5262
|
|
| MD5 |
7dc7afe27f7d3f52fe82d075c3a64b12
|
|
| BLAKE2b-256 |
66e879a47eb3bbffd8be462df273aab198db915278bc99f7c01f635e99cc95a3
|
File details
Details for the file openget-0.3.5-py3-none-any.whl.
File metadata
- Download URL: openget-0.3.5-py3-none-any.whl
- Upload date:
- Size: 81.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.0.0 CPython/3.11.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
5afa8ff8636a91666b829f570c7498d0663306c301bbb9d1d0440723dd3998f9
|
|
| MD5 |
0c3d1e47d915a5cb21d9098c35ddea04
|
|
| BLAKE2b-256 |
d226e29d09d4eed5f7ac30b978ebf20f80debe9766c12b870314675502bf8ad6
|
