A Spider FrameWork
Project description
openget
A Spider FrameWork.
Installation
You can install openget by simply doing:
pip install openget
pip install git+https://github.com/dytttf/openget.git@main
pip install git+https://github.com/dytttf/openget.git@dev
Usage
Very Important Thing This line must be the first line in code
from openget.spiders import *
命令行支持
- 生成配置文件样例
# 环境变量类型的配置文件
python -m openget --gen_env_example
About Environment Variable
优先级
- $(pwd)/.env
- ~/.openget/.env
docker build
cd docekr
bash build.sh
TODO
- ftp download
- auto add task
- statis crawl speed
- browser support: zhipin.com tmall.com
- sqlite.db rename
- use cmd to create spider template
- 金融许可证 卡死问题处理
- 环境变量定义优化
- 新项目创建优化
- 极简模式、完整模式 减少包的依赖
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
openget-0.3.4.tar.gz
(70.5 kB
view details)
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
openget-0.3.4-py3-none-any.whl
(81.4 kB
view details)
File details
Details for the file openget-0.3.4.tar.gz.
File metadata
- Download URL: openget-0.3.4.tar.gz
- Upload date:
- Size: 70.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.0.0 CPython/3.11.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
235e92bb521c67126a76ef6c519a3ef4bdd69174d839ce86e798345db0009953
|
|
| MD5 |
dd612146ae23474c1e2430310a8c5cce
|
|
| BLAKE2b-256 |
f6dcca5dcb99abdeed9fb0437d842f31d349c0ad518ecca1526fd16fbe3d08b5
|
File details
Details for the file openget-0.3.4-py3-none-any.whl.
File metadata
- Download URL: openget-0.3.4-py3-none-any.whl
- Upload date:
- Size: 81.4 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.0.0 CPython/3.11.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
2e9bc646c4a30872a7f5e63e8ea95131d2c8aca5e926fc64939fa39f02299500
|
|
| MD5 |
b32ade0599705922354d0bae286cf0ca
|
|
| BLAKE2b-256 |
1dc17926dd1b832a2a31ccd1c63a8326944648bf664c45ab8dfc797e3076d9d0
|
