A Spider FrameWork
Project description
openget
A Spider FrameWork.
Installation
You can install openget by simply doing:
pip install openget
pip install git+https://github.com/dytttf/openget.git@main
pip install git+https://github.com/dytttf/openget.git@dev
Usage
Very Important Thing This line must be the first line in code
from openget.spiders import *
命令行支持
- 生成配置文件样例
# 环境变量类型的配置文件
python -m openget --gen_env_example
- 查看配置文件内容
# 环境变量类型的配置文件
python -m openget --show_env_config
About Environment Variable
优先级
- $(pwd)/.env
- ~/.openget/.env
docker build
cd docekr
bash build.sh
TODO
- ftp download
- auto add task
- statis crawl speed
- browser support: zhipin.com tmall.com
- sqlite.db rename
- use cmd to create spider template
- 金融许可证 卡死问题处理
- 环境变量定义优化
- 新项目创建优化
- 极简模式、完整模式 减少包的依赖
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
openget-0.3.7.tar.gz
(70.5 kB
view details)
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
openget-0.3.7-py3-none-any.whl
(81.6 kB
view details)
File details
Details for the file openget-0.3.7.tar.gz.
File metadata
- Download URL: openget-0.3.7.tar.gz
- Upload date:
- Size: 70.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.0.0 CPython/3.11.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
1253a54bf49a1a5a33924f31b9de9a924c389000a75398b7c726d42dbeb63c44
|
|
| MD5 |
228804fa1e26c88a6b2085d8ba40f4fb
|
|
| BLAKE2b-256 |
5dcd0c8716af1c418c612e298adbc298c039a58b8a353bfb5aeb4a88ad336fb1
|
File details
Details for the file openget-0.3.7-py3-none-any.whl.
File metadata
- Download URL: openget-0.3.7-py3-none-any.whl
- Upload date:
- Size: 81.6 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.0.0 CPython/3.11.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
f7e063e7f359d23c7db2bad1858ddce96fea3db7e70689e817922db38380b88b
|
|
| MD5 |
f48994b03a2993e0579d9b6a3ef7f8fb
|
|
| BLAKE2b-256 |
776536b6d60b6cae5c92768be03f9e689c8ee0c668bb1813b1ddd5033519e9be
|
