CrawlFrame: A Web Crawling framework
Behold, the power of CrawlFrame:
crawlf createapp <appname> -url http://<website>/
crawlf createpro <projectname>
crawlf start app:<name>
crawlf start project
crawlf stop app:<name>
crawlf stop project
crawlf stop <pid>
CrawlFrame officially supports python3.5+ .
Installation
To install CrawlFrame, simply use pip:
$ pip install crawlframe
Documentation
In the construction...
Changed
crawlframe-1.0.3
下载器并发方式变更
crawlframe-1.0.2
CRAWLFRAME_SURVIVE_SWITCH = True
CRAWLFRAME_SURVIVE_MAX = int
reload spider
CRAWLFRAME_SURVIVE_SWITCH = True CRAWLFRAME_SURVIVE_MAX = int 设置两个参数 发出一定请求数之后, 自动重启爬虫. 避免内存溢出. 修复middle不能加载以及logger不能创建的bug. BaseLogger改为单例模式.
crawlframe-1.0.1
crawlf createapp <appname> -url http://<website>/
crawlf createpro <projectname>
crawlf start app:<name>
crawlf start project
crawlf stop app:<name>
crawlf stop project
crawlf stop <pid>
实际上以上命令只实现了start 与 stop. 但是已经能够在项目中使用了. 为了方便自己安装使用, 就先上传到 pypi 与 github 了.
1 maintainer
Metadata
1 maintainer
Release files for crawlframe 1.0.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| crawlframe-1.0.3.tar.gz | 27.4 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| crawlframe-1.0.3-py2.py3-none-any.whl | Python 2, Python 3 | none | any | Details |
Total release size: 73.5 kB
Release files / crawlframe-1.0.3.tar.gz
| Download URL | crawlframe-1.0.3.tar.gz |
|---|---|
| Size | 27.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
619b1431411c9ea76d1b3787ee03fdfd606802b8918f8e374b02378646baa453
|
|
BLAKE2b-256 checksum How to use checksums |
2431886f70a99116dcd9d3376583286574222d97c4e431791c2f826d8fed7e3a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.13.0 pkginfo/1.5.0.1 requests/2.18.4 setuptools/40.9.0 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.6.2
|
Release files / crawlframe-1.0.3-py2.py3-none-any.whl
| Download URL | crawlframe-1.0.3-py2.py3-none-any.whl |
|---|---|
| Size | 46.1 kB |
| Tags | Python 2 Python 3 |
|
SHA-256 checksum How to use checksums |
3f3c5feb3b20cb97ed81d4b36eba48a4300fb0b2dc30c4523e370e89e57c2c18
|
|
BLAKE2b-256 checksum How to use checksums |
a3e5c15adce5b73fb9c4b38bd09c07e30e4b1e200946d619acc65bbbe760f2b4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.13.0 pkginfo/1.5.0.1 requests/2.18.4 setuptools/40.9.0 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.6.2
|
