A Simple Web Crawling and Web Scraping framework
Project description
Crwy
简介
Crwy是一个轻量级的爬虫抓取框架，参考Scrapy框架结构开发而来。该框架提供了实用的爬虫模板，旨在帮助大家快速实现爬虫任务，高效开发。并为scrapy使用者提供通用轮子.。新增了gevent，使爬虫异步执行，速度更快。
运行环境
Python2 & Python3
Works on Linux, Mac OSX
依赖包
beautifulsoup4>=4.5.1
requests>=2.20.0
configparser>=3.5.0
SQLAlchemy>=1.0.14
pyssdb>=0.1.2
redis>=2.10.5,<3.0.0
gevent>=1.2.1
retrying>=1.3.3
imapclient>=2.0.0
安装
快速安装
pip install crwy
or 前往下载: https://pypi.python.org/pypi/Crwy/
使用手册
友情链接
更新日志
TODO
完善scrapy_plugs
完善selenium_api
兼容python3
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file Crwy-1.5.6.tar.gz.
File metadata
- Download URL: Crwy-1.5.6.tar.gz
- Upload date:
- Size: 30.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.15.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.2 requests-toolbelt/0.9.1 tqdm/4.40.2 CPython/2.7.15
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
647eafab54089337ea2283f38ca8ba89178a95df16c0716e2710bacbf2f935e0
|
|
| MD5 |
37ba4f3b4be88f821063197c2bab940a
|
|
| BLAKE2b-256 |
a89528c093cef1a9e89970143c71885fb821bfc0e9784d17afc4d8517764a888
|
File details
Details for the file Crwy-1.5.6-py3.6.egg.
File metadata
- Download URL: Crwy-1.5.6-py3.6.egg
- Upload date:
- Size: 112.6 kB
- Tags: Egg
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.2 requests-toolbelt/0.9.1 tqdm/4.40.2 CPython/3.6.7
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
a5ba948f979a79e176b921843d76240d7df0d27f289356971e563c37c47ddee9
|
|
| MD5 |
4623bc2dde22d7d9ff6d2225a582964d
|
|
| BLAKE2b-256 |
091aa35f99cfa9e21051b10e9b90919f2a47c09efd1e3b958a035cf04b999dd6
|
File details
Details for the file Crwy-1.5.6-py2.7.egg.
File metadata
- Download URL: Crwy-1.5.6-py2.7.egg
- Upload date:
- Size: 110.7 kB
- Tags: Egg
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.15.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.2 requests-toolbelt/0.9.1 tqdm/4.40.2 CPython/2.7.15
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
98444e716dae6f056b94380f047b2bc79971dcc33e09178a3b741c6673c0d113
|
|
| MD5 |
eae934305452154f219305af200d656c
|
|
| BLAKE2b-256 |
8fad6ed6e57efe66a410e40dfde9d4d7a8bc31167531293abe4165e9fc394d2b
|
