A Simple Web Crawling and Web Scraping framework
Project description
Crwy
简介
Crwy是一个轻量级的爬虫抓取框架,参考Scrapy框架结构开发而来。该框架提供了实用的爬虫模板,旨在帮助大家快速实现爬虫任务,高效开发。并为scrapy使用者提供通用轮子.。新增了gevent,使爬虫异步执行,速度更快。
运行环境
Python2 & Python3
Works on Linux, Mac OSX
依赖包
beautifulsoup4>=4.5.1
requests>=2.20.0
configparser>=3.5.0
SQLAlchemy>=1.0.14
pyssdb>=0.1.2
redis>=2.10.5,<3.0.0
gevent>=1.2.1
retrying>=1.3.3
imapclient>=2.0.0
安装
快速安装
pip install crwy
or 前往下载: https://pypi.python.org/pypi/Crwy/
使用手册
友情链接
更新日志
TODO
完善scrapy_plugs
完善selenium_api
兼容python3
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Crwy-1.5.5.tar.gz
(30.7 kB
view hashes)
Built Distributions
Crwy-1.5.5-py3.6.egg
(112.7 kB
view hashes)
Crwy-1.5.5-py2.7.egg
(110.6 kB
view hashes)