This release is a pre-release and may not be stable for production use.
crawl-me
crawl-me是一个基于plugin的轻量级快速网页图片下载工具。crawl-me通过简单的命令行就可以用你想要的方式下载各个网站下的图片。目前暂时只支持gamersky(游明星空), pixiv（P站），更多plugin尽请期待，欢迎为它添加新的plugin。
Fork me on Github :) https://github.com/nyankosama/crawl-me
Features
支持Http range-headers 并发分段下载，速度更快
支持添加plugin自定义新的行为，例如添加对微博的支持
Available plugins
pixiv : P站图片下载插件，支持下载某P主所有作品
gamersky : 游明星空图库下载插件，支持下载游民星空图库中的某一个专题的所有图片
Installation
通过pip安装
首先请确保已安装python2.7和pip
由于代码依赖了pyquery，linux下请先确保libxslt-devel libxml2-devel已安装，windows下选择对应的`lxml installer <https://pypi.python.org/pypi/lxml/3.3.5#downloads>`_下载并安装。
以上步骤完成后，只需执行下面的命令
$ pip install crawl-me
windows平台下使用crawl-me命令之前，请确保把{$python-home}/Scripts/ 加入Windows环境变量中
通过git安装
1. Ubuntu下安装
由于代码依赖了pyquery，安装前请确保libxslt-devel libxml2-devel已被安装
sudo apt-get install libxml2-dev sudo apt-get install libxslt1-dev
然后请确保安装了`setuptools <https://pypi.python.org/pypi/setuptools#downloads>`_, Ubuntu下你可以：
sudo apt-get install python-setuptools
然后从github clone source到本地
$ git clone https://github.com/nyankosama/crawl-me.git $ cd crawl-me/ $ sudo python setup.py install
2. Windows下安装
首先你需要安装`python2.7 <https://www.python.org/download/releases/2.7.7/>`_和`pip <https://pip.pypa.io/en/latest/installing.html>`_，python2.7可以通过windows installer安装。安装pip首先下载`get-pip.py <https://bootstrap.pypa.io/get-pip.py>`_， 然后执行下面命令。
python get-pip.py
然后，你需要安装pyquery的所依赖的lxml，选择对应的`lxml installer <https://pypi.python.org/pypi/lxml/3.3.5#downloads>`_下载并安装
最后从github clone 到本地
$ git clone https://github.com/nyankosama/crawl-me.git $ cd crawl-me/ $ sudo python setup.py install
在使用crawl-me之前，请确保把{$python-home}/Scripts/ 加入Windows环境变量中
Usage
Examples
下载gamersky下的http://www.gamersky.com/ent/201404/352055.shtml 的第1页到第10页的所有图片到当前目录的gamersky-crawl文件夹下
crawl-me gamersky http://www.gamersky.com/ent/201404/352055.shtml ./gamersky-crawl 1 10
一键下载P站藤原桑的所有作品到pixiv-crawl文件夹（藤原桑的id是27517）
crawl-me pixiv 27517 ./pixiv-crawl <your pixiv loginid> <your password>
Command line options
general help
$ crawl-me -h usage: crawl-me [-h] plugin positional arguments: plugin plugin the crawler uses optional arguments: -h, --help show this help message and exit available plugins: ----gamersky ----pixivgamersky
$ crawl-me gamersky -h usage: crawl-me [-h] plugin url savePath beginPage endPage positional arguments: plugin plugin the crawler uses url your url to crawl savePath the path where the imgs ars saved beginPage the page where we start crawling endPage the page where we end crawling optional arguments: -h, --help show this help message and exitpixiv
$ crawl-me pixiv -h usage: crawl-me [-h] plugin authorId savePath pixivId password positional arguments: plugin plugin the crawler uses authorId the author id you want to crawl savePath the path where the imgs ars saved pixivId your pixiv login id password your pixiv login password optional arguments: -h, --help show this help message and exit
TODO
Functions:
支持断点续传
English Version README
Plugins:
weibo
qq zone
Licenses
ChangeLog
0.1.8dev-20140612
Date: 2014-06-12
add -v –version option for main runnable file to show the package version
0.1.7
Date: 2014-06-11
add the http range header support auto-check
0.1.6
Date: 2014-06-11
bug fix: terminal without colour doesnt display syslog prefix
0.1.5
Date: 2014-06-11
bug fix:pip install bug in windows platform
0.1.5dev-20140611
Date: 2014-06-11
bug fix:pypi data_files
0.1.4
Date: 2014-06-11
the latest release
0.1.4dev-20140611
Date: 2014-06-11
modify the README.md. Now it is consistent with rst format to display on pypi
0.1.4dev-20140610
Date: 2014-06-10
add support for installing from pip
0.1.4dev3
Date: 2014-06-10
bug fix
fix the binary write problem in windows platform
0.1.4dev2
Date: 2014-06-10
add setuptools install support
0.1.4dev1
Date: 2014-06-09
bug fix
rangedownloader:http range-headers may not be supported
0.1.3
Date: 2014-06-07
do some refactory
add conf dictionary
0.1.2
Date: 2014-06-06
add plugin
pixiv
0.1.1
Date: 2014-06-05
add plugin
gamersky
0.0.1
Date: 2014-06-05
init the project
1 maintainer
Metadata
1 maintainer
Release files for crawl-me 0.1.8dev-20140612
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| crawl-me-0.1.8dev-20140612.tar.gz | 12.4 kB | Without provenance | Details |
Release files / crawl-me-0.1.8dev-20140612.tar.gz
| Download URL | crawl-me-0.1.8dev-20140612.tar.gz |
|---|---|
| Size | 12.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
90452567f6193e314fd7bf729a1eaedc8c2b13ac1e9a9d0ab1338dd2db1952bf
|
|
BLAKE2b-256 checksum How to use checksums |
d875981320e08dee8f3035ef5dd08256af72db3ab0c17a07c6f9f682a8ac132c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
Security for crawl-me 0.1.8dev-20140612
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
