utils for spider, like middlewares, pipelines, common design implementation, etc
Project description
spider-common
Common code used in python spider.
Changelog
See CHANGELOG
Project Structure
-
common_utils
通用全局函数
-
clue
clue(爬虫线索)模块
-
notify
事件、信号通知
-
persistent
通用数据模型
抓取数据持久化
代码结构:
├── clue
│ ├── api.py
│ ├── constants.py
│ ├── __init__.py
│ ├── models.py
│ └── scrapy
│ ├── extensions.py
│ ├── __init__.py
│ ├── items.py
│ ├── middlewares.py
│ ├── pipelines.py
│ └── spider.py
├── common_utils
│ ├── exceptions.py
│ ├── __init__.py
│ └── time.py
├── __init__.py
├── notify
│ ├── constants
│ │ ├── __init__.py
│ │ └── signals.py
│ ├── __init__.py
│ ├── __pycache__
│ └── scrapy
│ ├── extensions
│ │ ├── __init__.py
│ │ └── signal_handler.py
│ └── __init__.py
├── persistent
│ ├── __init__.py
│ ├── items
│ │ ├── base.py
│ │ ├── __init__.py
│ │ └── shop.py
│ └── scrapy
│ ├── dw_logger.py
│ ├── __init__.py
│ └── pipelines
│ ├── dw_pipeline.py
│ ├── __init__.py
│ └── mysql_pipeline.py
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
No source distribution files available for this release.See tutorial on generating distribution archives.
Built Distribution
spider_common-0.0.1-py3.6.egg
(59.8 kB
view details)
File details
Details for the file spider_common-0.0.1-py3.6.egg
.
File metadata
- Download URL: spider_common-0.0.1-py3.6.egg
- Upload date:
- Size: 59.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.6.3 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.6.7
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5991b457b60ec399b4423001413c2975b311df46cd133f29c11c22aa1eb2c98f |
|
MD5 | 79aa67e1554eb38c9441811f26f6d9e4 |
|
BLAKE2b-256 | 4788a0078eefda77fa43b71bf5c517659c42abc7f742977905533149c1c63c26 |