Skip to main content

utils for spider, like middlewares, pipelines, common design implementation, etc

Project description

spider-common

Common code used in python spider.

Changelog

See CHANGELOG

Project Structure

  • common_utils

    通用全局函数

  • clue

    clue(爬虫线索)模块

  • notify

    事件、信号通知

  • persistent

    通用数据模型

    抓取数据持久化

代码结构:

├── clue
│   ├── api.py
│   ├── constants.py
│   ├── __init__.py
│   ├── models.py
│   └── scrapy
│       ├── extensions.py
│       ├── __init__.py
│       ├── items.py
│       ├── middlewares.py
│       ├── pipelines.py
│       └── spider.py
├── common_utils
│   ├── exceptions.py
│   ├── __init__.py
│   └── time.py
├── __init__.py
├── notify
│   ├── constants
│   │   ├── __init__.py
│   │   └── signals.py
│   ├── __init__.py
│   ├── __pycache__
│   └── scrapy
│       ├── extensions
│       │   ├── __init__.py
│       │   └── signal_handler.py
│       └── __init__.py
├── persistent
│   ├── __init__.py
│   ├── items
│   │   ├── base.py
│   │   ├── __init__.py
│   │   └── shop.py
│   └── scrapy
│       ├── dw_logger.py
│       ├── __init__.py
│       └── pipelines
│           ├── dw_pipeline.py
│           ├── __init__.py
│           └── mysql_pipeline.py

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distribution

spider_common-0.0.1-py3.6.egg (59.8 kB view details)

Uploaded Source

File details

Details for the file spider_common-0.0.1-py3.6.egg.

File metadata

  • Download URL: spider_common-0.0.1-py3.6.egg
  • Upload date:
  • Size: 59.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.6.3 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.6.7

File hashes

Hashes for spider_common-0.0.1-py3.6.egg
Algorithm Hash digest
SHA256 5991b457b60ec399b4423001413c2975b311df46cd133f29c11c22aa1eb2c98f
MD5 79aa67e1554eb38c9441811f26f6d9e4
BLAKE2b-256 4788a0078eefda77fa43b71bf5c517659c42abc7f742977905533149c1c63c26

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page