tkit_scrapy_mongo

use mongodb instead of redis and rewrite scrapy_redis py3 module


    # 这里覆盖默认的settings配置,custom_settings settings.py
    custom_settings = dict(
        #MONGODB_URI="mongodb://192.168.1.18:27017/",
        #REDIS_URL = 'redis://192.168.1.18:6379',  # 指定redis的地址
        #POOL_PROXY = "http://192.168.1.18:5010"
        CONCURRENT_REQUESTS_PER_IP = 4,
        CONCURRENT_REQUESTS_PER_DOMAIN = 4,
        DUPEFILTER_CLASS = 'pet.tkit_scrapy_mongo.dupefilter.RFPDupeFilter',
        # 去重复
        SPIDER_MIDDLEWARES={
            # "scrapy_splash.SplashDeduplicateArgsMiddleware": 100,
            "pet.tkit_scrapy_mongo.dupefilter.RFPDupeFilter":101
        }


    )

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tkit_scrapy_mongo-0.0.0.116654862.tar.gz (23.1 kB view details)

Uploaded Oct 11, 2022 Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

tkit_scrapy_mongo-0.0.0.116654862-py2.py3-none-any.whl (25.3 kB view details)

Uploaded Oct 11, 2022 Python 2Python 3

File details

Details for the file tkit_scrapy_mongo-0.0.0.116654862.tar.gz.

File metadata

Download URL: tkit_scrapy_mongo-0.0.0.116654862.tar.gz
Upload date: Oct 11, 2022
Size: 23.1 kB
Tags: Source
Uploaded using Trusted Publishing? No
Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/34.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.9 tqdm/4.62.3 importlib-metadata/4.8.1 keyring/23.1.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.7

File hashes

Hashes for tkit_scrapy_mongo-0.0.0.116654862.tar.gz
Algorithm	Hash digest
SHA256	`d4ba3e9b797c5259af1d363ac863bc7d9f9d8f5497f3cba0ace1a708ac1c268f`
MD5	`85e4e096c94557e2d59acaaffe72ed83`
BLAKE2b-256	`fd838813087686c6ac0bcf5993fe502bae8430987b0b436159218ed210775d87`

See more details on using hashes here.

File details

Details for the file tkit_scrapy_mongo-0.0.0.116654862-py2.py3-none-any.whl.

File metadata

Download URL: tkit_scrapy_mongo-0.0.0.116654862-py2.py3-none-any.whl
Upload date: Oct 11, 2022
Size: 25.3 kB
Tags: Python 2, Python 3
Uploaded using Trusted Publishing? No
Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/34.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.9 tqdm/4.62.3 importlib-metadata/4.8.1 keyring/23.1.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.7

File hashes

Hashes for tkit_scrapy_mongo-0.0.0.116654862-py2.py3-none-any.whl
Algorithm	Hash digest
SHA256	`398ab1cedba514b549b204836e561c9a4c2307bb3822b89b8e177867c27e6a30`
MD5	`e8598831dba35c1b838317c200b94ed8`
BLAKE2b-256	`aa58ea51d68ced8f1dec47d689ab7016c315a002382d90ac30081342770ba4a7`