tkitScrapyMongoPipeline,
Project description
tkitScrapyMongoPipeline
MongoPipeline
https://docs.scrapy.org/en/latest/topics/item-pipeline.html
数据存储到mongodb
# settings
# 1、设置MongoDB 的数据库地址
MONGO_URI = "mongodb://192.168.123.117:27017/"
MONGO_DATABASE="test"
# # 2、启用中间件MongoPipeline
ITEM_PIPELINES = {
# 'base.pipelines.DuplicatesPipeline': 100,
'base.pipelines.MongoPipeline': 100,
}
# item 字段示例
item={
#设置强制去重复字段
"dup_id":url,
"title": title, "url": url, "content": text, "site": "playbarkrun.com", "content_type": "content",
# 设置表名
"collection_name":"test11"
}
详细参考
dev.md
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for tkitScrapyMongoPipeline-0.0.0.116515135.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1ade996c5762e4915027875ad372a016ac76b4d12397761581500b3c03df7954 |
|
MD5 | 83380af027fa0de781878025f91ec29d |
|
BLAKE2b-256 | 3bbc981a058c6debd545bc8676ffa83d85a59118e4d6d967915bd8bf151e523c |
Close
Hashes for tkitScrapyMongoPipeline-0.0.0.116515135-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cc287319403e18c3192939e1f587c71adf4b496ff04c7b2eeca2cc23d213e380 |
|
MD5 | 6e34dd70534bee196c2282d6f6cb24d7 |
|
BLAKE2b-256 | b40210528fdc19f710bcf6515faffaedd32ae0f33968f412f9e1c16c5357513d |