tkitScrapyMongoPipeline,
Project description
tkitScrapyMongoPipeline
MongoPipeline
https://docs.scrapy.org/en/latest/topics/item-pipeline.html
数据存储到mongodb
# settings
import tkitScrapyMongoPipeline
# 1、设置MongoDB 的数据库地址
MONGO_URI = "mongodb://192.168.123.117:27017/"
MONGO_DATABASE="test"
# # 2、启用中间件MongoPipeline
ITEM_PIPELINES = {
# 'base.pipelines.DuplicatesPipeline': 100,
'base.pipelines.MongoPipeline': 100,
}
# item 字段示例
item={
#设置强制去重复字段
"unique_id":url,
"title": title, "url": url, "content": text, "site": "playbarkrun.com", "content_type": "content",
# 设置表名
"collection_name":"test11"
}
详细参考
dev.md
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for tkitScrapyMongoPipeline-0.0.0.116522771.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 636e680a66e6c8c918c4bf92aa696d502eea6549e63c09cb0a0050fe48868845 |
|
MD5 | 0161dd2bed6d864a5e244630a013704c |
|
BLAKE2b-256 | 286e8af1139d78732ad1140b67d8e635240c387ddad72120889f3b9ddb3be72c |
Close
Hashes for tkitScrapyMongoPipeline-0.0.0.116522771-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 85f4d08d7070374fda6a8e3edc264d8671e8abc61078ce4fb046747b468a9e87 |
|
MD5 | 86ecfcfb4289ed3fb70a8fd542fc0364 |
|
BLAKE2b-256 | 87ea2d4966b6a59a7670ddee701c2cc94e2b1cfea58cdd5c8f4a6f1d33de0400 |