Skip to main content

一个分布式异步爬虫

Project description

# Dcrawl(这次不装逼,用中文)
一个分布式异步爬虫,采用生产Producer和消费Consumer模式

# producer_demo

from producer import Producer
p = Producer()
data = {
"name": 'baidu',
"url": "https://baidu.com",
"headers": {"Content-Type": "application/json"},
"data": {},
"method": "get"
}
p.add_task(data)
p.run()

# consumer_demo

from consumer import Consumer
cons = Consumer()

# 数据处理方法 todo 可以根据不同的name写不同的handle
@Consumer.handle
def result(resp):
print('test')
print(resp)

cons.loop_task()

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for chuwt_Dcrawl, version 0.1.1
Filename, size File type Python version Upload date Hashes
Filename, size chuwt_Dcrawl-0.1.1.tar.gz (1.6 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page