Skip to main content
Donate to the Python Software Foundation or Purchase a PyCharm License to Benefit the PSF! Donate Now

一个分布式异步爬虫

Project description

# Dcrawl(这次不装逼,用中文)
一个分布式异步爬虫,采用生产Producer和消费Consumer模式

# producer_demo

from producer import Producer
p = Producer()
data = {
"name": 'baidu',
"url": "https://baidu.com",
"headers": {"Content-Type": "application/json"},
"data": {},
"method": "get"
}
p.add_task(data)
p.run()

# consumer_demo

from consumer import Consumer
cons = Consumer()

# 数据处理方法 todo 可以根据不同的name写不同的handle
@Consumer.handle
def result(resp):
print('test')
print(resp)

cons.loop_task()

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
chuwt_Dcrawl-0.1.1.tar.gz (1.6 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page