Skip to main content

采集中文小说网站的爬虫

Project description

python 环境

  • python 2.7

  • mysql 5.7.9

项目安装

pip install novelSpider

MySQL 配置

# 加入 mysqld.cnf
# * Encode
init_connect='SET collation_connection = utf8_unicode_ci'
init_connect='SET NAMES utf8'
character-set-server=utf8
collation-server=utf8_unicode_ci
skip-character-set-client-handshake

项目运行

from novelSpider.task import Task

class Config(object):
    '''
    @desc:数据库配置
    @param:username 数据库用户名
    @param:password 数据库密码
    '''
    def __init__(self):
        self.username = 'root'
        self.password = 'root'
        self.database = 'novel'

# 创建任务实例
task = Task()

# 创建爬虫实例
spider = task.createDownloader(Config)

# 下载小说书目信息、章节列表
spider.getCharptList(novelNum=1)

# 下载小说书目信息、章节列表
spider.getCharptContent(novelId=0, charptNum=1)

版本日志

0.0.13 版本

# 支持下载小说书目信息、章节列表、章节内容

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

novelSpider-0.0.13.tar.gz (4.7 kB view hashes)

Uploaded Source

Built Distribution

novelSpider-0.0.13-py2-none-any.whl (8.2 kB view hashes)

Uploaded Python 2

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page