mathgenproject-jgolden17 0.1.0

Mathematics Genealogy Project Spider

mathgenproject

Mathematics Genealogy Project Spider

A webspider for the Mathematics Genealogy Project.

Installation

Use the package manager pip to install the spider.

pip install mathgenproject

Usage

Define a pipeline through which to process each mathematician returned from the spider.

class MyPipeline(object):
    def open_spider(self, spider):
        ...

    def process_item(self, item, spider):
        print(item['name'])
        return item

    def close_spider(self, spider):
        ...

Run the spider using scrapy's CrawlerProcess, passing in the mathematician's MGP ID.

from scrapy.crawler import CrawlerProcess
from mathgenproject.spiders import MathGenProjectSpider

process = CrawlerProcess(settings={
    'FEED_FORMAT': 'json',
    'FEED_URI': 'items.json',
    'ITEM_PIPELINES': {
        'MyPipeline': 300,
    },
})

process.crawl(MathGenProjectSpider, mgp_id='216087')
process.start()

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

License

MIT