Skip to main content

Lightweight and expandable spider framework.

Project description

charlotte

Build Status

Lightweight and expandable spider framework.

Install

pip install charlotte

Example

import json
import asyncio
from charlotte.spider import BaseSpider


class Spider(BaseSpider):

    async def on_start(self):
        response = await self.fetch('https://blog.qiaohong.org/api/v1/articles')

        items = json.loads(response.body)

        await asyncio.wait(
            [self.fetch('https://blog.qiaohong.org/api/v1/articles/' + item['slug'],
                        parser=self.parse_detail) for item in items['data']['articles']])

    def parse_detail(self, res):
        print(res.body.decode('utf-8'))


if __name__ == '__main__':
    Spider().run()

Documentation

Check wiki for details.

LICENSE

MIT

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

charlotte-0.2.4.tar.gz (5.2 kB view details)

Uploaded Source

File details

Details for the file charlotte-0.2.4.tar.gz.

File metadata

  • Download URL: charlotte-0.2.4.tar.gz
  • Upload date:
  • Size: 5.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/3.7.3

File hashes

Hashes for charlotte-0.2.4.tar.gz
Algorithm Hash digest
SHA256 34a9198246374f12a1d03708cb1bdab177577a084855d3bedcc876862c518cb7
MD5 7263f5537814530b3a9476cb02862e35
BLAKE2b-256 ed0f2e07981f24da2ed7de2979a8f9ffe9c9d6350536cd255cc95dfdb24accc8

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page