Okami

Okami is a high-level web scraping framework built entirely for Python 3.6+ using asynchronous model provided by standard library asyncio module with aiohttp as a networking layer and lxml for parsing data.

Architecture is entirely modular and main components can be swapped out and replaced with custom implementations.

Features

• complete website-wide page processing
• full scraping mode or delta mode scraping only unvisited pages
• immediate, on-demand or real-time page processing over HTTP API
• single page processing via command line
• lots of pipelines, middlewares and signals

Spiders are very simple implementations. Take a look at an example here.

Quick start

• Install okami

  • pip install okami

• Run example web server

  • OKAMI_SETTINGS=okami.cfg.example okami example server

Open localhost:8000 and browse around a little. Quite a remarkable website. We will run our example spider against this website shortly and process few items.

• Run example spider

  • OKAMI_SETTINGS=okami.cfg.example okami example spider

Our example spider started and you can see it processing pages. Take a look at an example spider implementation here.

Documentation

Read the rest of documentation here.

License

Okami is licensed under a three clause BSD License. Full license text can be found here.