Ruya crawler
Project description
Ruya is a Python-based breadth-first, level-, delayed, event-based-crawler for crawling English, Japanese websites. It is targeted solely towards developers who want crawling functionality in their projects using API, and crawl control.
You can view the online documentation at http://ruya.sourceforge.net/
