Cabu

Cabu is a simple microservice framework to remotely crawl websites. It’s built on Flask and Selenium, contains a virtual display wrapper and few methods.

Full documentation here

Usage

@app.route('/gizmodo_last_articles_links')
def gizmodo_last_articles():
    app.webdriver.get('http://www.gizmodo.com')
    articles_links = [i.get_attribute('href') for i in app.webdriver.find_elements_by_css_selector('h1.headline>a')]

    return jsonify({'articles': articles_links})

Installing

$ pip install cabu

Features

• Selenium configuration out of the box

• Flask wrapping

• Crawling methods included

• AWS S3 Export

• FTP / FTPS

• Cookies persistence

• Link extractor

• Proxy configuration

• Headless optional for local debug

• Docker pre-configured distributed environment

• Database handler

• Compatible with most Flask extensions (Flask-Admin, Flask-Mail, Flask-OAuth, …)

• 12 Factors compliance

(Likely to come soon)

• CouchDB support

• Couchbase support

• Mobile drivers

• SFTP

• HtmlUnit web driver

• Remote webdriver wrapper

• Parallelization

• Neural Network plugins

Testing

All tests were written using Docker services instead of Mocks. Alternative mocks will be added soon ;)

$ pip install -r requirements-dev.txt
$ py.test cabu/tests

Contributing

Please see the Contribute page.

Copyright

Cabu is an open source project by Théotime Lévèque.