Iob

https://travis-ci.org/lorien/iob.png?branch=master

https://coveralls.io/repos/lorien/iob/badge.svg?branch=master

https://pypip.in/download/iob/badge.svg?period=month

https://landscape.io/github/lorien/iob/master/landscape.png

Web scraping framework based on py3 asyncio & aiohttp libraries.

Usage Example

import re
from itertools import islice

from iob import Crawler, Request

RE_TITLE = re.compile(r'<title>([^<]+)</title>', re.S | re.I)

class TestCrawler(Crawler):
    def task_generator(self):
        for host in islice(open('var/domains.txt'), 100):
            host = host.strip()
            if host:
                yield Request('http://%s/' % host, tag='page')

    def handler_page(self, req, res):
        print('Result of request to {}'.format(req.url))
        try:
            title = RE_TITLE.search(res.body).group(1)
        except AttributeError:
            title = 'N/A'
        print('Title: {}'.format(title))

bot = TestCrawler(concurrency=10)
bot.run()

Installation

pip install iob

Dependencies

Python>=3.4
aiohttp

Metadata

Release files for iob 0.0.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for iob 0.0.2
File	Size	Uploaded	Provenance	Actions
iob-0.0.2.tar.gz	6.7 kB	Feb 19, 2015	Without provenance	Details for iob-0.0.2.tar.gz

Release files / iob-0.0.2.tar.gz

Download URL	iob-0.0.2.tar.gz
Size	6.7 kB
Tags	Source
SHA-256 checksum How to use checksums	`a86440277b1d91996c1f6c1c33a1219790962a504350adfc1f14771936bc3a77`
BLAKE2b-256 checksum How to use checksums	`d69b4d2dadb823bdd27fe2ca45f19e33dce4d3aac3d02efa3a6087bd561bc145`
Upload date	Feb 19, 2015
Uploaded using Trusted Publishing? What is trusted publishing?	No

Security for iob 0.0.2

Report project as malware

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.0.2 This release

Feb 19, 2015 1 release file No provenance

0.0.1

Feb 19, 2015

iob 0.0.2

Iob