noizze-crawler 12

A web page crawler which returns (title, og:image, og:description).

NOIZZE Crawler

A web page crawler PyPI Package which returns (title(og, head), image(og, meta), description(og, meta)).

Dependency

• BeautifulSoup4

Installation

Run the folowing to install:

pip install noizze-crawler

Usage

import noizze_crawler as nc
import sys


if __name__ == '__main__':
    url = 'https://dvdprime.com/g2/bbs/board.php?bo_table=comm&wr_id=20525678'

    try:
        (title, desc, image_url, html) = nc.crawler(url)

    except nc.HostNotFound as e:
        print("Host Not Found")
        sys.exit(1)
    except nc.HTTPError as e:
        print("HTTP {}".format(e))
        sys.exit(1)

    print(title, desc, image_url)  # html

ChangeLog

• v11: Fixed bugs #3 #8
• v10: Fixed bugs
• v9: Youtube crawler with Google API #4
• v8: Changed PyPI dependency - bs4
• v7: PEP8 passed codes
• v6: HostNotFound, HTTPError exceptions