Skip to main content

Simple, powerful and pythonic web page search results crawler.

Project description

PageFLow

PageFlow is a Python (2 and 3) library for webpage search result crawler. It provides a simple API and support Google, Baidu, Bing search engines. [https://pypi.org/project/pageflow/]

Features

  • support pages argument instead of just the first pate result.
  • support redirect pages information extraction.

Installation

1. using pip

pip install pageflow

2. using setup.py

git clone https://github.com/Lapis-Hong/PageFlow.git 
cd PageFlow
pip setup.py install

Usage

from pageflow import PageFlow

query = "python"
pages = 1  # search results total pages

pf = PageFlow("baidu", proxies=None)


# Get search page html.
html = pf.get_html(query=query, pages=pages)


# The following results are all generator of SearchResult obj.
# Get search result urls.
url = pf.get_url(query=query, pages=pages)

# Get search result titles.
title = pf.get_title(query=query, pages=pages)

# Get search result abstract.
abstract = pf.get_abstract(query=query, pages=pages)

# Get search result redirect html.
redirect_html = pf.get_redirect_html(query=query, pages=pages)

# Get search result redirect content.
redirect_content = pf.get_redirect_content(query=query, pages=pages)

# Get search result title, abstract and url.
result = pf.get(query=query, pages=pages)

# Get search result title, abstract, url, redirect html and redirect content.
result_all = pf.get_all(query=query, pages=pages)

References

https://github.com/howie6879/magic_google https://github.com/meibenjin/GoogleSearchCrawler https://github.com/chrislinan/cx-extractor-python

Project details


Release history Release notifications

This version

0.1

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for pageflow, version 0.1
Filename, size File type Python version Upload date Hashes
Filename, size pageflow-0.1-py2-none-any.whl (12.8 kB) File type Wheel Python version py2 Upload date Hashes View hashes
Filename, size pageflow-0.1.tar.gz (10.0 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page