Skip to main content

Search Engines Scraper

Project description

\n# search_engines
A Python library that queries Google, Bing, Yahoo and other search engines and collects the results from multiple search engine results pages.
Please note that web-scraping may be against the TOS of some search engines, and may result in a temporary ban.

Supported search engines

Google
Bing
Yahoo
Duckduckgo
Startpage
Aol
Dogpile
Ask
Mojeek
Brave
Torch

Features

  • Creates output files (html, csv, json).
  • Supports search filters (url, title, text).
  • HTTP and SOCKS proxy support.
  • Collects dark web links with Torch.
  • Easy to add new search engines. You can add a new engine by creating a new class in search_engines/engines/ and add it to the search_engines_dict dictionary in search_engines/engines/__init__.py. The new class should subclass SearchEngine, and override the following methods: _selectors, _first_page, _next_page.
  • Python2 - Python3 compatible.

Requirements

Python 2.7 - 3.x with
Requests and
BeautifulSoup

Installation

Run the setup file: $ python setup.py install.
Done!

Usage

As a library:

from search_engines import Google

engine = Google()
results = engine.search("my query")
links = results.links()

print(links)

As a CLI script:

$ python search_engines_cli.py -e google,bing -q "my query" -o json,print

Other versions

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

search_engines_kit-1.0.tar.gz (18.4 kB view hashes)

Uploaded Source

Built Distribution

search_engines_kit-1.0-py3-none-any.whl (27.9 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page