Search Engines Scraper
Project description
\n# search_engines
A Python library that queries Google, Bing, Yahoo and other search engines and collects the results from multiple search engine results pages.
Please note that web-scraping may be against the TOS of some search engines, and may result in a temporary ban.
Supported search engines
Google
Bing
Yahoo
Duckduckgo
Startpage
Aol
Dogpile
Ask
Mojeek
Brave
Torch
Features
- Creates output files (html, csv, json).
- Supports search filters (url, title, text).
- HTTP and SOCKS proxy support.
- Collects dark web links with Torch.
- Easy to add new search engines. You can add a new engine by creating a new class in
search_engines/engines/
and add it to thesearch_engines_dict
dictionary insearch_engines/engines/__init__.py
. The new class should subclassSearchEngine
, and override the following methods:_selectors
,_first_page
,_next_page
. - Python2 - Python3 compatible.
Requirements
Python 2.7 - 3.x with
Requests and
BeautifulSoup
Installation
Run the setup file: $ python setup.py install
.
Done!
Usage
As a library:
from search_engines import Google
engine = Google()
results = engine.search("my query")
links = results.links()
print(links)
As a CLI script:
$ python search_engines_cli.py -e google,bing -q "my query" -o json,print
Other versions
- async-search-scraper A really cool asynchronous implementation, written by @soxoj
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
search_engines_kit-0.0.17.tar.gz
(18.4 kB
view hashes)
Built Distribution
Close
Hashes for search_engines_kit-0.0.17.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8cc6c1e5e0d63108b507eb5ee6a590b8b9fc49617181519711e81aaeb60c7e68 |
|
MD5 | 50a66e4ecab79bc19596658a1ffdbb54 |
|
BLAKE2b-256 | 7489a1e2df5aaf1fdf0c6a23004884fb70d44dc1cf029c43ca581e53fb45c1bf |
Close
Hashes for search_engines_kit-0.0.17-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3c44ca23fdcc83190be4a0b834a22bc0c75b442a326b6e03f81a315a08824d92 |
|
MD5 | 8761b8095c6d72afc814a184f12be82d |
|
BLAKE2b-256 | f726c33aac7b608d07ca2fb1724312078d4d979fcb2398dbcff8c57cbfe50021 |