Search google, bing, yahoo, and other search engines with python
Project description
search_engines
A Python library that queries Google, Bing, Yahoo and other search engines and collects the results from multiple search engine results pages.
Please note that web-scraping may be against the TOS of some search engines, and may result in a temporary ban.
Supported search engines
Google
Bing
Yahoo
Duckduckgo
Startpage
Aol
Dogpile
Ask
Mojeek
Brave
Torch
Features
- Creates output files (html, csv, json).
- Supports search filters (url, title, text).
- HTTP and SOCKS proxy support.
- Collects dark web links with Torch.
- Easy to add new search engines. You can add a new engine by creating a new class in
search_engines/engines/
and add it to thesearch_engines_dict
dictionary insearch_engines/engines/__init__.py
. The new class should subclassSearchEngine
, and override the following methods:_selectors
,_first_page
,_next_page
. - Python2 - Python3 compatible.
Requirements
Python 2.7 - 3.x with
Requests and
BeautifulSoup
Installation
Run the setup file: $ python setup.py install
.
Done!
Usage
As a library:
from search_engines import Google
engine = Google()
results = engine.search("my query")
links = results.links()
print(links)
As a CLI script:
$ python search_engines_cli.py -e google,bing -q "my query" -o json,print
Other versions
- async-search-scraper A really cool asynchronous implementation, written by @soxoj
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for Search-Engines-Scraper-Tasos-0.6.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2f2cfbe5cbb30f9b2a1bdfcef341bc2ad961f0f9fc40730636c80180d06493cd |
|
MD5 | 5aca5b3d5861aa362896736a39173115 |
|
BLAKE2b-256 | 8d15c1acd6f3a6fa34560f6ef43b2962bd050648a1b1b1cd7cc60c1b9b63d2cb |
Close
Hashes for Search_Engines_Scraper_Tasos-0.6-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 40e213e666a23a94866bb9b0a6f71a13534dbeac815d45b56db02b6ce8a6ed12 |
|
MD5 | 2733696a505b241c3ba6992b3e38353f |
|
BLAKE2b-256 | 39d8625779ee8333f33c85eab9e5b12fff2f600ea8c8d94bff77859aaac26b8a |