Search google, bing, yahoo, and other search engines with python
Project description
search_engines
A Python library that queries Google, Bing, Yahoo and other search engines and collects the results from multiple search engine results pages.
Please note that web-scraping may be against the TOS of some search engines, and may result in a temporary ban.
Supported search engines
Google
Bing
Yahoo
Duckduckgo
Startpage
Aol
Dogpile
Ask
Mojeek
Brave
Torch
Features
- Creates output files (html, csv, json).
- Supports search filters (url, title, text).
- HTTP and SOCKS proxy support.
- Collects dark web links with Torch.
- Easy to add new search engines. You can add a new engine by creating a new class in
search_engines/engines/
and add it to thesearch_engines_dict
dictionary insearch_engines/engines/__init__.py
. The new class should subclassSearchEngine
, and override the following methods:_selectors
,_first_page
,_next_page
. - Python2 - Python3 compatible.
Requirements
Python 2.7 - 3.x with
Requests and
BeautifulSoup
Installation
Run the setup file: $ python setup.py install
.
Done!
Usage
As a library:
from search_engines import Google
engine = Google()
results = engine.search("my query")
links = results.links()
print(links)
As a CLI script:
$ python search_engines_cli.py -e google,bing -q "my query" -o json,print
Other versions
- async-search-scraper A really cool asynchronous implementation, written by @soxoj
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file Search-Engines-Scraper-Tasos-0.6.tar.gz
.
File metadata
- Download URL: Search-Engines-Scraper-Tasos-0.6.tar.gz
- Upload date:
- Size: 18.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.8.13
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2f2cfbe5cbb30f9b2a1bdfcef341bc2ad961f0f9fc40730636c80180d06493cd |
|
MD5 | 5aca5b3d5861aa362896736a39173115 |
|
BLAKE2b-256 | 8d15c1acd6f3a6fa34560f6ef43b2962bd050648a1b1b1cd7cc60c1b9b63d2cb |
File details
Details for the file Search_Engines_Scraper_Tasos-0.6-py3-none-any.whl
.
File metadata
- Download URL: Search_Engines_Scraper_Tasos-0.6-py3-none-any.whl
- Upload date:
- Size: 28.9 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.8.13
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 40e213e666a23a94866bb9b0a6f71a13534dbeac815d45b56db02b6ce8a6ed12 |
|
MD5 | 2733696a505b241c3ba6992b3e38353f |
|
BLAKE2b-256 | 39d8625779ee8333f33c85eab9e5b12fff2f600ea8c8d94bff77859aaac26b8a |