Search and extract google results.
Project description
degoogle
search Google and extract result urls directly. skip all the click-through links and other sketchiness
contributions welcome
install with pip:
pip install degoogle
or
git clone
cd degoogle
pip install .
command line usage | script usage |
---|---|
degoogle "query here" |
make a dg object, execute queries with run() |
usage: degoogle [-h] [-o OFFSET] [-p PAGES] [-t TIME_WINDOW] [-j] query
Search and extract google results.
positional arguments:
query search query
optional arguments:
-h, --help show this help message and exit
-o OFFSET, --offset OFFSET
page offset to start from
-p PAGES, --pages PAGES
specify multiple pages
-t TIME_WINDOW, --time-window TIME_WINDOW
time window
-j, --exclude-junk exclude junk (yt, fb, quora)
note that time window
follows a syntax used by google's tbs
parameter with the qdr
option (read someone explain how it works here)
examples:
- find
.txt
files on.edu
sites within the past3 months
:
degoogle "site:edu filetype:txt" -t m3
- with one
dg
instance, updatequery
in a loop to perform multiple searches:
from degoogle import dg
degoogler = dg()
queries = ["site:edu", "site:gov", "filetype:txt"]
for query in queries:
print(query)
degoogler.query = query
results = degoogler.run()
for result in results:
print(result)
print()
- one liner: make
dg
instance (query set in constructor), search across 5months, format + print results, end:
[print(result['desc'],": ",result['url'],"\n") for result in (dg("intext:'begin rsa' site:*.edu.*",time_window="m5").run())]
this is an experiment meant to have benefits for both user privacy and broadened osint capabilities
idea: when you search google normally, your results will appear to be direct links to the target site, but what you're really getting is more like this:
https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=3ahUWEwjVn87AHIEHeyMAsIQFjAZegQIWARN&url=https%3A%2F%2Fexample.com%2F&usg=BOvWas1Dcw1x9iNBCBHvWL8rWGgJO4
by using a link like this to access example.com
, you are providing more information about yourself than necessary. they don't need to know what is done with the results after they are served; their job is done, the click-through is merely a suggestion.
example.com
can identify that google was your referer; the page you're clicking through from includes your search query as part of the URL, so they probably know what you searched for too. whether or not they're looking at it is another question
if you navigate to example.com
directly, without a referer like google, example.com
knows that you have visited, but not where you came from, and google knows even less - even if you originally found out about example.com
by spotting it in a google search at some point.
google will obviously always know that the search took place, and which results were served, but I prefer to access results directly and not give anyone that click-through confirmation.
there are also utility benefits here for dorking - you might find a super juicy link on page 10 with a bunch of strange parameters cached, but when you follow google's click-through you're redirected to an index page or 404 without even having a chance to copy the link from the result to research more.
there are times where visually inspecting a URL is just as valuable as accessing the link. this tool ganks the good stuff, URL decodes and normalizes, and returns just the scraped URL + description
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file degoogle-1.0.1.tar.gz
.
File metadata
- Download URL: degoogle-1.0.1.tar.gz
- Upload date:
- Size: 6.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/53.0.0 requests-toolbelt/0.9.1 tqdm/4.56.2 CPython/3.9.1
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1c31407b032899b5d643c2fbb7c4c28181a339448b5599ea52182cf62a8d07fc |
|
MD5 | 5d0360aa36fb56bb00b2cc58c8009333 |
|
BLAKE2b-256 | cea20b6dda904d3b4b3ac238de6284cbdd7638f8c9158c68ea473bf37ab4a62e |
File details
Details for the file degoogle-1.0.1-py3-none-any.whl
.
File metadata
- Download URL: degoogle-1.0.1-py3-none-any.whl
- Upload date:
- Size: 7.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/53.0.0 requests-toolbelt/0.9.1 tqdm/4.56.2 CPython/3.9.1
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7d8db5ce188eb91f4cc5ae80c2b9e2314f5c382152c3bad0cedb58b9f15b4063 |
|
MD5 | 2625858dd9f2ba3f9cb3d5187b62a75a |
|
BLAKE2b-256 | b6c93249d29e919e35bac498f990d3f145dbfa7fdffce361d8088a55420e2f24 |