Skip to main content

Search and extract google results.

Project description

degoogle

search Google and extract result urls directly. skip all the click-through links and other sketchiness

contributions welcome


install with pip: pip install degoogle

or

git clone
cd degoogle
pip install .

command line usage script usage
degoogle "query here" make a dg object, execute queries with run()
usage: degoogle [-h] [-o OFFSET] [-p PAGES] [-t TIME_WINDOW] [-j] query

Search and extract google results.

positional arguments:
  query                 search query

optional arguments:
  -h, --help            show this help message and exit
  -o OFFSET, --offset OFFSET
                        page offset to start from
  -p PAGES, --pages PAGES
                        specify multiple pages
  -t TIME_WINDOW, --time-window TIME_WINDOW
                        time window
  -j, --exclude-junk    exclude junk (yt, fb, quora)

note that time window follows a syntax used by google's tbs parameter with the qdr option (read someone explain how it works here)

examples:

  1. find .txt files on .edu sites within the past 3 months:

degoogle "site:edu filetype:txt" -t m3

image

  1. with one dg instance, update query in a loop to perform multiple searches:
from degoogle import dg
degoogler = dg()
queries = ["site:edu", "site:gov", "filetype:txt"]
for query in queries:
	print(query)
	degoogler.query = query
	results = degoogler.run()
	for result in results:
		print(result)
	print()

image

  1. one liner: make dg instance (query set in constructor), search across 5months, format + print results, end:

[print(result['desc'],": ",result['url'],"\n") for result in (dg("intext:'begin rsa' site:*.edu.*",time_window="m5").run())]

image


this is an experiment meant to have benefits for both user privacy and broadened osint capabilities

idea: when you search google normally, your results will appear to be direct links to the target site, but what you're really getting is more like this:

https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=3ahUWEwjVn87AHIEHeyMAsIQFjAZegQIWARN&url=https%3A%2F%2Fexample.com%2F&usg=BOvWas1Dcw1x9iNBCBHvWL8rWGgJO4

by using a link like this to access example.com, you are providing more information about yourself than necessary. they don't need to know what is done with the results after they are served; their job is done, the click-through is merely a suggestion.

example.com can identify that google was your referer; the page you're clicking through from includes your search query as part of the URL, so they probably know what you searched for too. whether or not they're looking at it is another question

if you navigate to example.com directly, without a referer like google, example.com knows that you have visited, but not where you came from, and google knows even less - even if you originally found out about example.com by spotting it in a google search at some point.

google will obviously always know that the search took place, and which results were served, but I prefer to access results directly and not give anyone that click-through confirmation.

there are also utility benefits here for dorking - you might find a super juicy link on page 10 with a bunch of strange parameters cached, but when you follow google's click-through you're redirected to an index page or 404 without even having a chance to copy the link from the result to research more.

there are times where visually inspecting a URL is just as valuable as accessing the link. this tool ganks the good stuff, URL decodes and normalizes, and returns just the scraped URL + description

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

degoogle-1.0.1.tar.gz (6.2 kB view details)

Uploaded Source

Built Distribution

degoogle-1.0.1-py3-none-any.whl (7.2 kB view details)

Uploaded Python 3

File details

Details for the file degoogle-1.0.1.tar.gz.

File metadata

  • Download URL: degoogle-1.0.1.tar.gz
  • Upload date:
  • Size: 6.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/53.0.0 requests-toolbelt/0.9.1 tqdm/4.56.2 CPython/3.9.1

File hashes

Hashes for degoogle-1.0.1.tar.gz
Algorithm Hash digest
SHA256 1c31407b032899b5d643c2fbb7c4c28181a339448b5599ea52182cf62a8d07fc
MD5 5d0360aa36fb56bb00b2cc58c8009333
BLAKE2b-256 cea20b6dda904d3b4b3ac238de6284cbdd7638f8c9158c68ea473bf37ab4a62e

See more details on using hashes here.

File details

Details for the file degoogle-1.0.1-py3-none-any.whl.

File metadata

  • Download URL: degoogle-1.0.1-py3-none-any.whl
  • Upload date:
  • Size: 7.2 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/53.0.0 requests-toolbelt/0.9.1 tqdm/4.56.2 CPython/3.9.1

File hashes

Hashes for degoogle-1.0.1-py3-none-any.whl
Algorithm Hash digest
SHA256 7d8db5ce188eb91f4cc5ae80c2b9e2314f5c382152c3bad0cedb58b9f15b4063
MD5 2625858dd9f2ba3f9cb3d5187b62a75a
BLAKE2b-256 b6c93249d29e919e35bac498f990d3f145dbfa7fdffce361d8088a55420e2f24

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page