Skip to main content

A module to scrape and extract links, titles and descriptions from Google search results

Project description

## GoogleScraper - Scraping the Google Search Engine

### Table of Contents

1. [Installation](#install)
2. [About](#about)
3. [Usage with Python](#usage)
4. [Command line usage (read this!)](#cli-usage)
5. [Contact](#contact)

<a name="install" \>
### Installation

GoogleScraper is written in Python 3. Therefore install at least python 3.3
Furthermore, you need to install the Chrome Browser, maybe even the ChromeDriver for Selenium (I didn't have to).

From now on (august 2014), you can just install with pip:

pip install GoogleScraper

#### Alternatively install from Github:

First clone and change in project tree.

Begin with installing the following third party modules:

bs4 [try beautifulsoup4]

You can do so with:

`pip3 install module1, module2, ..`

If you want to install GoogleScraper locally, do as follows (Run all commands in the directory):

virtualenv --no-site-packages .venv
source .venv/bin/activate
pip install -r requirements.txt
# Now test it

<a name="about" />
### What does

GoogleScraper parses Google search engine results easily and in a performant way. It allows you to extract all found
links and their titles and descriptions programmatically which enables you to process scraped data further.

There are unlimited *usage scenarios*:

+ Quickly harvest masses of [google dorks][1].
+ Use it as a SEO tool.
+ Discover trends.
+ Compile lists of sites to feed your own database.
+ Many more use cases...

First of all you need to understand that GoogleScraper uses **two completely different scraping approaches**:
+ Scraping with low level networking libraries such as `urllib.request` or `requests` modules. This simulates the http packets sent by real browsers.
+ Scrape by controlling a real browser with Python

Whereas the first approach was implemented first, the second approach looks much more promising in comparison.
Effective: Development for the second approach started around 10.03.2014

GoogleScraper is implemented with the following techniques/software:

+ Written in Python 3.4
+ Uses multithreading/asynchroneous IO. (two possible approaches, currently only multi-threading is implemented)
+ Supports parallel google scraping with multiple IP addresses.
+ Provides proxy support using [socksipy][2] and built in browser proxies:
* Socks5
* Socks4
* HttpProxy
+ Support for additional google search features like news/image/video search.

### How does GoogleScraper maximize the amount of extracted information per IP address?

Scraping is a critical and highly complex subject. Google and other search engine giants have a strong inclination
to make the scrapers life as hard as possible. There are several ways for the Google Servers to detect that a robot is using
their search engine:

+ The User-Agent is not one of a browser.
+ The search params are not identical to the ones that browser used by a human sets:
* Javascript generates challenges dynamically on the client side. This might include heuristics that try to detect human behaviour. Example: Only humans move their mouses and hover over the interesting search results.
+ Robots have a strict requests pattern (very fast requests, without a random time between the sent packets).
+ Dorks are heavily used
+ No pictures/ads/css/javascript are loaded (like a browser does normally) which in turn won't trigger certain javascript events

So the biggest hurdle to tackle is the javascript detection algorithms. I don't know what Google does in their javascript, but I will soon investigate it further and then decide if it's not better to change strategies and
switch to a **approach that scrapes by simulating browsers in a browserlike environment** that can execute javascript. The networking of each of these virtual browsers is proxified and manipulated such that it behaves like
a real physical user agent. I am pretty sure that it must be possible to handle 20 such browser sessions in a parallel way without stressing resources too much. The real problem is as always the lack of good proxies...

### How to overcome difficulties of low level (http) scraping?

As mentioned above, there are several drawbacks when scraping with `urllib.request` or `requests` modules and doing the networking on my own:

Browsers are ENORMOUSLY complex software systems. Chrome has around 8 millions line of code and firefox even 10 LOC. Huge companies invest a lot of money to push technology forward (HTML5, CSS3, new standards) and each browser
has a unique behaviour. Therefore it's almost impossible to simulate such a browser manually with HTTP requests. This means Google has numerous ways to detect anomalies and inconsistencies in the browsing usage. Alone the
dynamic nature of Javascript makes it impossible to scrape undetected.

This cries for an alternative approach, that automates a **real** browser with Python. Best would be to control the Chrome browser since Google has the least incentives to restrict capabilities for their own native browser.
Hence I need a way to automate Chrome with Python and controlling several independent instances with different proxies set. Then the output of result grows linearly with the number of used proxies...

Some interesting technologies/software to do so:
+ [Selenium](
+ [Mechanize](

<a name="usage" \>
### Example Usage
Here you can learn how to use GoogleScrape from within your own Python scripts.

Keep in mind that the bottom example source uses the not very powerful *http* scrape method. Look [here](#cli-usage) if you
need to unleash the full power of GoogleScraper.

import GoogleScraper
import urllib.parse


if __name__ == '__main__':

results = GoogleScraper.scrape('Best SEO tool', num_results_per_page=50, num_pages=3, offset=0, searchtype='normal')

for page in results:
for link_title, link_snippet, link_url, *rest in page['results']:
# You can access all parts of the search results like that
# link_url.scheme => URL scheme specifier (Ex: 'http')
# link_url.netloc => Network location part (Ex: '')
# link_url.path => URL scheme specifier (Ex: ''help/Python.html'')
# link_url.params => Parameters for last path element
# link_url.query => Query component
print(urllib.parse.unquote(link_url.geturl())) # This reassembles the parts of the url to the whole thing

# How many urls did we get on all pages?
print(sum(len(page['results']) for page in results))

# How many hits has google found with our keyword (as shown on the first page)?

### Example Output

This is a example output of the above **. You can execute it by just firing `python` in the project directory:

[nikolai@niko-arch GoogleScraper]$ python
About 14,100,000 results

<a name="cli-usage" \>
### Direct command line usage

Probably the best way to use GoogleScrape is to use it from the command line and fire a command such as
the following:
python sel --keyword-file path/to/keywordfile

Here *sel* marks the scraping mode as 'selenium'. This means scrapes with real browsers. This is pretty powerful, since
you can scrape long and a lot of sites (Google has a hard time blocking real browsers). The argument of the flag `--keyword-file` must be a file with keywords separated by
newlines. So: For every google query one line. Easy, isnt' it?

Example keyword-file:
keyword number one
how to become a good rapper
allintext:"You have a Mysql Error in your"
intitle:"admin config"
Best brothels in atlanta

By default, *sel* mode only requests the first 10 results for each keyword. But you can specify on how many Google result pages
you want to scrape every keyword. Just use the **-p** parameter as shown below:

# searches all keywords in the keywordfile on 10 result pages
python sel --keyword-file path/to/keywordfile -p 10

By now, you have 10 results per page by default (google returns up to 100 results per page), but this will also be configurable in the near future. *http* mode
supports up to 100 results per page.

After the scraping you'll automatically have a new sqlite3 database in the project directory (with a date time string as file name). You can open the database with any sqlite3 command
line tool or other software.

It shouldn't be a problem to scrape **_10'000 keywords in 2 hours_**, if you are really crazy, set the maximal browsers in the config a little
bit higher (in the top of the script file).

If you want, you can specify the flag `--proxy-file`. As argument you need to pass a file with proxies in it and with the following format:

protocol proxyhost:proxyport username:password
socks5 blabla:12345
socks4 elite:js@fkVA3(Va3)

That's basically all for the *sel* modeHave fun.

In case you want to use in *http* mode (which means that raw http headers are sent), use it as follows:

python http -p 1 -n 25 -q "keywords separated by whitespaces"

<a name="contact" \>
### Contact

If you feel like contacting me, do so and send me a mail. You can find my contact information on my [blog][3].

[1]: "Google Dorks"
[2]: "Socksipy Branch"
[3]: "Contact with author"

Project details

Release history Release notifications

History Node


History Node


History Node


History Node


History Node


History Node


History Node


History Node


History Node


History Node


History Node


History Node


History Node


History Node


History Node


History Node


History Node


History Node


History Node


History Node


History Node


History Node


History Node


History Node


History Node


History Node


History Node


History Node


History Node


History Node


History Node


History Node


History Node


History Node


History Node


History Node


This version
History Node


History Node


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
GoogleScraper-0.0.2.dev1.tar.gz (46.8 kB) Copy SHA256 hash SHA256 Source None Aug 12, 2014

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging CloudAMQP CloudAMQP RabbitMQ AWS AWS Cloud computing Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page