The maintained successor of GoogleScraper is the general purpose crawling infrastructure

GoogleScraper - Scraping search engines professionally

Scrapeulous.com - Scraping Service

GoogleScraper is a open source tool and will remain a open source tool in the future.

Also the modern successor of GoogleScraper, the general purpose crawling infrastructure, will remain open source and free.

Some people however would want to quickly have a service that lets them scrape some data from Google or any other search engine. For this reason, I created the web service scrapeulous.com.

Switching from Python to Javascript/puppeteer

Last State: Feburary 2019

The successor of GoogleScraper can be found here

This means that I won't maintain this project anymore. All new development goes in the above project.

There are several reasons why I won't continue to put much effort into this project.

1. Python is not the language/framework for modern scraping. Node/Javascript is. The reason is puppeteer. puppeteer is the de-facto standard for controlling and automatizing web browsers (especially Chrome). This project uses Selenium. Selenium is kind of old and outdated.
2. Scraping in 2019 is almost completely reduced to controlling webbrowsers. There is no more need to scrape directly on the HTTP protocol level. It's too bugy and too easy to fend of by anit-bot mechanisms. And this project still supports raw http requests.
3. Scraping should be parallelized in the cloud or among a set of dedicated machines. GoogleScraper cannot handle such use cases without significant effort.
4. This project is extremely buggy.

For this reason I am going to continue developing a scraping library named https://www.npmjs.com/package/se-scraper in Javascript which runs on top of puppeteer.

You can download the app here: https://www.npmjs.com/package/se-scraper

It supports a wide range of different search engines and is much more efficient than GoogleScraper. The code base is also much less complex without threading/queueing and complex logging capabilities.

August/September 2018

For questions you can contact me on my wegpage and write me an email there.

This project is back to live after two years of abandonment. In the coming weeks, I will take some time to update all functionality to the most recent developments. This encompasses updating all Regexes and changes in search engine behavior. After a couple of weeks, you can expect this project to work again as documented here.

Table of Contents

1. Installation
2. Quick Start
3. Asynchronous mode
4. Testing
5. About
6. Command line usage
7. Contact

Installation

GoogleScraper is written in Python 3. You should install at least Python 3.6. The last major development was all done with Python 3.7. So when using Ubuntu 16.04 and Python 3.7 for instance, please install Python 3 from the official packages. I use the Anaconda Python distribution, which does work very well for me.

Furthermore, you need to install the Chrome Browser and also the ChromeDriver for Selenium mode. Alternatively install the Firefox Browser and the geckodriver for Selenium Mode. See instructions below.

You can also install GoogleScraper comfortably with pip:

virtualenv --python python3 env
source env/bin/activate
pip install GoogleScraper

Right now (September 2018) this is discouraged. Please install from latest Github sources.

Alternatively install directly from Github

Sometimes the newest and most awesome stuff is not available in the cheeseshop (That's how they call https://pypi.python.org/pypi/pip). Therefore you maybe want to install GoogleScraper from the latest source that resides in this Github repository. You can do so like this:

virtualenv --python python3 env
source env/bin/activate
pip install git+git://github.com/NikolaiT/GoogleScraper/

Please note that some features and examples might not work as expected. I also don't guarantee that the app even runs. I only guarantee (to a certain degree at least) that installing from pip will yield a usable version.

Chromedriver

Download the latest chromedriver from here: https://sites.google.com/a/chromium.org/chromedriver/downloads

Unzip the driver and save it somewhere and then update the chromedriver_path in the GoogleScraper configuration file scrape_config.py to the path where you saved the driver chromedriver_path = 'Drivers/chromedriver'

Geckodriver

Download the latest geckodriver from here: https://github.com/mozilla/geckodriver/releases

Unzip the driver and save it somewhere and then update the geckodriver_path in the GoogleScraper configuration file scrape_config.py to the path where you saved the driver geckodriver_path = 'Drivers/geckodriver'

Update the settings for selenium and firefox/chrome

Update the following settings in the GoogleScraper configuration file scrape_config.py to your values.

# chrome driver executable path
# get chrome drivers here: https://chromedriver.storage.googleapis.com/index.html?path=2.41/
chromedriver_path = 'Drivers/chromedriver'

# geckodriver executable path
# get gecko drivers here: https://github.com/mozilla/geckodriver/releases
geckodriver_path = 'Drivers/geckodriver'

# path to firefox binary
firefox_binary_path = '/home/nikolai/firefox/firefox'

# path to chromium browser binary
chrome_binary_path = '/usr/bin/chromium-browser'

Quick Start

Install as described above. Make sure that you have the selenium drivers for chrome/firefox if you want to use GoogleScraper in selenium mode.

See all options

GoogleScraper -h

Scrape the single keyword "apple" with http mode:

GoogleScraper -m http --keyword "apple" -v info

Scrape all keywords that are in the file SearchData/5words in selenium mode using chrome in headless mode:

GoogleScraper -m selenium --sel-browser chrome --browser-mode headless --keyword-file SearchData/5words -v info

Scrape all keywords that are in

• keywords.txt
• with http mode
• using 5 threads
• scrape in the search engines bing and yahoo
• store the output in a JSON file
• increase verbosity to the debug level

GoogleScraper -m http --keyword-file SearchData/some_words.txt --num-workers 5 --search-engines "bing,yahoo" --output-filename threaded-results.json -v debug

Do an image search for the keyword "K2 mountain" on google:

GoogleScraper -s "google" -q "K2 mountain" -t image -v info

Asynchronous mode

This is probably the most awesome feature of GoogleScraper. You can scrape with thousands of requests per second if either

• The search engine doesn't block you (Bing didn't block me when requesting 100 keywords / second)
• You have enough proxies

Example for Asynchronous mode:

Search the keywords in the keyword file SearchData/marketing-models-brands.txt on bing and yahoo. By default asynchronous mode spawns 100 requests at the same time. This means around 100 requests per second (depends on the actual connection...).

GoogleScraper -s "bing,yahoo" --keyword-file SearchData/marketing-models-brands.txt -m http-async -v info -o marketing.json

The results (partial results, because there were too many keywords for one IP address) can be inspected in the file Outputs/marketing.json.

Testing GoogleScraper

GoogleScraper is hugely complex. Because GoogleScraper supports many search engines and the HTML and Javascript of those Search Providers changes frequently, it is often the case that GoogleScraper ceases to function for some search engine. To spot this, you can run functional tests.

For example the test below runs a scraping session for Google and Bing and tests that the gathered data looks more or less okay.

python -m pytest Tests/functional_tests.py::GoogleScraperMinimalFunctionalTestCase

What does GoogleScraper.py?

GoogleScraper parses Google search engine results (and many other search engines _) easily and in a fast way. It allows you to extract all found links and their titles and descriptions programmatically which enables you to process scraped data further.

There are unlimited usage scenarios:

• Quickly harvest masses of [google dorks][1].
• Use it as a SEO tool.
• Discover trends.
• Compile lists of sites to feed your own database.
• Many more use cases...
• Quite easily extendable since the code is well documented

First of all you need to understand that GoogleScraper uses two completely different scraping approaches:

• Scraping with low level http libraries such as urllib.request or requests modules. This simulates the http packets sent by real browsers.
• Scrape by controlling a real browser with the selenium framework

Whereas the former approach was implemented first, the later approach looks much more promising in comparison, because search engines have no easy way detecting it.

GoogleScraper is implemented with the following techniques/software:

• Written in Python 3.7
• Uses multithreading/asynchronous IO.
• Supports parallel scraping with multiple IP addresses.
• Provides proxy support using [socksipy][2] and built in browser proxies:
  • Socks5
  • Socks4
  • HttpProxy
• Support for alternative search modes like news/image/video search.

What search engines are suppported ?

Currently the followi