pygooglenews

If Google News had a Python library

Created by Artem from newscatcherapi.com but you do not need anything from us or from anyone else to get the software going, it just works out of the box.

My blog post about how I did it

Demo

You might also like to check our Google News API or Financial Google News API

Table of Contents

• About
• Examples of Use Cases
• Working with Google News in Production
• Motivation
• Installation
• Quickstart
• Documentation
• Advanced Query Search Examples
• About me
• Change Log

<a name="about"/>

About

A python wrapper of the Google News RSS feed.

Top stories, topic related news feeds, geolocation news feed, and an extensive full text search feed.

This work is more of a collection of all things I could find out about how Google News functions.

How is it different from other Pythonic Google News libraries?

1. URL-escaping user input helper for the search function
2. Extensive support for the search function that makes it simple to use:
   • exact match
   • in title match, in url match, etc
   • search by date range (from_ & to_), latest published (when)
3. Parsing of the sub articles. Almost always, all feeds except the search one contain a subset of similar news for each article in a feed. This package takes care of extracting those sub articles. This feature might be highly useful to ML task when you need to collect a data of similar article headlines

<a name="usecase"/>

Examples of Use Cases

1. Integrating a news feed to your platform/application/website
2. Collecting data by topic to train your own ML model
3. Search for latest mentions for your new product
4. Media monitoring of people/organizations — PR

<a name="production"/>

Working with Google News in Production

Before we start, if you want to integrate Google News data to your production then I would advise you to use one of the 3 methods described below. Why? Because you do not want your servers IP address to be locked by Google. Every time you call any function there is an HTTPS request to Google's servers. Don't get me wrong, this Python package still works out of the box.

1. NewsCatcher's Google News API — all code is written for you, clean & structured JSON output. Low price. You can test it yourself with no credit card. Plus, financial version of API is also available.
2. ScrapingBee API which handles proxy rotation for you. Each function in this package has scraping_bee parameter where you paste your API key. You can also try it for free, no credit card required. See example
3. Your own proxy — already have a pool of proxies? Each function in this package has proxies parameter (python dictionary) where you just paste your own proxies.

<a name="motivaion"/>

Motivation

I love working with the news data. I love it so much that I created my own company that crawls for hundreds of thousands of news articles, and allow you to search it via a news API. But this time, I want to share with the community a Python package that makes it simple to get the news data from the best search engine ever created - Google.

Most likely, you know already that Google has its own news service. It is different from the usual Google search that we use on a daily basis (sorry DuckDuckGo, maybe next time).

This package uses the RSS feed of the Google News. The top stories page, for example.

RSS is an XML page that is already well structured. I heavily rely on Feedparser package to parse the RSS feed.

Google News used to have an API but it was deprecated many years ago. (Unofficial) information about RSS syntax is decentralized over the web. There is no official documentation. So, I tried my best to collect all this informaion in one place.

<a name="installation"/>

Installation

$ pip install pygooglenews --upgrade

<a name="quickstart"/>

Quickstart

from pygooglenews import GoogleNews

gn = GoogleNews()

Top Stories

top = gn.top_news()

Stories by Topic

business = gn.topic_headlines('business')

Geolocation Specific Stories

headquaters = gn.geo_headlines('San Fran')

Stories by a Query Search

# search for the best matching articles that mention MSFT and 
# do not mention AAPL (over the past 6 month
search = gn.search('MSFT -APPL', when = '6m')

---

<a name="documentation"/>

Documentation - Functions & Classes

GoogleNews Class

from pygooglenews import GoogleNews
# default GoogleNews instance
gn = GoogleNews(lang = 'en', country = 'US')

To get the access to all the functions, you first have to initiate the GoogleNews class.

It has 2 required variables: lang and country

You can try any combination of those 2, however, it does not exist for all. Only the combinations that are supported by GoogleNews will work. Check the official Google News page to check what is covered:

On the bottom left side of the Google News page you may find a Language & region section where you can find all of the supported combinations.

For example, for country=UA (Ukraine), there are 2 languages supported:

• lang=uk Ukrainian
• lang=ru Russian

---

Top Stories

top = gn.top_news(proxies=None, scraping_bee = None)

top_news() returns the top stories for the selected country and language that are defined in GoogleNews class. The returned object contains feed (FeedParserDict) and entries list of articles found with all data parsed.

---

Stories by Topic

business = gn.topic_headlines('BUSINESS', proxies=None, scraping_bee = None)

The returned object contains feed (FeedParserDict) and entries list of articles found with all data parsed.

Accepted topics are:

• WORLD
• NATION
• BUSINESS
• TECHNOLOGY
• ENTERTAINMENT
• SCIENCE
• SPORTS
• HEALTH

However, you can find some other topics that are also supported by Google News.

For example, if you search for corona in the search tab of en + US you will find COVID-19 as a topic.

The URL looks like this: https://news.google.com/topics/CAAqIggKIhxDQkFTRHdvSkwyMHZNREZqY0hsNUVnSmxiaWdBUAE?hl=en-US&gl=US&ceid=US%3Aen

We have to copy the text after topics/ and before ?, then you can use it as an input for the top_news() function.

from pygooglenews import GoogleNews

gn = GoogleNews()
covid = gn.topic_headlines('CAAqIggKIhxDQkFTRHdvSkwyMHZNREZqY0hsNUVnSmxiaWdBUAE')

However, be aware that this topic will be unique for each language/country combination.

---

Stories by Geolocation

gn = GoogleNews('uk', 'UA')
kyiv = gn.geo_headlines('kyiv', proxies=None, scraping_bee = None)
# or 
kyiv = gn.geo_headlines('kiev', proxies=None, scraping_bee = None)
# or
kyiv = gn.geo_headlines('киев', proxies=None, scraping_bee = None)
# or
kyiv = gn.geo_headlines('Київ', proxies=None, scraping_bee = None)

The returned object contains feed (FeedParserDict) and entries list of articles found with all data parsed.

All of the above variations will return the same feed of the latest news about Kyiv, Ukraine:

geo['feed'].title

# 'Київ - Останні - Google Новини'

It is language agnostic, however, it does not guarantee that the feed for any specific place will exist. For example, if you want to find the feed on LA or Los Angeles you can do it with GoogleNews('en', 'US').

The main (en, US) Google News client will most likely find the feed about the most places.

---

Stories by a Query

gn.search(query: str, helper = True, when = None, from_ = None, to_ = None, proxies=None, scraping_bee=None)

The returned object contains feed (FeedParserDict) and entries list of articles found with all data parsed.

Google News search itself is a complex function that has inherited some features from the standard Google Search.

The official reference on what could be inserted

The biggest obstacle that you might have is to write the URL-escaping input. To ease this process, helper = True is turned on by default.

helper uses urllib.parse.quote_plus to automatically convert the input.

For example:

• 'New York metro opening' --> 'New+York+metro+opening'
• 'AAPL -MSFT' --> 'AAPL+-MSFT'
• '"Tokyo Olimpics date changes"' --> '%22Tokyo+Olimpics+date+changes%22'

You can turn it off and write your own query in case you need it by helper = False

when parameter (str) sets the time range for the published datetime. I could not find any documentation regarding this option, but here is what I deducted:

• h for hours.(For me, worked for up to 101h). when=12h will search for only the articles matching the search criteri and published for the last 12 hours
• d for days
• m for month (For me,