fbcrawl

Fbcrawl is an advanced crawler for Facebook, written in python, based on the Scrapy framework.

UNMAINTAINED

For an undefined period I will be unable to review issues, fix bugs and merge pull requests. As I have been the sole contributor to the project, it's likely that the code will remain frozen at the current stage.

Anybody who is skilled enough and willing to partecipate, may open a dedicated issue or contact at my email address: rugantio AT gmail DOT com

I will be back, but in the meantime I'd appreciate if this becomes a community project.

DONATIONS

Fbcrawl is free software. It is not "free as beer" nor "free as speech", it is "free as a toilet": it is always available and working, but someone as to keep it clean and tidy, and I am the only one at the moment, it is not a community project. Please consider make a donation, it will keep this project alive and if I see actual interest from people I will get on with the TODO list. One of the my long-term goal is to refactor the framework with a gui, connections to databases and graph vizualitations. These tasks would take at least a couple of months of work, and I will be able to afford them only with your support! Thank you :)

DISCLAIMER

This software is not authorized by Facebook and doesn't follow Facebook's robots.txt. Scraping without Facebook explicit written is a violation of the terms and conditions on scraping and can potentially cause a lawsuit

This software is provided as is, for educational purposes, to show how a crawler can be made to recursively parse a facebook page. Use at your own risk.

Introduction

<div style="text-align:center"> <img src="./trump.png" alt="Donald Trump" width="1080"> </div>

EDIT: fbcrawl can now crawl comments! check out the "how to crawl comments" section!

What features can fbcrawl obtain? Everything that you see in the table is crawled by default. I decided to simplify the timestamp feature, leaving out the hour and to ignore comments and commentators, which are going to be parsed post-by-post by another crawler.

Fbcrawl makes use of an static mobile version of facebook, unknown to many: https://mbasic.facebook.com because it's all plain HTML and we can navigate easily through the pages without having to emulate a browser or inject javascript code.

Installation

Requirements are: python3 (python2 is also supported) and the scrapy framework, that should pull other needed libs as dependencies (twisted, libxml2 etc.).

Scrapy can be installed through the package manager of the distribution (in my arch box is simply called "scrapy") or through internal python package system, typing:

pip install scrapy

Architecture

The way scrapy works is through an engine that manages granularly every step of the crawling process.

<img src="https://docs.scrapy.org/en/latest/_images/scrapy_architecture_02.png" width="800">

The project is thus divided in several files that serve different purposes:

\fbcrawl <br />     README.md -- this file <br />     scrapy.cfg -- ini-style file that defines the project <br />     \fbcrawl <br />         _init.py_ <br />         items.py -- defines the fields that we want to export <br />         middlewares.py <br />         pipelines.py -- defines how we handle each item (the set of fields) <br />         settings.py -- all the parameter settings of the project <br />         \spiders <br />             _init.py_ <br />             fbcrawl.py -- implements the spider for posts <br />             comments.py -- implements the spider for comments

How to crawl a page (fbcrawl.py)

The core of the crawler is this spider class, fbcrawl. On init, it navigates to mbasic.facebook.com and logs into facebook according to the provided credentials, passed as parameters at execution time (see "How to use"). Then the parse_page method is called with the page name given at runtime and the crawling process begins recursively retrieving all the posts found in every page. For each of the post it retrieves all the features, using the callback parse_post, and all the reactions, using parse_reactions.

The webpage are parsed and the fields are extracted using XPath selectors. These selectors are implemented on the python lib lxml so they are very fast.

Thanks to XPath, scrapy can navigate the webpage in a DOM model, as one would navigate a filesystem, with several features of pattern matching. If you know nothing about XPath this guide and this cheatsheet can be helpful. Other resources are the original W3C docs and XPath functions.

The XPath are easy to obtain using Firefox's or Chromium's dev tools, but sometimes the field relative to a property changes location, which is something to keep in mind. For example, notice how I had to handle the source field using the pipe | that is the OR operator: new.add_xpath('source', '//span/strong/a/text() | //div/a/strong/text() | //td/div/h3/strong/a/text()'). This kind of juggling is helpful to maintain consistency of the data in our table. The control on the data and the policy to use is often implemented in the Item Pipeline.

So the parse methods populates Item fields (to be explained in the next section) and pass control over to the Item Loader.

Refer to Scrapy's Spider documentation for more info.

Items (items.py)

This file defines an Item class, so that the fields that we have extracted can be grouped in Items and organized in a more concise manner. Item objects are simple containers used to collect the scraped data. They provide a dictionary-like API with a convenient syntax for declaring their available fields.

I have extracted every field present in the post elements and add a few local ones. Namely for each article we have:

source      -    name of the post publisher, if it's shared it's the original one
shared_from -    if the post is shared, is the name profile of the original post creator
date        -    timestamp in datetime.date() format
text        -    full text of the post, if empty it's a pic or a video
reactions   -    total number of reactions
likes       -    number of likes 
ahah        -    number of ahah
love        -    number of love
wow         -    number of wow
sigh        -    number if sigh
grrr        -    number of grrr
comments    -    number of comments
url         -    relative link to the post

Notice that this file is also used to modify the fields that we want to change before deciding what to do with the items. To accomplish these kinds of tasks, scrapy provides a series of built-in "processors" (such as the input_processor) and functions (such as TakeFirst()) that we can use to adjust the fields we want. These are explained in the official Item Loaders section of the documentation.

Also Refer to Scrapy's Item documentation for more info.

Settings (settings.py)

Scrapy is a very powerful framework and it allows complex tweaking to be put in place. In this project we changed just only a handful of settings, but keep in mind that there are a lot of them. To make the crawler synchronous and get all the items one-by-one so that they are chronologically ordered in the final CSV you can set CONCURRENT_REQUESTS = 1 in settings.py.

Pipelines are useful methods to manipulate items as you can see from the official guide. In our project I have prepared a pipeline to drop all the posts that were made before a certain date, you can check out the code in pipelines.py. Pipelines are not initialized by default, they need to be declared here. Since we can define more than one of them a number in the 0-1000 range is used to indicate priority (lower is first). This is why we have set:

ITEM_PIPELINES = {
    'fbcrawl.pipelines.FbcrawlPipeline': 300,
}

Besides dropping our items according to timestamp we can also export it locally to a CSV or a JSON. In case we choose to create a CSV file we need to specify the order of the columns by explicitly setting:

FEED_EXPORT_FIELDS = ["source", "date", "text", "reactions","likes","ahah","love","wow","sigh","grrr","comments","url"]

Scrapy's default behavior is to follow robots.txt guidelines, so we need to disable this by setting ROBOTSTXT_OBEY = False.

How to use

Make sure that scrapy is installed, and clone this repository. Navigate through the project's top level directory and launch scrapy with:

scrapy crawl fb -a email="EMAILTOLOGIN" -a password="PASSWORDTOLOGIN" -a page="N