ckanext-extractor #################

.. image:: https://travis-ci.org/stadt-karlsruhe/ckanext-extractor.svg?branch=master :target: https://travis-ci.org/stadt-karlsruhe/ckanext-extractor

.. image:: https://coveralls.io/repos/github/stadt-karlsruhe/ckanext-extractor/badge.svg?branch=master :target: https://coveralls.io/github/stadt-karlsruhe/ckanext-extractor

A CKAN_ extension for automatically extracting text and metadata from datasets.

ckanext-extractor automatically extracts text and metadata from your resources and adds them to the search index so that they can be used to find your data.

.. _CKAN: https://www.ckan.org

Requirements

ckanext-extractor has been developed and tested with CKAN 2.6 and later. Other versions may or may not work.

Since ckanext-extractor relies on the background job system introduced in CKAN 2.7, users of earlier CKAN versions need to also install ckanext-rq_.

.. _ckanext-rq: https://github.com/ckan/ckanext-rq

Installation

Note: The following steps assume a standard CKAN source installation.

Install Python Package

Activate your CKAN virtualenv::

. /usr/lib/ckan/default/bin/activate

Install the latest development version of ckanext-extractor and its dependencies::

cd /usr/lib/ckan/default
pip install -e git+https://github.com/stadt-karlsruhe/ckanext-extractor#egg=ckanext-extractor
pip install -r src/ckanext-extractor/requirements.txt

On a production system you'll probably want to pin a certain release version_ of ckanext-extractor instead::

pip install -e git+https://github.com/stadt-karlsruhe/ckanext-extractor@v0.4.0#egg=ckanext-extractor

.. _release version: https://github.com/stadt-karlsruhe/ckanext-extractor/releases

Configure CKAN

Open your CKAN configuration file (e.g. /etc/ckan/default/production.ini) and add extractor to the list of plugins::

ckan.plugins = ... extractor

Initialize the database::

paster --plugin=ckanext-extractor init -c /etc/ckan/default/production.ini

Start Background Worker

ckanext-extractor uses background jobs to perform the extraction asynchronously so that they do not block the web server. You therefore need to make sure that a CKAN background worker is running::

paster --plugin=ckan jobs worker --config=/etc/ckan/default/production.ini

See the CKAN documentation_ for more information on background jobs and for tips on how to run workers in production environments.

.. _CKAN documentation: http://docs.ckan.org/en/latest/maintaining/background-tasks.html

Configure Solr

For the actual extraction CKAN's Apache Solr server is used. However, the necessary Solr plugins are deactivated by default. To enable them, find your main Solr configuration file (usually /etc/solr/conf/solrconfig.xml) and add/uncomment the following lines::

<lib dir="../../dist/" regex="apache-solr-cell-\d.*\.jar" />
<lib dir="../../contrib/extraction/lib" regex=".*\.jar" />

Note: The Solr packages on Ubuntu are broken_ and do not contain the necessary files. You can simply download an official release_ of the same version, unpack it to a suitable location (without installing it) and adjust the dir arguments in the configuration lines above accordingly. For example, if you have unpacked the files to /var/lib/apache-solr, then you would need to put the following lines into solrconfig.xml::

<lib dir="/var/lib/apache-solr/dist/" regex="apache-solr-cell-\d.*\.jar" />
<lib dir="/var/lib/apache-solr/contrib/extraction/lib" regex=".*\.jar" />

.. _broken: https://bugs.launchpad.net/ubuntu/+source/lucene-solr/+bug/1565637 .. _official release: http://archive.apache.org/dist/lucene/solr

Once the text and metadata have been extracted they need to be added to the Solr index, which requires appropriate Solr fields. To set them up add the following lines to your Solr schema configuration (usually /etc/solr/conf/schema.xml)::

# Directly before the line that says "</fields>"
<dynamicField name="ckanext-extractor_*" type="text" indexed="true" stored="false"/>

# Directly before the line that says "</schema>"
<copyField source="ckanext-extractor_*" dest="text"/>

Make sure to restart Solr after you have applied the changes. For example, if you're using Jetty as an application server for Solr, then

::

sudo service jetty restart

Restart CKAN

Finally, restart your CKAN server::

sudo service apache2 restart

Test your Installation

The installation is now complete. To verify that everything is working open the URL /api/3/action/extractor_list, e.g. via

::

wget -qO - http://localhost/api/3/action/extractor_list

The output should look like this (in particular, success should be true)::

{"help": "http://localhost/api/3/action/help_show?name=extractor_list", "success": true, "result": []}

You're Done!

Your installation of ckanext-extractor is now complete, and new/updated resources will have their metadata automatically indexed. You may want to adapt the configuration to your needs, see below for details. Once that is done you may also want to extract metadata from your existing resources::

. /usr/lib/ckan/default/bin/activate
paster --plugin=ckanext-extractor extract all -c /etc/ckan/default/production.ini

This and other paster administration commands are explained below in more detail.

Configuration

ckanext-extractor can be configured via the usual CKAN configuration file (e.g. /etc/ckan/default/production.ini). You must restart your CKAN server after updating the configuration.

Formats for Extraction

While Solr can extract text and metadata from many file formats not all of them might be of interest to you. You can therefore configure for which formats extraction is performed via the ckanext.extractor.indexed_formats option. It takes a list of space-separated formats, where the format is the one specified in a resource's CKAN metadata (and not the file extension or MIME type)::

ckanext.extractor.indexed_formats = pdf txt

Formats are case-insensitive. You can use wildcards (* and ?) to match multiple formats. To extract data from all formats simply set

::

ckanext.extractor.indexed_formats = *

By default, extraction is only enabled for the PDF format::

ckanext.extractor.indexed_formats = pdf

Fields for Indexing

Once text and metadata have been extracted they can be added to the search index. Again, Solr supports more metadata fields than one usually needs. You can therefore configure which fields are indexed via the ckanext.extractor.indexed_fields option. It takes a space-separated list of field names::

ckanext.extractor.indexed_fields = fulltext author

The full text of a document is available via the fulltext field. Field names are case-insensitive. You can use wildcards (* and ?) to match multiple field names. To index all fields simply set

::

ckanext.extractor.indexed_fields = *

By default, only the full text of a document is indexed::

ckanext.extractor.indexed_fields = fulltext

Note: ckanext-extractor normalizes the field names reported by Solr by replacing underscores (_) with hyphens (-). In addition, multiple values for the same field in the same document are collapsed into a single value.

Paster Commands

In general, ckanext-extractor works automatically: whenever a new resource is created or an existing resource changes, its metadata is extracted and indexed. However, for administration purposes, metadata can also be managed from the command line using the paster_ tool.

.. _paster: http://docs.ckan.org/en/latest/maintaining/paster.html

Note: You have to activate your virtualenv before you can use these commands::

. /usr/lib/ckan/default/bin/activate

The general form for a paster command is

::

paster --plugin=ckanext-extractor COMMAND ARGUMENTS --config=/etc/ckan/default/production.ini

Replace COMMAND and ARGUMENTS as described below. For example::

paster --plugin=ckanext-extractor extract all --config=/etc/ckan/default/production.ini

• delete (all | ID [ID [...]]): Delete metadata. You can specify one or more resource IDs or a single all argument (in which case all metadata is deleted).

• extract [--force] (all | ID [ID [...]]): Extract metadata. You can specify one or more resource IDs or a single all argument (in which case metadata is extracted from all resources with appropriate formats). An optional --force argument can be used to force extraction even if the resource is unchanged, or if another extraction job already has been scheduled for that resource.

  Note that this command only schedules the necessary extraction background tasks. A background jobs worker has to be running for the extraction to actually happen.

• init: Initialize the database tables for ckanext-extractor. You only need to use this once (during the installation).

• list: List the IDs of all resources for which metadata has been extracted.

• show (all | ID [ID [...]]): Show extracted metadata. You can specify one or more resource IDs or a single all argument (in which case all metadata is shown).

API

Metadata can be managed via the standard CKAN API_. Unless noted otherwise all commands are only available via POST requests to authenticated users.

.. _CKAN API: http://docs.ckan.org/en/latest/api/index.html

extractor_delete

Delete metadata.

Only available to administrators.

Parameters:

:id: ID of the resource for which metadata should be deleted.

extractor_extract

Extract metadata.

This function schedules a background task for extract