Iaso : a geospatial data management platform

Full documentation is at docs.openiaso.com

Introduction

Iaso is a georegistry and data collection web platform structured around trees of organization units (also known a master lists)

The main tasks it allows accomplishing are:

• Data collection using XLSForm forms linked to org units through a mobile application.
• Import, comparison and merging of multiple Org Units' trees, both through a web application and an API allowing manipulation through data science tools like Jupyter notebooks.
• Validation of received data for Org Units' trees and forms.
• Exports of the Org Units' trees and form data, either in csv, xlsx, GeoPackage or through an api.

Video presentation of IASO at FOSDEM 2021, with slides.

Long Intro

Iaso is a platform created to support geo-rich data collection efforts, mainly in public health in emerging countries. The key feature that it supports is that any survey is linked to an organizational unit that is part of a canonical hierarchy. Each one of these org. units can have a location and a territory. The mobile data collection tool can be used to enrich this hierarchy with additional GPS coordinates, names corrections, etc ... which can then be validated by officials of the organizations in question through the web dashboard. This leads to continuous improvements of the geographic references available through the routine activities already planned. e.g. locating and registering health facilities while investigating malaria cases.

The tool has been used in multiple data collection efforts, notably in the domain of Performance Based Financing of health services in D.R. Congo, Niger, Cameroon and Nigeria and is more and more used to compare multiple versions of official organisational hierarchies when a canonical one needs to be rebuilt (for example to rebuild a school map for DRC). To help for this type of project, we provide location selection interfaces, multiple levels of audits and an API open to data scientists for analysis and mass edits.

Iaso has been created by the company Bluesquare, specialised in software and services for public health, and has become open source under the MIT License in November 2020.

Iaso is made of a white labeled Android application using Java/Kotlin, reusing large parts of the ODK projects, and a web platform programmed using Python/GeoDjango on top of PostGIS. Frontend is mainly React/Leaflet. One of the aims is the ease of integration with other platforms. We already have csv and geopackage imports and exports and target easy integration with OSM.

Structure

This repository contains Iaso frontend and backend, respectively in Python Django and JS React. They interact via an API implemented via Django rest framework, all data is stored in Postgresql or the media/ directory.

A companion mobile app for Android allow submitting Form and creating org unit.

Form can also be filled in a web interface via the Enketo companion service. Both Iaso and Enketo need to be configured to work together. It is possible to run an Enketo service locally, see Enketo section below.

More documentation on the Front End part is present in hat/assets/README.rst

Data Model / Glossary

IASO inspired its terminology both from ODK and from DHIS2. We will highlight some equivalences that might help you.

This is not (yet) the complete Data Model, but here are the main concepts/model in Iaso:

• Iaso is multi tenant. Tenant are called and represented by the model Account. It represents roughly one client org or country. It also represents the natural limit of right for a user.
• Each Django's User has a linked Iaso Profile that link it to an Account and store extra parameters for the user.
• Each tenant can have multiple Project. Projects are linked to one android version App via the app_id. We use the link to control what a user can see from that app.
• DHIS2 is a standard server application and web UI in the industry to handle Health Data. Iaso can import and export data (forms and org unit) to it.
• OrgUnit (Organizational Unit) is a Node of the GeoRegistry tree. e.g a particular Country, City or Hospital. each belonging to each other via a parent relationship.
  • They can have a type OrgUnitType e.g. Country, City, Hospital
  • they can belong to multiple Group, e.g. Urban Region or Campaign 2017
  • DHIS2 has the concept of Group but not Type so when importing from a DHIS2 Instance all the type will be Unknown and OrgUnit will belong to group like Clinic
  • GroupSet are Group of group. Used when we export Group to DHIS2
  • OrgUnit may have a position in space, it can be an area, the geom field is then used, or just a Point, the location field is then used.
    • It's technically possible to have both
    • a OrgUnit may have no geographical info
    • a OrgUnit may geographically be outside its parent.
• DataSource links OrgUnit and Group imported from the same source, e.g a DHIS2 instance, a CSV or a GeoPackage.
  • A source_ref on the imported instance is used to keep the reference from the original source, so we can match it again in the future (when updating the import or exporting it back)
  • SourceVersion is used to keep each version separated. e.g. each time we import from DHIS2 we create a new version.
  • OrgUnit (for their parent) and Group should only reference other OrgUnit and Group in the same version. (This is not enforced everywhere yet)
• Task are asynchronous function that will be run by a background worker in production. eg: Importing Data from DHIS2. see Worker section below for more info.
• Form is the definition of a Form (list of question and their presentation).
  • The model contain the metadata, the actual definition is done in a XSLForm as an attached file.
  • Form are linked to one or more Project. This is used to know which Form are presented in the Mobile App.
  • Form can have multiple versions
• Instance or Form instance is the Submission of a form. A form that has actually been filed by a user.
  • Instance can be GeoTagged and/or linked to a OrgUnit
  • Note: We are moving to use Submission everywhere in the UI, but it is still in progress. please submit PR.
  • Submission cannot be done via the Iaso UI itself but through Enketo or the Mobile App.
• EntityType represents a type of person or object to which we want to attach multiple submissions to track said submissions in time and across OrgUnits.
• Entity represents an actual person or object, defined by its EntityType. A concrete example is given in the docstrings of iaos.models.entity
• APIImport are used to log some request from the mobile app, so we can replay them in case of error. See vector_control Readme
• audit.Modification are used to keep a history of modification on some models (mainly orgunit). See audit readme
• Link are used to match two OrgUnit (in different sources or not) that should be the same in the real world. Links have a confidence score indicating how much we trust that the two OrgUnit are actually the same.

They are usually generated via AlgorithmRun, or the matching is done in a Notebook and uploaded via the API.

Development environment

Setup

A running local instance for development can be spin up via docker compose which will install and configure all dep in separate container. As such your computer should only need:

• git
• docker
• docker compose

If docker compose give you trouble, make sure it can connect to the docker daemon.

If you use an Apple Silicon Mac, ensure export DOCKER_DEFAULT_PLATFORM=linux/amd64 is set.

A pgdata-iaso folder, containing the database data, will be created in the parent directory of the git repository.

1. Environment variables

The docker-compose.yml file contains sensible defaults for the Django application.

Other environment variables can be provided by a .env file.

As a starting point, you can copy the sample .env.dist file and edit it to your needs.

cp .env.dist .env

note

all the commands here need to be run in the project directory, where you cloned the repository

2. Build the containers

This will build and download the containers.

docker compose build

3. Start the database

docker compose up db

(if you get this message: "Database is uninitialized and superuser password is not specified. You must specify POSTGRES_PASSWORD" you can set POSTGRES_PASSWORD=postgres in the .env file )

4. Run migrations

In a separate bash (without closing yet the started db), launch the migrations

docker compose run --rm iaso manage migrate

(If you get a message saying that the database iaso does not exist, you can connect to your postgres instance using

psql -h localhost -p 5433 -U postgres

then type

create database iaso;

to create the missing database.)

(If you get a message saying that /opt/app/entrypoint.sh does not exist, you need to disable