Geoip2Influx

---

A python script that will parse the nginx access.log and send geolocation metrics and log metrics to InfluxDB

For the linuxserver/letsencrypt docker mod, click here : https://github.com/linuxserver/docker-mods/tree/swag-geoip2influx

---

Usage

Enviroment variables:

These are the default values for all envs. Add the ones that differ on your system.

| Environment Variable | Example Value | Description | | -------------------- | ------------- | ----------- | | NGINX_LOG_PATH | /config/log/nginx/access.log | Container path for Nginx logfile , defaults to the example. | | GEO_MEASUREMENT | geoip2influx | InfluxDB measurement name for geohashes. Optional, defaults to the example. | | LOG_MEASUREMENT | nginx_access_logs | InfluxDB measurement name for nginx logs. Optional, defaults to the example. | | SEND_NGINX_LOGS | true | Set to false to disable nginx logs. Optional, defaults to true. | | GEOIP2INFLUX_LOG_LEVEL | info | Sets the log level in geoip2influx.log. Use debug for verbose logging Optional, defaults to info. | | GEOIP2INFLUX_LOG_PATH | /config/log/geoip2influx/geoip2influx.log | Optional. Defaults to example. | | GEOIP_DB_PATH | /config/geoip2db/GeoLite2-City.mmdb | Optional. Defaults to example. | | MAXMINDDB_LICENSE_KEY | xxxxxxx | Add your Maxmind licence key | | MAXMINDDB_USER_ID | xxxxxxx| Add your Maxmind account id |

InfluxDB v1.8.x values

| Environment Variable | Example Value | Description | | -------------------- | ------------- | ----------- | | INFLUX_HOST | localhost | Host running InfluxDB. | | INFLUX_HOST_PORT | 8086 | Optional, defaults to 8086. | | INFLUX_DATABASE | geoip2influx | Optional, defaults to geoip2influx. | | INFLUX_USER | root | Optional, defaults to root. | | INFLUX_PASS | root | Optional, defaults to root. | | INFLUX_RETENTION | 7d | Sets the retention for the database. Optional, defaults to example.| | INFLUX_SHARD | 1d | Set the shard for the database. Optional, defaults to example. |

InfluxDB v2.x values

| Environment Variable | Example Value | Description | | -------------------- | ------------- | ----------- | | USE_INFLUXDB_V2 | true | Required if using InfluxDB2. Defaults to false | | INFLUXDB_V2_TOKEN | secret-token | Required | | INFLUXDB_V2_URL | http://localhost:8086 | Optional, defaults to http://localhost:8086 | | INFLUXDB_V2_ORG | geoip2influx | Optional, defaults to geoip2influx. Will be created if not exists. | | INFLUXDB_V2_BUCKET | geoip2influx | Optional, defaults to geoip2influx. Will be created if not exists. | | INFLUXDB_V2_RETENTION | 604800 | Optional, defaults to 604800. 7 days in seconds | | INFLUXDB_V2_DEBUG | false | Optional, defaults to false. Enables the debug mode for the influxdb-client package. | | INFLUXDB_V2_BATCHING | true | Optional, defaults to false. Enables batch writing of data. | | INFLUXDB_V2_BATCH_SIZE | 100 | Optional, defaults to 10. | | INFLUXDB_V2_FLUSH_INTERVAL | 30000 | Optional, defaults to 15000. How often in milliseconds to write a batch |

INFLUXDB_V2_TOKEN

If the organization or bucket does not exist, it will try and create them with the token.

[!NOTE] The minimim level of rights needed is write access to the bucket.

MaxMind Geolite2

Default download location is /config/geoip2db/GeoLite2-City.mmdb

Get your licence key here: https://www.maxmind.com/en/geolite2/signup

InfluxDB

InfluxDB v2.x and v1.8x is supported.

Note: The Grafana dashboard currently only supports InfluxDB v1.8.x

The InfluxDB database/bucket and retention rules will be created automatically with the name you choose.

-e INFLUX_DATABASE=geoip2influx or -e INFLUXDB_V2_BUCKET=geoip2influx

Docker

docker create \
  --name=geoip2influx \
  -e PUID=1000 \
  -e PGID=1000 \
  -e TZ=Europe/Oslo \
  -e INFLUX_HOST=<influxdb host> \
  -e INFLUX_HOST_PORT=<influxdb port> \
  -e MAXMINDDB_LICENSE_KEY=<license key>\
  -e MAXMINDDB_USER_ID=<account id>\
  -v /path/to/appdata/geoip2influx:/config \
  -v /path/to/nginx/accesslog/:/config/log/nginx/ \
  --restart unless-stopped \
  ghcr.io/gilbn/geoip2influx

Docker compose

version: "2.1"
services:
  geoip2influx:
    image: ghcr.io/gilbn/geoip2influx
    container_name: geoip2influx
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Europe/Oslo
      - INFLUX_HOST=<influxdb host>
      - INFLUX_HOST_PORT=<influxdb port>
      - MAXMINDDB_LICENSE_KEY=<license key>
      - MAXMINDDB_USER_ID=<account id>
    volumes:
      - /path/to/appdata/geoip2influx:/config
      - /path/to/nginx/accesslog/:/config/log/nginx/
    restart: unless-stopped

InfluxDB2 examples

docker create \
  --name=geoip2influx \
  -e PUID=1000 \
  -e PGID=1000 \
  -e TZ=Europe/Oslo \
  -e INFLUXDB_V2_URL=<influxdb url> \
  -e INFLUXDB_V2_TOKEN=<influxdb token> \
  -e USE_INFLUXDB_V2=true \
  -e MAXMINDDB_LICENSE_KEY=<license key>\
  -e MAXMINDDB_USER_ID=<account id>\
  -v /path/to/appdata/geoip2influx:/config \
  -v /path/to/nginx/accesslog/:/config/log/nginx/ \
  --restart unless-stopped \
  ghcr.io/gilbn/geoip2influx

version: "2.1"
services:
  geoip2influx:
    image: ghcr.io/gilbn/geoip2influx
    container_name: geoip2influx
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Europe/Oslo
      - INFLUXDB_V2_URL=<influxdb url>
      - INFLUXDB_V2_TOKEN=<influxdb token>
      - USE_INFLUXDB_V2=true
      - MAXMINDDB_LICENSE_KEY=<license key>
      - MAXMINDDB_USER_ID=<account id>
    volumes:
      - /path/to/appdata/geoip2influx:/config
      - /path/to/nginx/accesslog/:/config/log/nginx/
    restart: unless-stopped

---

Grafana dashboard:

Use nginx_logs_geo_map.json

Note

Currently only supports InfluxDB 1.8.x.

---

Sending Nginx log metrics

Nginx needs to be compiled with the geoip2 module: https://github.com/leev/ngx_http_geoip2_module

1. Add the following to the http block in your nginx.conf file:

geoip2 /config/geoip2db/GeoLite2-City.mmdb {
auto_reload 5m;
$geoip2_data_country_iso_code country iso_code;
$geoip2_data_city_name city names en;
}

log_format custom '$remote_addr - $remote_user [$time_local]'
           '"$request" $status $body_bytes_sent'
           '"$http_referer" $host "$http_user_agent"'
           '"$request_time" "$upstream_connect_time"'
           '"$geoip2_data_city_name" "$geoip2_data_country_iso_code"';

2. Set the access log use the custom log format.

access_log /config/log/nginx/access.log custom;

Multiple log files

If you separate your nginx log files but want this script to parse all of them you can do the following:

As nginx can have multiple access log directives in a block, just add another one in the server block.

Example

	access_log /config/log/nginx/technicalramblings/access.log custom;
	access_log /config/log/nginx/access.log custom;

This will log the same lines to both files.

Then use the /config/log/nginx/access.log file in the NGINX_LOG_PATH variable.

---

Updates

18.08.24 - Rename env from USE_INFLUX_V2 to USE_INFLUXDB_V2.

10.08.24 - Add support for InfluxDB2.

06.08.24 - Complete refactor of the python code. Deprecate the old geoip2influx.py file.

28.07.24 - Refactor to alpine 3.20. New env required. MAXMINDDB_USER_ID.

21.06.20 - Added $host(domain) to the nginx log metrics. This will break your nginx logs parsing, as you need to update the custom log format.

06.06.20 - Added influx retention policy to try and mitigate max-values-per-tag limit exceeded errors.

• -e INFLUX_RETENTION Default 30d
• -e INFLUX_SHARD Default 2d
• It will only add the retention policy if the database doesn't exist.

30.05.20 - Added logging. Use -e GEOIP2INFLUX_LOG_LEVEL to set the log level.

15.05.20 - Removed GEOIP2_KEY and GEOIP_DB_PATHvariables. With commit https://github.com/linuxserver/docker-letsencrypt/commit/75b9685fdb3ec6edda590300f289b0e75dd9efd0 the letsencrypt container now natively supports downloading and updating(weekly) the GeoLite2-City database!

---

Adapted source: https://github.com/ratibor78/geostat