• jsonformatter -- for python log json
  • Installation
  • Contributing
  • Basic Usage
    • Case 1. Initial root logger like logging.basicConfig
    • Case 2. Complete config in python code
    • Case 3. Use config file
    • Case 4. In Flask project, add LogRecord attribute for auto output
    • Case 5. In Django project, config LOGGING
  • More Usage
    • Case 1. Mix extra to output
    • Case 2. Output multiple attributes in one key
    • Case 3. Support json.dumps all optional parameters
    • Case 4. Solve cumtom LogRecord attribute is not JSON serializable
  • LogRecord Attributes

jsonformatter -- for python log json

jsonformatter is a formatter for python output json log, e.g. output LogStash needed log.

Easily custom(add/replace) LogRecord attribute, e.g. in Flask web project, add username attribute to LogRecord for auto output username.

Python 2.7 and python 3 are supported from version 0.2.X, if you are using a version lower than 0.2.X, Only python 3 is supported.

Installation

jsonformatter is available on PyPI. Use pip to install:

$ pip install jsonformatter

or:

$ git clone https://github.com/MyColorfulDays/jsonformatter.git
$ cd jsonformatter
$ python setup.py install

or python >= 3.7:

$ git clone https://github.com/MyColorfulDays/jsonformatter.git
$ cd jsonformatter
$ pip install .

Contributing

Download source code

$ git clone https://github.com/MyColorfulDays/jsonformatter.git
$ cd jsonformatter
$ pip install -e .

Run tests

$ python -m unittest tests/test.py
$ python -m unittest tests/test_windows.py

Build

$ pip install build
$ python -m build

Basic Usage

Case 1. Initial root logger like logging.basicConfig

import logging

from jsonformatter import basicConfig

# default keyword parameter `format`: """{"levelname": "levelname", "name": "name", "message": "message"}"""
basicConfig(level=logging.INFO)
logging.info('hello, jsonformatter')

output:

{"levelname": "INFO", "name": "root", "message": "hello, jsonformatter"}

Case 2. Complete config in python code

import logging

from jsonformatter import JsonFormatter

# `format` can be `json`, `OrderedDict`, `dict`.
# If `format` is `dict` and python version < 3.7.0, the output order is sorted keys, otherwise will same as defined order.
# key: string, can be whatever you like.
# value: `LogRecord` attribute name.
STRING_FORMAT = '''{
    "Name":            "name",
    "Levelno":         "levelno",
    "Levelname":       "levelname",
    "Pathname":        "pathname",
    "Filename":        "filename",
    "Module":          "module",
    "Lineno":          "lineno",
    "FuncName":        "funcName",
    "Created":         "created",
    "Asctime":         "asctime",
    "Msecs":           "msecs",
    "RelativeCreated": "relativeCreated",
    "Thread":          "thread",
    "ThreadName":      "threadName",
    "Process":         "process",
    "Message":         "message"
}'''

root = logging.getLogger()
root.setLevel(logging.INFO)

formatter = JsonFormatter(STRING_FORMAT)

sh = logging.StreamHandler()
sh.setFormatter(formatter)
sh.setLevel(logging.INFO)

root.addHandler(sh)

root.info("test %s format", 'string')

output:

{"Name": "root", "Levelno": 20, "Levelname": "INFO", "Pathname": "test.py", "Filename": "test.py", "Module": "test", "Lineno": 75, "FuncName": "test_string_format", "Created": 1588185267.3198836, "Asctime": "2020-04-30 02:34:27,319", "Msecs": 319.8835849761963, "RelativeCreated": 88.2880687713623, "Thread": 16468, "ThreadName": "MainThread", "Process": 16828, "Message": "test string format"}

Case 3. Use config file

config file:

$ cat logger_config.ini
[loggers]
keys=root

[logger_root]
level=INFO
handlers=infohandler


###############################################

[handlers]
keys=infohandler

[handler_infohandler]
class=StreamHandler
level=INFO
formatter=form01
args=(sys.stdout,)

###############################################

[formatters]
keys=form01

[formatter_form01]
class=jsonformatter.JsonFormatter
format={"name": "name","levelno": "levelno","levelname": "levelname","pathname": "pathname","filename": "filename","module": "module","lineno": "lineno","funcName": "funcName","created": "created","asctime": "asctime","msecs": "msecs","relativeCreated": "relativeCreated","thread": "thread","threadName": "threadName","process": "process","message": "message"}

python code:

import logging
import os
from logging.config import fileConfig

fileConfig(os.path.join(os.path.dirname(__file__), 'logger_config.ini'))
root = logging.getLogger('root')
root.info('test file config')

output:

{"name": "root", "levelno": 20, "levelname": "INFO", "pathname": "test.py", "filename": "test.py", "module": "test", "lineno": 315, "funcName": "test_file_config", "created": 1588185267.3020294, "asctime": "2020-04-30 02:34:27", "msecs": 302.0293712615967, "relativeCreated": 70.4338550567627, "thread": 16468, "threadName": "MainThread", "process": 16828, "message": "test file config"}

Case 4. In Flask project, add LogRecord attribute for auto output

flask_demo.py

import datetime
import json
import logging
import random
from collections import OrderedDict

from jsonformatter import JsonFormatter
from flask import Flask, has_request_context, request, session
from flask.logging import default_handler

app = Flask(__name__)

# the key will add/replace `LogRecord` attribute.
# the value must be `callable` type and not support positional paramters, the returned value will be as the `LogRecord` attribute value.
RECORD_CUSTOM_ATTRS = {
    # no parameters
    'url': lambda: request.url if has_request_context() else None,
    'username': lambda: session['username'] if has_request_context() and ('username' in session) else None,
    # Arbitrary keywords parameters
    'status': lambda **record_attrs: 'failed' if record_attrs['levelname'] in ['ERROR', 'CRITICAL'] else 'success'
}

RECORD_CUSTOM_FORMAT = OrderedDict([
    # custom record attributes start
    ("Url", "url"),
    ("Username", "username"),
    ("Status", "status"),
    # custom record attributes end
    ("Name", "name"),
    ("Levelno", "levelno"),
    ("Levelname", "levelname"),
    ("Pathname", "pathname"),
    ("Filename", "filename"),
    ("Module", "module"),
    ("Lineno", "lineno"),
    ("FuncName", "funcName"),
    ("Created", "created"),
    ("Asctime", "asctime"),
    ("Msecs", "msecs"),
    ("RelativeCreated", "relativeCreated"),
    ("Thread", "thread"),
    ("ThreadName", "threadName"),
    ("Process", "process"),
    ("Message", "message")
])


formatter = JsonFormatter(
    RECORD_CUSTOM_FORMAT,
    record_custom_attrs=RECORD_CUSTOM_ATTRS
)

default_handler.setFormatter(formatter)
app.logger.warning('hello, jsonformatter')

output:

{"Url": null, "Username": null, "Status": "success", "Name": "flask_demo", "Levelno": 30, "Levelname": "WARNING", "Pathname": "flask_demo.py", "Filename": "flask_demo.py", "Module": "flask_demo", "Lineno": 54, "FuncName": "<module>", "Created": 1595781463.3557186, "Asctime": "2020-07-27 00:37:43,355", "Msecs": 355.71861267089844, "RelativeCreated": 858.7081432342529, "Thread": 15584, "ThreadName": "MainThread", "Process": 17560, "Message": "hello, jsonformatter"}

Case 5. In Django project, config LOGGING

settings.py

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'standard': {
            'class': 'jsonformatter.JsonFormatter',
            'format': OrderedDict([
                ("Name", "name"),
                ("Levelno", "levelno"),
                ("Levelname", "levelname"),
                ("Pathname", "pathname"),
                ("Filename", "filename"),
                ("Module", "module"),
                ("Lineno", "lineno"),
                ("FuncName", "funcName"),
                ("Created", "created"),
                ("Asctime", "asctime"),
                ("Msecs", "msecs"),
                ("RelativeCreated", "relativeCreated"),
                ("Thread", "thread"),
                ("ThreadName", "threadName"),
                ("Process", "process"),
                ("Message", "message")
            ])
        },
    },
    'handlers': {
        'console': {
            'level': 'INFO',
            'formatter': 'standard',
            'class': 'logging.StreamHandler',
        },
    },
    'loggers': {
        'django': {
            'handlers': ['console'],
            'level': 'INFO',
            'propagate': False
        },
    }
}

More Usage

Case 1. Mix extra to output

import logging

from jsonformatter import JsonFormatter

root = logging.getLogger()
root.setLevel(logging.INFO)

sh = logging.StreamHandler()
formatter = JsonFormatter(
    ensure_ascii=False, 
    mix_extra=True,
    mix_extra_position='tail' # optional: head, mix
)
sh.setFormatter(formatter)
sh.setLevel(logging.INFO)
root.addHandler(sh)

root.info(
    'test mix extra in fmt',
    extra={
        'extra1': 'extra content 1',
        'extra2': 'extra content 2'
    })
root.info(
    'test mix extra in fmt',
    extra={
        'extra3': 'extra content 3',
        'extra4': 'extra content 4'