JData - NeuroJSON client with fast parsers for JSON, binary JSON, NIFTI, GIFTI, SNIRF, CSV/TSV, HDF5 data files

Copyright: (C) Qianqian Fang (2019-2026) <q.fang at neu.edu>
License: Apache License, Version 2.0
Version: 0.9.3
URL: https://github.com/NeuroJSON/pyjdata
Acknowledgement: This project is supported by US National Institute of Health (NIH) grant U24-NS124027

Introduction
File formats
Submodules
How to install
How to build
How to use
Advanced interfaces
Reading JSON via REST-API
Using JSONPath to access and query complex datasets
Downloading and caching _DataLink_ referenced external data files
Utility
How to contribute
Test

Introduction

jdata is a lightweight and fast neuroimaging data file parser, with built in support for NIfTI-1/2 (.nii, .nii.gz), two-part Analyze 7.5 (.img/.hdr, .img.gz), HDF5 (.h5), SNIRF (.snirf), MATLAB .mat files (.mat), CSV/TSV (.csv, .csv.gz, .tsv, .tsv.gz), JSON (.json), and various binary-JSON data formats, including BJData (.bjd), UBJSON (.ubj), and MessagePack (.msgpack) formats. jdata can load data files both from local storage and REST-API via URLs. To maximize portability, the outputs of jdata data parsers are intentionally based upon only the native Python data structures (dict/list/tuple) plus numpy arrays. The entire package is less than 60KB in size and is platform-independent.

jdata highly compatible to the JSONLab toolbox for MATLAB/Octave, serving as the reference library for Python for the JData Specification, The JData Specification defines a lightweight language-independent data annotation interface enabling easy storing and sharing of complex data structures across different programming languages such as MATLAB, JavaScript, Python etc. Using JData formats, a complex Python data structure, including numpy objects, can be encoded as a simple dict object that is easily serialized as a JSON/binary JSON file and share such data between programs of different languages.

Since 2021, the development of the jdata module and the underlying data format specificaitons JData and BJData have been funded by the US National Institute of Health (NIH) as part of the NeuroJSON project (https://neurojson.org and https://neurojson.io).

The goal of the NeuroJSON project is to develop scalable, searchable, and reusable neuroimaging data formats and data sharing platforms. All data produced from the NeuroJSON project will be using JSON/Binary JData formats as the underlying serialization standards and the lightweight JData specification as language-independent data annotation standard.

File formats

The supported data formats can be found in the below table. All file types support reading and writing, except those specified below.

| Format | Name | | Format | Name | | ------ | ------ | --- |-----------------------------------| ------ | | JSON-compatible files | | | Binary JSON (same format) [1] | | | ✅ .json | ✅ JSON files | | ✅ .bjd | ✅ binary JSON (BJD) files | | ✅ .jnii | ✅ JSON-wrapper for NIfTI data (JNIfTI)| | ✅ .bnii | ✅ BJD-wrapper for NIfTI data | | ✅ .jnirs | ✅ JSON-wrapper for SNIRF data (JSNIRF)| | ✅ .bnirs | ✅ BJD-wrapper for SNIRF data | | ✅ .jmsh | ✅ JSON-encoded mesh data (JMesh) | | ✅ .bmsh | ✅ BJD-encoded for mesh data | | ✅ .jdt | ✅ JSON files with JData annotations | | ✅ .jdb | ✅ BJD files with JData annotations | | ✅ .jdat | ✅ JSON files with JData annotations | | ✅ .jbat | ✅ BJD files with JData annotations | | ✅ .jbids | ✅ JSON digest of a BIDS dataset | | ✅ .pmat | ✅ BJD encoded .mat files | | NIfTI formats | | | CSV/TSV formats | | | ✅ .nii | ✅ uncompressed NIfTI-1/2 files | | ✅ .csv | ✅ CSV files | | ✅ .nii.gz | ✅ compressed NIfTI files | | ✅ .csv.gz | ✅ compressed CSV files | | ✅ .img/.hdr | ✅ Analyze 7.5 two-part files | | ✅ .tsv | ✅ TSV files | | ✅ .img.gz | ✅ compressed Analyze files | | ✅ .tsv.gz | ✅ compressed TSV files | | HDF5 formats [2] | | | Other formats (read-only) | | | ✅ .h5 | ✅ HDF5 files | | ✅ .mat | ✅ MATLAB .mat files [3] | | ✅ .hdf5 | ✅ HDF5 files | | ✅ .bval | ✅ EEG .bval files | | ✅ .snirf | ✅ HDF5-based SNIRF data | | ✅ .bvec | ✅ EEG .bvec files | | ✅ .nwb | ✅ HDF5-based NWB files | | ✅ .msgpack| ✅ Binary JSON MessagePack format [4] |

[1] requires bjdata Python module when needed, pip install bjdata
[2] requires h5py Python module when needed, pip install h5py
[3] requires scipy Python module when needed, pip install scipy
[4] requires msgpack Python module when needed, pip install msgpack

Submodules

The jdata module further partition the functions into smaller submodules, including

jdata.jfile provides loadjd, savejd, load, save, loadt, savet, loadb, saveb, loadts, loadbs, jsoncache, jdlink, ...
jdata.jdata provides encode, decode, jdataencode, jdatadecode, {zlib,gzip,lzma,lz4,base64}encode, {zlib,gzip,lzma,lz4,base64}decode
jdata.jpath provides jsonpath
jdata.jnifti provides load{jnifti,nifti}, save{jnifti,nifti,jnii,bnii}, nii2jnii, jnii2nii, nifticreate, jnifticreate, niiformat, niicodemap
jdata.jgifti provides load{jgifti,gifti}, save{jgifti,gifti}, gii2jgii, jgii2gii, jgifticreate, giicodemap, JGifti
jdata.neurojson provides neuroj, neurojgui
jdata.njprep provides dataset2json
jdata.h5 provides loadh5, saveh5, regrouph5, aos2soa, soa2aos, jsnirfcreate, snirfcreate, snirfdecode

All these functions can be found in the MATLAB/GNU Octave equivalent, JSONLab toolbox. Each function can be individually imported

# individually imported
from jdata.jfile import loadjd
data=loadjd(...)

# import everything
from jdata import *
data=loadjd(...)

# import under jdata namespace
import jdata as jd
data=jd.loadjd(...)

How to install

Github: download from https://github.com/NeuroJSON/pyjdata
PIP: run pip install jdata see https://pypi.org/project/jdata/

This package can also be installed on Ubuntu 21.04 or Debian Bullseye via

sudo apt-get install python3-jdata

On older Ubuntu or Debian releases, you may install jdata via the below PPA:

sudo add-apt-repository ppa:fangq/ppa
sudo apt-get update
sudo apt-get install python3-jdata

Dependencies:

numpy: PIP: run pip install numpy or sudo apt-get install python3-numpy
(optional) bjdata: PIP: run pip install bjdata or sudo apt-get install python3-bjdata, see https://pypi.org/project/bjdata/, only needed to read/write BJData/UBJSON files
(optional) lz4: PIP: run pip install lz4, only needed when encoding/decoding lz4-compressed data
(optional) h5py: PIP: run pip install h5py, only needed when reading/writing .h5 and .snirf files
(optional) scipy: PIP: run pip install scipy, only needed when loading MATLAB .mat files
(optional) msgpack: PIP: run pip install msgpack, only needed when loading MessagePack .msgpack files
(optional) blosc2: PIP: run pip install blosc2, only needed when encoding/decoding blosc2-compressed data
(optional) backports.lzma: PIP: run sudo apt-get install liblzma-dev and pip install backports.lzma (needed for Python 2.7), only needed when encoding/decoding lzma-compressed data
(optional) python3-tk: run sudo apt-get install python3-tk to install the Tk support on a Linux in order to run neurojgui function

Replacing pip by pip3 if you are using Python 3.x. If either pip or pip3 does not exist on your system, please run

sudo apt-get install python3-pip

Please note that in some OS releases (such as Ubuntu 20.04), python2.x and python-pip are no longer supported.

How to build

One can also install this module from the source code. To do this, you first check out a copy of the latest code from Github by

git clone https://github.com/NeuroJSON/pyjdata.git
cd pyjdata

then install the module to your local user folder by

python3 setup.py install --user

or, if you prefer, install to the system folder for all users by

sudo python3 setup.py install

Instead of installing the module, you can also import the jdata module directly from your local copy by cd the root folder of the unzipped pyjdata package, and run

import jdata as jd

How to use

The jdata module provides a unified data parsing and saving interface: jd.loadjd() and jd.savejd(). These two functions supports all file format described in the above "File formats" section. The jd.loadjd() function also supports loading online data via URLs.

import jdata as jd
nii = jd.loadjd('/path/to/img.nii.gz')
snirf = jd.loadjd('/path/to/mydata.snirf')
nii2 = jd.loadjd('https://example.com/data/vol.nii.gz')
jsondata = jd.loadjd('https://example.com/rest/api/')
matlabdata = jd.loadjd('matlabdata.mat')
jd.savejd(matlabdata, 'newdata.mat')

Pyjdata

Install / Use

README