jtc - cli tool to extract, manipulate and transform source JSON

jtc stand for: JSON transformational chains (used to be JSON test console).

jtc offers a powerful way to select one or multiple elements from a source JSON and apply various actions on the selected elements at once (wrap selected elements into a new JSON, filter in/out, sort elements, update elements, insert new elements, remove, copy, move, compare, transform, swap around and many other operations).

Enhancement requests and/or questions are more than welcome: ldn.softdev@gmail.com

Content:

1. Short description

2. Compilation and installation options

   • Linux and MacOS precompiled binaries
   • Installing via MacPorts
   • Installation on Linux distributions
   • Manual installation
   • Release Notes

3. Quick-start guide

   • list all URLs
   • dump all bookmark names
   • dump all URL's names
   • dump all the URLs and corresponding names
   • Debugging and validating JSON

4. Complete User Guide

5. C++ class and interface usage primer

6. jtc vs jq

   • utility ideology
   • learning curve
   • handling irregular JSONs
   • solutions input invariance
   • programming model
   • JSON numerical fidelity
   • performance
   • compare jtc based solutions with jq's

Short description

- jtc is a simple yet very powerful and efficient cli utility tool to process and manipulate JSON data

jtc offers following features (a short list of main features):

• simple user interface allowing applying a bulk of changes in a single or chained sets of commands
• featured walk-path interface lets extracting any combination of data from sourced JSON trees
• extracted data is representable either as found, or could be encapsulated in JSON array/object or transformed using templates
• support Regular Expressions when searching source JSON
• fast and efficient processing of very large JSON files (various built-in search caches)
• insert/update operations optionally may undergo shell cli evaluation
• support in-place modifications of the input/source JSON file
• features namespaces, facilitating interpolation of preserved JSON values in templates
• supports buffered and streamed modes of input read
• sports concurrent input JSON reading/parsing (on multi-core CPU)
• written entirely in C++14, no dependencies (STL only, idiomatic C++, no memory leaks)
• extensively debuggable
• conforms JSON specification (json.org)

The walk-path feature is easy to understand - it's only made of 2 kinds of lexemes traversing JSON tree, which could be mixed up in any order:

• subscripts - enclosed into [, ], subscripts let traversing JSON tree downwards (towards the leaves) and upwards (towards the root)
• search lexemes - encased as <..> or >..< (for a recursive and non-recursive searches respectively); search lexemes facilitate various match criteria defined by an optional suffix and/or quantifier

There's also a 3rd kind of lexemes - directives: they typically facilitate other functions like working with namespaces, controlling walk-path execution, etc; directives are syntactically similar to the search lexemes

All lexemes can be iterable:

• iterable subscripts let iterating over children of currently addressed JSON iterables nodes (arrays/objects),
• while iterable search lexemes let iterating over all (recursive) matches for a given search criteria

A walk-path may have an arbitrary number of lexemes -the tool accepts a virtually unlimited number of walk paths. See below more detailed explanation with examples

Compilation and installation options

For compiling, c++14 (or later) is required. To compile under different platforms:

• MacOS/BSD:
  • c++ -o jtc -Wall -std=c++14 -Ofast jtc.cpp
• Linux:
  • non-relocatable (dynamically linked) image:
    • c++ -o jtc -Wall -std=gnu++14 -Ofast -pthread -lpthread jtc.cpp
  • relocatable (statically linked) image:
    • c++ -o jtc -Wall -std=gnu++14 -Ofast -static -Wl,--whole-archive -lrt -pthread -lpthread -Wl,--no-whole-archive jtc.cpp
• Debian:
  • c++ -o jtc -Wall -std=c++14 -pthread -lpthread -Ofast jtc.cpp (ensure c++ poits to clang++-6.0 or above)

Following debug related flags could be passed to jtc when compiling:

• -DNDEBUG: compile w/o debugs, however it's unadvisable - there's no performance gain from doing so
• -DNDBG_PARSER: disable debugs coming from parsing JSON (handy when deep debugging huge JSONs and want to skip parsing debugs)
• -DBG_FLOW: all debuggable function/method calls will disply an entry and exit points
• -DBG_mTS, -DBG_uTS: display absolute time-stamps in the debug: with millisecond accuracy and with microsecond accuracy respectively
• -DBG_dTS: used with either of 2 previous flags: makes time-stamp to display delta (since last debug message) instead of absolute stamp
• -DBG_CC: every call to a copy-constructor in Jnode class will reveal itself (handy for optimization debugging)

Linux and MacOS precompiled binaries are available for download

Choose the latest precompiled binary:

• latest macOS

  if you don't want to go through macOS security hurdle, then remove the quarantine attribute from the file after binary download, e.g. (assuming you opened terminal in the folder where downloaded binary is):

  bash $ mv ./jtc-macos-64.latest ./jtc
  bash $ chmod 754 ./jtc
  bash $ xattr -r -d com.apple.quarantine ./jtc
  

• latest linux 64 bit
• latest linux 32 bit

Rename the downloaded file and give proper permissions. E.g., for the latest macOS:

mv jtc-macos-64.latest jtc
chmod 754 jtc

Packaged installations:

Installing via MacPorts

On MacOS, you can install jtc via the MacPorts package manager:

$ sudo port selfupdate
$ sudo port install jtc

Installation on Linux distributions

jtc is packaged in the following Linux distributions and can be installed via the package manager.

• Fedora: jtc is present in Fedora 31 and later:

$ dnf install jtc

• openSUSE: jtc can be installed on openSUSE Tumbleweed via zypper:

$ zypper in jtc

or on Leap 15.0 and later by adding the utilities repository and installing jtc via zypper.

Manual installation:

download jtc-master.zip, unzip it, descend into unzipped folder, compile using an appropriate command, move compiled file into an install location.

here're the example steps for MacOS:

• say, jtc-master.zip has been downloaded to a folder and the terminal app is open in that folder:
• unzip jtc-master.zip
• cd jtc-master
• c++ -o jtc -Wall -std=c++17 -Ofast jtc.cpp
• sudo mv ./jtc /usr/local/bin/

Release Notes

See the latest Release Notes

Quick-start guide:

• run the command jtc -g to read the mini USER-GUIDE, with walk path syntax, usage notes, short examples
• read the examples just below
• see stackoverflow-json for lots of worked examples based on Stack Overflow questions
• refer to the complete User Guide for further examples and guidelines.

Consider a following JSON (a mockup of a bookmark container), stored in a file Bookmarks:

{
   "Bookmarks": [
      {
         "children": [
            {
               "children": [
                  { "name": "The New York Times", "stamp": "2017-10-03, 12:05:19", "url": "https://www.nytimes.com/" },
                  { "name": "HuffPost UK", "stamp": "2017-11-23, 12:05:19", "url": "https://www.huffingtonpost.co.uk/" }