YAML Path and Command-Line Tools

Along with providing a standard for defining YAML Paths, this project aims to provide generally-useful command-line tools which implement YAML Paths. These bring intuitive YAML, EYAML, JSON, and compatible data parsing and editing capabilties to the command-line. It is also a Python library for other projects to readily employ YAML Paths.

Contents

1. Introduction
2. Illustration
3. Supported YAML Path Segments
4. Installing
   1. Requirements
   2. Using pip
      1. Very Old Versions of pip or its setuptools Dependency
   3. Installing EYAML (Optional)
5. Based on ruamel.yaml
6. The Files of This Project
   1. Command-Line Tools
   2. Libraries
7. Basic Usage

   1. Basic Usage: Command-Line Tools

      1. Rotate Your EYAML Keys
      2. Get the Differences Between Two Documents
      3. Get a YAML/JSON/Compatible Value
      4. Search For YAML Paths
      5. Change a YAML/JSON/Compatible Value
      6. Merge YAML/JSON/Compatible Files
      7. Validate YAML/JSON/Compatible Documents

   2. Basic Usage: Libraries

      1. Initialize ruamel.yaml and These Helpers
      2. Searching for YAML Nodes
      3. Changing Values
      4. Merging Documents

Introduction

This project presents and utilizes YAML Paths, which are a powerful, intuitive means of identifying one or more nodes within YAML, EYAML, or compatible data structures like JSON. Both dot-notation (inspired by Hiera) and forward-slash-notation (influenced by XPath) are supported. The libraries (modules) and several command-line tool implementations are provided. With these, you can build YAML Path support right into your own application or easily use its capabilities right away from the command-line to retrieve, update, merge, validate, and scan YAML/JSON/Compatible data.

This implementation of YAML Path is a query language in addition to a node descriptor. With it, you can describe or select a single precise node or search for any number of nodes that match some criteria. Keys, values, elements, anchors, and aliases can all be searched at any number of levels within the data structure using the same query. Collectors can also be used to gather and further select from otherwise disparate parts of the source data.

The project Wiki provides a deeper dive into these concepts.

Illustration

To illustrate some of these concepts, consider these samples:

---
hash:
  child_attr:
    key: 5280

This value, 5280, can be identified via YAML Path as any of:

1. hash.child_attr.key (dot-notation)
2. hash.child_attr[.=key] (search all child keys for one named, key, and yield its value)
3. /hash/child_attr/key (same as 1 but in forward-slash notation)
4. /hash/child_attr[.=key] (same as 2 but in forward-slash notation)

---
aliases:
  - &first_anchor Simple string value

With YAML Path, you can select this anchored value by any of these equivalent expressions:

1. aliases[0] (explicit array element number)
2. aliases.0 (implicit array element number in dot-notation)
3. aliases[&first_anchor] (search by Anchor name)
4. aliases[.^Simple] (search for any elements starting with "Simple")
5. aliases[.%string] (search for any elements containing "string")
6. aliases[.$value] (search for any elements ending with "value")
7. aliases[.=~/^(\b[Ss][a-z]+\s){2}[a-z]+$/] (search for any elements matching a complex Python Regular Expression, which happens to match the example)
8. /aliases[0] (same as 1 but in forward-slash notation)
9. /aliases/0 (same as 2 but in forward-slash notation)
10. /aliases[&first_anchor] (same as 3 but in forward-slash notation)

---
users:
  - name: User One
    password: ENC[PKCS7,MIIBiQY...Jk==]
    roles:
      - Writers
  - name: User Two
    password: ENC[PKCS7,MIIBiQY...vF==]
    roles:
      - Power Users
      - Editors

With an example like this, YAML Path enables:

• selection of single nodes: /users/0/roles/0 = Writers
• all children nodes of any given parent: /users/1/roles = ["Power Users", "Editors"]
• searching by a child attribute: /users[name="User One"]/password = Some decrypted value, provided you have the appropriate EYAML keys
• pass-through selections against arrays-of-hashes: /users/roles = ["Writers"]\n["Power Users", "Editors"] (each user's list of roles are a separate result)
• collection of disparate results: (/users/name) = ["User One", "User Two"] (all names appear in a single result instead of one per line)

For a deeper exploration of YAML Path's capabilities, please visit the project Wiki.

Supported YAML Path Segments

A YAML Path segment is the text between separators which identifies zero or more parent or leaf nodes within the data structure. For dot-notation, a path like hash.key identifies two segments: hash (a parent node) and key (a leaf node). The same path in forward-slash notation would be: /hash/key.

YAML Path understands these segment types:

• Top-level Hash key selection: key
• Explicit top-level array element selection: [#] where # is the zero-based element number; # can also be negative, causing the element to be selected from the end of the Array
• Implicit array element selection or numbered hash key selection: # where # is the 0-based element number or exact name of a hash key which is itself a number
• Top-level (Hash) Anchor lookups: &anchor_name (the & is required to indicate you are seeking an Anchor by name)
• Hash sub-keys: hash.child.key or /hash/child/key
• Demarcation for dotted Hash keys: hash.'dotted.child.key' or hash."dotted.child.key" (not necessary when using forward-slash notation, /hash/dotted.child.key)
• Named Array element selection: array[#], array.#, /array[#], or /array/# where array is the name of the Hash key containing Array data and # is the 0-based element number
• Anchor lookups in named Arrays: array[&anchor_name] where array is the name of the Hash key containing Array data and both of the [] pair and & are required to indicate you are seeking an Anchor by name within an Array
• Array slicing: array[start#:stop#] where start# is the first inclusive, zero-based element and stop# is the last exclusive element to select; either or both can be negative, causing the elements to be selected from the end of the Array; when start# and stop# are identical, it is the same as array[start#]
• Hash slicing: hash[min:max] where min and max are alphanumeric terms between which the Hash's keys are compared
• Escape symbol recognition: hash.dotted\.child\.key, /hash/whacked\/child\/key, and keys_with_\\slashes
• Hash attribute searches (which can return zero or more matches):
  • Exact match: hash[name=admin]
  • Starts With match: hash[name^adm]
  • Ends With match: hash[name$min]
  • Contains match: hash[name%dmi]
  • Less Than match: hash[access_level<500]
  • Greater Than match: hash[access_level>0]
  • Less Than or Equal match: hash[access_level<=100]
  • Greater Than or Equal match: hash[access_level>=0]
  • Python Regular Expression matches: hash[access_level=~/^\D+$/] (the / Regular Expression delimiter can be substituted for any character you need, except white-space; note that / does not interfere with forward-slash notation and it does not need to be escaped because the entire search expression is contained within a [] pair)
  • Invert any match with !, like: hash[name!=admin] or even hash[!name=admin] (the former syntax is used when YAML Paths are stringified but both forms are equivalent)
  • Demarcate and/or escape expression operands, like: hash[full\ name="Some User\'s Name"] (note that em