Faconne
Data Restructuring DSL in Clojure. You write declarative specifications for complicated data transformations and faconne generates the functions automatically. Published on Clojars and used in production code for years.
Install / Use
/learn @turtlegrammar/FaconneREADME
Logo courtesy of Clayton Belcher. This README is based on demo.clj. Play with it in a REPL!
(ns demo
(:require [faconne.core :as f]))
See the change log
Note: 1.1.0 -> 1.1.1 may require a migration, but 1.0.x -> 1.1.1 does not.
Include
Lein: [faconne "1.1.1"] Import for Clojure:
(ns demo
(:require [faconne.core :as f]))
Import for Clojurescript: clj (ns demo (:require [faconne.core :as f :include-macros true]))
Overview/Motivation
Frequently in Clojure, we find ourselves with a deeply nested collection that we wish had a different shape. So, we write an ugly nest of map, filter, reduce, group-by, etc. that takes our collection and re-arranges it, maps deep into it, inverts part of it, introduces or changes groupings in it, filters deep into it, or tags child collection with metadata. This is hard to read, and difficult to update when the shape of the data changes. Faconne lets you declaratively transform nested collections into other nested collections by visually specifying the shape of the input and the desired shape of the output. For example:
(def student-data
;; As we might pull from a database with JDBC
[{:student "john", :grade 97, :course "math", :campus "east"}
{:student "john", :grade 90, :course "english", :campus "east"}
{:student "john", :grade 70, :course "history", :campus "east"}
{:student "dave", :grade 80, :course "math", :campus "east"}
{:student "dave", :grade 100 ,:course "english", :campus "east"}
{:student "mary", :grade 90, :course "math", :campus "west"}
{:student "mary", :grade 92, :course "english", :campus "west"}
{:student "mary", :grade 94, :course "history", :campus "west"}])
(defn average [xs] (/ (reduce + 0 xs) (count xs)))
(def organized-student-data
(f/transform student-data
[{:keys [student grade course campus]}]
;; ^:expand denotes that the collection should be fully built up
;; before the surrounding code is executed.
{campus {:number-students (count ^:expand #{student})
:avg-grade-per-course {course (average ^:expand [grade])}
:student-grades {student {course grade}}}}))
;; (Note that `count`, and `average` are normal Clojure functions
;; called on values built up in the transform.)
;; =>
{"east" {:number-students 2,
:avg-grade-per-course {"math" 177/2,
"english" 95,
"history" 70},
:student-grades {"john" {"math" 97,
"english" 90,
"history" 70},
"dave" {"math" 80,
"english" 100}}},
"west" {:number-students 1,
:avg-grade-per-course {"math" 90,
"english" 92,
"history" 94},
:student-grades {"mary" {"math" 90,
"english" 92,
"history" 94}}}}
;; We can transform back to the original collection with:
(f/transform organized-student-data
{campus {:student-grades {student {course grade}}}}
[{:campus campus, :student student, :course course, :grade grade}])
;; If we need to do an action for its side effects at each node, we can use
;; `f/for-each` instead of `f/transform`. Instead of specifying the desired
;; shape of the collection, we specify any Clojure expression.
(f/for-each organized-student-data
{campus {:student-grades {student {course grade}}}}
(prn (str "Congratulations to " student " at " campus " campus!"
" They got a " grade " in " course "!"))
;; See the section on filtering for more details
:where [(>= grade 95)])
;; =>
;; "Congratulations to john at east campus! They got a 97 in math!"
;; "Congratulations to dave at east campus! They got a 100 in english!"
Faconne is great for tranforming data pulled from a database or API into something that's easier for your application to work with.
Examples
I've used Faconne in a production code-base for several years. Here's some usages I've encountered using Faconne 'in the wild.'
Un-nesting / un-grouping
(def store-data
{"Gas Station" {1 ["Hot Dog", "Nachos", "Cola"]
2 ["Hot Dog", "Toothpaste", "Deoderant"]}
"Grocer" {1 ["Celery" "Onion" "Carrot"]
2 ["Orange" "Apple"]
3 ["Salmon"]}})
(f/transform store-data
{store {aisle [product]}}
{store #{product}})
;; =>
{"Gas Station" #{"Nachos" "Toothpaste" "Cola" "Deoderant" "Hot Dog"}
"Grocer" #{"Carrot" "Onion" "Celery" "Salmon" "Orange" "Apple"}}
(f/transform store-data
{store {aisle [product]}}
{store [{:aisle aisle, :product product}]})
;; =>
{"Gas Station" [{:aisle 1, :product "Hot Dog"}
{:aisle 1, :product "Nachos"}
{:aisle 1, :product "Cola"}
{:aisle 2, :product "Hot Dog"}
{:aisle 2, :product "Toothpaste"}
{:aisle 2, :product "Deoderant"}],
"Grocer" [{:aisle 1, :product "Celery"}
{:aisle 1, :product "Onion"}
{:aisle 1, :product "Carrot"}
{:aisle 2, :product "Orange"}
{:aisle 2, :product "Apple"}
{:aisle 3, :product "Salmon"}]}
Nesting / Grouping
(def event-data
[{:day "2018-08-10", :type "add-user" :handled? false :data ["steve"]}
{:day "2018-08-10", :type "add-user" :handled? true :data ["george"]}
{:day "2018-08-10", :type "remove-user" :handled? false :data ["janice"]}
{:day "2018-08-11", :type "add-user" :handled? true :data ["jocelyn"]}
{:day "2018-08-11", :type "remove-user" :handled? false :data ["steve"]}])
(f/transform
event-data
[{:keys [day type handled? data]}]
{day {(if handled? :handled :unhandled) {type data}}})
;; =>
{"2018-08-10"
{:unhandled
{"add-user" ["steve"],
"remove-user" ["janice"]},
:handled {"add-user" ["george"]}}
, "2018-08-11"
{:handled {"add-user" ["jocelyn"]},
:unhandled {"remove-user" ["steve"]}}}
Reducing
;; Suppose we want to find the number of classes each student is in:
(f/transform {"math" {2062 #{"John", "Mary", "Paul", "Susan"}
4001 #{"Mary", "Tone", "Mike"}}
"history" {6000 #{"John", "Paul", "Susan", "Tone"}
3052 #{"Tone", "Mike", "Susan"}}}
{category {course-number #{student}}}
{student (count ^:expand #{course-number})})
;; =>
{"Susan" 3, "Mary" 2, "John" 2, "Paul" 2, "Tone" 3, "Mike" 2}
;; Faconne typically works by evaluating the range of the transform
;; (in this case, `{student (count ^:expand #{course-number})}`)
;; at each leaf of the domain
;; (in this case, `{category {course-number #{student}}}`)
;; then deep merging the results.
;; But sometimes you want to only partially evaluate the range
;; at each leaf, and wait until a collection is fully built up
;; before applying a function -- like `count` -- to it.
;; To denote this, tag the collection with `^:expand` metadata.
;; Faconne makes you opt-in for reducing logic, because I've found that,
;; while both are useful, wanting to evaluate at each leaf and deep merge
;; is more common than wanting to reduce over a collection built up in the transform. An example of the former:
(f/transform {"math" {2062 #{"John", "Mary", "Paul", "Susan"}
4001 #{"Mary", "Tone", "Mike"}}
"history" {6000 #{"John", "Paul", "Susan", "Tone"}
3052 #{"Tone", "Mike", "Susan"}}}
{category {course-number #{student}}}
{student (if (< 4000 course-number)
{:undergraduate #{course-number}}
{:graduate #{course-number}})})
;; =>
{"Susan" {:graduate #{2062 3052}, :undergraduate #{6000}},
"Mary" {:graduate #{2062}, :undergraduate #{4001}},
"John" {:graduate #{2062}, :undergraduate #{6000}},
"Paul" {:graduate #{2062}, :undergraduate #{6000}},
"Tone" {:undergraduate #{4001 6000}, :graduate #{3052}},
"Mike" {:undergraduate #{4001}, :graduate #{3052}}}
Inverting
(f/transform {"GYU-6749" 1
"JEI-1353" 2
"JNMK-194" 3}
{license-plate parking-space}
{parking-space license-plate})
;; =>
{1 "GYU-6749", 2 "JEI-1353", 3 "JNMK-194"}
(f/transform {"Grocer" #{"Hot Dog", "Celery", "Tooth Brush"}
"Gas Station" #{"Hot Dog", "Tooth Brush", "Beer"}}
{store #{product}}
{(clojure.string/lower-case product)
#{(clojure.string/lower-case store)}})
;; =>
{"celery" #{"grocer"},
"tooth brush" #{"gas station" "grocer"},
"hot dog" #{"gas station" "grocer"},
"beer" #{"gas station"}}
Mapping and Merging
(f/transform {"First Baseman" [{:first-name "Steve" :last-name "White"}]
"first baseman" [{:first-name "Mark" :last-name "Smith"}]
"second Baseman" [{:first-name "George" :last-name "Brown"}]}
{position [{:first-name f :last-name l}]}
{(-> position
clojure.string/lower-case
(clojure.string/replace " " "-")
keyword)
[(str l ", " f)]})
;; =>
{:first-baseman ["White, Steve" "Smith, Mark"],
:second-baseman ["Brown, George"]}
Filtering
(def franchise-info
[{:franchise "Laundry Store"
:location {:name "West Location"}
:managers [{:name "Ruth", :months-worked 15}
{:name "Bruno", :months-worked 1}]
:employees [{:name "Luke", :months-worked 0}]}
{:franchise "Laundry Store"
:location {:name "East Location"}
:managers [{:name "Tomas", :
