Dif

dif - a preprocessing front end to meld / gvimdiff / kdiff3 / tkdiff / kompare

'dif' compares files after it preprocesses them.
Preprocessing options include:

• remove comments, whitespace, timestamps
• search/replace
• keep/ignore certain lines
• json/yaml reformatting
• parse values from xls spreadsheets
• many other options (see far below)

'dif' can also be used to compare directories recursively, after optionally preprocessing each file

Overview

The graphical compare tools meld, gvimdiff, kdiff3, tkdiff, and kompare are used to compare text files on Linux

In many cases, it is difficult and time-consuming to visually compare large files because of the large number of differences

For example:

• different versions of code may differ only in comments or whitespace
• log files are often many MB of text, with some "don't care" information such as timestamps or temporary filenames
• json or yaml files may have ordering differences due to the library used to write the file
• xls spreadsheets cannot be compared easily because of the file format

Purpose

'dif' preprocesses input text files with a wide variety of options

Afterwards, it runs the Linux tools meld, gvimdiff, kdiff3, tkdiff, or kompare on these intermediate files

This allows you to concentrate on the important differences, and ignore the rest

Solutions

Problem: differences in whitespace or comments or case cause mismatches

Solution: Use options -white or -nowhite or -comments or -case

Problem: files both need to be filtered using regexes, to strip out certain characters or sequences

Solution 1: Use -grep <regex> or -ignore <regex> to filter in or out

Solution 2: Use -search <regex> -replace <regex> to supply one instance of substitution and replacement

Solution 3: Use -replaceTable <file> to supply a file with many substitution/replacement regexes

Solution 4: Use -replaceDates to remove dates and timestamps

Problem: need to view your changes to a file on Perforce or SVN or GIT

Solution: 'dif file' will show the differences between the head revision and the local file

Problem: need to recursively compare directories

Solution 1: 'dif dir1 dir2' will iteratively compare pairs of files

Solution 2: 'dif dir1 dir2 -report' will open a GUI to compare the directories

Any preprocessing option (-comments, -white, -sort, -grep, etc) can be used when comparing directories

Usage examples

• dif file1 file2
• dif file1 file2 -white -case
• dif file1 file2 file3 -comments
• dif file1 file2 -search 'foo' -replace 'bar'
• dif file1.xls file2.xls
• dif dir1 dir2 -report

Options

Filtering options:    
   -comments          Remove any comments such as // or # or single-line */ /*.  Also removes trailing whitespace

                      To remove comments in other languages, use the search/replace options:
                      For example, to replace comments (marked with ';') in assembly language:
                          -search '\s*(;.*)?$' -replace ''

   -white             Remove blank lines and leading/trailing whitespace
                      Condense multiple whitespace to a single space
                      Remove any non-printable characters
   
   -noWhite           Remove all whitespace and non-printable characters

   -case              Convert files to lowercase before comparing
   
   -grep 'regex'      Only display lines which match the user-specified Perl regex
                      Multiple regexs can be specified, for example:  -grep '(regexA|regexB)'
                      To display lines above/below matches, see the help text for option -externalPreprocessScript

   -ignore 'regex'    Ignore any lines which match the user-specified regex
                      This is the opposite of the -grep function

   -search 'regex'    On each line, do a global regex search and replace
   -replace 'regex'   
                      For example, to replace temporary filenames such as '/tmp/foo123456/bar.log' with '/tmp/file':
                          -search '/tmp/\S+' -replace '/tmp/file'

                      Since the search/replace terms are interpreted as regex,
                      remember to escape any parentheses
                          Exception:  if you are using regex grouping, 
                                      do not escape the parentheses
                          For example:
                              -search '(A|B|C)'  -replace 'D'

                      Since the replace term is run through 'eval', make sure to escape any $ dollar signs
                      Make sure to use 'single-quotes' instead of double-quotes
                      For example, to convert all spaces to newlines, use:
                          -search '\s+'  -replace '\n'

                      If case-insensitive search is needed, also use option -case

   -replaceTable file     Specify a two-column file which will be used for search/replace
                          The delimiter is any amount of spaces
                          Terms in the file are treated as regular expressions
                          The replace term is run through eval

   -replaceDates      Remove dates and times, for example:
                           17:36:34
                           Monday July 20 17:36:34 PDT 2020
                           Dec  3  2019
                           Jul 10 17:42
                           1970.01.01
                           1/1/1970

   -fields N          Compare only field(s) N
                      Multiple fields may be given, separated by commas (-fields N,M)
                      Field numbers start at 0
                      Fields in the input files are assumed to be separated by spaces,
                          unless the filename ends with .csv (separated by commas)
                      Example:  -fields 2
                      Example:  -fields 0,2      (fields 0 and 2)
                      Example:  -fields -1       (last field)
                      Example:  -fields 2+       (field 2 and above)
                      Example:  -fields not2+    (ignore fields 2 and above)
                      Example:  -fields not0,5+  (ignore fields 0, 5, and above)

   -fieldSeparator regex    Only needed if default field separators above are not sufficient
                            Example:  -fieldSeparator ':'
                            Example:  -fieldSeparator '[,=]' 
   
   -fieldJustify      Make all fields the same width, right-justified

   -split             Splits each line on whitespace
   
   -splitChar 'char'  Splits each line on 'char'
                      For example:  -splitChar ',' to split on comma

   -splitWords        Splits on whitespace.  Each word will be on its own line.
                      Identical to -splitChar '\s+'

   -trim              Trims each line to 105 characters, discarding the overflow
                      Useful when lines are very long, and the important information is near the beginning
   
   -trimChars N       Trims with specified number of characters, instead of 105
   
   -head              Compare only the first 10% of the file,
                        with a minimum of 50, and a maximum of 10000 lines
   
   -headLines N       Compare only the first N lines
                      If a negative number is used, ignore the first -N lines

   -tail              Compare only the last 10% of the file
                        with a minimum of 50, and a maximum of 10000 lines
   
   -tailLines N       Compare only the last N lines
                      If a negative number is used, ignore the last -N lines
   
   -yaml              Compare two yaml files, sorting the keys
   
   -json              Compare two json files, sorting the keys

   -removeDictKeys 'regex'
                      For use with -yaml or -json
                      Removes all dictionary keys matching the regex
                      Removes all dictionary keys matching the regex

   -flatten           For use with -yaml or -json
                      Flatten nested dictionary and array structures

   -basenames         Convert path/file to file
                      This can be useful when comparing log files which contain temporary directories

   -extensions        Convert path/file.extension to .extension
   
   -removeExtensions  Convert path/file.extension to path/file

   -lineWordSort      Sort the words in each line (space delimited)
   
   -round 'string'    Round all numbers according to the sprintf string
                      For example -round '%0.2f'
   
   -dos2unix          Run all files through dos2unix

   -lsl               Useful when comparing previously captured output of 'ls -l'
                      Compares only names and file sizes

   -tartv             Compare tarfiles using tar -tv, and compare the names and file sizes
                      If file sizes are not desired in the comparison (names only), also use -fields 1
      
   -perlEval          The input file is a perl hashref
                      Print the keys in alphabetical order

   -perlDump          Useful when comparing previously captured output of Data::Dumper
                      filter out all SCALAR/HASH/ARRAY/REF/GLOB/CODE addresses from output of Dumpvalue,