Build Status:

Master
Develop

AppCompatProcessor (Beta)

"Evolving AppCompat/AmCache data analysis beyond grep"

AppCompatProcessor has been designed to extract additional value from enterprise-wide AppCompat / AmCache data beyond the classic stacking and grepping techniques.

If you don't fancy reading check the SANS Threat Hunting 2017 presentation here

Note: Windows platform support has been temporarily removed (expect to see it back shortly though)

Installation:

OSX You need Python 2.7+, libregf and pyregf (python bindings) from https://github.com/libyal/libregf

-Option A Source distribution package from https://github.com/libyal/libregf/releases

./configure --enable-python && make
sudo make install
python setup.py build
python setup.py install

-Option B Direct from source

git clone https://github.com/libyal/libregf.git
cd libregf/
./synclibs.sh
./autogen.sh
./configure --enable-python && make
sudo make install
python setup.py build
python setup.py install

The rest of the requirements you can handle with 'pip install -r requirements.txt'.

Linux You need Python 2.7+ and 'sudo pip install -r requirements.txt' should take care of everything for you. If you have issues with libregf or pyregf (python bindings for libregf) you can find them here: https://github.com/libyal/libregf

Note: There seems to be an issue with some libregf versions on some linux environments in which libregf apparently provides file paths encoded in UTF16 (breaking sql inserts for us) when it used to do UTF8. Still trying to pinpoint when and why that happens to handle it correctly.

SIFT v3 users Sift comes pre-packaged with libregf v20141030 so 'sudo pip install -r requirements.txt' will add the few missing pieces easily.

Windows Note: ACP is currently broken on Windows due to differences in implementation of multiprocessing!

Ingestion Modules

The following ingestion formats are currently supported:

AppCompat in CSV format as produced by ShimCacheParser.py

Use flags -t -o and store as one file per host. File naming convention that you'll need to follow: <HOSTNAME>.csv. Note that BOM is not currently supported so avoid the '--bom' flag for the time being.

AppCompat in Redline format

Redline creates a folder named after the hostname in the AnalysisSession1/Audits folder. Aggregate all those folders into a single folder that must be called 'RedlineAudits' and ingest that folder to load everything up.

AppCompat from raw SYSTEM hives

File names must begin by "SYSTEM". Host name is extracted from the hive itself.

AmCache from raw AmCache hives

File naming convention that you'll need to follow: <HOSTNAME>.hve.

Shim Shady in-memory extraction of ShimCache records (no enrichment)
AppCompat Mir RegistryAudit (XML)
AppCompat Mir LUA script (XML)
AppCompat from SYSTEM hives retrieved through a Mir FileAcquisition audit
AmCache from AmCache.hve hives retrieved through a Mir FileAcquisition audit
Zip files containing any of the above

Modules:

load 'path': Load (or add new) AppCompat/AmCache data from 'path'

Check [Ingestion Modules](#Ingestion Modules) for a list of supported formats

./AppCompatProcessor.py ./database.db load ./path/to/load

Load will recurse down any available folders to identify files to load.

./AppCompatProcessor.py ./database.db load ./path/to/file.zip

Load will do in-memory processing of the zip file and load its contents.

status: Print status of database

Provides a Host and Entry/Instance count.

list: List hosts in database

Lists all hosts in database including Recon scoring if available.

search -f 'regex': Search for regular expression 'regex' (remember to provide shell escaping)

Search for a regex term. Search space is limited to FilePath\FileName. Output is written to 'Output.txt' if the -o flag isn't specified to write to a custom file.

search -F 'string': Search for literal 'string'

Search for a literal term. Search space is limited to FilePath\FileName.

search ['file']: Search for KnownBad expressions.

Designed to perform massive searches for known bad and methodology regex terms, supports filtering to reduce FPs. Search space is limited to FilePath\FileName. If no file is provided as an argument it will search for KnownBad expressions shipped with ACP, otherwise the provided KnownBad file will be used. Bundled known bad expressions and filters are provided in AppCompatSearch.txt. When installed through setuptools the bundled 'AppCompatSearch.txt' will be deployed in /etc/AppCompatProcessor

Additional files matching 'AppCompatSearch-.*' can be created to supplement the default set of regular expressions with your own sets, these will automatically picked up by ACP.

Most modules from AppCompatProcessor have been optimized and refactored to enable them to take advantage of modern multi-core processors.

./AppCompatProcessor.py test.db search
Searching for known bad list: ./AppCompatSearch.txt (130 search terms) - SearchSpace: (FilePath || '\' || FileName) => Output.txt
Using 6 cores for searching / 1 cores for dumping results
Searching: [#########################] 100.0% Done...
Finishing dumping results to disk: [#########################] 100.0% Done...
Hit histogram:
\\..\..{1,3}$                                      120
\\Start Menu\\Programs\\Startup                    93
\\RarSFX0\\.*\.exe                                 91
\\tsclient\\                                       40
\\(rundll32|cmd|taskeng|conhost|powershell)\.exe   29
C:\\Windows\\setup\.exe                            19
\\.\..{1,3}$                                       13
\.(log|txt|dat)$                                   9
\\Music\\[^\\]*\.                                  8
\\rar\.exe                                         4
\\ProgramData\\[^\\]*\.                            4

The search module will produce two files along with a hit histogram:

Output.txt: Raw dump of hits
Output.mmd: Dump of hits using MultiMarkDown notation to highlight expressions matched or each entry.

fsearch 'field/list' [--sql] (-f/-F) '[<=>]regex/string': Field Search, same principle as the Search module but operating on a user supplied DB field.

'fsearch list' will print out the fields available in the Entries table against which you can search. --sql can be used to build creative search spaces against which to search

./AppCompatProcessor.py ./database.db fsearch list
['rowid', 'hostid', 'entrytype', 'rownumber', 'lastmodified', 'lastupdate', 'filepath', 'filename', 'size', 'execflag', 'sha1', 'filedescription', 'firstrun', 'created', 'modified1', 'modified2', 'linkerts', 'product', 'company', 'pe_sizeofimage', 'version_number', 'version', 'language', 'header_hash', 'pe_checksum', 'switchbackcontext', 'recon', 'reconsession']

./AppCompatProcessor.py ./database.db fsearch FileName -F "cmd.exe"
Will search the FileName field for anything that contains 'cmd.exe' 

./AppCompatProcessor.py ./database.db fsearch FileName -F "=cmd.exe"
Will search the FileName field for anything that exactly matches 'cmd.exe' 

./AppCompatProcessor.py ./database.db fsearch Size -F "4096"
Will find files whose size contains "4096" 

./AppCompatProcessor.py ./database.db fsearch Size -F "=4096"
Will find files whose size _is_ "4096" 

./AppCompatProcessor.py ./database.db fsearch Size -F ">4096"
Will find files whose size is bigger than 4096 bytes (and has Size data of course: XP appcompat or AmCache data)

./AppCompatProcessor.py ./test-AmCache.db fsearch Product -F "Microsoft@"
Will find files for some attackers that regularly screwed the trademark symbol on the versioning information on their tools.

./AppCompatProcessor.py ./delete.db fsearch FirstRun -F ">2015-01-18<2015-01-21"
(nope sorry, just use sqlite if you want to get that fancy!: "SELECT * FROM Csv_Dump WHERE LastModified BETWEEN '2015-01-18' and '2015-01-21'")

./AppCompatProcessor.py ./database.db fsearch FileName -f "=cm[ad].exe"
Will search the FileName field for anything that exactly matches against the regular expression '^cmd[ad].exe$' 

./AppCompatProcessor.py ./database.db fsearch --sql "(FilePath || '\\' || FileName)" -f "Windows\\hkcmd.exe"
Will search for entries who's fullpath contains Windows\hkcmd.exe. This sql tweak is exactly what happens by default with the Search module BTW.
Note: The weird syntax there is what SQL expect you to use to concatenate two fields with a backslash separator. You can use this as an example of how to build custom search spaces.

filehitcount 'file': Count # of FileName hits from 'file'

Provides a quick count of hits for a set of filenames stored in a file. Search space if limited to FileName only.

./AppCompatProcessor.py ./database.db filehitcount ./path/to/file/file.txt
FileName              HitCount
cmd.exe               4098
wmiprvse.exe          2973
net.exe               2528
net1.exe              2392
schtasks.exe          2285
WMIC.exe              1791
netsh.exe             1624
ARP.EXE               1413
HOSTNAME.EXE          1364
PING.EXE              1346
ipconfig.exe          1185
vds.exe               1019
vdsldr.exe            1015
CompMgmtLauncher.exe  940
ceipdata.exe          912
ceiprole.exe          837
NETSTAT.EXE           728
ServerManagerCmd.exe  494
whoami.exe            352
tasklist.exe          284
at.exe                139
winver.exe            118
TRACERT.EXE           110
systeminfo.exe        91
quser.exe             50
dsget.exe             11
dsquery.exe           11

Not

Appcompatprocessor

Install / Use

README

AppCompatProcessor (Beta)

Installation:

Ingestion Modules

Modules: