Binary Refinery

[ Documentation ][docs] [][tests] [][codecov] [][pypi]

  __     __  High Octane Triage Analysis          __
  ||    _||______ __       __________     _____   ||
  ||    \||___   \__| ____/   ______/___ / ____\  ||
==||=====||  | __/  |/    \  /==|  / __ \   __\===]|
  '======||  |   \  |   |  \_  _| \  ___/|  |     ||
         ||____  /__|___|__/  / |  \____]|  |     ||
=========''====\/=========/  /==|__|=====|__|======'
                         \  /
                          \/

The Binary Refinery™ is a collection of Python scripts that implement transformations of binary data such as compression and encryption. We will often refer to it simply by refinery, which is also the name of the corresponding package. The scripts are designed to exclusively read input from stdin and write output to stdout. The main philosophy is that every script should be a unit in the sense that it does one job, and individual units can be combined into pipelines with the piping operator | on the commandline to perform more complex tasks. The project's main focus is malware triage, and is an attempt to implement something like CyberChef on the commandline.

Short Version

Make a Python virtual environment. You need Python 3.8 or later. Install refinery like this:

python -m pip install -U pip
python -m pip install -U binary-refinery[extended]

Run units with -h to learn how they work, grep through the [docs][] or use the command binref to find them. Watch [a recent video][VOD3] if you want to see it in action. But also, read the rest of this readme.

Release Schedule

There is no release schedule, but releases happen very frequently and it is recommended to update periodically. Bugfixes are not documented outside of GIT, but all other changes (i.e. new features) are documented in the changelog. Follow me on [Mastodon][] for updates about particularly impactful releases.

Documentation

The help text that is displayed when executing a unit with the -h or --help switch is its main documentation. The [automatically generated documentation][docs] contains a compilation of that output for each unit at the top level, but also contains the specification for the three fundamental concepts of the toolkit: [framing][frame], [multibin arguments][argformats], and [meta variables][meta]. Full-text search of the description and help text for every unit is also available on the command line, via the provided binref command. In recognition of the fact that reference documentation can be somewhat dry, there is an ongoing effort to produce a series of tutorials; I very much recommend to check them out. On top of that, I collect additional resources (including some produced by third parties) below.

[!NOTE]
Refinery is still in alpha and the interface can sometimes change, i.e. units and parameters can be removed or renamed. Hence, it can happen that specific command lines from older videos and blog posts don't work any more.

[2021/08] [OALabs][OA] was kind enough to let me [demo the toolkit in a feature video][VOD1]. In the video, I essentially work through the contents of the first tutorial.
[2021/11] [Johannes Bader][JB] wrote an amazing [blog post][BLOG] about analyzing malspam with binary refinery.
[2024/03] [Malware Analysis For Hedgehogs][MH] made [a video about unpacking an XWorm sample][VOD2] using refinery.
[2024/11] [the CyberYeti][CY] had me [on stream presenting refinery][VOD3].
[2025/06] I was [on stream again][VOD4] with [the CyberYeti][CY], this one is a little more raw. All bugs you can see here were fixed. 😉

Showcases again include samples from the example section below and the tutorials.

License

The Binary Refinery is (c) 2019 Jesko Hüttenhain, and published under a [3-Clause BSD License][license]. This repository also contains a copy of the full license text. If you want to do something with it that's not covered by this license, please feel free to contact the author.

Warnings & Advice

The refinery requires at least Python 3.8. It is recommended to install it into its own [virtual environment][venv]: The package can pull in a lot of dependencies, and installing it into your global Python is somewhat prone to version conflicts. Also, since the toolkit introduces a large number of new commands, there is a good chance that some of these will clash on some systems, and keeping them in their own separate virtual environment is one way to prevent that.

If you want to have all refinery commands available in your shell at all times (i.e. without having to switch to a custom virtual environment), you also have the option to choose a prefix for the installation, which will be put in front of every command shim that is installed. For example, if you choose r. as your prefix, then the [emit][] unit will be installed as the command r.emit. An added benefit is that you can type r. and hammer <kbd>Tab</kbd> twice to get a list of all available refinery commands. Note however that no prefix is assumed in documentation and it is a development goal of refinery to not clash on most systems. The author does not use a prefix and provides this option as a safety blanket.

Installation

The most straightforward way to install and update refinery is via pip. Make sure you run the latest version first:

python -m pip install -U pip

Then just install the refinery package:

pip install -U binary-refinery

If you want to choose a prefix for all units, you can specify it via the environment variable REFINERY_PREFIX. For example, the following command will install refinery into the current Python environment with prefix r. on Linux:

REFINERY_PREFIX=r. pip install -U binary-refinery

On Windows, you would have to run the following commands:

set REFINERY_PREFIX=r.
pip install -U binary-refinery

Specifying the special prefix ! will have the effect that no shell commands are created at all, and binary refinery will be installed only as a library. If you want to install the current refinery HEAD, you can repeat all of the above steps, specifying this repository instead of the pip package. For example, the following will install the very latest refinery commit:

pip install -U git+git://github.com/binref/refinery.git

Finally, if you are using [REMnux][remnux-main], you can use their [refinery docker container][remnux].

Claude Skill Installation

If you want to teach your local malware analysis [claude][] to use binary refinery, check out the [Binary Refinery Skill][agent].

Shell Support

The following is a summary of how well various shell environments are currently supported:

| Shell | Platform | State | Comment | |:-----------|:---------|:----------------|:-----------------------------------------------------------------| | Bash | Posix | 🔵 Good | Used occasionally by the author. | | CMD | Windows | 🔵 Good | Used extensively by the author. | | PowerShell | Windows | 🟡 Reasonable | It [just works if the PowerShell version is at least 7.4.][psh1] | | Zsh | Posix | 🟠 Minor Issues | Following a [discussion][zsh1], there is a [fix][zsh2]. | | Fish | Posix | 🟠 Minor Issues | See issue [#55][fsh1] and discussion [#22][fsh2]. |

If you are using a different shell and have some feedback to share, please let me know!

Modular Dependencies

There are some very situational units with (sometimes large) external dependencies. For example, [stego][] is a unit that requires the image parsing library Pillow. Some libraries are not installed by default to keep the installation time for refinery at a reasonable level for first-time users. The corresponding units will tell you what to do when their dependency is missing:

$ emit config.png | stego RG
(13:37:00) failure in stego: dependency Pillow is missing; run pip install Pillow

You can then install these missing dependencies manually. If you do not want to be bothered by missing dependencies and don't mind a long refinery installation, you can install the package as follows:

pip install -U binary-refinery[all]

which will install all dependencies on top of the required ones. More precisely, there are the following extra categories available:

| Name | Included Dependencies | |-----------:|:------------------------------------------------------------------| | default | recommended selection of reasonable dependencies, author's choice | | extended | an extended selection, excluding only the most obscure ones | | all | all dependencies for all refinery units |

These are listed here in ascending order, i.e. extended will install everything that default would install.

Bleeding Edge

Alternatively, you can clone this repository and use the scripts update.sh (on Linux) or update.ps1 (on Windows) to install the refinery package into a local virtual environment. The installation and update process for this method is to simply run the script:

it pulls the repository,
activates the virtual environment,
uninstalls binary-refinery,
and then installs binary-refinery[all].

Generating Documentation

You can also genera

Refinery

Install / Use

README