Dendrol
🌴 The STIX2 Pattern expression parser for humans
Install / Use
/learn @PerchSecurity/DendrolREADME
This iconic STIX2 Pattern visualization is based upon this example expression:
(
[ipv4-addr:value = '198.51.100.1/32' OR
ipv4-addr:value = '203.0.113.33/32' OR
ipv6-addr:value = '2001:0db8:dead:beef:dead:beef:dead:0001/128']
FOLLOWEDBY [
domain-name:value = 'example.com']
) WITHIN 600 SECONDS
Using the formal STIX2 Pattern grammar, that expression is converted into this parse tree:
dendrol will convert that expression (by way of that parse tree) into this more human-readable and machine-actionable form:
pattern:
expression:
join: FOLLOWEDBY
qualifiers:
- within:
value: 600
unit: SECONDS
expressions:
- observation:
objects:
? ipv4-addr
? ipv6-addr
join: OR
qualifiers:
expressions:
- comparison:
object: ipv4-addr
path: [value]
negated:
operator: '='
value: 198.51.100.1/32
- comparison:
object: ipv4-addr
path: [value]
negated:
operator: '='
value: 203.0.113.33/32
- comparison:
object: ipv6-addr
path: [value]
negated:
operator: '='
value: 2001:0db8:dead:beef:dead:beef:dead:0001/128
- observation:
objects: {domain-name}
join:
qualifiers:
expressions:
- comparison:
object: domain-name
path: [value]
negated:
operator: '='
value: example.com
How do I use it?
dendrol provides an interface for parsing STIX2 Pattern Expressions much like cti-pattern-validator, with the dendrol.Pattern class. This class has a method, to_dict_tree(), which converts the ANTLR parse tree to a dict-based tree structure, PatternTree.
from dendrol import Pattern
pattern = Pattern("[domain-name:value = 'http://xyz.com/download']")
assert pattern.to_dict_tree() == {
'pattern': {
'observation': {
'objects': {'domain-name'},
'join': None,
'qualifiers': None,
'expressions': [
{'comparison': {
'object': 'domain-name',
'path': ['value'],
'negated': None,
'operator': '=',
'value': 'http://xyz.com/download',
}}
]
}
}
}
A specialized YAML representation is also proposed, to make visualization of this data a little less cumbersome:
from dendrol import Pattern
pattern = Pattern("[domain-name:value = 'http://xyz.com/download']")
assert str(pattern.to_dict_tree()) == '''\
pattern:
observation:
objects: {domain-name}
join:
qualifiers:
expressions:
- comparison:
object: domain-name
path: [value]
negated:
operator: '='
value: http://xyz.com/download
'''
For more info, read The Spec below, or check out the tests.
Development
To develop dendrol and run its tests, first clone the repo. Then install the dev and testing dependencies:
pip install .[dev] .[test]
pytest is used for testing:
py.test
Reported issues and pull requests welcomed! From new features and suggestions to typo fixes and poor naming choices, fresh eyes bolster software eternally in development.
If submitting a pull request, please add yourself to the CONTRIBUTORS file for a piece of that sweet, sweet street cred!
<a name="the-spec" href="#the-spec">The Spec</a>
Brief
A PatternTree begins with a 'pattern' key. Below it is an observation expression, with an 'observation' or 'expression' key (which may contain more observation expressions joined by AND/OR/FOLLOWEDBY). Below 'observation' keys are comparison expressions, marked by a 'comparison' or 'expression' key (which may contain more comparison expressions joined by AND/OR). 'comparison' keys denote a single comparison between an object property and a literal value.
<a name="spec-pattern" href="#spec-pattern">Pattern</a>
{'pattern': {...}}
pattern:
...
A PatternTree is a dict with one top-level key, 'pattern'. This paradigm of a dict with a single key identifying its contents is seen throughout this spec.
The value of this 'pattern' key is an observation expression.
<a name="spec-observation-expressions" href="#spec-observation-expressions">Observation Expressions</a>
An Observation Expression is a dict with a single key of either 'expression' or 'observation'. An 'expression' SHALL contain two or more observation expressions joined by AND/OR/FOLLOWEDBY, whereas an 'observation' SHALL contain only comparison expressions.
<a name="spec-observation-expressions-expression" href="#spec-observation-expressions-expression">Expression</a>
{'expression': {
'join': oneOf('AND', 'OR', 'FOLLOWEDBY'),
'qualifiers': [...],
'expressions': [...],
}}
expression:
join: AND | OR | FOLLOWEDBY
qualifiers:
expressions:
- a
- b
- ...
An 'expression' is a container for other observation expressions, joined by an observation operator in 'join'. It MAY have a list of qualifiers in the 'qualifiers' key, or None if there are none.
Its children are in 'expressions', whose values SHALL be dicts with single keys (of either 'observation' or 'expression').
<a name="spec-observation-expressions-observation" href="#spec-observation-expressions-observation">Observation</a>
{'observation': {
'objects': {'ipv4-addr', 'ipv6-addr', ...},
'join': oneOf('AND', 'OR'),
'qualifiers': [...],
'expressions': [...],
}}
observation:
objects:
? ipv4-addr
? ipv6-addr
? ...
join: AND | OR
qualifiers:
expressions:
- a
- ...
An 'observation' is analogous to square brackets in STIX2 Pattern Expressions, e.g.: [ipv4-addr:value = '1.2.3.4']. Children of an observation (in the 'expressions' key) SHALL only be comparisons or comparison expressions.
An 'observation' MAY have qualifiers, but its children MUST NOT.
An 'observation' MAY have a join method, which denotes how its child comparison expressions are to be joined. This method MAY be AND or OR, but MUST NOT be FOLLOWEDBY, because the join method applies to comparison expressions, not observation expressions. If there is only a single child comparison expression, 'join' MAY be None.
An 'observation' SHALL contain a set of all the object types of its child comparison expressions. This is mainly for human consumption. A STIX2 observation is allowed to contain comparisons on disparate object types, provided they're joined by OR— this is why 'objects' is a set, not a single string.
If 'objects' contains only a single object type, it MAY be compacted into set literal form:
observation:
objects: {ipv4-addr}
join: AND | OR
qualifiers:
expressions:
- a
- ...
<a name="spec-observation-expressions-qualifiers" href="#spec-observation-expressions-qualifiers">Qualifiers</a>
A Qualifier is a dict having a single key identifying its Qualifier type. Currently, this SHALL be one of:
<a name="spec-observation-expressions-qualifiers-start-stop" href="#spec-observation-expressions-qualifiers-start-stop">Start/Stop Qualifier</a>
{'start_stop': {
'start': datetime(2018, 10, 7, 0, 0, tzinfo=tzutc()),
'stop': datetime(2018, 10, 7, 23, 59, tzinfo=tzutc()),
}}
start_stop:
start: 2018-10-07T00:00:00Z
stop: 2
