Connect the open source telemetry engine VAST with Threat Bus, the open source intelligence platform
Project description
PyVAST Threat Bus App
Threat Bus is a publish-subscribe broker for threat intelligence. It is expected
that applications register themselves at the bus. Since VAST cannot do so on its
own (yet), pyvast-threatbus.py
implements that functionality in the meantime.
The application provides a thin layer around PyVAST, VAST's Python CLI bindings. It facilitates message exchange between Threat Bus and a VAST instance.
Installation
Install pyvast-threatbus
via pip
. Optionally, use a virtual environment.
virtualenv venv # optional
source venv/bin/activate # optional
python -m pip install pyvast-threatbus
Development
Use the dev-mode
command from the Makefile
to install the project in
development mode.
We recommend to use a virtual environment for development.
virtualenv venv
source venv/bin/activate
make dev-mode
Quick Start
You can configure the app either via a YAML config file or via command line
arguments. See config.yaml.example
for an example configuration file that uses
fever alertify to transform sighting contexts
before they get printed to STDOUT
. See the section
Features section for
details and --help
for command line usage. Here are some command line options
to get you started.
Start with a config file:
./pyvast_threatbus.py -c config.yaml
Startup with debug logging and customized endpoints for Threat Bus and VAST:
./pyvast_threatbus.py --vast-binary=/opt/tenzir/bin/vast --vast=localhost:42000 --threatbus=localhost:13370 --loglevel=DEBUG
Request an intelligence snapshot of the past 50 days and match it retrospectively against the entire VAST database:
./pyvast_threatbus.py --snapshot=50 --retro-match
Features
This section explains the most important features and CLI options of
pyvast-threatbus
.
IoC Matching
VAST can match IoCs either live or retrospectively via usual queries.
Live Matching
VAST's
live matching
works as
continuous query.
pyvast-threatbus
subscribes to those continuous query results and reports all
new IoC matches from VAST to Threat Bus as Sightings
.
Live matching is the default mode of operation for pyvast-threatbus
. Note that
the VAST node must support this feature.
Retro Matching
pyvast-threatbus
supports retro matching via the command line option
--retro-match
. This instructs the application to translate IoCs from Threat
Bus to normal VAST queries instead of feeding the IoCs to a live matcher.
Each result from an IoC query is treated as Sighting
of that IoC and reported
back to Threat Bus. You can limit the maximum amount of results returned from
VAST via the option --retro-match-max-events
.
Sighting Context Transformation
The app provides a command line option to invoke another program for parsing
Sighting context
data.
The option --transform-context "cmd args"
translates the context
field of a Sighting via the specified utility. For example, pass the context
object to DCSO/fever alertify
:
apps/vast/pyvast-threatbus.py --retro-match --transform-context "fever alertify --alert-prefix 'MY PREFIX' --extra-key my-ioc --ioc %ioc"
A Sighting
object is structured as follows:
{
"ts": "2020-09-04 11:21:13.663164",
"reference": "threatbus__86",
"ioc": "foobar.com",
"context": {
...
}
The context
field can contain arbitrary data. For example, retro matches from
VAST contain the full query result in the context field (like a Suricata EVE
entry or a Zeek conn.log entry).
Note that the cmd
string passed to --transform-context
is treated as
template string. The placeholder %ioc
is replaced with the contents of the
actually matched IoC.
Custom Sinks for Sightings
The pyvast-threatbus
offers to send Sighting context
to a configurable sink
instead of reporting them back to Threat Bus. This can be configured via the
--sink
parameter. The special placeholder STDOUT
can be used to print the
Sighting context to STDOUT
. Example:
./pyvast_threatbus.py --sink stdout
A custom sink is useful to forward Sightings
to another process, like
syslog
, or forward STDOUT via a UNIX pipe. Note that it may be desirable to
disable logging in that case (--loglevel NOTSET
).
Note that only the context
field is printed, and not the object structure of
the Sighting
. The Sighting
itself is a Threat Bus internal format and most
likely not usable for most applications.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for pyvast-threatbus-2020.11.26.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | e05d1ca1d27dd361c9a579bcbfaf2bf5c239c4c3ce33abbd4bffe88ef58e1b6a |
|
MD5 | c72b8e17858f7bd31a26f88e5c576e00 |
|
BLAKE2b-256 | 424127a21aea08df2539277ac7f6602f6624d15d08a5ce5803af3a63fee4fcc3 |
Hashes for pyvast_threatbus-2020.11.26-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | da0d889520b9bebd55a38415e3bb61948c940fee3791a701a0b79d29dc8ec6b5 |
|
MD5 | b6c17a9f5c80c4ef9e52b8c75d333175 |
|
BLAKE2b-256 | 9c1a0c5eb29958c76565d444a6d4545343e264057ef1d76d593610d06f05e2e5 |