Skip to main content

Manipulate McAfee ATD appliance

Project description

Robust ATD CLI tools
================


[![GitHub release](https://img.shields.io/github/release/shadowbq/robust-atd.svg?style=for-the-badge)](https://github.com/shadowbq/robust-atd/releases)
[![license](https://img.shields.io/github/license/shadowbq/robust-atd.svg?style=for-the-badge)](/LICENSE)
[![GitHub Release Date](https://img.shields.io/github/release-date/shadowbq/robust-atd.svg?style=for-the-badge)](https://github.com/shadowbq/robust-atd/releases)
[![Code](https://img.shields.io/badge/Language-Python--2.7-ff69b4.svg?style=for-the-badge)](/README.md)


"Robust" is a set of tools to leverage the HTTPS REST API of the [McAfee Advanced Threat Detection](http://www.mcafee.com/us/products/advanced-threat-defense.aspx) 3.8 - 4.x appliance.

## Tools Overview

* `robust` : basic cli submission of a single piece of malware to a MATD server.
* `robust-profiles` : list the available MATD profiles
* `robust-search` : search MATD via MD5 for a report.
* `robust-watchdog` : monitor a directory for files and submit `multithreaded` to MATD
* `robust-convict` : submit `multithreaded` a directory filled with samples and sort into malicious, clean, error, etc.
* `robust-reporter` : parse offline the json files returned during large batch submissions.
* `robust-version-checker` : Check the MATD Server Version

## Important

This is *not a supported or official application of McAfee*. This work is based off of publicly available published documentation for integrating with the McAfee ATD REST API 3.6.x to 4.x

Official API Documentation is available here:

* https://support.mcafee.com/ServicePortal/faces/knowledgecenter?q=api&v=&p=Advanced+Threat+Defense

## McAfee ATD - Advanced Threat defense

McAfee ATD is a commercial grade enterprise security sandbox analysis appliance. It main function is to provide advanced detection for stealthy, zero-day malware. McAfee Advanced Threat Defense is available as an on-premises appliance or a virtual form factor, with support for both private and public cloud with availability in the Azure Marketplace.

* https://www.mcafee.com/us/products/advanced-threat-defense.aspx
* https://www.mcafee.com/us/resources/data-sheets/ds-advanced-threat-defense.pdf

## Install

Req: Python 2.7.x.

Bug #5: https://github.com/shadowbq/robust-atd/issues/5 - ~~`pip install robust-atd`~~

Note: Python 3.x is not supported.

### PKG Download & Manual Install Alternative

Note: `python setup.py install` will attempt to install dependencies from the internet via `pip`.

For offline runtime installation, please download the pip packages listed in the `requirements.txt`.

### Virutalenv

It is recommended to install virtualenv & virtualenvwrapper via `Virtualenv Burrito`.

See: [README_PYTHON_UP.md](/README_PYTHON_UP.md)

```
$> mkvirtualenv robust
$> workon robust
$(robust)> wget https://github.com/shadowbq/robust-atd/archive/master.zip
$(robust)> unzip master.zip
$(robust)> cd master
$(robust)> python setup.py install
```

-or-

```
$> mkvirtualenv --python=python2.7 robust
$> workon robust
$(robust)> pip install robust-atd
```

### Robust (DOT) Configuration file

Robust will use the `~\.robust` configuration file to load defaults into the scripts.

The configuration file is broken into multiple sections. If you use a section you must define all the settings in that section.

* [auth]
* [connection]
* [convict]

It is recommended to set the file to `read-write` only for the current user, and remove all world `(-)rwx` permissions.

Authentication Section `[auth]` :

```shell
$(robust)> cat ~/.robust
[auth]
user: admin
password: password.
```

Connection Detail Section `[connection]` :

```shell
$(robust)> cat ~/.robust
[connection]
ip: atd.localhost.localdomain
skipssl: true
maxthreads: 15
```

Data Storage Section `[storage]`:

Note: Datastorage locations will be created if they do not exist.

```shell
$(robust)> cat ~/.robust
[storage]
severity: 3
cleandir: ~/robust/clean
dirtydir: ~/robust/dirty
reportdir: ~/robust/reports
errordir: ~/robust/errors
```

This file is expanded via the `os` module is compliant with windows user directories.


## Robust:

Using `robust` for submitting samples.

```
usage: robust.py [-h] [-u USER] [-p PASSWORD] [-i ATD IP] [-n] -s
FILE_TO_UPLOAD -a ANALYZER_PROFILE [-v] [--version]

Robust McAfee ATD Python CLI tool

optional arguments:
-h, --help show this help message and exit
--version show program's version number and exit
-v, --verbosity increase output verbosity
(default: None)
-q, --quiet (q)uiet all output
(default: False)


Authentication parameters:
-u USER (u)sername for the API of the ATD
(default: admin)
-p PASSWORD (p)assword for username
(default: password.)
-i ATD IP (i)p or hostname address of ATD
(default: atd.localhost.localdomain)
-n do (n)ot verify the SSL certificate for the communications
(default: False)

Sample parameters:
-s FILE_TO_UPLOAD (s)ample or file to be analyzed
(default: None)
-a ANALYZER_PROFILE (a)nalyzer profile id to be used during analysis
(default: None)

Examples:
robust.py -u admin -p password. -i atd.localhost.localdomain -s /usr/local/bin/file_to_scan -a 1
```

### Submitting a Sample

A sample can be submitted via cli with full flags, `.robust` configuration file, or interrupt passwords.

```shell
$(robust)> robust.py -u admin -p password. -i atd.localhost.localdomain -s /home/malware/non-malicious-container/putty_upx_7.exe
```

Using interrupt (interactive) passwords:

```shell
$(robust)> robust-profiles.py -n -l
Password: <input password>
ATD profiles: 1
Profile id: 26
Name: Win XP Down Select (Online)
OS: winXPsp3
Run all down selects?: Off
******************
```

### Managing Outputs

Using System Return codes with `-q` Quiet output flag. When the quiet flag is
used for submitting samples or searching reports the *severity* of the application
is returned as a system exit/return code. Negative return codes indicate *faults*
or failure during submission.

```
(robust)>$ robust.py -n -a 26 -s ./.samples/Sample.exe -q
(robust)>$ echo $?
2
```

Common Fault codes:

```
-1 ---> Error connecting to the ATD Server
-2 ---> Error uploading file to the ATD Server
-3 ---> Analysis failed
-4 ---> Error getting report
-5 ---> Error Obtaining vmprofilelist
```

Malware ranking:
(If the severity level of the sample is 3 and above it is generally regarded a threat)
```
N/A -> Sample did not run
-1 --> Sample is white listed
0 ---> No malicious activity detected (None)
1 ---> Sample is slightly suspicious (Low)
2 ---> Sample is somewhat/probably is suspicious
3 ---> Sample is malicious (Medium)
4 ---> Sample is malicious
5 ---> Sample is malicious (Very High)
```
## robust-version-checker

You can quickly test your connection settings in the CLI.

```
$(robust)> robust-version-checker.py -u robust -p password. -i atd.example.com -n
Connection successful...

Session Value: g7aenj99pfp0gbrogfbqsd9085
User ID: 57
ATD ver: 4.2.2.16
ATD Box heartbeat: 1519939175
```

## robust-profiles

A tool designed to pull the *Analyzer Profile* policy list available to a specific user.

Pulling the Policy List - In order to submit a sample using `robust` you must identify the Analyzer Profile ID. `robust-profiles` assists in identifying the available profiles your user can submit samples to.

```
$(robust)> robust-profiles.py -n -l
ATD profiles: 10
Profile id: 1
Name: Android
OS: android
Run All Selected?: Off
******************
Profile id: 26
Name: Win XP Down Select (Online)
OS: winXPsp3
Run All Selected?: Off
******************
Profile id: 25
Name: Windows XP Full Run (Offline)
OS: winXPsp3
Run All Selected?: On
******************
Profile id: 24
Name: Windows XP Full Run (Online)
OS: winXPsp3
Run All Selected?: On
******************
```

## robust-search

A tool designed to search and return reports for a specific md5 hash.

```shell
(robust)$> $ robust-search.py -m 2F7568342339CDB8321B52FF7BEBE661 -n
(Sample.exe:2F7568342339CDB8321B52FF7BEBE661) = 2: "Sample probably is suspicious"
```
### Help details

`robust-search` has the options `-w` and `-t` to collect the proper report on the submission.

```
usage: robust-search.py [-h] [-u USER] [-p PASSWORD] [-i ATD IP] [-n] -m MD5
[-t {html,txt,xml,zip,json,ioc,stix,pdf,sample}]
[-w FILENAME] [--version] [-v | -q]

Robust McAfee ATD Python CLI tool

optional arguments:
-h, --help show this help message and exit
--version show program's version number and exit
-v, --verbosity increase output (v)erbosity
(default: None)
-q, --quiet (q)uiet all output
(default: False)

Authentication parameters:
-u USER (u)sername for the API of the ATD
(default: admin)
-p PASSWORD (p)assword for username
(default: password.)
-i ATD IP (i)p or hostname address of ATD
(default: atd.localhost.localdomain)
-n do (n)ot verify the SSL certificate for the communications
(default: False)

Search parameters:
-m MD5 (m)d5 32bit hash of the sample to search
(default: None)

Reporting parameters:
-t {html,txt,xml,zip,json,ioc,stix,pdf,sample}
(t)ype of report requested
(default: None)
-w FILENAME (w)rite filename for saving the requested report
(default: None)
```


## robust-watchdog

A tool that watches a directory recursively for any new files to submit.

Example CLI
```
usage: robust-watchdog.py [-h] -u USER [-p PASSWORD] -i ATD IP [-n] -a
ANALYZER_PROFILE -d DIRECTORY [-e] [-j MAXTHREADS]
[--version] [-v | -q]

Robust McAfee ATD Python CLI tool

optional arguments:
-h, --help show this help message and exit
--version show program's version number and exit
-v, --verbosity increase output (v)erbosity
(default: None)
-q, --quiet (q)uiet all output
(default: False)

Authentication parameters:
-u USER (u)sername for the API of the ATD
(default: None)
-p PASSWORD (p)assword for username
(default: None)
-i ATD IP (i)p or hostname address of ATD
(default: None)
-n do (n)ot verify the SSL certificate for the communications
(default: False)

Watch parameters:
-f (f)ollow and watch the directory for new files to submit
(default: True)
-a ANALYZER_PROFILE (a)nalyzer profile id to be used during analysis
(default: None)
-d DIRECTORY (d)irectory to watch for events
(default: None)
-e (e)xisting files in directory will be submitted
(default: False)
-j MAXTHREADS (j) max number of threads
(default: 1)
```

Let it run in a shell and open another one or the file browser to create files in the /path/to/directory. Since the handler is printing the results, the output will reflect the flags chosen similar to `robust.py`:

The `-e` flag can be passed to cause all existing files in the directory (recurisively) to be submitted upon start.

```shell
(robust)$> robust-watchdog.py -a 26 -d ./ -n -e
.
...
.
.....
```

## robust-convict

`robust-convict` is a tool designed like `robust-watchdog` but its purpose is to help sort large directories of malware samples into directories, while downloading their corresponding reports.

Example Usage

```
robust-convict.py -n -a 26 -c ./tmp/clean/ -x ./tmp/dirty/ -r ./tmp/reports/ -z ./tmp/errors/ -d ./tmp/preprocess -j 10 -t zip -q
```

Options

```
usage: robust-convict.py [-h] [-u USER] [-p PASSWORD] [-i ATD IP] [-n] -a
ANALYZER_PROFILE -d DIRECTORY [-e] [-y SEVERITY]
[-c CLEANDIR] [-x DIRTYDIR] [-r REPORTDIR]
[-z ERRORDIR]
[-t {html,txt,xml,zip,json,ioc,stix,pdf,sample}]
[-j MAXTHREADS] [--version] [-v | -q]

Robust McAfee ATD Python CLI tool

optional arguments:
-h, --help show this help message and exit
--version show program's version number and exit
-v, --verbosity increase output (v)erbosity
(default: None)
-q, --quiet (q)uiet all output
(default: False)

Authentication parameters:
-u USER (u)sername for the API of the ATD
(default: robust)
-p PASSWORD (p)assword for username
(default: ****<.robust>*****)
-i ATD IP (i)p or hostname address of ATD
(default: atd.localhost.localdomain)
-n do (n)ot verify the SSL certificate for the communications
(default: True)

Watch parameters:
-f (f)ollow and watch the directory for new files to submit
(default: False)
-a ANALYZER_PROFILE (a)nalyzer profile id to be used during analysis
(default: None)
-d DIRECTORY (d)irectory to watch for events
(default: None)
-e (e)xisting files in directory will be submitted
(default: False)
-j MAXTHREADS (j) max number of threads
(default: 1)

Convict parameters:
-y SEVERITY (y) treat sample as dirty with this severity [0-5] or higher
(default: 3)
-c CLEANDIR (c) move clean files to this directory
(default: ~/robust/clean/)
-x DIRTYDIR (x) move processed dirty files to this directory
(default: ~/robust/malware/)
-r REPORTDIR (r) save reports to this directory
(default: ~/robust/reports/)
-z ERRORDIR (z) move error or skip files to this directory
(default: ~/robust/errors/)
-t {html,txt,xml,zip,json,ioc,stix,pdf,sample}
(t)ype of report requested
(default: None)
```

## robust-reporter

`robust-reporter` is a tool designed to quickly summarize the downloaded `*.json` files in your 'reports' directory.

Options

```
usage: robust-reporter.py [-h] [-r REPORTDIR] [--version] [-v | -q]

Robust McAfee ATD Python CLI tool

optional arguments:
-h, --help show this help message and exit
--version show program's version number and exit
-v, --verbosity increase output (v)erbosity
(default: None)
-q, --quiet (q)uiet all output
(default: False)

Reporter parameters:
-r REPORTDIR (r) reports are processed or stored using this directory
(default: ~/robust/reports/)
```
Sample Run

```
$ robust-reporter.py
82344C9864B0F1D120C0D1AB7F7C54C3 (---) : Somewhat/probably is suspicious(Low-Medium) - 24sec
D012492123E4CF0CFB3A017A2E92C077 (Malware.Dynamic) : Malicious(High) - 194sec
DB273A97C54E3E23F411EA7C9B5A82DA (Malware.Dynamic) : Malicious (Medium) - 53sec
165A36C02B3FAAF4DE38F93A3DCB821B (---) : Somewhat/probably is suspicious(Low-Medium) - 36sec
D10195670651A40C46C22972CD839E89 (Artemis!D10195670651) : Malicious (Very High) - 32sec
8271093E0E78574428BBDDDA6F34A980 (Malware.Dynamic) : Malicious(High) - 192sec
86DAFA0262BF217F5344A3B057C0DB06 (Malware.Dynamic) : Malicious(High) - 193sec
8DA4CDC3E2EE16021F237EA7A043DA8E (Malware.Dynamic) : Malicious(High) - 191sec
```

## Tunning for Linux File Watchers

### iNotify Tuning Parameters

The inotify(7) subsystem has three important tunings that impact robust's directory watching.

```
/proc/sys/fs/inotify/max_user_instances impacts how many different root dirs you can watch.
/proc/sys/fs/inotify/max_user_watches impacts how many dirs you can watch across all watched roots.
/proc/sys/fs/inotify/max_queued_events impacts how likely it is that your system will experience a notification overflow.
```

You obviously need to ensure that `max_user_instances` and `max_user_watches` are set so that the system is capable of keeping track of your files.

`max_queued_events` is important to size correctly; if it is too small, the kernel will drop events and robust won't be able to report on them. Making this value bigger reduces the risk of this happening.

# Developers

## Install Development

```
$(robust)> pip install -r devel-requirements.txt

```

## Development Tasks

```shell
(robust)$> invoke -l
Available tasks:

build Build the setup.py
clean Clean up docs, bytecode, and extras
codestats Run flake8 PeP8 tests for code stats
release ``version`` should be a string like '0.4' or '1.0'.
smell Run flake8 PeP8 tests
test Run Unit tests

```

### Running the Test Suite

Nose is run via `invoke test`

```
Clearing rm -rf build
Clearing rm -rf dist
Clearing rm -rf *.egg-info
Clearing rm -rf pyclient.log
Clearing rm -rf **/**/*.pyc
Clearing rm -rf **/*.pyc
Clearing rm -rf ./*.pyc
...................
-----------------------------------------------------------------------------
19 tests run in 0.3 seconds (19 tests passed)
117 E501 line too long (97 > 79 characters)
7 F401 'ratd' imported but unused
1 F841 local variable 'rb_rtnv' is assigned to but never used
1 N802 function name should be lowercase
5 W601 .has_key() is deprecated, use 'in'
```

### Additional LICENSE information

A modified Fork of `atdcli.py` (Carlos Munoz - 2014).

https://pypi.python.org/pypi/atd

## VX Workshop Appliance Option

There is a fully operational Xubuntu 14.04 liveCD that includes:

* robust - https://github.com/shadowbq/robust-atd
* maltrieve - https://github.com/shadowbq/maltrieve
* vxcage - https://github.com/shadowbq/vxcage

It also includes

* hexeditors
* static analysis tools
* google chrome
* vmtools
* etc..

xubuntu-14.04.4-desktop-x86_64-VX-Workshop-0.4.iso (~ 1.2 GB)

Available to download with READMEs here: https://goo.gl/flcvew


Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
robust_atd-0.5.2-py2-none-any.whl (32.0 kB) Copy SHA256 hash SHA256 Wheel py2
robust-atd-0.5.2.tar.gz (26.3 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page