robust-atd 0.4.1

Manipulate McAfee ATD appliance

Robust ATD CLI tools
================

"Robust" is a set of tools to leverage the HTTPS REST API of the [McAfee Advanced Threat Detection](http://www.mcafee.com/us/products/advanced-threat-defense.aspx) 3.8 - 4.x appliance.


## Important

This is *not a supported or official application of McAfee*. This work is based off of publicly available published documentation for integrating with the McAfee ATD REST API 3.6.x to 4.x

Official API Documentation is available here:

* https://support.mcafee.com/ServicePortal/faces/knowledgecenter?q=api&v=&p=Advanced+Threat+Defense

## McAfee ATD - Advanced Threat defense

McAfee ATD is a commercial grade enterprise security sandbox analysis appliance. It main function is to provide advanced detection for stealthy, zero-day malware. McAfee Advanced Threat Defense is available as an on-premises appliance or a virtual form factor, with support for both private and public cloud with availability in the Azure Marketplace.

* https://www.mcafee.com/us/products/advanced-threat-defense.aspx
* https://www.mcafee.com/us/resources/data-sheets/ds-advanced-threat-defense.pdf

## Install

Req: Python 2.7.x

~~pip install robust-atd~~

Note: robust-atd is not published on pypi at this time.

### PKG Download & Manual Install Alternative

Note: `python setup.py install` will attempt to install dependencies from the internet via `pip`.

For offline runtime installation, please download the pip packages listed in the `requirements.txt`.

### Virutalenv

It is recommended to install virtualenv & virtualenvwrapper via Virtualenv Burrito.

See: README_PYTHON_UP.md

```
$> mkvirtualenv robust
$> workon robust
$(robust)> wget https://github.com/shadowbq/robust-atd/archive/master.zip
$(robust)> unzip master.zip
$(robust)> cd master
$(robust)> python setup.py install
```

### Robust (DOT) Configuration file

Robust will use the `~\.robust` configuration file to load defaults into the scripts.

The configuration file is broken into multiple sections. If you use a section you must define all the settings in that section.

* [auth]
* [connection]
* [convict]

It is recommended to set the file to `read-write` only for the current user, and remove all world `(-)rwx` permissions.

Authentication Section `[auth]` :

```shell
$(robust)> cat ~/.robust
[auth]
user: admin
password: password.
```

Connection Detail Section `[connection]` :

```shell
$(robust)> cat ~/.robust
[connection]
ip: atd.localhost.localdomain
skipssl: true
maxthreads: 15
```

Data Storage Section `[storage]`:

Note: Datastorage locations will be created if they do not exist.

```shell
$(robust)> cat ~/.robust
[storage]
severity: 3
cleandir: ~/robust/clean
dirtydir: ~/robust/dirty
reportdir: ~/robust/reports
errordir: ~/robust/errors
```

This file is expanded via the `os` module is compliant with windows user directories.

## Tools Overview

* robust
* robust-profiles
* robust-search
* robust-watchdog
* robust-convict

## Robust:

Using `robust` for submitting samples.

```
usage: robust.py [-h] [-u USER] [-p PASSWORD] [-i ATD IP] [-n] -s
FILE_TO_UPLOAD -a ANALYZER_PROFILE [-v] [--version]

Robust McAfee ATD Python CLI tool

optional arguments:
-h, --help show this help message and exit
--version show program's version number and exit
-v, --verbosity increase output verbosity
(default: None)
-q, --quiet (q)uiet all output
(default: False)


Authentication parameters:
-u USER (u)sername for the API of the ATD
(default: admin)
-p PASSWORD (p)assword for username
(default: password.)
-i ATD IP (i)p or hostname address of ATD
(default: atd.localhost.localdomain)
-n do (n)ot verify the SSL certificate for the communications
(default: False)

Sample parameters:
-s FILE_TO_UPLOAD (s)ample or file to be analyzed
(default: None)
-a ANALYZER_PROFILE (a)nalyzer profile id to be used during analysis
(default: None)

Examples:
robust.py -u admin -p password. -i atd.localhost.localdomain -s /usr/local/bin/file_to_scan -a 1
```

### Submitting a Sample

A sample can be submitted via cli with full flags, `.robust` configuration file, or interrupt passwords.

```shell
$(robust)> robust.py -u admin -p password. -i atd.localhost.localdomain -s /home/malware/non-malicious-container/putty_upx_7.exe
```

Using interrupt (interactive) passwords:

```shell
$(robust)> robust-profiles.py -n -l
Password: <input password>
ATD profiles: 1
Profile id: 26
Name: Win XP Down Select (Online)
OS: winXPsp3
Run all down selects?: Off
******************
```

### Managing Outputs

Using System Return codes with `-q` Quiet output flag. When the quiet flag is
used for submitting samples or searching reports the *severity* of the application
is returned as a system exit/return code. Negative return codes indicate *faults*
or failure during submission.

```
(robust)>$ robust.py -n -a 26 -s ./.samples/Sample.exe -q
(robust)>$ echo $?
2
```

Common Fault codes:

```
-1 ---> Error connecting to the ATD Server
-2 ---> Error uploading file to the ATD Server
-3 ---> Analysis failed
-4 ---> Error getting report
-5 ---> Error Obtaining vmprofilelist
```

Malware ranking:
(If the severity level of the sample is 3 and above it is generally regarded a threat)
```
N/A -> Sample did not run
-1 --> Sample is white listed
0 ---> No malicious activity detected (None)
1 ---> Sample is slightly suspicious (Low)
2 ---> Sample is somewhat/probably is suspicious
3 ---> Sample is malicious (Medium)
4 ---> Sample is malicious
5 ---> Sample is malicious (Very High)
```
## robust-version-checker

You can quickly test your connection settings in the CLI.

```
$(robust)> robust-version-checker.py -u robust -p password. -i atd.example.com -n
Connection successful...

Session Value: g7aenj99pfp0gbrogfbqsd9085
User ID: 57
ATD ver: 4.2.2.16
ATD Box heartbeat: 1519939175
```

## robust-profiles

A tool designed to pull the *Analyzer Profile* policy list available to a specific user.

Pulling the Policy List - In order to submit a sample using `robust` you must identify the Analyzer Profile ID. `robust-profiles` assists in identifying the available profiles your user can submit samples to.

```
$(robust)> robust-profiles.py -n -l
ATD profiles: 10
Profile id: 1
Name: Android
OS: android
Run All Selected?: Off
******************
Profile id: 26
Name: Win XP Down Select (Online)
OS: winXPsp3
Run All Selected?: Off
******************
Profile id: 25
Name: Windows XP Full Run (Offline)
OS: winXPsp3
Run All Selected?: On
******************
Profile id: 24
Name: Windows XP Full Run (Online)
OS: winXPsp3
Run All Selected?: On
******************
```

## robust-search

A tool designed to search and return reports for a specific md5 hash.

```shell
(robust)$> $ robust-search.py -m 2F7568342339CDB8321B52FF7BEBE661 -n
(Sample.exe:2F7568342339CDB8321B52FF7BEBE661) = 2: "Sample probably is suspicious"
```
### Help details

`robust-search` has the options `-f` and `-t` to collect the proper report on the submission.

```
usage: robust-search.py [-h] [-u USER] [-p PASSWORD] [-i ATD IP] [-n] -m MD5
[-t {html,txt,xml,zip,json,ioc,stix,pdf,sample}]
[-f FILENAME] [--version] [-v | -q]

Robust McAfee ATD Python CLI tool

optional arguments:
-h, --help show this help message and exit
--version show program's version number and exit
-v, --verbosity increase output (v)erbosity
(default: None)
-q, --quiet (q)uiet all output
(default: False)

Authentication parameters:
-u USER (u)sername for the API of the ATD
(default: admin)
-p PASSWORD (p)assword for username
(default: password.)
-i ATD IP (i)p or hostname address of ATD
(default: atd.localhost.localdomain)
-n do (n)ot verify the SSL certificate for the communications
(default: False)

Search parameters:
-m MD5 (m)d5 32bit hash of the sample to search
(default: None)

Reporting parameters:
-t {html,txt,xml,zip,json,ioc,stix,pdf,sample}
(t)ype of report requested
(default: None)
-f FILENAME (f)ilename for saving the requested report
(default: None)
```


## robust-watchdog

A tool that watches a directory recursively for any new files to submit.

Example CLI
```
usage: robust-watchdog.py [-h] -u USER [-p PASSWORD] -i ATD IP [-n] -a
ANALYZER_PROFILE -d DIRECTORY [-e] [-j MAXTHREADS]
[--version] [-v | -q]

Robust McAfee ATD Python CLI tool

optional arguments:
-h, --help show this help message and exit
--version show program's version number and exit
-v, --verbosity increase output (v)erbosity
(default: None)
-q, --quiet (q)uiet all output
(default: False)

Authentication parameters:
-u USER (u)sername for the API of the ATD
(default: None)
-p PASSWORD (p)assword for username
(default: None)
-i ATD IP (i)p or hostname address of ATD
(default: None)
-n do (n)ot verify the SSL certificate for the communications
(default: False)

Watch parameters:
-a ANALYZER_PROFILE (a)nalyzer profile id to be used during analysis
(default: None)
-d DIRECTORY (d)irectory to watch for events
(default: None)
-e (e)xisting files in directory will be submitted
(default: False)
-j MAXTHREADS (j) max number of threads
(default: 1)
```

Let it run in a shell and open another one or the file browser to create files in the /path/to/directory. Since the handler is printing the results, the output will reflect the flags chosen similar to `robust.py`:

The `-e` flag can be passed to cause all existing files in the directory (recurisively) to be submitted upon start.

```shell
(robust)$> robust-watchdog.py -a 26 -d ./ -n -e
.
...
.
.....
```

## robust-convict

`robust-convict` is a tool designed like `robust-watchdog` but its purpose is to help sort large directories of malware samples into directories, while downloading their corresponding reports.

Example Usage

```
robust-convict.py -n -a 26 -c ./tmp/clean/ -x ./tmp/dirty/ -r ./tmp/reports/ -z ./tmp/errors/ -d ./tmp/preprocess -j 10 -t zip -q
```

Options

```
usage: robust-convict.py [-h] [-u USER] [-p PASSWORD] [-i ATD IP] [-n] -a
ANALYZER_PROFILE -d DIRECTORY [-e] [-y SEVERITY]
[-c CLEANDIR] [-x DIRTYDIR] [-r REPORTDIR]
[-z ERRORDIR]
[-t {html,txt,xml,zip,json,ioc,stix,pdf,sample}]
[-j MAXTHREADS] [--version] [-v | -q]

Robust McAfee ATD Python CLI tool

optional arguments:
-h, --help show this help message and exit
--version show program's version number and exit
-v, --verbosity increase output (v)erbosity
(default: None)
-q, --quiet (q)uiet all output
(default: False)

Authentication parameters:
-u USER (u)sername for the API of the ATD
(default: robust)
-p PASSWORD (p)assword for username
(default: ****<.robust>*****)
-i ATD IP (i)p or hostname address of ATD
(default: atd.localhost.localdomain)
-n do (n)ot verify the SSL certificate for the communications
(default: True)

Watch parameters:
-a ANALYZER_PROFILE (a)nalyzer profile id to be used during analysis
(default: None)
-d DIRECTORY (d)irectory to watch for events
(default: None)
-e (e)xisting files in directory will be submitted
(default: False)
-j MAXTHREADS (j) max number of threads
(default: 1)

Convict parameters:
-y SEVERITY (y) treat sample as dirty with this severity [0-5] or higher
(default: 3)
-c CLEANDIR (c) move clean files to this directory
(default: ~/robust/clean/)
-x DIRTYDIR (x) move processed dirty files to this directory
(default: ~/robust/malware/)
-r REPORTDIR (r) save reports to this directory
(default: ~/robust/reports/)
-z ERRORDIR (z) move error or skip files to this directory
(default: ~/robust/errors/)
-t {html,txt,xml,zip,json,ioc,stix,pdf,sample}
(t)ype of report requested
(default: None)
```

## robust-reporter

`robust-reporter` is a tool designed to quickly summarize the downloaded *.json files in your 'reports' directory.

Options

```
usage: robust-reporter.py [-h] [-r REPORTDIR] [--version] [-v | -q]

Robust McAfee ATD Python CLI tool

optional arguments:
-h, --help show this help message and exit
--version show program's version number and exit
-v, --verbosity increase output (v)erbosity
(default: None)
-q, --quiet (q)uiet all output
(default: False)

Reporter parameters:
-r REPORTDIR (r) reports are processed or stored using this directory
(default: ~/robust/reports/)
```
Sample Run

```
$ robust-reporter.py
82344C9864B0F1D120C0D1AB7F7C54C3 (---) : Somewhat/probably is suspicious(Low-Medium) - 24sec
D012492123E4CF0CFB3A017A2E92C077 (Malware.Dynamic) : Malicious(High) - 194sec
DB273A97C54E3E23F411EA7C9B5A82DA (Malware.Dynamic) : Malicious (Medium) - 53sec
165A36C02B3FAAF4DE38F93A3DCB821B (---) : Somewhat/probably is suspicious(Low-Medium) - 36sec
D10195670651A40C46C22972CD839E89 (Artemis!D10195670651) : Malicious (Very High) - 32sec
8271093E0E78574428BBDDDA6F34A980 (Malware.Dynamic) : Malicious(High) - 192sec
86DAFA0262BF217F5344A3B057C0DB06 (Malware.Dynamic) : Malicious(High) - 193sec
8DA4CDC3E2EE16021F237EA7A043DA8E (Malware.Dynamic) : Malicious(High) - 191sec
```

## Tunning for Linux File Watchers

### iNotify Tuning Parameters

The inotify(7) subsystem has three important tunings that impact robust's directory watching.

```
/proc/sys/fs/inotify/max_user_instances impacts how many different root dirs you can watch.
/proc/sys/fs/inotify/max_user_watches impacts how many dirs you can watch across all watched roots.
/proc/sys/fs/inotify/max_queued_events impacts how likely it is that your system will experience a notification overflow.
```

You obviously need to ensure that `max_user_instances` and `max_user_watches` are set so that the system is capable of keeping track of your files.

`max_queued_events` is important to size correctly; if it is too small, the kernel will drop events and robust won't be able to report on them. Making this value bigger reduces the risk of this happening.

# Developers

## Install Development

```
$(robust)> pip install -r devel-requirements.txt

```

## Development Tasks

```shell
(robust)$> invoke -l
Available tasks:

build Build the setup.py
clean Clean up docs, bytecode, and extras
codestats Run flake8 PeP8 tests for code stats
release ``version`` should be a string like '0.4' or '1.0'.
smell Run flake8 PeP8 tests
test Run Unit tests

```

### Running the Test Suite

Nose is run via `invoke test`

```
Clearing rm -rf build
Clearing rm -rf dist
Clearing rm -rf *.egg-info
Clearing rm -rf pyclient.log
Clearing rm -rf **/**/*.pyc
Clearing rm -rf **/*.pyc
Clearing rm -rf ./*.pyc
...................
-----------------------------------------------------------------------------
19 tests run in 0.3 seconds (19 tests passed)
117 E501 line too long (97 > 79 characters)
7 F401 'ratd' imported but unused
1 F841 local variable 'rb_rtnv' is assigned to but never used
1 N802 function name should be lowercase
5 W601 .has_key() is deprecated, use 'in'
```

### Additional LICENSE information

A modified Fork of `atdcli.py` (Carlos Munoz - 2014).

https://pypi.python.org/pypi/atd

## VX Workshop Appliance Option

There is a fully operational Xubuntu 14.04 liveCD that includes:

* robust - https://github.com/shadowbq/robust-atd
* maltrieve - https://github.com/shadowbq/maltrieve
* vxcage - https://github.com/shadowbq/vxcage

It also includes

* hexeditors
* static analysis tools
* google chrome
* vmtools
* etc..

xubuntu-14.04.4-desktop-x86_64-VX-Workshop-0.4.iso (~ 1.2 GB)

Available to download with READMEs here: https://goo.gl/flcvew