faraday-agent-dispatcher 2.3.0

Faraday agent dispatcher to communicate an agent to faraday

Faraday Agents Dispatcher helps user develop integrations with Faraday written in any language.

Installation

Just run pip3 install faraday_agent_dispatcher and you should see the faraday-dispatcher command in your system.

To setup a development environment (this is, to change code of the dispatcher itself, not to write your own integrations), you should clone this repo and run pip install -e ..

Running Faraday Agent Dispatcher for first time

1. Generate a configuration file running faraday-dispatcher config-wizard.

2. Run the agent with faraday-dispatcher run command. The config file that it creates will be located in ~/.faraday/config/dispatcher.ini if you do not pass a custom path.

You should complete the agent configuration with your registration token, located at http://localhost:5985/#/admin/agents. Check that the server section has the correct information about your Faraday Server instance. Then, complete the agent section with the desired name of your agent. Finally, add the executors

Executors

Creating your own executors

An executor is a script that prints out single-line JSON data to stdout. Remember that if you print any other data to stdout, the dispatcher will trigger an error. If you want to print debugging or logging information you should use stderr for that.

Every line written to stdout by the executor will be decoded by the dispatcher and sent to Faraday using the Bulk Create endpoint. Therefore, the JSON you print must have the schema that the endpoint requires (this schema is detailed below). Otherwise, the dispatcher will complain because you supplied invalid data to it.

If you want to debug your executor, the simplest way to do it is by running it directly instead of running with the Dispatcher. Since the executor just prints JSON data to stdout, you will be able to see all information it wants to send to Faraday, but without actually sending it.

Configuring a executor

After writing your executor, you have to add it with the faraday-dispatcher config-wizard within the executor section, adding its name, command to execute and the max size of the JSON to send to Faraday Server. Additionally, you can configure the Environment variables and Arguments in their proper section.

Running a executor

To run an executor use the faraday-dispatcher config-wizard command, and play it from the Faraday Server. The executor will use the environment variables set and ask for the arguments.

Bulk Create JSON format

The data published to faraday must correspond to the bulk_create endpoint of the Faraday's REST API

Running multiple dispatchers

If you want to have more than one dispatcher, each one runninng its own executors, the preferred of doing this is to create different configuration files for each one (for example, ~/.faraday/config/dispatcher-1.ini and ~/.faraday/config/dispatcher-2.ini). Then, you can run two different Dispatcher instances with faraday-dispatcher --config-file PATH_TO_A_CONFIG_FILE.

Executors

Inside the executors directory you can find the already created executors.

Official

The official executors are the collection of ready-to-go executors (with minimum configuration with the wizard). They have a manifest JSON file, which gives details about the uses of the executor and helps with the configuration of them.

The current official executors are:

• Arachni
• Burp
• CrackMapExec
• Nessus
• Nikto
• Nmap
• Nuclei
• Openvas
• Report processor: Consumes a local report where the dispatcher is with the faraday plugins
• Sublist3r
• W3af
• Wpscan
• Zap

Development

The development executors are the collection of executors we do not fully maintain, we have examples of use, conceptual, and in development executors. The most important of them are:

• basic_example.py: The Hello World of Faraday executors. It will create a host with an associeted vulnerability to it
• heroku_discovery_agent.py: Load host and service information from your Heroku account
• prowlerSample.py: Run the prowler command and send its output to Faraday
• brainfuck.sh: A proof-of-concept to demonstrate you can create an executor in any programming language, including Brainfuck!

Roadmap

We are currently working on new executors, apart from improving the experience using the agents.

We would like to give some agents read access to their workspace, so they can benefit of the existing data in order to find more valuable information.

Documentation

For more info you can check our documentation

2.3.0 [Sep 5th, 2022]:

• Now InsighVM's executer will executa a scan if a site_id is provided
• Add tags for plugins
• Add installation in docker file for nmap script: vulners
• Now the api-key from zap is a enviroment variable
• [FIX] Change AGENT_CONFIG_HOSTNAME_RESOLUTION por AGENT_CONFIG_RESOLVE_HOSTNAME
• Update docs

2.2.0 [Jul 25th, 2022]:

• Add timeout parameter to arachni's executor
• Add python2.7, w3af and its dependencies to docker image
• Add ignore_info and hostname_resolution options for most executors.
• Nessus now list in the logs the available templates and uses posixpath.join instead of concat strings. Nikto now uses only requieres TARGET_URL argument.
• Fix logs and change .format to fstrings
• Remove ws from dispatcher.yaml.
• Now faraday-dispatcher send the parameters of the executors when it connects to faraday server. Also it checks if there are new enviroment variables defined in the manifest file and warn the user.

2.1.3 [Dec 13th, 2021]:

• Add --api-token --random-user-agent to wpscan
• Move shodan executor to official and change logic to work with plugins

2.1.2 [Oct 19th, 2021]:

• ADD script to nmap logic

2.1.1 [Aug 18th, 2021]:

• ADD option via configuration YAML file to ignore ssl errors
• MOD Wizard connection ports defaults vary if SSL value has changed in the previous configuration
• Faraday versions: 3.16.0, 3.16.1, 3.16.2, 3.17.0, 3.17.1, 3.17.2

2.1.0 [Aug 9th, 2021]:

• ADD Reminder message to run --token command after wizard
• FIX start_date and end_date to be sent as UTC
• FIX Receiving API error when faraday license is expired
• ADD Executor for insightvm
• REMOVE Host and api from burp executor parameters
• Faraday versions: 3.16.0, 3.16.1, 3.16.2, 3.17.0, 3.17.1, 3.17.2

2.0.0 [Jun 29th, 2021]:

• ADD Executor parameter typing
• ADD versioning for manifests from typing package
• FIX typo in wizard
• Faraday versions: 3.16.0, 3.16.1, 3.16.2, 3.17.0, 3.17.1, 3.17.2

1.5.1 [May 6th, 2021]:

• FIX Burp executor parse the IP
• Faraday versions: 3.14.3, 3.15.0, 3.15.1

1.5.0 [Mar 30th, 2021]:

• ADD having at least a executor is mandatory, if not it will not save the configuration
• UPD executor pagination, now each executor have a "unique" id
• MOD Update all reference to faraday to API v3
• MOD Connectivity endpoint is now /_api/v3/info
• MOD Now registration token is needed within the run command. Only needed the first time
• MOD setting host in the wizard now accepts full urls, such as https://my.server.com:12345
• ADD new plugin to support newer OpenVas/gvm versions (gvm_openvas). The old openvas executor was renamed to "openvas_legacy"
• Faraday versions: 3.14.3, 3.15.0, 3.15.1

1.4.2 [Feb 26th, 2021]:

• MOD Update faraday-plugins version, improving nessus plugin process
• Faraday versions: 3.14.0, 3.14.1, 3.14.2

1.4.1 [Feb 17th, 2021]:

• MOD Various UX improves in wizard:
  • ADD special character control in name executor
  • ADD More verbose info
  • It is possible to exit wizard if its misconfigurated (will not be saved)
  • FIX Not choosing executor (Using Q) generates correct config file
  • MOD max data sent to server option is a manual edit configuration
  • MOD more extensive default list of official executors
  • MOD change color for options "next page" "don't choose"
• ADD new WPScan executor that does not need docker anymore
• FIX in nuclei_exclude parameter for nuclei executor
• Faraday versions: 3.14.0, 3.14.1, 3.14.2

1.4.0 [Dec 23rd, 2020]:

• A base_route can be added before the root of the server (e.g: https://my.company.com/faraday/ as / of faraday)
• Add duration to bulk_create to be set correctly
• The new official executors are:
  • nuclei
  • reports: local report consumed by faraday-plugins
• Add new flags for nmap executor: -sC,-sV,-Pn,--script-timeout,--host-timeout,--top-ports
• Fix bug nmap and nessus executors to execute with the dispatcher environment
• Fix nmap executor when http(s) scheme passed as target
• Faraday versions: 3.14.0, 3.14.1, 3.14.2

1.3.1 [Sep 7th, 2020]:

• Add proxy setup by HTTP_PROXY or HTTPS_PROXY environment variables
• Fix default report name with the nessus executor
• Faraday versions: 3.12.0

1.3.0 [Sep 3rd, 2020]:

• An Agent can post data to multiples workspaces
• The run command tries to migrate the configuration to the latest version from others as the config-wizard does
• Improve agent signal management and server disconnection, affecting the exit code
• The wizard page size can be customized (See: faraday-dispatcher config-wizard --help )
• The new official executors are:
  • burp
  • crackmapexec
• Arachni executor generates reports in /tmp now
• Nmap executor updates use of nmap plugin (byte-string to string response)
• Faraday versions: 3.12.0

1.2.1 [Jun 22nd, 2020]:

• Now the dispatcher runs the check commands before running an executor
• Fix error when connects with faraday fails when trying to access with SSL to not SSL server
• Fix error when connects with faraday fails when server does not respond
• Fix error when connects with faraday fails when SSL verification fails
• Fix error attempting to create an executor with a comma in the name
• Now the wizard ask if you want use the default SSL behavior
• Started the process of documentation
• The new official executors are:
  • arachni
  • openvas
  • zap
• Nmap executor now acepted multi target
• Fix W3af executor now uses python2
• Escape user-controlled executor parameters in order to prevent OS argument injection (not command injection)
• Faraday versions: 3.11, 3.11.1, 3.11.2

1.2 [May 27th, 2020]:

• Now we have official executors, packaged with the dispatcher
• Fix error when killed by signal
• Fix error when server close connection
• Fix error when ssl certificate does not exists
• Fix error when folder ~/.faraday does not exists, creating it
• The new official executors are:
  • nessus
  • nikto
  • nmap
  • sublist3r
  • wpscan
  • w3af
• Faraday versions: 3.11, 3.11.1, 3.11.2

1.1 [Apr 22th, 2020]:

• The dispatcher now runs with a faraday-dispatcher run command
• faraday-dispatcher wizard command added which generates configuration .ini file
• Manage execution_id within WS and API communication
• The route of Faraday ws comunication change from / to /websockets
• Better error management, now shows error and exceptions depending on log levels
• Better management of invalid token errors
• Add ssl support
• Faraday versions: 3.11, 3.11.1, 3.11.2

1.0 [Dec 17th, 2019]:

• You can add fixed parameters than shouldn't came by the web (e.g. passwords) are set in the dispatcher.ini
• Now its possible to manage multiple executors within one agent
• Now is possible to receive params from the Faraday server
• Faraday versions: 3.10, 3.10.1, 3.10.2

0.1 [Oct 31th, 2019]:

• First beta version published
• Basic structure implemented, with executor with fixed values
• Faraday versions: 3.9.2, 3.9.3