search-vulns 0.8.2

Search for known vulnerabilities in software using software titles or a CPE 2.3 string.

search_vulns

Search for known vulnerabilities in software using software titles or a CPE 2.3 string.

About

search_vulns can be used to search for known vulnerabilities in software. To achieve this, the tool utilizes a locally built vulnerability database, currently containing:

• CVE information from the National Vulnerability Database (NVD)
• Enhanced NVD information from VulnCheck NVD++
• Exploit information from the Exploit-DB (EDB)
• Exploit information from PoC-in-GitHub
• Vulnerability information from the GitHub Security Advisory Database
• Software currency information from endoflife.date
• Backpatch information from the Debian Security Bug Tracker

Since search_vulns is designed in a modular fashion, new data sources and extensions can be integrated easily.

Using the search_vulns tool, this local information can be queried, either by providing software titles like 'Apache 2.4.39' or by providing a CPE 2.3 string like cpe:2.3:a:sudo_project:sudo:1.8.2:*:*:*:*:*:*:*. You can also search for vulnerabilities like CVE-2023-1234 or GHSA-xx68-jfcg-xmmf directly by using a comma-separated list of IDs.

search_vulns can either be used as a CLI tool or via a web server. It is recommended to use the CLI tool for automated workflows that might be resource-constrained. Otherwise, using the web server is recommended, because it offers more features and flexibility. This includes the ability to achieve more complete results. Also, the presentation of results is clearer and results can be exported for further use.

Quick Links

• Public instance of the web server: https://search-vulns.com
• Black Hat Arsenal recorded demo
• The Surprising Complexity of Finding Known Vulnerabilities - A blog post detailing the challenges and motivations behind search_vulns.
• search_vulns: Simplifying the Surprising Complexity of Finding Known Vulnerabilities - A blog post introducing search_vulns and describing its features.
• search_vulns: A Deep Dive into its Technologies and Approaches - A blog post detailing how search_vulns works on a technical level, including its novel approaches.

Installation

The core of search_vulns can be installed as a lightweight Python package, optionally with a web server component. An extended installation can be performed, which enables you to build the local databases yourself instead of pulling them from the latest release on GitHub and to use MariaDB as database backend. As of now, there are no other functional differences.

Lightweight Python Package Installation

To install search_vulns, you have to have Python and pip installed beforehand. Then you can install the search_vulns Python package from PyPI like so:

pip install search_vulns

Note that you may have to include --break-system-packages, or use a virtualenv or pipx.

To install the required packages for the optional web server component, you can run:

pip install search_vulns[web]

You can also clone this repository, build the Python package yourself and keep all data editable and in the cloned repository (beneficial for development purposes):

git clone https://github.com/ra1nb0rn/search_vulns
pip install -e .

After installing search_vulns, you need to pull the prebuilt database files from GitHub like so:

search_vulns -u

Lastly, you can run search_vulns or start the search_vulns web server if the web dependencies are installed:

$ search_vulns -q 'jquery 2.1.3'
$ python3 -m search_vulns.web_server

Complete / Full Installation of search_vulns

You can perform a full installation like so (see notes above regarding pip installation):

pip install search_vulns
search_vulns --full-install

Note that this installs some required system packages, as specified in the install.sh files throughout the code.

Thereafter, you can download the database files from GitHub as shown above or build the databases yourself:

search_vulns --full-update

Note, however, that you should supply API keys via a config file, e.g. src/search_vulns/config.json or as environment variables. The API keys are free and you just need to register with the NVD or VulnCheck, for example.

Dockerfile

There's also a Dockerfile you can use:

docker build -t search_vulns .
docker run -p 127.0.0.1:5000:5000 -it search_vulns bash

The port forwarding is optional, in case you do not intend on using the web server component. If you do, make sure to adjust the listening socket at the end of src/search_vulns/web_server.py accordingly.

Usage

search_vulns's usage information is shown in the following:

usage: search_vulns [-h] [-u] [--full-update] [--full-install] [-a] [-f {json,txt}] [-o OUTPUT]
                    [-q QUERY] [-c CONFIG] [-V] [--cpe-search-threshold CPE_SEARCH_THRESHOLD]
                    [--ignore-general-product-vulns] [--include-single-version-vulns]
                    [--use-created-product-ids] [--include-patched]

Search for known vulnerabilities in software -- Created by Dustin Born (ra1nb0rn)

options:
  -h, --help            show this help message and exit
  -u, --update          Download the latest version of the the local vulnerability and software database
  --full-update         Fully (re)build the local vulnerability and software database
  --full-install        Fully install search_vulns, including all dependencies (python packages, system
                        packages etc.)
  -a, --artifacts       Print JSON list of artifacts created during full update
  -f {json,txt}, --format {json,txt}
                        Output format, either 'txt' or 'json' (default: 'txt')
  -o OUTPUT, --output OUTPUT
                        File to write found vulnerabilities to
  -q QUERY, --query QUERY
                        A query, either software title like 'Apache 2.4.39' or a product ID string (e.g.
                        CPE 2.3)
  -c CONFIG, --config CONFIG
                        A config file to use (default: config.json)
  -V, --version         Print the version of search_vulns
  --cpe-search-threshold CPE_SEARCH_THRESHOLD
                        Similarity threshold used for retrieving a CPE via the cpe_search tool
  --ignore-general-product-vulns
                        Ignore vulnerabilities that only affect a general product (i.e. without version)
  --include-single-version-vulns
                        Include vulnerabilities that only affect one specific version of a product when
                        querying a lower version
  --use-created-product-ids
                        If no matching product ID exists in the software database, automatically use
                        matching ones created by search_vulns
  --include-patched     Include vulnerabilities reported as (back)patched, e.g. by Debian Security
                        Tracker, in results

Note that when querying software with -q you have to put the software information in quotes if it contains any spaces. Also, you can use -q multiple times to make multiple queries at once. For one, a query can be a software name / title like 'Apache 2.4.39' or 'Wordpress 5.7.2'. Furthermore, a query can also be a CPE 2.3 string.

Here are some examples:

• Query Sudo 1.8.2 for known vulnerabilities:

  $ search_vulns -q 'Sudo 1.8.2'
  
  [+] Sudo 1.8.2 (cpe:2.3:a:sudo_project:sudo:1.8.2:*:*:*:*:*:*:*/cpe:2.3:a:todd_miller:sudo:1.8.2:*:*:*:*:*:*:*)
  CVE-2019-14287 (CVSSv3.1/8.8): In Sudo before 1.8.28, an attacker with access to a Runas ALL sudoer account can bypass certain policy blacklists and session PAM modules, and can cause incorrect logging, by invoking sudo with a crafted user ID. For example, this allows bypass of !root configuration, and USER= logging, for a "sudo -u \#$((0xffffffff))" command.
  Exploits:  https://www.exploit-db.com/exploits/47502
             [...]
  Reference: https://nvd.nist.gov/vuln/detail/CVE-2019-14287, 2019-10-17
  CVE-2017-1000368 (CVSSv3.0/8.2): Todd Miller's sudo version 1.8.20p1 and earlier is vulnerable to an input validation (embedded newlines) in the get_process_ttyname() function resulting in information disclosure and command execution.
  Reference: https://nvd.nist.gov/vuln/detail/CVE-2017-1000368, 2017-06-05
  [...]
  

• Query Moodle 3.4.0 for known vulnerabilities:

  $ search_vulns -q 'Moodle 3.4.0'
  
  [+] Moodle 3.4.0 (cpe:2.3:a:moodle:moodle:3.4.0:*:*:*:*:*:*:*)
  CVE-2018-14630 (CVSSv3.0/8.8): moodle before versions 3.5.2, 3.4.5, 3.3.8, 3.1.14 is vulnerable to an XML import of ddwtos could lead to intentional remote code execution. When importing legacy 'drag and drop into text' (ddwtos) type quiz questions, it was possible to inject and execute PHP code from within the imported questions, either intentionally or by importing questions from an untrusted source.
  Reference: https://nvd.nist.gov/vuln/detail/CVE-2018-14630, 2018-09-17
  CVE-2018-1133 (CVSSv3.0/8.8): An issue was discovered in Moodle 3.x. A Teacher creating a Calculated question can intentionally cause remote code execution on the server, aka eval injection.
  Exploits:  https://www.exploit-db.com/exploits/46551
  Reference: https://nvd.nist.gov/vuln/detail/CVE-2018-1133, 2018-05-25
  [...]
  

Modularity

search_vulns' search engine is designed in a modular manner. Therefore, new databases can be integrated easily. For example, modules can help in finding product IDs, vulnerabilities, extra information about vulnerabilities and extra information about the queried product. Examples of the latter two are CVSS scores or software recency information. Furthermore, modules can classify identified vulnerabilities as patched if they store and utilize special information related to the query, for example.

Have a look at the template module to get started writing your own modules: src/search_vulns/modules/template/search_vulns_template.py.

Running a Web Server

It is also possible to run a web server that provides this tool's functionality to clients over the network. src/search_vulns/web_server.py contains a working example using Flask. Depending on your environment, you may want to modify the server IP and port at the end of this file. To run a simple Flask web server, just run:

python3 -m search_vulns.web_server

Furthermore, you can use gunicorn to make the web server more scalable; for example by running:

gunicorn --worker-class=gevent --worker-connections=50 --workers=3 --bind '0.0.0.0:8000' search_vulns.wsgi:app

You can read more about choosing good gunicorn settings for your system here. Note, however, that this tool is quite CPU intensive, meaning that scalability is somewhat limited.

Finally, you can also use Nginx as a reverse proxy. A sample configuration file is provided in web_server_files/nginx.conf.sample. Again, you may have to adjust this to your needs. When using Nginx, make sure you have the app running at the configured endpoint(s). For the sample configuration file, for example, you would have to run something similar to the following:

gunicorn --worker-class=gevent --worker-connections=50 --workers=3 --bind 'unix:/tmp/gunicorn.sock' search_vulns.wsgi:app

MariaDB as Alternative Database Option

search_vulns can be configured to use MariaDB as an alternative to the preconfigured SQLite mechanism. A sample configuration file for MariaDB is provided in src/search_vulns/config_mariadb.json.

Make sure that you adjust the values for MariaDB in the configuration file to your MariaDB deployment (user, password, host and port).

To use MariaDB instead of SQLite for the webserver, simply change the CONFIG_FILE variable in web_server.py to your config file (e.g. src/search_vulns/config_mariadb.json).

To improve the performance of search_vulns with MariaDB, it is recommend to add the following settings to your MariaDB configuration file (e.g. /etc/mysql/my.cnf):

[mariadb]
query_cache_type = 1
query_cache_size = 192M
innodb_buffer_pool_size = 8G
thread_handling = pool-of-threads

innodb_buffer_pool_size should be set to approximately 80% of available memory (see MariaDB's official documentation).

License

search_vulns is licensed under the MIT license, see here.

View the licenses of the included data sources here.