Skip to main content

AppThreat's vulnerability database and package search library with a built-in sqlite based storage. OSV, CVE, GitHub, npm are the primary sources of vulnerabilities.

Project description

Introduction

This repo is a vulnerability database and package search for sources such as AppThreat vuln-list, OSV, NVD, and GitHub. Vulnerability data are downloaded from the sources and stored in a sqlite based storage with indexes to allow offline access and efficient searches.

Why vulnerability db?

A good vulnerability database must have the following properties:

Multiple upstream sources are used by vdb to improve accuracy and reduce false negatives. SQLite database containing data in CVE 5.1 schema format is precompiled and distributed as files via ghcr to simplify download. With automatic purl prefix generation even for git repos, searches on the database can be performed with purl, cpe, or even http git url string. Every row in the database uses an open specification such as CVE 5.0 or Package URL (purl and vers) thus preventing the possibility of vendor lock-in.

Vulnerability Data sources

  • Linux vuln-list (Forked from AquaSecurity)
  • OSV (1)
  • NVD
  • GitHub

1 - We exclude Linux and oss-fuzz feeds by default. Set the environment variable OSV_INCLUDE_FUZZ=true to include them. 2 - Malware feeds are included by default, thus increasing the db size slightly. Set the environment variable OSV_EXCLUDE_MALWARE=true to exclude them.

Linux distros

  • AlmaLinux
  • Debian
  • Alpine
  • Amazon Linux
  • Arch Linux
  • RHEL/CentOS
  • Rocky Linux
  • Ubuntu
  • OpenSUSE
  • Photon
  • Chainguard
  • Wolfi OS

Installation

pip install appthreat-vulnerability-db>=6.2.0

To install vdb with optional dependencies such as oras use the [oras] or [all] dependency group.

pip install appthreat-vulnerability-db[all]

NOTE: VDB v6 is a major rewrite to use SQLite database. Current users of depscan v5 must continue using version 5.8.x

pip install appthreat-vulnerability-db==5.8.0

Usage

This package is ideal as a library for managing vulnerabilities. This is used by owasp-dep-scan, a free open-source dependency audit tool. However, there is a limited cli capability available with few features to test this tool directly.

Option 1: Download pre-built database (Recommended)

To download a pre-built SQLite database (refreshed every 6 hours) containing all application vulnerabilities (~ 700MB). This step is recommended for all users.

# pip install appthreat-vulnerability-db[all]
vdb --download-image

You can execute this command daily or when a fresh database is required.

To perform containers and OS scans, download the full image (~ 7.5GB) which includes all application and OS vulnerabilities.

vdb --download-full-image

Use any sqlite browser or cli tools to load and query the two databases.

data.index.vdb6 - index db with purl prefix and vers

index

data.vdb6 - Contains source data in CVE 5.1 format stored as a jsonb blob.

database

Option 2: Download pre-built database (ORAS)

Using ORAS cli might be slightly faster.

export VDB_HOME=$HOME/vdb
oras pull ghcr.io/appthreat/vdbxz:v6.4.x -o $VDB_HOME
tar -xvf *.tar.xz
rm *.tar.xz

Option 3: Use HuggingFace cli

Download one of the databases.

pip install -U "huggingface_hub[cli]"

app only database

export VDB_HOME=$(pwd)/app
huggingface-cli download AppThreat/vdb --include "app/*.vdb6" --repo-type dataset --local-dir .

app only 10 year database

export VDB_HOME=$(pwd)/app-10y
huggingface-cli download AppThreat/vdb --include "app-10y/*.vdb6" --repo-type dataset --local-dir .

app and os database

export VDB_HOME=$(pwd)/app-os
huggingface-cli download AppThreat/vdb --include "app-os/*.vdb6" --repo-type dataset --local-dir .

app and os 10 year database

export VDB_HOME=$(pwd)/app-os-10y
huggingface-cli download AppThreat/vdb --include "app-os-10y/*.vdb6" --repo-type dataset --local-dir .

Citation

Use the below citation in your research.

@misc{vdb,
  author = {Team AppThreat},
  month = Feb,
  title = {{AppThreat vulnerability-db}},
  howpublished = {{https://huggingface.co/datasets/AppThreat/vdb}},
  year = {2025}
}

Option 4: Manually create the vulnerability database (ADVANCED users)

Cache application vulnerabilities

vdb --cache

The typical size of this database is over 700 MB.

Cache from just OSV

vdb --cache --only-osv

It is possible to customize the cache behavior by increasing the historic data period to cache by setting the following environment variables.

  • NVD_START_YEAR - Default: 2018. Supports up to 2002
  • GITHUB_PAGE_COUNT - Default: 2. Supports up to 20

Cache application and OS vulnerabilities

vdb --cache-os

Note the size of the database with OS vulnerabilities is over 7.5 GB. It is possible to ignore/exlude specific OS distros using environment variables.

Example to ignore almalinux and ubuntu data from getting included, set the below environment variables:

export VDB_IGNORE_ALMALINUX=true
export VDB_IGNORE_UBUNTU=true

Refer to the variable LINUX_DISTRO_VULN_LIST_PATHS in config.py for the full list of distro strings supported.

CLI Usage

usage: vdb [-h] [--clean] [--cache] [--cache-os] [--only-osv] [--only-aqua] [--only-ghsa] [--search SEARCH] [--list-malware] [--bom BOM_FILE]

AppThreat's vulnerability database and package search library with a built-in sqlite based storage.

options:
  -h, --help       show this help message and exit
  --clean          Clear the vulnerability database cache from platform specific user_data_dir.
  --cache          Cache vulnerability information in platform specific user_data_dir.
  --cache-os       Cache OS vulnerability information in platform specific user_data_dir.
  --only-osv       Use only OSV as the source. Use with --cache.
  --only-aqua      Use only Aqua vuln-list as the source. Use with --cache.
  --only-ghsa      Use only recent ghsa as the source. Use with --cache.
  --search SEARCH  Search for the package or CVE ID in the database. Use purl, cpe, or git http url.
  --list-malware   List latest malwares with CVE ID beginning with MAL-.
  --bom BOM_FILE   Search for packages in the CycloneDX BOM file.
  --download-image      Downloaded pre-created vdb image to platform specific user_data_dir. Application vulnerabilities only.
  --download-full-image
                        Downloaded pre-created vdb image to platform specific user_data_dir. All vulnerabilities including OS.
  --print-vdb-metadata  Display metadata about the current vdb in user_data_dir.

CLI search

It is possible to perform a range of searches using the cli.

vdb --search pkg:pypi/xml2dict@0.2.2

# Search based on a purl prefix
vdb --search pkg:pypi/xml2dict

# Full url and short form for swift
vdb --search "pkg:swift/github.com/vapor/vapor@4.39.0"

vdb --search "pkg:swift/vapor/vapor@4.89.0"

# Search by cpe
vdb --search "cpe:2.3:a:npm:gitblame:*:*:*:*:*:*:*:*"

# Search by colon separated values
vdb --search "npm:gitblame:0.0.1"

# Search by vulnerability id (CVE, GHSA, ALSA, DSA, etc.)
vdb --search CVE-2024-25169

# Search with wildcard for CVE
vdb --search CVE-2024-%

# Search by git url
vdb --search "https://github.com/electron/electron"

# Search by CycloneDX SBOM
vdb --bom bom.json

List recent malware

To list malware entries with the MAL- prefix, use the following command.

vdb --list-malware

Library usage

Refer to cli.py for usage. Below is a sample code snippet.

from vdb.lib import config, db6 as db_lib, search
from vdb.lib.cve_model import CVE

# Perform search
results = search.search_by_any("purl or cpe or url", with_data=True)

# Download prebuilt database
from vdb.lib.orasclient import download_image

download_image(config.VDB_DATABASE_URL, config.DATA_DIR)

# Check if database needs updating
if db_lib.needs_update():
    print("Database needs to be updated")
    download_image(...)

Model Context Protocol (MCP) server

Refer to the readme

Read .vdb6 files in other languages

.vdb6 files are standard SQLite database files. Use any modern sqlite library to read and query them. There is a mini deno example in this repo for demonstration.

Local development

Setup uv by following the official documentation.

uv sync --all-extras --dev
uv run depscan --help
uv run pytest

License

MIT

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

appthreat_vulnerability_db-6.4.1.tar.gz (92.0 kB view details)

Uploaded Source

Built Distribution

appthreat_vulnerability_db-6.4.1-py3-none-any.whl (88.2 kB view details)

Uploaded Python 3

File details

Details for the file appthreat_vulnerability_db-6.4.1.tar.gz.

File metadata

File hashes

Hashes for appthreat_vulnerability_db-6.4.1.tar.gz
Algorithm Hash digest
SHA256 46cf18836d550e5884624b9ac2f125dcf926b95a26f6a343c2e0f8f654d11cda
MD5 f37447c166e94ca392aac7fe98e3ad1a
BLAKE2b-256 5bd2cf704f47154d37d7897f2c3df0f3327ce27b3b120486725b35f043365660

See more details on using hashes here.

File details

Details for the file appthreat_vulnerability_db-6.4.1-py3-none-any.whl.

File metadata

File hashes

Hashes for appthreat_vulnerability_db-6.4.1-py3-none-any.whl
Algorithm Hash digest
SHA256 bc27d24ff77e8f484f6cc1456046d17c172cad0167b663d03a53fedcd9d2badb
MD5 0f6f70b3edd03a611a13611624276310
BLAKE2b-256 573fb6921b59fd643a8533f615e551995fca5bff3bdb845969a8aeb71b39381b

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page