greenbone-feed-sync 23.1.1

Synchronization for the Greenbone Community Feed

greenbone-feed-sync

New script for syncing the Greenbone Community Feed

• Installation
  • Requirements
  • Install using pip
• Usage
• Settings
  • verbose
  • quiet
  • config
  • private-directory
  • compression-level
  • type
  • feed-url
  • destination-prefix
  • gvmd-data-destination
  • gvmd-data-url
  • notus-destination
  • notus-url
  • nasl-destination
  • nasl-url
  • scap-data-destination
  • scap-data-url
  • cert-data-destination
  • cert-data-url
  • report-formats-destination
  • report-formats-url
  • scan-configs-destination
  • scan-configs-url
  • port-lists-destination
  • port-lists-url
  • gvmd-lock-file
  • openvas-lock-file
  • fail-fast
  • no-wait
  • wait-interval
  • rsync-timeout
• Config
• Development
• Maintainer
• Contributing
• License

Installation

Requirements

Python 3.9 and later is supported.

greenbone-feed-sync requires the rsync tool being installed and available within the PATH.

On Debian based Distributions like Ubuntu and Kali rsync can be installed via

apt install rsync

Install using pip

You can install the latest stable release of greenbone-feed-sync from the Python Package Index (pypi) using pip

python3 -m pip install greenbone-feed-sync

Usage

Most of the time you should just run the script without any arguments to download the new data for all necessary feed types

greenbone-feed-sync

To get verbose progress output during the data download you might increase the verbosity

greenbone-feed-sync -vvv

If the script is run in a cron job the output can be turned of via

greenbone-feed-sync --quiet

Settings

The greenbone-feed-sync script is adjustable for all kind of purposes and very flexible which content gets downloaded. Most likely you will never need to adjust the settings because the defaults will suffice. Changing the settings is only required for experts and testing purposes.

verbose

Name	Value
CLI Argument	--verbose, -v
Config Variable	verbose
Environment Variable	GREENBONE_FEED_SYNC_VERBOSE
Default Value	2
Description	Log verbosity. -vvv for maximum verbosity.

quiet

Name	Value
CLI Argument	--quiet
Config Variable
Environment Variable
Default Value
Description	Disable all log output. Same as setting verbose or GREENBONE_FEED_SYNC_VERBOSE to 0

config

Name	Value
CLI Argument	--config, -c
Config Variable
Environment Variable
Default Value	~/.config/greenbone-feed-sync.toml and /etc/gvm/greenbone-feed-sync.toml
Description	TOML config file to load settings from.

private-directory

Name	Value
CLI Argument	--private-directory
Config Variable	private-directory
Environment Variable	GREENBONE_FEED_SYNC_PRIVATE_DIRECTORY
Default Value
Description	(Sub-)Directory to exclude from the sync which will never get deleted automatically.

compression-level

Name	Value
CLI Argument	--compression-level
Config Variable	compression-level
Environment Variable	GREENBONE_FEED_SYNC_COMPRESSION_LEVEL
Default Value	9
Description	rsync compression level 0-9. (0 - no compression, 9 - high compression)

type

Name	Value
CLI Argument	--type
Config Variable
Environment Variable
Default Value	all
Description	Specifies which feed data should be downloaded.

feed-url

Name	Value
CLI Argument
Config Variable	feed-url
Environment Variable	GREENBONE_FEED_SYNC_URL
Default Value	rsync://feed.community.greenbone.net/community
Description	URL to download the feed data from. Other URLs will be relative to this URL by default. For example using rsync://example.com as feed url the notus url will be rsync://example.com/vulnerability-feed/22.04/vt-data/notus/.

destination-prefix

Name	Value
CLI Argument
Config Variable	destination-prefix
Environment Variable	GREENBONE_FEED_SYNC_DESTINATION_PREFIX
Default Value	/var/lib/
Description	Directory prefix to use for default feed data download destinations. Other download destinations will be relative to this path by default. For example using /opt/lib as destination prefix will change the default of the notus destination to /opt/lib/notus.

gvmd-data-destination

Name	Value
CLI Argument	--gvmd-data-destination
Config Variable	gvmd-data-destination
Environment Variable	GREENBONE_FEED_SYNC_GVMD_DATA_DESTINATION
Default Value	/var/lib/gvm/data-objects/gvmd/22.04/
Description	Destination of the downloaded gvmd data.

gvmd-data-url

Name	Value
CLI Argument	--gvmd-data-url
Config Variable	gvmd-data-url
Environment Variable	GREENBONE_FEED_SYNC_GVMD_DATA_URL
Default Value	rsync://feed.community.greenbone.net/community/data-feed/22.04/
Description	URL to download the gvmd data from. This includes scan-configs, report-formats and port-lists.

notus-destination

Name	Value
CLI Argument	--notus-destination
Config Variable	notus-destination
Environment Variable	GREENBONE_FEED_SYNC_NOTUS_DESTINATION
Default Value	/var/lib/notus
Description	Destination of the downloaded notus data.

notus-url

Name	Value
CLI Argument	--notus-url
Config Variable	notus-url
Environment Variable	GREENBONE_FEED_SYNC_NOTUS_URL
Default Value	rsync://feed.community.greenbone.net/community/vulnerability-feed/22.04/vt-data/notus/
Description	URL to download the notus data from.

nasl-destination

Name	Value
CLI Argument	--nasl-destination
Config Variable	nasl-destination
Environment Variable	GREENBONE_FEED_SYNC_NASL_DESTINATION
Default Value	/var/lib/openvas/plugins
Description	Destination of the downloaded nasl data.

nasl-url

Name	Value
CLI Argument	--nasl-url
Config Variable	nasl-url
Environment Variable	GREENBONE_FEED_SYNC_NASL_URL
Default Value	rsync://feed.community.greenbone.net/community/vulnerability-feed/22.04/vt-data/nasl/
Description	URL to download the nasl data from.

scap-data-destination

Name	Value
CLI Argument	--scap-data-destination
Config Variable	scap-data-destination
Environment Variable	GREENBONE_FEED_SYNC_SCAP_DATA_DESTINATION
Default Value	/var/lib/gvm/scap-data
Description	Destination of the downloaded SCAP data.

scap-data-url

Name	Value
CLI Argument	--scap-data-url
Config Variable	scap-data-url
Environment Variable	GREENBONE_FEED_SYNC_SCAP_DATA_URL
Default Value	rsync://feed.community.greenbone.net/community/vulnerability-feed/22.04/scap-data
Description	URL to download the SCAP data from.

cert-data-destination

Name	Value
CLI Argument	--cert-data-destination
Config Variable	cert-data-destination
Environment Variable	GREENBONE_FEED_SYNC_CERT_DATA_DESTINATION
Default Value	/var/lib/gvm/cert-data
Description	Destination of the downloaded CERT data.

cert-data-url

Name	Value
CLI Argument	--cert-data-url
Config Variable	cert-data-url
Environment Variable	GREENBONE_FEED_SYNC_CERT_DATA_URL
Default Value	rsync://feed.community.greenbone.net/community/vulnerability-feed/22.04/cert-data
Description	URL to download the CERT data from.

report-formats-destination

Name	Value
CLI Argument	--report-formats-destination
Config Variable	report-formats-destination
Environment Variable	GREENBONE_FEED_SYNC_REPORT_FORMATS_DESTINATION
Default Value	/var/lib/gvm/data-objects/gvmd/22.04/report-formats
Description	Destination of the downloaded report format data.

report-formats-url

Name	Value
CLI Argument	--report-formats-url
Config Variable	report-formats-url
Environment Variable	GREENBONE_FEED_SYNC_REPORT_FORMATS_URL
Default Value	rsync://feed.community.greenbone.net/community/data-feed/22.04/report-formats
Description	URL to download the report format data from.

scan-configs-destination

Name	Value
CLI Argument	--scan-configs-destination
Config Variable	scan-configs-destination
Environment Variable	GREENBONE_FEED_SYNC_SCAN_CONFIGS_DESTINATION
Default Value	/var/lib/gvm/data-objects/gvmd/22.04/scan-configs
Description	Destination of the downloaded scan config data.

scan-configs-url

Name	Value
CLI Argument	--scan-configs-url
Config Variable	scan-configs-url
Environment Variable	GREENBONE_FEED_SYNC_SCAN_CONFIGS_URL
Default Value	rsync://feed.community.greenbone.net/community/data-feed/22.04/scan-configs
Description	URL to download the scan config data from.

port-lists-destination

Name	Value
CLI Argument	--port-lists-destination
Config Variable	port-lists-destination
Environment Variable	GREENBONE_FEED_SYNC_PORT_LISTS_DESTINATION
Default Value	/var/lib/gvm/data-objects/gvmd/22.04/port-lists
Description	Destination of the downloaded port list data.

port-lists-url

Name	Value
CLI Argument	--port-lists-url
Config Variable	port-lists-url
Environment Variable	GREENBONE_FEED_SYNC_PORT_LISTS_URL
Default Value	rsync://feed.community.greenbone.net/community/data-feed/22.04/port-lists
Description	URL to download the port list data from.

gvmd-lock-file

Name	Value
CLI Argument	--gvmd-lock-file
Config Variable	gvmd-lock-file
Environment Variable	GREENBONE_FEED_SYNC_GVMD_LOCK_FILE
Default Value	/var/lib/openvas/feed-update.lock
Description	File to use for locking the feed synchronization for data loaded by the gvmd daemon. Used to avoid that more then one process accesses the feed data at the same time.

openvas-lock-file

Name	Value
CLI Argument	--openvas-lock-file
Config Variable	openvas-lock-file
Environment Variable	GREENBONE_FEED_SYNC_OPENVAS_LOCK_FILE
Default Value	/var/lib/gvm/feed-update.lock
Description	File to use for locking the feed synchronization for data loaded by the openvas scanner. Used to avoid that more then one process accesses the feed data at the same time.

fail-fast

Name	Value
CLI Argument	--fail-fast, --failfast
Config Variable	fail-fast
Environment Variable	GREENBONE_FEED_SYNC_FAIL_FAST
Default Value	false
Description	Stop after a first error has occurred. Otherwise the script tries to download additional data if specified.

no-wait

Name	Value
CLI Argument	--no-wait
Config Variable	no-wait
Environment Variable	GREENBONE_FEED_SYNC_NO_WAIT
Default Value	false
Description	Fail directly if the lock file can't be acquired.

wait-interval

Name	Value
CLI Argument	--wait-interval
Config Variable	wait-interval
Environment Variable	GREENBONE_FEED_SYNC_LOCK_WAIT_INTERVAL
Default Value	5
Description	Time to wait in seconds after failed lock attempt before re-trying to lock the file.

rsync-timeout

Name	Value
CLI Argument	--rsync-timeout
Config Variable	rsync-timeout
Environment Variable	GREENBONE_FEED_SYNC_RSYNC_TIMEOUT
Default Value
Description	Maximum I/O timeout in seconds used for rsync. If no data is transferred for the specified time then rsync will exit. By default no timeout is set and the rsync default will be used.

Config

It is possible to use a config file for loading the settings of the greenbone-feed-sync script. The config file uses the TOML format. Without explicitly passing a config file, greenbone-feed-sync tries to load ~/.config/greenbone-feed-sync.toml and if that file doesn't exist afterwards /etc/gvm/greenbone-feed-sync.toml.

Example:

[greenbone-feed-sync]
destination-prefix = "/opt/greenbone-feed"
lock-file = "/opt/greenbone-feed.lock"
no-wait = true

Development

greenbone-feed-sync uses poetry for its own dependency management and build process.

First install poetry via pip

python3 -m pip install --user poetry

Afterwards run

poetry install

in the checkout directory of greenbone-feed-sync (the directory containing the pyproject.toml file) to install all dependencies including the packages only required for development.

Afterwards activate the git hooks for auto-formatting and linting via autohooks.

poetry run autohooks activate

Validate the activated git hooks by running

poetry run autohooks check

Maintainer

This project is maintained by Greenbone Networks GmbH

Contributing

Your contributions are highly appreciated. Please create a pull request on GitHub. Bigger changes need to be discussed with the development team via the issues section at GitHub first.

License

Copyright (C) 2022-2023 Greenbone Networks GmbH

Licensed under the GNU General Public License v3.0 or later.