nftlist 0.1.0

NFT complementing tool for blocking/allowing traffic based on domain names, IP or mac adresses.

Note: This is work in Progress

NFT List

Tool developed to provide an elegant way for defining access/deny lists for Nftables (nft). Access/deny lists can hold:

• Domain names
• IPv4 hosts and networks
• IPv6 hosts and networks
• Mac addresses

Configuration data is read from configuration files and loaded to appropriate NFT set. NFT configuration (/etc/nftables.*) and NFT List configuration is separated.

Domain names resolved by NFT List are resolved using CloudFlare 1.1.1.1 DOH (DNS over HTTP(s)) server. This is actually the most secure way to query domain names. Domain names are resolved to IP's from A and AAAA records.

IPs pointed by DNS can change over time, thus NFT List runs periodic checks to keep firewall coherent with an actual DNS state.

NOTE: Usage of this tool requires knowledge regarding Linux NFT firewall framework. For more information check out NFT Quick reference in 10 minutes. Additional NFT guides and an examples can be found in Useful links section at the bottom of this page.

Table of contents

• Overview
• NFT in short
• Project's scope
• HowTo Use
  • Part I: Firewall
    • NFT sets
    • NFT Structure
  • Part II: Installation and Configuration
    • Installation
    • Configuration files and directories
      • File format
    • Directives
    • Command line
    • Periodic runs
    • White List example
  • Troubleshooting
• Summary
• Useful links

Project overview

Modern Linux systems use NFT firewall - successor of Iptables. It comes with a new C structure alike configuration format (look: NFT in short 🠋).

Firewall is a crucial security element, thus its configuration should be kept straightforward.

TODO: move it configuration: The NFT List uses "available-enabled pattern" (look: Configuration files and directories 🠋🠋🠋) configuration, that is well known from Apache or Nginx. List resource, such as IPs are read from simple 'line-delimited' text files.

The idea is to avoid mixing NFT configuration with resource lists. Instead, keep them separate, similar to good program design where algorithms and data remain distinct and are not intertwined.

Next points describe how to configure NFT, you can also jump straight to Examples.

NFT sets

Nftables provides so-called sets. Set is a set of elements of the same type, type can be one of: ipv4_addr, ipv6_addr, ether_addr and more (Named sets specifications).

The NFT Sets are an 'anchors' where NFT List attaches the list of elements (IP addresses, macs, domain names).

Non-atomic issue

Advantage of NFT over IPtables is 'atomic' operation. The command nft -f <filename> loads configuration 'on a side'. When configfile is successfully loaded NFT switches in an atomic manner to a new configuration. This means there is no single moment when firewall is partially configured. As NFT List operates after nft command, this might bring security issues, such as access to machine from banned IP (for very short time).

The solution is to cork a sets with 0.0.0.0/0 and ::/128 masks. See example below:

table inet filtering_machine {
    set allow_ip_list {
        type ipv4_addr;
        flags timeout;
    }

    set allow_ip6_list {
        type ipv6_addr;
        flags timeout;
    }

    set ban_ip_list {
        type ipv4_addr;
        elements = { 0.0.0.0/0 }
    }

    set ban_ip6_list {
        type ipv6_addr;
        elements = { ::/128 }
    }

    set allow_mac_list {
        type ether_addr;
        flags timeout;
    }
}

The allow lists should be left empty (no elements), for filling in bit later by NFT List. In this case there us no security thread as firewall would not allow packages (empty set) to pass through. You can put in NFT configuration IP of administrative machine if you suspect the scenario where NFT List does not load, to ensure access to machine.

Ban lists are logical opposites to 'allow' lists. To ensure there is no single moments when 'bad' packege can 'sneak under', mask 0.0.0.0/0 and ::/128 should be used to cork all ipv4_addr and ipv6_addr traffic. NFT List will remove that addresses after ban list is loaded.

TODO: remove Note, timeout flag of NFT set can be used to define elements to expire automatically. It might be a good feature for a long, constantly being updated lists. More information in TODO.

NFT List at each boot chceks if lists are propperly sealed. If a ste is bound to drop or reject, but no '0.0.0.0/0 cork' is added, warning is issued.

NFT List configuration

...

Simple example

TODO: move to an end Simple example can be foundin: example1.1-workstation.nft and NFT List example1.1-workstation.list.

Once when NFT with this configuration is loaded (note flush ruleset - that purges all previous settings) allowed_hosts is empty. Therefore, no other host is allowed to let in, access can be open from command line:

nft add element inet my_table allowed_hosts { 172.22.0.2 }

or NFT List can do this for you. Define file such as:

# /etc/nftdef/allowed_hosts.list
# host that are allowed for connecting this box

@set inet my_table allowed_hosts # load bellow addresses into 'allowed_hosts' set
172.22.0.2 # Peer 2

and load:

nftlist.sh update /etc/nftdef/allowed_hosts.list

NFT List comes with OpenRC init script to load settings a boot time, and cron script for periodic reloads. More in Part II: Installation and Configuration.

HowTo Use

Part I: Firewall

This part explains an interaction of NFT List with Nftables user-space tools (firewall).

NFT sets

NFT comes with NFT sets that allow on grouping elements of the same type together. Example definition of NFT set looks as follows:

table inet user_defined_table_name {
       set user_defined_set_name {
               type ipv4_addr;
               flags timeout, interval;
               auto-merge;

               elements = { 10.0.0.0/8 }
       }
}

Sets can be anonymous, or named. Above example defines named (user_defined_set_name) set. To have NFT List to feed sets with a data it must be obviously named.

NFT defines various types of elements, for instance ipv4_addr is for keeping IPv4 addresses, ether_addr for mac addresses while ifname is to group network interfaces (e.g. enp7s0, br0).

NFT List operates on a Sets of the following types:

• ipv4_addr for IPv4,
• ipv6_addr for IPv6 elements
• ether_addr for mac addresses.

Nftables also provides typeof keyword that as documentation states:

allows you to use a high level expression, then let nftables resolve the base type for you

NFT List can operate only on sets declared with type. Trial of loading data into typeof, or type but of not supported type will fail.

Set can be complemented by flags that specifies more precisely behaviour and format of an elements:

• timeout - specifies timeout when a set element is set for expiration, note that various elements within one set can have a different timeouts
• constant - set content can not be changed once when it has been bound to a rule
• interval - set elements can be intervals, that is IP addresses with network prefixes or address ranges in format: -, e.g. 192.168.100-192.168.199

NFT List validates these flags and acts as follows:

• timeout - NFT List resolves IP addresses and adds them to 'timeout set' setting up a default timeout that is 3 days. If the list is refreshed, timeout is updated. If given domain name does not resolve to a certain IP anymore, it will simpli expire and eventually disappear from the set.
• constant - NFT List acts normally if set is not bound with rule.
• interval - NFT List extends accepted format by network prefixes and address ranges.

auto-merge option of NFT Set combines a set elements into intervals, this enables auto-packing addresses into groups. For instance adding 10.2.0.0/16 and 11.3.0.0/16 merges into one network: 10.2.0.0/15. With auto-merge defined the result of nftlist update might not only be insertions and removals, but also merge, split actions.

In NFT there is no type inet_addr that would match both types ipv4_addr and ipv6_addr. Therefore, one set can not hold mixed IPv4 and IPv6 elements. If set has been defined of a type ipv4_addr NFT-List will resolve A DNS records to acquire IP. If set is of type ipv6_addr AAAA DNS records are queried.

NFT Structure

If you inspect an example from Quick introduction there are three noticeable aspects:

• Sets belong to table
• where chains and finally rules are defined
• rules refer to set to apply a certain policy for a given network resources

Sets are a meeting point. NFT List fills in Set with a desired IP/mac addresses. NFT List does not create, modify or delete chains, rules or other 'structural' firewall settings. It only operates within sets, 'NFT List' can only:

• add, or
• remove elements of an indicated NFT sets.

It's administrator's job to define a firewall configuration. NFT List is a tool designed for filling and keeping set elements up to date.

Part II: Installation and Configuration

This part is about how to configure the tool.

Installation

NFT list is in an early development phase, there is no packages developed yet, you can install it by unpacking release file into / directory:

tar -xzvf nfthelper-1.2.9-alpha.tar.gz -C /

this will simply extract nftlist.sh to /usr/local/bin/ and nftlist to etc/init.d/.

At this point, only OpenRC is supported (no systemd).

Add nftlist to OpenRC's default runlevel:

rc-update add nftlist default

To launch NFT List simply start the service:

service nftlist start

this reads configuration from /etc/nftlists/available/ and loads NFT sets. Just after installation as configuration is empty nothing will happen.

Configuration files and directories

Refer to your Linux distribution guide to learn how to handle NFT configuration under your system. Usually (Debian, Alphine) main configuration file is /etc/nftables.nft while directory /etc/nftables.d/ is used for drop in additional settings.

Nft List uses /etc/nftlists/ as a storage for network resource lists.

It holds available and enabled directories:

# tree /etc/nftlists/
/etc/nftlists/
├── available
│   ├── inbound_filter.list
│   └── outbound_filter.list
├── enabled
│   └── outbound_filter.list -> ../available/outbound_filter.list
└── included
    └── phising_list.txt.gz

3 directories, 4 files

nftlist by default, if no configuration file or directory is passed, loads all files ended with .list extension that are located in /etc/nftlists/enabled/ (no recursive search).

File format

The file format expected by NFT List is a text file holding domain/address list, such as:

# /etc/nftlists/avaliable/blacklisted_domain.list

@set inet filter blacklist

bad-domain.xxx.com # don't go there
your-bank.login.space # no comment
# aha ...
specialoffernow.info

Comments are marked by \# symbol.

Directives

Resource list should or can be proceeded by a directives:

@set <family>|- <table name>|- <set name>|-

Indicates NFT target set for filling, addresses placed below this directive will be loaded to an indicated set. The directive must define table family, name, and set name. Minus sign (-) (copy sign) if is used for one or more \@set parameters instructs to apply previous settings (copy previous parameter) e.g.:

@set ipv6 - -

# set will apply to: set ipv6 filter allow_outbound
@set - filter allow_outbound

... addresses ...

# set will apply to: set ipv6 filter allow_inbound
@set - - allow_inbound

... addresses ...

@query <selector>

Do additional DNS query if domain name is encountered, selector indicates query type:

• @query subdomain. - query also subdomain, e.g.:

  @query www.

  also checks www. subdomain for each encountered domain in a current set

@include <file name>

Include file, file that is to be included must be located inside /etc/nftlists/included. It does not need to have .list extension, it can be compressed, the compression algorithm must be indicated by a correct extension: .gz, .zip ...

@onpanic keep|discard

This directive defines an action in the case of panic signal. If such an event happen, probably white lists should be discarded from sets while black addresses keeped. Panic signal can be issued with:

nftlist panic

command.

The only required directive is \@set that indicates target set. However, file's \@set directive can be replaced with a command line --set (or short -s) option that does the same. In this case target set does not need to be defined in *.list configuration file. For more options see command line usage.

Command line

NFT Sets are updated by NFT List each time when revise is started, or reloaded:

service nftlist start
service nftlist reload

NFT List can also be called directly from command line:

# nftlist
Following updates has been found
Id:    action:
 1     add 4 IPv4 elements
        to 'blacklist' set of 'filter' inet table

proceed with update?
      yes    / no / details / help or (y    /n/d/h) - if yes,     update all
  or  yes Ids/ no / details / help or (y Ids/n/d/h) - if yes Ids, update chosen set
 :

Typing yes, or y will proceed with update, update can be limited to a chosen set by passing Id indicating actions just after yes, e.g. yes 1, yes 2 5.

The prompt can be skipped with:

# Update Sets without prompt
nftlist update
# or shorter
nftlist u

This will proceed with updates without prompting.

By default, configuration from /etc/nftlists/available/ is loaded, but it can be overwritten:

nftlist update /etc/my_list.list
# or
nftlist update /etc/my_list_directory/
# additionally include directory can be indicated:
nftlist update /etc/my_list.list --includedir /etc/incl_lists/
# or shortly:
nftlist update /etc/my_list.list -D /etc/incl_lists/

Set can be indicated as a command line argument:

nftlist u --set inet filter allow
# or use short '-s' flag
nftlist u -s inet filter allow

If it is provided as command line argument it does not need to be declared in .list file.

nftlist comes with panic option that will apply policies defined by \@onpanic directive.

Periodic runs

A or AAAA records can change over time, therefore firewall shall be updated periodically. It is advised to add to daily cron service nftlist reload so configuration shall be keep in sync with DNS entries.

White List example

One of the example of whitelisting is to allow outgoing connections to be only possible with a very limited number of servers. This can be a list of a domain names that are used for system updates. The configuration might look as bellow:

# file: /etc/nftlists/available/repo_devuan.list

# Devuan repositories
deb.devuan.org
deb.debian.org

# file: /etc/nftlists/enabled/

@set inet filter allow_outgoing
@include repo_devuan.list

We can whitelist more resources:

# file: /etc/nftlists/enabled/

@set inet filter allow_outgoing
@include repo_devuan.list

10.2.0.100
example.com

Note that by splitting configuration to a set of files it become more manageable. Also note, that you can mix domain names with IP address.

NFT rule bound to allow_outgoing set would be defined in forward chain. The default policy for that chain might be drop, the rule might be as follows:

iifname br0 oifname eth0 ip daddr @allow_outgoing tcp dport { 80, 443 } accept

Troubleshooting

In the case of troubles you run:

nftlist --verbose

And check for any detailed error messages. If it does not help analyze what exactly is under the hood:

# Stop firewall
service nftlist stop
service nftables stop

# checkout defined rules
# it should be empty
nft list ruleset

# start firewall
service nftables stop

# analyze configuration
nft list ruleset

# start nftlist
service nftlist start

# and verify if sets has been updated
nft list ruleset

also you can:

# List all sets
nft list sets

# Add set elements manually
nft add element inet my_table allowed_hosts { 172.22.0.2 }

NFT provides counters, it can give you some information if traffic is passing through a certain firewall rule or hook. In tha case of troubles tools such as tcpdump and WireShark come in handy.

NFT advanced features

NFT by design provides sophisticated functions such as timeouts, intervals, different data types. Additionally, IPs resolved from DNS might change over time. Domain name lists (especially blacklists) do change also, etc. One of the 'musts' for a well-made software is a capability to reload configuration keeping in mind all these sophisticated functions of NFT. Thus, the program requirements has grow comparing original 'read list -> resolve names -> load firewall' concept.

Summary

NFT List can be used in:

• virtualise environments to is useful with containerization/virtualization technics where running VMs can be limited to exact network resources they need. It has been developed as a lite, robust and straightforward solution for various firewall configuration cases.

Useful links

HowTo:

• Nftables Wiki here
• Gentoo Nftables guide (nice and compact) here
• and Arch Linux NFT (alike Gentoo's wiki, but nothing about howto compile kernel :) wiki

Theory:

• Netfilter framework at Wikipedia

Administration

• Using iptables-nft: a hybrid Linux firewall at Redhat.com
• serverfault: nftables: referencing a set from another table

Bad addresses databases

• Phishing Database: github: mitchellkrogza/Phishing.Database

Programming

• fw4 Filtering traffic with IP sets by DNS Openwrt.org
• Python Nftables tutorial: GitHub