anycastd 0.1.12

A daemon to manage anycasted services based on status checks.

anycastd

anycastd functions as a daemon managing the announcement of network prefixes employed by redundant services using multiple backends that share a common set of service prefixes. Each prefix is announced individually to the network, forming a load-balancing strategy with redundancy, commonly referred to as Anycast. This tool ensures that service prefixes are exclusively announced when all underlying service components are confirmed to be in a healthy state. By doing so, anycastd prevents the attraction of traffic to service instances that may be malfunctioning, avoiding service diruption.

Table of Contents

• Usage Example
• Services
  • Prefixes
    • FRRouting
  • Health Checks
    • Cabourotte
• Configuration
  • Schema

Usage Example

In the following example, we will use anycastd to manage the prefixes of two dual-stacked services commonly run on the same host. FRRouting is used to announce the prefixes of both services which are health checked through Cabourotte.

anycastd configuration

To configure the two services in anycastd, we create the /etc/anycastd/config.toml configuration file with the following contents.

[services.dns]
prefixes.frrouting = ["2001:db8::b19:bad:53", "203.0.113.53"]
checks.cabourotte = ["dns"]

[services.ntp]
prefixes.frrouting = [
    { "prefix" = "2001:db8::123:7e11:713e", "vrf" = "123" },
    { "prefix" = "203.0.113.123", "vrf" = "123" },
]
checks.cabourotte = [
    { "name" = "ntp_v6", "interval" = 1 },
    { "name" = "ntp_v4", "interval" = 1 },
]

The first service, aptly named "dns", simply configures a DNS resolver service that announces the prefixes 2001:db8::b19:bad:53/128 & 203.0.113.53/32 through FRRouting as long as the Cabourotte health check dns is reported as healthy.

The second service, "ntp" is similar in functionality, although its configuration is a bit more verbose. Rather than omitting values that have a preconfigured default, a VRF as well as a health check interval are explicitly specified.

FRRouting configuration

Next, we need to configure FRRouting so that anycastd can add and remove prefixes based on the services health checks. To do this, we create the /etc/frr/frr.conf with the following minimal configuration.

!
router bgp 65536
 bgp router-id 203.0.113.179
 neighbor unnumbered peer-group
 neighbor unnumbered remote-as external
 neighbor unnumbered capability extended-nexthop
 neighbor eth0 interface peer-group unnumbered
 !
 address-family ipv4 unicast
  redistribute static
 !
 address-family ipv6 unicast
  redistribute static
  neighbor fabric activate
  neighbor fabric nexthop-local unchanged
!
router bgp 65537 vrf 123
 bgp router-id 203.0.113.181
 neighbor unnumbered peer-group
 neighbor unnumbered remote-as external
 neighbor unnumbered capability extended-nexthop
 neighbor eth1 interface peer-group unnumbered
 !
 address-family ipv4 unicast
  redistribute static
 !
 address-family ipv6 unicast
  redistribute static
  neighbor fabric activate
  neighbor fabric nexthop-local unchanged
!

This creates two BGP instances, AS65536 in the default VRF and AS65537 in VRF 123. Both of them have a single unnumbered session that will be used to advertise the service prefixes. The most important statement here is redistribute static for both IPv4 and IPv6, instructing FRRouting to redistribute the static routes containing the service prefixes that will later be created by anycastd.

Cabourotte configuration

The last thing we have to configure is Cabourotte, which performs the actual health checks. We create the following /etc/cabourotte/config.yml.

---
http:
  host: 127.0.0.1
  port: 9013

dns-checks:
  # Assumes that the DNS service is used as system wide resolver.
  - name: dns
    domain: check.local
    timeout: 1s
    interval: 5s
    expected-ips: ["2001:db8::15:600d"]

command-checks:
  - name: ntp_v6
    timeout: 3s
    interval: 5s
    command: ntpdate
    arguments: ["-q", "2001:db8::123:7e11:713e"]
  - name: ntp_v4
    timeout: 3s
    interval: 5s
    command: ntpdate
    arguments: ["-q", "203.0.113.123"]

This sets up two fairly rudimentary health checks. The first renders healthy if a request to the DNS service for the check.local name returns the IPv6 address 2001:db8::15:600d in the form of an AAAA record. The other two checks, ntp_v6 and ntp_v4 use the ntpdate CLI utility to determine if a date is returned by the NTP service.

Starting services

To finish up, we need to start our services. For this example we assume that both services as well as Cabourotte are run using systemd while anycastd is run directly for the purposes of this example.

So, to start the DNS, NTP and Cabourotte services we run

$ systemctl start dns.service ntp.service cabourotte.service

After which we can start anycastd itself.

$ anycastd run
2024-03-25T15:17:23.783539Z [info     ] Reading configuration from /etc/anycastd/config.toml. config_path=/etc/anycastd/config.toml
2024-03-25T15:17:23.785613Z [info     ] Starting service "dns".      service_health_checks=['dns'] service_healthy=False service_name=dns service_prefixes=['2001:db8::b19:bad:53', '203.0.113.53']
2024-03-25T15:17:23.785613Z [info     ] Starting service "ntp".      service_health_checks=['ntp_v4', 'ntp_v6'] service_healthy=False service_name=ntp service_prefixes=['2001:db8::123:7e11:713e', '203.0.113.123']
2024-03-25T15:17:23.797760Z [info     ] Service "dns" is now considered healthy, announcing related prefixes. service_health_checks=['dns'] service_healthy=True service_name=dns service_prefixes=['2001:db8::b19:bad:53', '203.0.113.53']
2024-03-25T15:17:23.812260Z [info     ] Service "ntp" is now considered healthy, announcing related prefixes. service_health_checks=['ntp_v4', 'ntp_v6'] service_healthy=True service_name=ntp service_prefixes=['2001:db8::123:7e11:713e', '203.0.113.123']

anycastd will execute the health checks and, since all of them pass, announce the configured service IPs, which we can verify by looking at the new FRRouting running configuration.

@@ -7,9 +7,11 @@
  neighbor eth0 interface peer-group unnumbered
  !
  address-family ipv4 unicast
+  network 203.0.113.53/32
   redistribute static
  !
  address-family ipv6 unicast
+  network 2001:db8::b19:bad:53/128
   redistribute static
   neighbor fabric activate
   neighbor fabric nexthop-local unchanged
@@ -22,9 +24,11 @@
  neighbor eth1 interface peer-group unnumbered
  !
  address-family ipv4 unicast
+  network 203.0.113.123/32
   redistribute static
  !
  address-family ipv6 unicast
+  network 2001:db8::123:7e11:713e/128
   redistribute static
   neighbor fabric activate
   neighbor fabric nexthop-local unchanged

Stopping services

anycastd will keep prefixes announced as long as health checks pass. To stop announcing prefixes, even though the underlying services are healthy, for example to perform maintenance, simply stop anycastd, causing all service prefixes to be denounced.

^C
2024-03-25T15:20:29.738135Z [info     ] Received SIGINT, terminating.
2024-03-25T15:20:29.817023Z [info     ] Service "dns" terminated.    service=dns
2024-03-25T15:20:29.819003Z [info     ] Service "ntp" terminated.    service=ntp

Services

Services are the main unit of abstraction within anycastd and are used to form a logical relationship between health checks and network prefixes containing IP addresses related to the underlying application represented by the service. They work by continuously monitoring defined health checks and announcing/denouncing their prefixes based on the combination of check results using the logic described below.

┌─[Service]─────────────┐                        ┌──────────┐
│                       │                   ┌──> │ HLTH CHK │
│           ┌───────────────────────────────┤    └──────────┘
│ IF healthy•:          │                   │    ┌──────────┐
│     announce prefixes │                   ├──> │ HLTH CHK │
│ ELSE:           •─────────────────────┐   │    └──────────┘
│     denounce prefixes │               │   │    ┌──────────┐
└───────────────────────┘               │   └──> │ HLTH CHK │
                                        │        └──────────┘
                                        │
┌─[Routing Daemon]────────────────┐     │
│ ┌──────────────────────────┐    │     │
│ │ Prefix                   │ <────────┤
│ │ 2001:db8::b19:bad:53/128 │    │     │
│ └──────────────────────────┘    │     │
│ ┌──────────────────────────┐    │     │
│ │ Prefix                   │ <────────┘
│ │ 203.0.113.53/32          │    │
│ └──────────────────────────┘    │
└─────────────────────────────────┘

Prefixes

Represents a BGP network prefix that can be announced or denounced as part of the service. Typically, these are networks containing "service IPs", meaning the IP addresses exposed by a particular service, serving as the points of contact for clients to make requests while being completely agnostic to the specifics of anycast.

anycastd does not come with its own BGP implementation, but rather aims to provide abstractions that interface with commonly used BGP daemons. Supported BGP daemons along with their configuration options are described below.

---

FRRouting

Free Range Routing, FRRouting, or simply FRR is a free and open source Internet routing protocol suite for Linux and Unix platforms. Amongst others, it provides a BGP implementation that can be used to announce BGP service prefixes dynamically.

Options

Option	Description	Default	Examples
prefix (required)	The network prefix to create when healthy.	null	2001:db8:4:387b::/64 192.0.2.240/28 2001:db8::b19:bad:53
vrf	A VRF to create the prefix in. If omitted, the default VRF is used.	None	EDGE
vtysh	The path to the vtysh binary used to configure FRRouting.	/usr/bin/vtysh	/usr/local/bin/vtysh

Supported Versions

While CI integration tests only target the latest version of FRRouting, we aim to support releases made within the last 6 months at minimum. anycastd is known to work with versions starting from 7.3.1, although older versions are likely to work as well.

Health Checks

Assessments on individual components constituting the service to ascertain the overall operational status of the service. A service is considered healthy as a whole if all of its health checks report a healthy status. Possible health check types along with their configuration options are described below.

---

Cabourotte

Cabourotte is a general purpose healthchecking tool written in Golang that can be configured to execute checks, exposing their results via API.

Options

Option	Description	Default	Examples
name (required)	The name of the health check, as defined in Cabourotte.	null	anycast-dns
url	The base URL of the Cabourotte API.	http://127.0.0.1:9013	https:://healthz.local
interval	The interval in seconds at which the health check should be executed.	5	2

---

Configuration

anycastd can be configured using a TOML configuration file located at /etc/anycastd/config.toml, or a path specified through the --configuration parameter. For a quick primer on TOML, see A Quick Tour of TOML.

Schema

[services] # A definition of services to be managed by `anycastd`.

  [services.<service-name>] # A service with a unique and recognizable name.
    [[prefixes.<prefix-type>]] # A prefix of the specified type.
      # Options related to the specified prefix type.

    [[checks.<check-type>]] # A check of the specified type.
      # Options related to the specified check type.

Contributing

Contributions of all sizes that improve anycastd in any way, be it DX/UX, documentation, performance or other are highly appreciated. To get started, please read the contribution guidelines. Before starting work on a new feature you would like to contribute that may impact simplicity, reliability or performance, please open an issue first.