cf-remote

cf-remote is a tool to deploy CFEngine. It works by contacting remote hosts with SSH and using ssh / scp to copy files and run commands. Commands for provisioning hosts in the cloud (AWS or GCP) are also available.

Requirements

• cf-remote requires python 3.6 or greater.
• SSH must be configured in such a way that cf-remote can login without a password.
• The account cf-remote logs in as must be root or be able to sudo. Passwordless sudo is not required, see Switching user on the remote hosts.
• An sftp server for transferring files on UNIX hosts. e.g. openssh-sftp-server for debian-based distributions.

Installation

Install with pipx:

pipx install cf-remote

Or pip3:

pip3 install cf-remote

Or pip:

pip install cf-remote

Examples

See information about remote host

The info command can be used to check basic information about a system. The --hosts/-H option accepts [user@]hostname[:port] for the hostname. In the case that hostname is an ipv6 address use literal square brackets as described in RFC-3986 (https://www.ietf.org/rfc/rfc3986.txt)

e.g. user@[FEDC:BA98:7654:3210:FEDC:BA98:7654:3210]:8022

$ cf-remote info -H 34.241.203.218

ubuntu@34.241.203.218
OS            : ubuntu (debian)
Architecture  : x86_64
CFEngine      : 3.12.1
Policy server : 172.31.42.192
Binaries      : dpkg, apt

(You must have ssh access).

Installing and bootstrapping CFEngine Enterprise Hub

The install command can automatically download and install packages as well as bootstrap both hubs and clients.

$ cf-remote install --hub 34.247.181.100 --bootstrap 172.31.44.146 --demo

ubuntu@34.247.181.100
OS            : ubuntu (debian)
Architecture  : x86_64
CFEngine      : Not installed
Policy server : None
Binaries      : dpkg, apt

Package already downloaded: '/Users/olehermanse/.cfengine/cf-remote/packages/cfengine-nova-hub_3.12.1-1_amd64.deb'
Copying: '/Users/olehermanse/.cfengine/cf-remote/packages/cfengine-nova-hub_3.12.1-1_amd64.deb' to '34.247.181.100'
Installing: 'cfengine-nova-hub_3.12.1-1_amd64.deb' on '34.247.181.100'
CFEngine 3.12.1 was successfully installed on '34.247.181.100'
Bootstrapping: '34.247.181.100' -> '172.31.44.146'
Bootstrap successful: '34.247.181.100' -> '172.31.44.146'
Transferring def.json to hub: '34.247.181.100'
Copying: '/Users/olehermanse/.cfengine/cf-remote/json/def.json' to '34.247.181.100'
Triggering an agent run on: '34.247.181.100'
Disabling password change on hub: '34.247.181.100'
Triggering an agent run on: '34.247.181.100'
Your demo hub is ready: https://34.247.181.100/ (Username: admin, Password: QxvTmKdLbRsWnp)

The username is always admin. The password is randomly generated for each hub. It is only shown in that last log message, so take note of it.

Note that this demo setup (--demo) is notoriously insecure. It has open access controls. Don't use it in a production environment.

Spawning instances in AWS EC2

cf-remote spawn can create cloud instances on demand, for example in AWS EC2, but you'll have to add some credentials and settings:

$ cf-remote spawn --init-config
Config file /home/olehermanse/.cfengine/cf-remote/cloud_config.json created, please complete the configuration in it.
$ cat /home/olehermanse/.cfengine/cf-remote/cloud_config.json
{
  "aws": {
    "key": "TBD",
    "secret": "TBD",
    "key_pair": "TBD",
    "security_groups": [
      "TBD"
    ],
    "region": "OPTIONAL (DEFAULT: eu-west-1)"
  },
  "gcp": {
    "project_id": "TBD",
    "service_account_id": "TBD",
    "key_path": "TBD",
    "region": "OPTIONAL (DEFAULT: europe-west1-b)"
  }
}

You can skip the gcp values if you will only be using AWS. After filling out those 4, it should just work:

$ cf-remote spawn --count 1 --platform ubuntu-20-04-x64 --role hub --name hub
Spawning VMs....DONE
Waiting for VMs to get IP addresses..........DONE
Details about the spawned VMs can be found in /home/olehermanse/.cfengine/cf-remote/cloud_state.json

You can now install nightlies, and use the --demo to make testing easier (Not secure for production use). Referring to the group names set by spawn, makes the commands a lot shorter and easier to script:

$ cf-remote --version master install --hub hub --bootstrap hub --demo

ubuntu@52.214.209.170
OS            : ubuntu (debian)
Architecture  : x86_64
CFEngine      : Not installed
Policy server : None
Binaries      : dpkg, apt

Downloading package: '/home/olehermanse/.cfengine/cf-remote/packages/cfengine-nova-hub_3.18.0a.a24173342~12762.ubuntu18_amd64.deb'
Copying: '/home/olehermanse/.cfengine/cf-remote/packages/cfengine-nova-hub_3.18.0a.a24173342~12762.ubuntu18_amd64.deb' to 'ubuntu@52.214.209.170'
Installing: 'cfengine-nova-hub_3.18.0a.a24173342~12762.ubuntu18_amd64.deb' on 'ubuntu@52.214.209.170'
CFEngine 3.18.0a.a24173342 (Enterprise) was successfully installed on 'ubuntu@52.214.209.170'
Bootstrapping: '52.214.209.170' -> '172.31.5.84'
Bootstrap successful: '52.214.209.170' -> '172.31.5.84'
Transferring def.json to hub: 'ubuntu@52.214.209.170'
Copying: '/home/olehermanse/.cfengine/cf-remote/json/def.json' to 'ubuntu@52.214.209.170'
Triggering an agent run on: '52.214.209.170'
Disabling password change on hub: 'ubuntu@52.214.209.170'
Triggering an agent run on: '52.214.209.170'
Your demo hub is ready: https://52.214.209.170/ (Username: admin, Password: hJmZqRtvBkNwdc)

Mission portal will be available at that IP, using the username and password from the last log message. The password is randomly generated, so it differs from the one above.

When you are done, you can decommission your spawned instance(s) using:

$ cf-remote destroy --all
Destroying all hosts

Deploying a version of masterfiles you're working on locally

The deploy command allows you to deploy your local checkout of masterfiles, to test policy while working on it:

$ cf-remote deploy --hub hub ~/code/northern.tech/cfengine/masterfiles

ubuntu@18.202.238.128
OS            : ubuntu (debian)
Architecture  : x86_64
CFEngine      : 3.18.0a.a24173342 (Enterprise)
Policy server : None
Binaries      : dpkg, apt

Copying: '/home/olehermanse/.cfengine/cf-remote/masterfiles.tgz' to 'ubuntu@18.202.238.128'
Running: 'systemctl stop cfengine3 && rm -rf /var/cfengine/masterfiles && mv masterfiles /var/cfengine/masterfiles && systemctl start cfengine3 && cf-agent -Kf update.cf && cf-agent -K'
$

Specify an SSH key

If you have more than one key in ~/.ssh you may need to specify which key cf-remote is to use.

$ export CF_REMOTE_SSH_KEY="~/.ssh/id_rsa.pub"

Switching user on the remote hosts

Most of what cf-remote does needs root, so unless it logs in as root it runs commands through sudo. If sudo asks for a password, use --ask-pass (-K) and cf-remote prompts for it once and uses it for all the hosts in the run:

$ cf-remote --ask-pass install --clients ubuntu@10.0.0.5
Password for switching user:

The password is written to the standard input of the ssh process, so it is never part of a command line and doesn't show up in the process list, in the shell history on the target host, or in the output of --log-level DEBUG. It is only sent to hosts where switching user actually asks for a password.

Where there is nobody to answer a prompt, such as in a script or a CI job, put the password on the first line of a file and point --password-file at it:

$ cf-remote --password-file ~/.cf-remote-password install --clients ubuntu@10.0.0.5

cf-remote refuses to read the file if others can read it, the same way ssh refuses to use a private key with too generous permissions, so chmod 600 it first.

Use --switch-user-command if sudo is not what you want to switch user with:

$ cf-remote --ask-pass --switch-user-command "doas /bin/sh -c" info -H bsd-host

The command to run is appended as a single quoted argument. The default is sudo -n bash -c, or sudo -S -p '' bash -c with --ask-pass, since sudo only reads the password from standard input when it is given -S. -n in the first is because there is no terminal to prompt on, so a sudo that wants a password should say so instead of trying to ask; it is left out of the second because it means never prompt, and sudo then refuses the password rather than reading it.

Whichever command is used, it is run with LC_ALL=C. cf-remote recognizes "this needs a password you didn't give me" by what the command said, and sudo says it in the caller's language on the distributions that ship its translations, which ssh carries over by default. sudo keeps LC_ALL, so the command being run is left in the C locale as well; commands run without switching user are not.

A password can only reach a command that reads it from standard input, which in practice means sudo -S and the tools that copy its interface, such as dzdo -S. doas and su read from a terminal instead, so they work with --switch-user-command where they need no password, but cannot be given one by cf-remote.

Working on the local host

cf-remote can work on the local host when the target host is localhost. In this case, it executes commands locally without connecting over SSH.

$ cf-remote info -H localhost

ubuntu@localhost
OS            : ubuntu (debian)
Architecture  : x86_64
CFEngine      : 3.12.1
Policy server : 172.31.42.192
Binaries      : dpkg, apt

When performing actions locally, cf-remote may require your password to run commands with sudo:

$ cf-remote install --clients localhost
ubuntu@localhost
OS            : debian
Architecture  : x86_64
CFEngine      : Not installed
Policy server :
Binaries      : dpkg, apt
Installing: '/home/ubuntu/.cfengine/cf-remote/packages/cfengine-nova_3.15.3-1.debian10_amd64.deb' on 'localhost'
[sudo] password for ubuntu:
CFEngine 3.15.3 (Enterprise) was successfully installed on 'localhost'

Contribute

Feel free to open pull requests to expand this documentation, add features or fix problems. You can also pick up an existing task or file an issue in our bug tracker.

Development

This project uses uv for managing the virtual environment, dependencies, building, etc. To set up a virtual environment with all dependencies and run all formatters, linters, and tests, use:

$ make check

To install cf-remote so that it reflects any changes in this source directory use:

$ pipx install --force --editable .

cloud_data.py tips

In order to find AWS images for a particular owner to work on cloud_data.py name_pattern list the names for an owner with the following aws command:

aws ec2 describe-images --region us-east-2 --owners 801119661308 --query 'Images[*].[Name]' --output text