Metadata proxy

currently undergoing some refactoring

Very simple project to present an AWS compatible metadata service to servers on-premise funneling all of the credential requests through a central server which either has AWS credentials or is running in AWS and has a host iam role.

Installation

Metadata Server

TODO - make docker container

Run the following docker container terrycain/SOMECONTAINER It listens on HTTP 8000 so put it behind a HTTPS loadbalancer / reverse proxy,

TODO - Create the following Dynamo Tables metadata-proxy-hosts metadata-proxy-containers metadata-proxy-users

Configuration

Here are some environment variables used to configure the metadata server, defaults are in parenthesis at the beginning.

• REG_KEY - Random string used for initial client registration.
• HOSTNAME_PREFIX - (ip) Hostnames will be generated like so: 192.168.0.1 -> HOSTNAME_PREFIX-192-168-0-1 .
• IAM_HOST_ROLE_CACHE_TTL - (120) Time in seconds to cache the list of IAM roles used for a dropdown list in the UI. This can take a while at times..

Currently Redis is a dependency but that can be a standalone redis ran alongside this container, will look to make it o ptional in the future.

• REDIS_HOST - (localhost) Redis hostname.
• REDIS_PORT - (6379) Redis port.
• REDIS_DB - (0) Redis DB.

STS credentials are cached (currently in redis) and they are encrypted with an AES key. This allows for multiple servers to be ran in a HA manner and also reduces STS.assume_role requests.

• STS_SEED - This should be a random secret that is long.

User authentication is either based on users in a DynamoDB table or via OpenID Connect.

• USER_AUTH_METHOD - (oidc) Either oidc or dynamo
• OIDC_CLIENT_ID - OpenID Connect client ID.
• OIDC_CLIENT_SECRET - Client secret.
• OIDC_BASE_URL - Base url of the OpenID server, at startup the server will hit the wellknown metadata url to get OpenID config.

DynamoDB configuration

• DYNAMODB_REGION - (eu-west-1) Region the DynamoDB tables are located
• DYNAMODB_HOSTS_TABLE - (metadata-proxy-hosts) Table that stores registered hosts and what role they have been assigned
• DYNAMODB_CONTAINER_TABLE - (metadata-proxy-containers) Table that contains
• DYNAMODB_USERS_TABLE - (metadata-proxy-users)

Metadata proxy

Todo - make pip package

Linux

Install the Python package (requires Python 3.5.3+)

sudo pip3 install terrycain-metadata-proxy[uvloop]

Create the following systemd unit file and associated users

sudo groupadd --system metadata-proxy
sudo useradd --system --gid metadata-proxy --home-dir /var/lib/metadata-proxy --shell /sbin/nologin metadata-proxy
sudo mkdir /var/lib/metadata-proxy
sudo chown metadata-proxy:metadata-proxy /var/lib/metadata-proxy
sudo chown 0700 /var/lib/metadata-proxy
cat << EOF | sudo tee /etc/systemd/system/metadata-proxy.service
[Unit]
Description=Metadata proxy
Requires=network.target
After=network.target

[Service]
Environment=PYTHONUNBUFFERED=1
TimeoutStartSec=0
Restart=always
User=metadata-proxy
Group=metadata-proxy
WorkingDirectory=/var/lib/metadata-proxy
PermissionsStartOnly=true
ExecStartPre=/sbin/iptables -t nat -I OUTPUT -p tcp -d 169.254.169.254 --dport 80 -j DNAT --to-destination 127.0.0.1:8000
ExecStart=/usr/local/bin/metadata-proxy
ExecStopPost=/sbin/iptables -t nat -D OUTPUT -p tcp -d 169.254.169.254 --dport 80 -j DNAT --to-destination 127.0.0.1:8000

[Install]
WantedBy=default.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now metadata-proxy

Test that the proxy is now running and the iptables rules are working

curl http://169.254.169.254/latest/meta-data/instance-type ; echo
# Should output:
# c5.16xlarge

Register the metadata service against the master server with the curl command from the UI

curl --noproxy '*' -XPOST http://169.254.169.254/register -H "Content-Type: application/json" --data '{"server_url": "https://metadata-eu.ficoccs-prod.net/api/v1/register", "key": "0753e6f9-5884-462d-bb46-d376f27047a1"}' 

Windows

Installing on Windows (Requires python 3.5.3+)

pip3 install terrycain-metadata-proxy

Create a service on Windows TODO

Register the metadata service against the master server with the curl command from the UI

TODO List

• Better logging
• Add support for cert.pem, key.pem, chain.pem ENV vars so that it can do HTTPS
• Finish documentation