jupyter-deploy-tf-aws-ec2-base 0.2.0

Base terraform template to deploy jupyter notebook on AWS EC2.

AWS EC2 instance running a Jupyter Server with GitHub OAuth access

---

This terraform project creates an EC2 instance in the default VPC and a route 53 record in a domain you own. Within the EC2 instance, it runs a jupyter service, a traefik service for proxy and an oauth sidecar for authentication and authorization.

The instance is configured so that you can access it using AWS SSM.

This project:

• places the instance in the first subnet of the default VPC
• selects the latest AL 2023 kernel default AMI for x86_64 architecture
• sets up an IAM role to enable SSM, route53 and (optionally) EFS access
• passes on the root volume of the AMI
• adds an EBS volume which will mount on the Jupyter Server container
• adds an Elastic IP (EIP) to keep the public IP of the instance stable
• writes docker service logs to disk at /var/log/services using fluent-bit
• configures automatic rotation for all log files using logrotate
• creates an SSM instance-startup script, which references several files:
  • cloudinit.sh.tftpl to configure the EC2 instance
  • docker-compose.yml.tftpl to configure the docker services
  • docker-startup.sh.tftpl to start the docker services
  • cloudinit-volumes.sh.tftpl to optionally mount additional elastic block store (EBS) or elastic file systems (EFS)
  • traefik.yml.tftpl to configure traefik
  • dockerfile.jupyter to build the Jupyter container
  • jupyter-start.sh to provide entrypoint script for the Jupyter container
  • jupyter-reset.sh to provide a fallback if the Jupyter container fails to start
  • pyproject.jupyter.toml to configure the Python dependencies of the base environment where the Jupyter server runs
  • jupyter_server_config.py to configure Jupyter server
  • dockerfile.logrotator to configure the sidecar container rotating log files on disk
  • logrotator-start.sh.tftpl to configure logrotate
  • fluent-bit.conf to configure the fluent-bit service writing docker service logs to /var/log/services
  • parsers.conf to configure the fluent-bit docker parsers
  • check-status-internal.sh to verify that the services are up and the TLS certificates are available
  • get-status.sh to translate the return code of check-status script to a human-readable status
  • update-auth.sh to update the authorized org, teams, and/or users
  • get-auth.sh to retrieve the authorized org, teams, and/or users
  • update-server.sh to update the services running within the host
  • refresh-oauth-cookie.sh to rotate the oauth cookie secret and invalidate all issued cookies
• creates an SSM association, which runs the startup script on the instance
• creates the Route 53 Hosted Zone for the domain unless it already exists
• adds the DNS record to the Route 53 Hosted Zone
• creates an AWS Secret to store the OAuth App client secret
• optionally creates or references EBS volumes or EFS and mount them to the home directory of the jupyter app
• provides two presets default values for the template variables:
  • defaults-all.tfvars comprehensive preset with all the recommended values
  • defaults-base.tfvars more limited preset; it will prompt user to select the instance type and volume size
• creates AWS SSM documents for jupyter-deploy commands

Prerequisites

• a domain that you own verifiable by route 53
  • instructions to register a domain
  • instructions to acquire a domain
• a GitHub OAuth App
  • instructions to create a new app
  • you'll need the app client ID and client secret
• at least one of the following for GitHub authorization:
  • a list of GitHub usernames to authorize
  • the name of a GitHub organization whose members to authorize; optionally restrict further by GitHub teams

Usage

This terraform project is meant to be used with jupyter-deploy.

Installation (with pip):

Create or activate a python environment.

pip install jupyter-deploy
pip install jupyter-deploy-tf-aws-ec2-base

Project setup

Consider making my-jupyter-deployment a git repository.

mkdir my-jupyter-deployment
cd my-jupyter-deployment

jd init . -E terraform -P aws -I ec2 -T base

Configure and create the infrastructure

jd config
jd up

Access your notebook

jd open

Manage access

** By GitHub users

jd users list
jd users add USERNAME1 USERNAME2
jd users remove USERNAME1

** By GitHub organization

jd organization get
jd organization set ORGANIZATION
jd organization unset

** Along with GitHub organization, by teams:

jd teams list
jd teams add TEAM1 TEAM2
jd team remove TEAM2

Temporarily stop your instance

jd host stop
jd host start
jd host status

Take down all the infrastructure

jd down

Requirements

Name	Version
terraform	>= 1.0
aws	>= 4.66

Providers

Name	Version
aws	>= 4.66

Modules

No modules.

Resources

Name	Type
aws_security_group	resource
aws_instance	resource
aws_iam_role	resource
aws_iam_role_policy_attachment	resource
aws_iam_instance_profile	resource
aws_ebs_volume	resource
aws_volume_attachment	resource
aws_ssm_document	resource
aws_ssm_association	resource
aws_route53_zone	resource
aws_route53_record	resource
aws_secretsmanager_secret	resource
aws_iam_policy	resource
aws_ssm_parameter	resource
null_resource	resource
aws_default_vpc	resource
aws_ebs_volume	resource
aws_efs_file_system	resource
aws_efs_mount_target	resource
aws_eip	resource
aws_subnets	data source
aws_subnet	data source
aws_ami	data source
aws_route53_zone	data source
aws_ebs_volume	data source
aws_efs_file_system	data source
aws_iam_policy	data source
aws_iam_policy_document	data source
local_file	data source

Inputs

Name	Type	Default	Description
region	string	us-west-2	The AWS region where to create the resources
instance_type	string	t3.medium	The type of instance to start
key_pair_name	string	null	The name of key pair
ami_id	string	null	The ID of the AMI to use for the instance
volume_size_gb	number	30	The size in GB of the EBS volume the Jupyter Server has access to
volume_type	string	gp3	The type of EBS volume the Jupyter Server will has access to
iam_role_prefix	string	Jupyter-deploy-ec2-base	The prefix for the name of the IAM role for the instance
oauth_app_secret_prefix	string	Jupyter-deploy-ec2-base	The prefix for the name of the AWS secret to store your OAuth app client secret
letsencrypt_email	string	Required	An email for letsencrypt to notify about certificate expirations
domain	string	Required	A domain that you own
subdomain	string	Required	A sub-domain of domain to add DNS records
oauth_provider	string	github	The OAuth provider to use
oauth_allowed_org	string	""	The GitHub organization to allowlist
oauth_allowed_teams	list(string)	[]	The list of GitHub teams to allowlist
oauth_allowed_usernames	list(string)	[]	The list of GitHub usernames to allowlist
oauth_app_client_id	string	Required	The client ID of the OAuth app
oauth_app_client_secret	string	Required	The client secret of the OAuth app
log_files_rotation_size_mb	number	50	The size in megabytes at which to rotate log files
log_files_retention_count	number	10	The maximum number of rotated log files to retain for a log group
log_files_retention_days	number	180	The maximum number of days to retain any log files
custom_tags	map(string)	{}	The custom tags to add to all the resources
additional_ebs_mounts	list(map(string))	[]	Elastic block stores to mount on the notebook home directory
additional_efs_mounts	list(map(string))	[]	Elastic file systems to mount on the notebook home directory

Outputs

Name	Description
jupyter_url	The URL to access your notebook app
auth_url	The URL for the OAuth callback - do not use directly
instance_id	The ID of the EC2 instance
ami_id	The Amazon Machine Image ID used by the EC2 instance
jupyter_server_public_ip	The public IP assigned to the EC2 instance
secret_arn	The ARN of the AWS Secret storing the OAuth client secret
region	The AWS region where the resources were created
server_status_check_document	Name of the SSM document to verify if the server is ready to serve traffic
server_update_document	Name of the SSM document to control server container operations
auth_org_unset_document	Name of the SSM document to remove the allowlisted organization
auth_check_document	Name of the SSM document to view authorized users, teams and organization
auth_users_update_document	Name of the SSM document to change the authorized users
auth_teams_update_document	Name of the SSM document to change the authorized teams
auth_org_set_document	Name of the SSM document to allowlist an organization
auth_org_unset_document	Name of the SSM document to remove the allowlisted organization