Skip to main content

Spawns JupyterHub single user servers in Docker containers running in AWS Fargate

Project description

fargatespawner

Spawns JupyterHub single user notebook servers in Docker containers running in AWS Fargate

Installation

pip install fargatespawner

Configuration

To configure JupyterHub to use FargateSpawner, you can add the following to your jupyterhub_config.py.

from fargatespawner import FargateSpawner
c.JupyterHub.spawner_class = FargateSpawner

You must also set the following settings on c.FargateSpawner in your jupyterhub_config.py. None of them are optional.

Setting Description Example
aws_region The AWS region in which the tasks are launched. 'eu-west-1'
aws_ecs_host The hostname of the AWS ECS API. Typically, this is of the form ecs.<aws-region>.amazonaws.com. 'ecs.eu-west-1.amazonaws.com'
notebook_port The port the notebook servers listen on. 8888
notebook_scheme The scheme used by the hub and proxy to connect to the notebook servers. At the time of writing 'https' will not work out of the box. However, users do not connect to the the notebook server directly, and does not, typically, allow incoming connections from the public internet. Instead, users connect to the proxy, which can be configured to listen on HTTPS independently of this setting. There is more information on setting up HTTPS for connections to the proxy at https://jupyterhub.readthedocs.io/en/stable/getting-started/security-basics.html. 'http'
get_run_task_args See below See below

The get_run_task_args argument must a callable that takes the spawner instance as a parameter, and returns a dictionary that is passed to the RunTask API call.

c.FargateSpawner.get_run_task_args = lambda spawner: {
    'cluster': 'jupyerhub-notebooks',
    'taskDefinition': 'jupyterhub-notebook:7',
    'overrides': {
        'taskRoleArn': 'arn:aws:iam::123456789012:role/notebook-task',
        'containerOverrides': [{
            'command': spawner.cmd + [f'--port={spawner.notebook_port}', '--config=notebook_config.py'],
            'environment': [
                {
                    'name': name,
                    'value': value,
                } for name, value in spawner.get_env().items()
            ],
            'name': 'jupyterhub-notebook',
        }],
    },
    'count': 1,
    'launchType': 'FARGATE',
    'networkConfiguration': {
        'awsvpcConfiguration': {
            'assignPublicIp': 'DISABLED',
            'securityGroups': ['sg-00026fc201a4e374b'],
            'subnets':  ['subnet-01fc5f15ac710c012'],
        },
    },
}

The parts of the return value that probably require change from the above example are:

  • cluster - The name of the ECS cluster into which the tasks are launched.

  • taskDefinition - The family:revision or full ARN of the task definition that runs the notebooks. Typically, this task definition would specify a docker image that builds on one from https://github.com/jupyter/docker-stacks.

  • taskRoleArn - The role the notebook tasks can assume. For example, in order for them to make requests to AWS, such as to use Jupyter S3 with role-based authentication.

  • securityGroups - The security groups associated with the Fargate task. They must allow communication to and from the hub/proxy. More information, such as the ports used, is at https://jupyterhub.readthedocs.io/en/stable/getting-started/networking-basics.html.

  • subnets - The list of possible subnets in which the task will start.

You must also, either, authenticate using a secret key, in which case you must have the following configuration

from fargatespawner import FargateSpawnerSecretAccessKeyAuthentication
c.FargateSpawner.authentication_class = FargateSpawnerSecretAccessKeyAuthentication

and the following settings on c.FargateSpawnerSecretAccessKeyAuthentication

Setting Description Example
aws_access_key_id The ID of the AWS access key used to sign the requests to the AWS ECS API. ommitted
aws_secret_access_key The secret part of the AWS access key used to sign the requests to the AWS ECS API . ommitted

or authenticate using a role in an ECS container, in which case you must have the following configuration

from fargatespawner import FargateSpawnerECSRoleAuthentication
c.FargateSpawner.authentication_class = FargateSpawnerECSRoleAuthentication

where FargateSpawnerECSRoleAuthentication does not have configurable options.

or authenticate using a IAM profile attached to EC2 instance, in which case you must have the following configuration:

from fargatespawner import FargateSpawnerEC2InstanceProfileAuthentication
c.FargateSpawner.authentication_class = FargateSpawnerEC2InstanceProfileAuthentication

Run-time dependencies

The spawner is deliberately written to not have any additional dependencies, beyond those that are required for JupyterHub.

Approximate minimum permissions

In order for the user to be able to start, monitor, and stop the tasks, they should have the below permissions.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": "ecs:RunTask",
      "Resource": "arn:aws:ecs:<aws_region>:<aws_account_id>:task-definition/<task_family>:*",
      "Condition": {
        "ArnEquals": {
          "ecs:cluster": "arn:aws:ecs:<aws_region>:<aws_account_id>:cluster/<cluster_name>"
        }
      }
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": "ecs:StopTask",
      "Resource": "arn:aws:ecs:<aws_region>:<aws_account_id>:task/*",
      "Condition": {
        "ArnEquals": {
          "ecs:cluster": "arn:aws:ecs:<aws_region>:<aws_account_id>:cluster/<cluster_name>"
        }
      }
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": "ecs:DescribeTasks",
      "Resource": "arn:aws:ecs:<aws_region>:<aws_account_id>:task/*",
      "Condition": {
        "ArnEquals": {
          "ecs:cluster": "arn:aws:ecs:<aws_region>:<aws_account_id>:cluster/<cluster_name>"
        }
      }
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": "iam:PassRole",
      "Resource": [
        "arn:aws:iam::<aws_account_id>:role/<task-execution-role>",
        "arn:aws:iam::<aws_account_id>:role/<task-role>"
      ]
    }
  ]
}

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fargatespawner-0.0.25.tar.gz (8.2 kB view details)

Uploaded Source

Built Distribution

fargatespawner-0.0.25-py3-none-any.whl (8.8 kB view details)

Uploaded Python 3

File details

Details for the file fargatespawner-0.0.25.tar.gz.

File metadata

  • Download URL: fargatespawner-0.0.25.tar.gz
  • Upload date:
  • Size: 8.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.4.2 requests/2.20.1 setuptools/47.1.1 requests-toolbelt/0.8.0 tqdm/4.28.1 CPython/3.7.1

File hashes

Hashes for fargatespawner-0.0.25.tar.gz
Algorithm Hash digest
SHA256 8f2aa3c11b85001373fa2ae2c658880b4a97d089c5868a8c270cc09112cac56b
MD5 58c78042a6bc6727bdd7409bd6c8d086
BLAKE2b-256 51b05d47af2cbabb2f0fe349ca9b5a1cb293543c1af6d343768d68b5f761c499

See more details on using hashes here.

File details

Details for the file fargatespawner-0.0.25-py3-none-any.whl.

File metadata

  • Download URL: fargatespawner-0.0.25-py3-none-any.whl
  • Upload date:
  • Size: 8.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.4.2 requests/2.20.1 setuptools/47.1.1 requests-toolbelt/0.8.0 tqdm/4.28.1 CPython/3.7.1

File hashes

Hashes for fargatespawner-0.0.25-py3-none-any.whl
Algorithm Hash digest
SHA256 e69808412f3a9371f6fb473c7a44b7ca217f1878d8b74c96b157417705dc8a60
MD5 3802137edae490be09c75cc034b54cfc
BLAKE2b-256 7e3effcf2b8ac3f4d46a113b75e8eb073cc018d5639b810704087e06103494a8

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page