opthub_runner_admin 0.4.6

Library for running OptHub problems and indicators.

日本語版 👈

OptHub Runner

Opthub Runner is a Python package that provides the following two features:

• Evaluator*: Feature to evaluate user-submitted solutions on Opthub using Docker Image.
• Scorer*: Feature to calculate scores from a series of evaluations using Docker Image.

Please refer to 4.競技の確認 in OptHub Tutorial for detailed explanations of the Evaluator and Scorer.

This repository describes how to install and run Opthub Runner.

Getting Started

Recommended Environment

We recommend using the following environments:

• AWS EC2.
• GCP Compute Engine.
• Sakura Cloud.
• Any other environment that supports Docker and can run continuously.

Prerequisites

• Install Python 3.10 or newer.
• Set up pip for package management.
• Install and start Docker.*

*For Mac users, you can install and start Docker Desktop.

1. Installation

Install opthub-runner-admin from PyPI.

pip install opthub-runner-admin

2. Configuring YAML File

Create a YAML file with the options needed to run the Evaluator and Scorer, based on config.default.yml. For details on the options to set in the file, please refer to YAML File Options.

3. Starting the Evaluator/Scorer

At first, start Docker. Please note that Docker must remain running while the Evaluator/Scorer is active.

Then, run the Evaluator/Scorer using the following command. Replace <evaluator|scorer> with either evaluator or scorer. For --config, specify the path to the YAML file you created in 2. Configuring YAML File. If not specified, the config.yml in the current directory will be used.

opthub-runner <evaluator|scorer> --config <yaml file path>

You will be prompted to enter the username and password. These must be the ones of the competition administrator.

Username: (your username)
Password: (your password)

Please note that the Evaluator/Scorer must remain running during the competition. If you have any other issues, please refer to Troubleshooting.

YAML File Options

The following table lists the options to include in the YAML file, along with their type, default values, and descriptions. Please confirm the default values for the following options with the OptHub representative. Contact information is here.

• evaluator_queue_url
• scorer_queue_url
• access_key_id
• secret_access_key
• region_name
• table_name

Option	Type	Default Value	Description
interval	int	2	Interval to fetch messages from Amazon SQS.
timeout	int	43200	Timeout for evaluation and score calculation using Docker Image.
rm	bool	True	Whether to remove the Docker container after evaluation and score calculation. True is recommended except for debugging. If set to False, the Docker containers created during evaluation and scoring will not be removed, accumulating over time.
log_level	[DEBUG, INFO, WARNING, ERROR, CRITICAL]	INFO	Log level to output.
force	bool	False	Whether to forcibly create a new flag file.
evaluator_queue_url	path	-	Amazon SQS queue URL used by the Evaluator.
scorer_queue_url	path	-	Amazon SQS queue URL used by the Scorer.
access_key_id	str	-	AWS Access Key ID.
secret_access_key	str	-	AWS Secret Access Key.
region_name	str	-	AWS default Region Name.
table_name	str	-	DynamoDB table name to store solutions, evaluations, and scores.

Troubleshooting

Docker is Not Running

The following error occurs when Docker is not running.

Error: Unable to communicate with Docker. Please ensure Docker is running and accessible. (Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory')))

For Mac, you can resolve this issue by starting Docker Desktop. You can install Docker Desktop here.

For Ubuntu, you can resolve this issue by installing Docker via the terminal with the following command.

sudo apt install docker.io

No Permission to Access Docker Socket

The following error occurs when the current user does not have permission to access the Docker socket.

Error: Unable to communicate with Docker. Please ensure Docker is running and accessible. (Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied')))

For Ubuntu, you can resolve this issue by adding the user to the Docker group and then logging in again. Specifically, follow these steps:

1. Run the following command in the terminal:

   sudo usermod -aG docker $USER
   

2. Either log in again or run the following command in the terminal:

   newgrp docker
   

Not Logged in with Competition Administrator Account

The following error occurs if you did not log in with the competition administrator account when starting the Evaluator/Scorer.

Traceback (most recent call last):
  ･･･
opthub_runner_admin.models.exception.DockerImageNotFoundError: Cannot access the Docker image. Please check your permissions. If you're not authenticated using the competition administrator's account, please do so.

You can resolve this issue by logging in with the competition administrator’s account when starting the Evaluator/Scorer. When the following prompt appears, enter the administrator’s username and password.

Note: Make sure to authenticate using the competition administrator's account.
Username: (The administrator's username)
Password: (The administrator's password)

YAML File Not Found

The following error occurs when the YAML file containing the Evaluator/Scorer settings does not exist.

Traceback (most recent call last):
  ･･･
FileNotFoundError: Configuration file not found: config.yml

Please create a config.yml file by referring to 2. Configuring YAML File.

Incorrect AWS Access Key ID

The following error occurs when the AWS Access Key ID (access_key_id) set in config.yml is incorrect.

Traceback (most recent call last):
  ･･･
botocore.exceptions.ClientError: An error occurred (InvalidClientTokenId) when calling the ReceiveMessage operation: The security token included in the request is invalid.

Please check the AWS Access Key ID set in config.yml and enter the correct value.

Incorrect AWS Secret Access Key

The following error occurs when the AWS Secret Access Key (secret_access_key) set in config.yml is incorrect.

Traceback (most recent call last):
  ･･･
botocore.exceptions.ClientError: An error occurred (SignatureDoesNotMatch) when calling the ReceiveMessage operation: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

Please check the AWS Secret Access Key set in config.yml and enter the correct value.

Incorrect SQS Queue Name

The following error occurs when the SQS queue name (evaluator_queue_name / scorer_queue_name) set in config.yml is incorrect.

Traceback (most recent call last):
  ･･･
botocore.errorfactory.QueueDoesNotExist: An error occurred (AWS.SimpleQueueService.NonExistentQueue) when calling the ReceiveMessage operation: The specified queue does not exist.

Please check the SQS queue name set in config.yml and enter the correct value.

Incorrect DynamoDB Table Name

The following error occurs when the Amazon DynamoDB table name (table_name) set in config.yml is incorrect.

Traceback (most recent call last):
  ･･･
botocore.errorfactory.ResourceNotFoundException: An error occurred (ResourceNotFoundException) when calling the GetItem operation: Requested resource not found

Please check the DynamoDB table name set in config.yml and enter the correct value.

Incorrect AWS Default Region Name

The following error occurs when the AWS Default Region Name (region_name) set in config.yml is incorrect.

Traceback (most recent call last):
  ･･･
urllib3.exceptions.NameResolutionError: <botocore.awsrequest.AWSHTTPSConnection object at 0x104d8bce0>: Failed to resolve 'sqs.ap-northeas-1.amazonaws.com' ([Errno 8] nodename nor servname provided, or not known)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  ･･･
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://sqs.ap-northeas-1.amazonaws.com/"

Please check the AWS Default Region Name set in config.yml and enter the correct value.

Error Inside the Docker Container

The following error is displayed when an error occurs inside the Docker container.

ERROR:opthub_runner_admin.evaluator.main:Error occurred while evaluating solution.
Traceback (most recent call last):
  File "/home/opthub/.venv/lib/python3.11/site-packages/opthub_runner_admin/evaluator/main.py", line 125, in evaluate
    evaluation_result = execute_in_docker(
                        ^^^^^^^^^^^^^^^^^^
  File "/home/opthub/.venv/lib/python3.11/site-packages/opthub_runner_admin/lib/docker_executor.py", line 88, in execute_in_docker
    raise RuntimeError(msg)
RuntimeError: Failed to parse stdout.

To investigate this error in detail, you need to check the Docker container logs. Modify the rm setting in config.yml to False, then restart the Evaluator/Scorer and, after execution, check the logs of the container. You can view the Docker container logs with the following command:

$ docker ps # Check Container ID
$ docker logs <Container ID> # View logs

Check the logs and modify the Docker image to ensure the Docker container output is in the correct format.

Additionally, the following container log shows an error that occurs when the built Docker image is run on an incompatible platform.

exec /usr/bin/sh: exec format error

To ensure compatibility, either match the environment used to build the Docker image with the environment used to run the Evaluator/Scorer, or use buildx to create a compatible Docker image.

For Contributors

Follow these steps to set up the environment:

1. Clone this repository.
2. Set up Poetry.
3. Run poetry install.
4. Download the recommended VSCode Extensions.
5. Disable the following VS Code Extensions for this workspace to avoid conflicts with other packages:
   • ms-python.pylint
   • ms-python.black-formatter
   • ms-python.flake8
   • ms-python.isort
6. Create config.yml with options based on config.default.yml.
   • Place config.yml directly under opthub-runner-admin.
   • For the values of the options commented out in config.default.yml, please contact the Opthub representative. Contact information is here.

Once you have completed the above setup, you can use the opthub-runner command in the project's root directory.

Contact

If you have any questions or concerns, please feel free to contact us (Email: [e-mail]).