Google Cloud Platform Condor Pool Manager (GCPM)
HTCondor pool manager for Google Cloud Platform.
Installation
Package installation
GCPM can be installed by pip:
$ pip install gcpm
Service file installation
To install as service, do:
$ gcpm install
:warning: Service installation is valid only for the system managed by Systemd.
If logrotate is installed, logrotation definition for /var/log/gcpm.log is also installed.
Configuration file
Configuration file path
The default configuration file is ~/.config/gcpm/gcpm.yml.
For service, the configuration file is /etc/gcpm.yml.
To change the configuration file, use --config option:
$ gcpm run --config /path/to/my/gcpm.yml
Configuration file content
A configuration file is YAML format.
| Name | Description | Default Value | Mandatory |
|---|---|---|---|
| config_dir | Directory for some gcpm related files. | ~/.config/gcpm/ (user) /var/cache/gcpm (service) |
No |
| oatuh_file | Path to OAuth information file for GCE/GCS usage. | <config_dir>/oauth | No |
| service_account_file | Service account JSON file for GCE/GCS usage. If not specified, OAuth connection is tried. |
- | No |
| project | Google Cloud Platform Project Name. | - | Yes |
| zone | Zone for Google Compute Engine. | - | Yes |
| machines | Array of machine settings. Each setting is array of [core, mem, disk, idle, image] (see below). |
[] | Yes |
| machines:core | Number of core of the machine type. | - | Yes |
| machines:mem | Memory (MB) of the machine type. | - | Yes |
| machines:swap | Swap memory (MB) of the machine type. | Same as mem | No |
| machines:disk | Disk size (GB) of the machine type. | - | Yes |
| machines:max | Limit of the number of instances for the machine type. | - | Yes |
| machines:idle | Number of idle machines for the machine type. | - | Yes |
| machines:image | Image of the machine type. | - | Yes |
| machines:<others> | Other any options can be defined for creating instance. | - | No |
| max_cores | Limit of the total number of cores of all instances. If it is set 0, no limit is applied. |
0 | No |
| static_wns | Array of instance names of static worker nodes, which are added as condor worker nodes. | [] | No |
| required_machines | Array of machines which should be running other than worker nodes. | [] | No |
| required_machines:name | Number of core of the machine type. | - | Yes |
| required_machines:mem | Memory (MB) of the machine type. | - | Yes |
| required_machines:swap | Swap memory (MB) of the machine type. | Same as mem | No |
| required_machines:disk | Disk size (GB) of the machine type. | - | Yes |
| required_machines:image | Image of the machine type. | - | Yes |
| required_machines:<others> | Other any options can be defined for creating instance. | - | No |
| primary_accounts | User accounts which jobs must run normal worker nodes. See below about primary accounts. | [] | No |
| prefix | Prefix of machine names. | gcp-wn | No |
| preemptible | 1 for preemptible machines, 0 for not. | 0 | No |
| off_timer | Second to send condor_off after starting. | 0 | No |
| startup_cmd | Additional commands at WN startup. | "" | No |
| shutdown_cmd | Additional commands at WN shutdown. | "" | No |
| network_tag | Array of GCP network tag. | [] | No |
| reuse | 1 to reused terminated instance. Otherwise delete and re-created instances. | 0 | No |
| interval | Second of interval for each loop. | 10 | No |
| clean_time | Time to clean up residual instances in starting/deleting status. | 600 | No |
| head_info | If head is empty, head node information is automatically taken for each option: hostname: Hostname ip: IP address gcp: Hostname |
gcp | No |
| head | Head node Hostname/IP address. | "" | No |
| port | HTCondor port. | 9618 | No |
| domain | Domain of the head node. Set empty to take it from hostnaem. |
"" | No |
| admin | HTCondor admin email address. | "" | Yes |
| owner | HTCondor owner name. | "" | Yes |
| wait_cmd | 1 to wait GCE commands result (create/start/stop/delete...). | 0 | No |
| bucket | Bucket name for pool_password file. | "" | Yes |
| storageClass | Storage class name of the bucket. | "REGIONAL" | No |
| location | Storage location for the bucket. If empty, it is decided from the zone. |
"" | No |
| log_file | Log file path. Empty to put it in stdout. | "" | No |
| log_level | Log level. (debug, info, warning, error, critical) | info | No |
Note:
- Primary accounts
If primary accounts are set, jobs of non-primary accounts can run on test worker nodes.
If there are already max number of 1 core worker nodes and idle jobs of non-primary accounts are there, test worker node named <prefix>-test-1core-XXXX will be launched and only non-primary account jobs can run on it.
This able to run such a test job w/o waiting for finishing any normal jobs.
Such test worker nodes can be launched until total cores are smaller than max_core.
To use this function effectively, set total of max of each core to less than max_core.
e.g.)
---
machines:
core: 1
max: 10
machines:
core: 8
max: 2
max_core: 20
primary_accounts:
- condor_primary
In this case, normal jobs can launch 10 1-core machines and 2 8-core machines, then 16 cores are used.
Even if there are a log of idle condor_primary's jobs, 1 core test jobs by other accounts can run: 4 jobs at most.
Puppet setup
A puppet module for GCPM.
A puppet example to create head (manager) node and worker node with puppet.
A puppet example to create frontier squid proxy server in GCP.
1 maintainer
Metadata
1 maintainer
Release files for gcpm 0.3.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| gcpm-0.3.1.tar.gz | 22.6 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| gcpm-0.3.1-py2.py3-none-any.whl | Python 2, Python 3 | none | any | Details |
Total release size: 45.9 kB
Release files / gcpm-0.3.1.tar.gz
| Download URL | gcpm-0.3.1.tar.gz |
|---|---|
| Size | 22.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
0166da5c506ff1e4efbbe3e49c3b430b3ab1a79e11f813243a61dbf9a10f04cb
|
|
BLAKE2b-256 checksum How to use checksums |
d64cee8ef50faf9f548c5a350df585372abd181d9053f0576bbfc3837d900f3a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.0.5 CPython/3.7.7 Linux/3.10.0-1062.4.3.el7.x86_64
|
Release files / gcpm-0.3.1-py2.py3-none-any.whl
| Download URL | gcpm-0.3.1-py2.py3-none-any.whl |
|---|---|
| Size | 23.3 kB |
| Tags | Python 2 Python 3 |
|
SHA-256 checksum How to use checksums |
71dfc61c15d9d6253aa04e04f51ca462d3a69cf9f770ab8de0e8c2f00a2b3fec
|
|
BLAKE2b-256 checksum How to use checksums |
6f17c1ccfb3f0a6baaa59dab63d91df5d4037b96f052ecd4be09c55614c335bb
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.0.5 CPython/3.7.7 Linux/3.10.0-1062.4.3.el7.x86_64
|
Security for gcpm 0.3.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
