orderly-web 1.0.0

Deploy scripts for OrderlyWeb

OrderlyWeb (Deploy)

This is the deploy scripts for OrderlyWeb. They are the only part of the system that runs directly on metal. I am not good at python packaging so some documentation for help me. These might not be the best ways to do things but they seem to work:

Installation

From local sources

python3 setup.py install --user

(you may need --upgrade to upgrade older versions of python packages).

This installs the package orderly_web for programmatic use, and a cli tool orderly-web for interacting with the package:

Usage

$ orderly-web --help
Usage:
  orderly-web start <path> [--extra=PATH] [--option=OPTION]... [--pull]
  orderly-web status <path>
  orderly-web stop <path> [--volumes] [--network] [--kill] [--force] [--extra=PATH] [--option=OPTION]...
  orderly-web admin <path> add-users <email>...
  orderly-web admin <path> add-groups <name>...
  orderly-web admin <path> add-members <group> <email>...
  orderly-web admin <path> grant <group> <permission>...

Options:
  --extra=PATH     Path, relative to <path>, of yml file of additional
                   configuration
  --option=OPTION  Additional configuration options, in the form key=value
                   Use dots in key for hierarchical structure, e.g., a.b=value
                   This argument may be repeated to provide multiple arguments
  --pull           Pull images before starting
  --volumes        Remove volumes (WARNING: irreversible data loss)
  --network        Remove network
  --kill           Kill the containers (faster, but possible db corruption)
  --force          Force stop even if containers are corrupted and cannot 
                   signal their running configuration, or if config cannot be parsed.
                   Use with extra and/or option to force stop with configuration options.

Here <path> is the path to a directory that contains a configuration file orderly-web.yml (more options will follow in future versions).

Examples

To deploy with 2 users:

orderly-web start ./config/basic
orderly-web admin ./config/basic add-users [e-mail] [e-mail]

To grant the users permissions on an individual basis:

orderly-web admin ./config/basic grant [e-mail] */reports.read
orderly-web admin ./config/basic grant [e-mail] */reports.read */reports.review */reports.run

Or to add 2 user groups, "funders" and "admin", and grant users permissions via group membership:

orderly-web admin ./config/basic add-groups funders admin
orderly-web admin ./config/basic grant funders */reports.read
orderly-web admin ./config/basic grant admin */reports.read */reports.review */reports.run
orderly-web admin ./config/basic add-members funders [e-mail]
orderly-web admin ./config/basic add-members admin [e-mail]

Development

To test changes during development often the best way is to try and run a deployment. To do this you will need to install the development version of orderly-web on a server. The best way to do this is to clone the repo, set the branch to your development branch and follow instructions above for installation.

Testing

Running integration tests requires an environment variable VAULT_TEST_GITHUB_PAT to be available. This needs to be a github pat for a user who is a member of the robots team in the VIMC org. This can be read from the vault vault read secret/vimc-robot/github-pat. Save it into an environment variable through e.g. your ~/.bashrc file so it is available to the tests.

Run

python3 setup.py test

This will take a while. There will be lots of warnings like:

ResourceWarning: unclosed <socket.socket fd=8,

which are out of our control (see the helper docker_client in docker_helpers.py for details).

Configuration

Configuration is a work in progress and will change as the tool progresses. See [config/complete/orderly-web.yml] for an annotated configuration that covers all the options.

Modified versions of configurations

It is possible to create sub-configurations that adapt a configuration. To do this, create a base configuration with shared options and save that as orderly-web.yml. Then, within the same directory, create secondary yml files (named however you want) that override options. For example if you have an orderly-web.yml that contains

web:
  port: 443
  name: OrderlyWeb
  dev_mode: false

(among other options), you could create a yaml file called testing.yml (in the same directory) that contains

web:
  port: 8000
  dev_mode: true

When run with

orderly-web path --extra testing.yml

the options in testing.yml will override the base configuration. The options that are not mentioned in the testing.yml are left unmodified (i.e, in this case we end up with

web:
  port: 8000
  name: OrderlyWeb
  dev_mode: true

It is also possible to change options by passing individual changes through with the --option flag, for example:

orderly-web path --option web.port=8000 --option web.dev_mode=true

Use . to indicate a level of nesting and do not use spaces around the =; the right-hand-side is parsed as if it was yaml.