usagi 0.3.1

Web API testing for haas

usagi is a plugin for haas that adds support for discovering Web API test cases descibed in YAML.

usagi requires haas v0.6.0 or later.

Current Features

• Describe web API tests using YAML.

• Template URLs to avoid repeating items.

• Variables and target hostname can be provided by environment variables.

• YAML format contains multiple test cases.

• Each test case is a grouping of related tests.

• Make flexible assertions about the server’s response.

• Contribute web API test runner functionality through plugins.

• Assert that the SHA256 of the response body matches an expected value.

• Filter JSON response bodies before comparison using jq filter syntax: http://stedolan.github.io/jq/

Plugins

• Var loaders

  • Load from environment

  • Template string based on other vars

  • Load from file, either plaintext or JSON

• Assertions

  • Assert status code

  • Assert header is present or matches value or regexp

TODO

• Add more assertions!

Test config format

• version: Currently required, but unverified (we are at v0.1.0.devN, after all).

• config: Common test case configuration.

  • host: The name (or IP) of the host to test.

    • Can come from env, template, file, like vars.

  • scheme: The scheme (http, https) to use to connect to host

  • vars: Common variable definitions for all test cases; formatted as a dictionary of var name to type and value.

    • Simple string vars are allowed.

    • Others are specified as a dictionary with key type to determine how to load.

• cases: Collection of test cases. Each case contains multiple tests

  • name: The name of the test case

  • tests: Collection of individual tests

    • name: The name of the test

    • url: The URI/path relative to host against which the test will be executed.

    • method: The HTTP method to use for the test.

    • assertions: List of assertions to make about the test.

Example Test

---
  version: '1.0'

  config:
    # Host is loaded as an environment variable
    host:
      type: env
      env: TEST_HOSTNAME
    vars:

      # Simple string var
      api_root: "/api/v1/json"

      # Template var
      metadata:
        type: template
        template: "{api_root}/metadata"

      # Variable loaded as JSON from file
      expected_index:
        type: file
        file: some_file.json
        format: json

  cases:
    - name: "Basic"
      tests:
        - name: "Test root URL"
          url: "/"
          assertions:
            - name: status_code
              expected: 200
            - name: header
              header: Content-Type
              value: text/plain

    - name: "A case"
      tests:
        - name: "Authentication failure"
          url:
            type: template
            template: "{metadata}/auth/required"
          parameters:
            method: GET
            headers:
              Content-Type: application/json
          assertions:
            - name: status_code
              expected: 401
            - name: header
              header: WWW-Authenticate
              regexp: "Basic realm=.*"
        - name: "POST json"
          url:
            type: template
            template: "{metadata}/post"
          parameters:
            method: POST
            body:
              format: json
              lookup-var: false
              value:
                some: ["json-compatible", "structure"]
          assertions:
            - name: status_code
              expected: 204