preacher 0.10.1

A web API verification tool.

Preacher verifies API servers, which requests to API servers and verify the response along to given scenarios.

Scenarios are written in YAML, bodies are analyzed jq or XPath queries and validation rules are based on Hamcrest (PyHamcrest) so that any developers can write without learning toughly.

Contents

• Development Policy

• Usage

• Writing Your Own Scenarios

• License

Development Policy

Preacher aims to automate tests using real backends: neither mocks or sandboxes. Supporting both automation and real backends has been challenging.

Preacher prefers:

• Flexible validation: Real backends causes fuzzy behavior. Matcher-based validation allow fuzziness caused by real backends.

• CI integration: CI tools are basic automation ways. CLI and YAML based test scenarios are suitable for CI.

• Simple GET requests: Testing with real backends often targets data fetching rather than HTTP interactions such as authorization. Development for complex HTTP interactions is less priored.

Comparing to other similar tools:

• Postman gives rich insights on Web APIs. On the other hand, testing with CLI tools and configuration files is more suitable for CI automation.

• Tavern is more suitable for testing HTTP interactions. It seems to be more suitable for testing simple systems or testing without real backends than Preacher because of simple validators.

Usage

Requirements

Supports only Python 3.7+.

pip install preacher
preacher-cli scenario.yml

# Help command is available.
preacher-cli --help

Writing Your Own Scenarios

Example

Here is a simple configuration example.

label: Scenario example
default:
  request:
    path: /path
  response:
    body:
      analyzed_as: json
cases:
  - label: Simple
    request: /path/to/foo
    response:
      status_code: 200
      body:
        - describe: .foo
          should:
            equal: bar
  - label: A Little Complecated
    request:
      path: /path/to/foo
      headers:
        user-agent: custom-value
      params:
        key1: value
        key2:
          - value1
          - value2
    response:
      status_code:
        - be_greater_than_or_equal_to: 200
        - be_less_than: 400
      headers:
        - describe: ."content-type"
          should:
            equal_to: application/xml
      body:
        analyzed_as: xml
        descriptions:
          - describe: /html/body/h1
            should:
              - start_with: x
              - end_with: y

Grammer

Scenario

A Scenario is written in YAML. A Scenario is a mapping that consists of below:

• label: String (Recommended)

  • A label of this scenario.

  • This field is actually optional but recommended to tell this scenario from another.

• default: Default (Optional)

  • Default of this scenario.

• cases: List<Case>

  • Test cases.

Case

A Case is a mapping that consists of below:

• label: String (Recommended)

  • A label of this case.

  • This field is actually optional but recommended to tell this case from another.

• request: Request (Optional)

  • A request.

• response: ResponseDescription (Optional)

  • A response description.

Request

A Request is a mapping or a string.

A mapping for Request has items below:

• path: String (Optional)

  • A request path. The default value is ''.

• Headers: Mapping<String, String> (Optional)

  • Request headers as a mapping of names to values.

• params: Mapping<String, String> (Optional)

  • Query parameters as a mapping of keys to values.

When given a string, that is equivalent to {"path": it}.

Response Decription

A ResponseDescription is a mapping that consists of below:

• status_code: Integer, Predicate or List<Predicate> (Optional)

  • Predicates that match a status code as an integer value.

  • When given a number, that is equivalent to {"equal": it}.

• headers:

  • Descriptions that descript the response headers.

  • Response headers are validated as a mapping of names to values and can be descripted by jq_ query (e.g. ."content-type"). Note that Names are lower-cased to normalize.

• body: BodyDescription (Optional)

  • A description that descript the response body.

Body Description

A BodyDescription is a mapping or a list.

A mapping for BodyDescription has items below.

• analyzed_as: String (Optional)

  • The method to analyze the body. The default value is json.

  • When given json, the body is analyzed as a JSON.

  • When given xml, the body is analyzed as an XML.

• descriptions: Description or List<Description>

  • Descriptions that descript the response body.

When given a list, that is equivalent to {"descritptions": it}.

Description

A Description is a mapping that consists of below:

• describe: Extraction

  • An extraction process.

• should: Predicate, or List<Predicate>> (Optional)

  • Predicates that match the descripted value.

Extraction

An Extraction is a mapping or a string.

A mapping for Extraction has one of below:

• jq: String

  • A jq query.

• xpath: String

  • A XPath query

• multiple: Boolean (Optional)

  • When given true, it returns the list of all extracted values.

  • When given false, it returns the first of extracted values.

  • The default value is false.

• cast_to: String (Optional)

  • When given, it returns the casted value.

  • Allowed values are int, float, string.

  • Casting does not affect null.

When given a string, that is equivalent to {“jq”: it}.

Note

The extraction must be compatible for the body analysis.

Body Analysis / Extraction	jq	xpath
JSON	o	x
XML	x	o

Predicate

A Predicate is a Matcher (can be extended in the future).

Matcher

A Matcher is a string or a mapping that has an item. Allowed matchers are below.

Note

A matchers taking none is given as a string like {"should": "be_null"}.

Note

A Value given as a Matcher is equivalent to {"equal": it}.

Object

• be: Matcher

  • Matches the given matcher.

• be_null: none

  • Matches if it is a null value.

• not_be_null: none

  • Matches if it is not a null value.

• equal: Value

  • Matches if it equals the given value.

• have_length: Integer

  • Matches if it has a length and its length is equal to the given value.

Comparable

• be_greater_than: Comparable

  • Matches if it is greater than the given value (it > argument).

• be_greater_than_or_equal_to: Comparable

  • Matches if it is greater than or equal to the given value (it >= argument).

• be_less_than: Comparable

  • Matches if it is less than the given value (it < argument).

• be_less_than_or_equal_to: Comparable

  • Matches if it is less than or equal to the given value (it < argument).

String

• contain_string: String

  • Matches if it is an string and contains the given value.

• start_with: String

  • Matches if it is an string and starts with the given value.

• end_with: String

  • Matches if it is an string and ends with the given value.

• match_regexp: String

  • Matches if it is an string and matches the given regular expression.

Datetime

• be_before: String

  • Matches if it is a datetime and before the given datetime.

  • When given now, then compares to the datetime just when the request starts.

  • When given an offset, then compares to the datetime when the request starts.

    • Days, hours, minutes and seconds offsets are available.

    • When given a positive offset like 1 day or +2 hours, then compares to the future datetime.

    • When given a negative offset like -1 minute or -2 seconds, then compares to the past datetime.

• be_after: String

  • Matches if it is a datetime and after the given datetime.

  • Usage is the same as be_before.

Note

Validated datetime values must be in ISO 8601 format like 2019-01-23T12:34:56Z.

Sequence

• be_empty: none

  • Matches an empty sequence.

• have_item: Matcher

  • Matches if it is a collection and has the given item.

• have_items: List<Matcher>

  • Matches if all given items appear in the list, in any order.

• contain: List<Matcher>

  • Exactly matches the entire sequence.

• contain_in_any_order: List<Matcher>

  • Matches the entire sequence, but in any order.

Logical

• not: Matcher

  • Matches if it doesn’t match the given matcher.

• all_of: List<Matcher>

  • Matches if it matches all of the given matchers.

• any_of: List<Matcher>

  • Matches if it matches any of the given matchers.

• anything: none

  • Matches anything.

Default

A Default is a mapping that consists of below:

• request: Request (Optional)

  • A request to overwrite the default request values.

• response: ResponseDescription (Optional)

  • A response description to overwrite the default response description values.

License

Copyright (c) 2019 Yu MOCHIZUKI