Fences

Fences is a python tool which lets you create test data based on schemas.

For this, it generates a set of valid samples which fullfil your schema. Additionally, it generates a set of invalid samples which intentionally violate your schema. You can then feed these samples into your software to test. If your software is implemented correctly, it must accept all valid samples and reject all invalid ones.

Unlike other similar tools, fences generate samples systematically instead of randomly. This way, the valid / invalid samples systematically cover all boundaries of your input schema (like placing fences, hence the name).

Installation

Use pip to install Fences:

python -m pip install fences

Fences is a self contained library without any external dependencies. It uses Lark for regex parsing, but in the standalone version where a python file is generated from the grammar beforehand (Mozilla Public License, v. 2.0).

Usage

Regular Expressions

Generate samples for regular expressions:

from fences import parse_regex

graph = parse_regex("a?(c+)b{3,7}")

for i in graph.generate_paths():
    sample = graph.execute(i.path)
    print("Valid:" if i.is_valid else "Invalid:")
    print(sample)

Valid:
cbbb
Valid:
acccbbbbbbb

JSON Schema

Generate samples for json schema:

from fences import parse_json_schema
import json

graph = parse_json_schema({
    'properties': {
        'foo': {
            'type': 'string'
        },
        'bar': {
            'type': 'boolean'
        }
    }
})

for i in graph.generate_paths():
    sample = graph.execute(i.path)
    print("Valid:" if i.is_valid else "Invalid:")
    print(json.dumps(sample, indent=4))

Valid:
{
    "foo": "",
    "bar": true
}
Valid:
{}
Valid:
{
    "foo": "",
    "bar": false
}
Valid:
""
Valid:
[
    "string"
]
Valid:
[
    42
]
Valid:
[
    null
]
Valid:
[
    true
]
Valid:
[
    false
]
Valid:
[
    {}
]
Valid:
[
    []
]
Valid:
true
Valid:
false
Valid:
0
Valid:
null
Invalid:
{
    "foo": 42
}
Invalid:
{
    "foo": null
}
Invalid:
{
    "foo": true,
    "bar": true
}
Invalid:
{
    "foo": false
}
Invalid:
{
    "foo": {},
    "bar": true
}
Invalid:
{
    "foo": []
}
Invalid:
{
    "bar": "string"
}
Invalid:
{
    "bar": 42
}
Invalid:
{
    "bar": null
}
Invalid:
{
    "bar": {}
}
Invalid:
{
    "bar": []
}

XML Schema

Generate samples for XML schema:

from fences import parse_xml_schema
from xml.etree import ElementTree
from xml.dom import minidom

et = ElementTree.fromstring("""<?xml version="1.0" encoding="UTF-8" ?>
    <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
        <xs:element name = 'class'>
            <xs:complexType>
                <xs:sequence>
                    <xs:element name = 'student' type = 'StudentType' minOccurs = '0' maxOccurs = 'unbounded' />
                </xs:sequence>
            </xs:complexType>
        </xs:element>
        <xs:complexType name = "StudentType">
            <xs:sequence>
                <xs:element name = "firstname" type = "xs:string"/>
                <xs:element name = "lastname" type = "xs:string"/>
                <xs:element name = "nickname" type = "xs:string"/>
                <xs:element name = "marks" type = "xs:positiveInteger"/>
            </xs:sequence>
            <xs:attribute name = 'rollno' type = 'xs:positiveInteger'/>
        </xs:complexType>
    </xs:schema>""")

graph = parse_xml_schema(et)
for i in graph.generate_paths():
    sample = graph.execute(i.path)
    s = ElementTree.tostring(sample.getroot())
    print("Valid:" if i.is_valid else "Invalid:")
    print(minidom.parseString(s).toprettyxml(indent="   "))

Valid:
<?xml version="1.0" ?>
<class/>

Valid:
<?xml version="1.0" ?>
<class>
   <student>
      <firstname>foo</firstname>
      <lastname>foo</lastname>
      <nickname>foo</nickname>
      <marks>780</marks>
   </student>
</class>

Valid:
<?xml version="1.0" ?>
<class>
   <student rollno="533">
      <firstname>x</firstname>
      <lastname>x</lastname>
      <nickname>x</nickname>
      <marks>780</marks>
   </student>
</class>

Invalid:
<?xml version="1.0" ?>
<class>
   <student>
      <firstname>foo</firstname>
      <lastname>foo</lastname>
      <nickname>foo</nickname>
      <marks>-10</marks>
   </student>
</class>

Invalid:
<?xml version="1.0" ?>
<class>
   <student rollno="533">
      <firstname>x</firstname>
      <lastname>x</lastname>
      <nickname>x</nickname>
      <marks>foo</marks>
   </student>
</class>

Invalid:
<?xml version="1.0" ?>
<class>
   <student rollno="-10">
      <firstname>foo</firstname>
      <lastname>foo</lastname>
      <nickname>foo</nickname>
      <marks>780</marks>
   </student>
</class>

Invalid:
<?xml version="1.0" ?>
<class>
   <student rollno="foo">
      <firstname>x</firstname>
      <lastname>x</lastname>
      <nickname>x</nickname>
      <marks>780</marks>
   </student>
</class>

Grammar

Generate samples for a grammar:

from fences.grammar.types import NonTerminal, CharacterRange
from fences import parse_grammar

number = NonTerminal("number")
integer = NonTerminal("integer")
fraction = NonTerminal("fraction")
exponent = NonTerminal("exponent")
digit = NonTerminal("digit")
digits = NonTerminal("digits")
one_to_nine = NonTerminal("one_to_nine")
sign = NonTerminal("sign")

grammar = {
    number:      integer + fraction + exponent,
    integer:     digit
                 | one_to_nine + digits
                 | '-' + digit
                 | '-' + one_to_nine + digits,
    digit:       '0'
                 | one_to_nine,
    digits:      digit*(1, None),
    one_to_nine: CharacterRange('1', '9'),
    fraction:    ""
                 | "." + digits,
    exponent:    ""
                 | 'E' + sign + digits
                 | "e" + sign + digits,
    sign:        ["", "+", "-"]
}

graph = parse_grammar(grammar, number)
for i in graph.generate_paths():
    sample = graph.execute(i.path)
    print(sample)

0
91.0901E0901
-0e+9
-10901.0
9E-0109

OpenAPI (Swagger)

You can use Fences to parse an OpenAPI specification and generate a set of sample requests:

from fences.open_api.generate import generate_all, SampleCache
from fences.open_api.open_api import OpenApi

description = {
    'info': {
        'title': 'Video API'
    },
    'paths': {
        '/videos': {
            'get': {
                'operationId': 'listVideos',
                'parameters': [{
                    'name': 'type',
                    'in': 'query',
                    'schema': {
                        'enum': ['public', 'private']
                    }
                }, {
                    'name': 'title',
                    'in': 'query',
                    'schema': {
                        'type': 'string',
                        'minLength': 3
                    }
                }],
                'responses': {}
            }
        },
        '/videos/{videoId}': {
            'patch': {
                'operationId': 'updateVideo',
                'parameters': [
                    {
                        'name': 'videoId',
                        'in': 'path',
                        'schema': {
                            'type': 'number'
                        }
                    }
                ],
                'requestBody': {
                    'content': {
                        'application/json': {
                            'schema': {
                                'type': 'object',
                                'properties': {
                                    'title': {
                                        'type': 'string',
                                        'minLength': 10
                                    }
                                },
                                'required': ['title']
                            }
                        }
                    }
                },
                'responses': {}
            }
        }
    }
}

open_api = OpenApi.from_dict(description)
sample_cache = SampleCache()
for operation in open_api.operations.values():
    graph = generate_all(operation, sample_cache)
    for i in graph.generate_paths():
        request = graph.execute(i.path)
        request.dump()

GET /videos
GET /videos?type=public&title=xxx
GET /videos?type=private
GET /videos?type=%23%23%23%23%23%23%23%23&title=xxx
PATCH /videos/0
  BODY: {"title": "xxxxxxxxxx"}
PATCH /videos/0
PATCH /videos/0
  BODY: {"title": 42}
PATCH /videos/0
  BODY: {"title": null}
PATCH /videos/0
  BODY: {"title": true}
PATCH /videos/0
  BODY: {"title": false}
PATCH /videos/0
  BODY: {"title": {}}
PATCH /videos/0
  BODY: {"title": []}
PATCH /videos/0
  BODY: {}
PATCH /videos/0
  BODY: "string"
PATCH /videos/0
  BODY: 42
PATCH /videos/0
PATCH /videos/0
  BODY: true
PATCH /videos/0
  BODY: false
PATCH /videos/0
  BODY: []

You can execute the generated tests using the request.execute() method. Please note, that you need to install the requests library for this.

Real-World Examples

Find some real-world examples in the examples folder.

Limitations

General:

Fences does not check if your schema is syntactically correct. Fences is designed to be as permissive as possible when parsing a schema but will complain if there is an aspect it does not understand.

For XML:

Python's default XML implementation xml.etree.ElementTree has a very poor support for namespaces (https://docs.python.org/3/library/xml.etree.elementtree.html#parsing-xml-with-namespaces). This might lead to problems when using the targetNamespace attribute in your XML schema.

For Grammars:

Fences currently does not generate invalid samples for grammars.

For OpenAPI:

The test cases generated by Fences are purely syntactic. They do not check for semantics, e.g. if retrieving a deleted resource returns 404.