valideer 0.3

Lightweight data validation and adaptation library for Python

Lightweight data validation and adaptation library for Python.

At a Glance:

• Supports both validation (check if a value is valid) and adaptation (convert a valid input to an appropriate output).

• Succinct: validation schemas can be specified in a declarative and extensible mini “language”; no need to define verbose schema classes upfront. A regular Python API is also available if the compact syntax is not your cup of tea.

• Batteries included: validators for most common types are included out of the box.

• Extensible: New custom validators and adaptors can be easily defined and registered.

• Informative, customizable error messages: Validation errors include the reason and location of the error.

• Agnostic: not tied to any particular framework or application domain (e.g. Web form validation).

• Well tested: Extensive test suite with 100% coverage.

• Production ready: Used for validating every access to the Podio API.

• Licence: MIT.

Installation

Valideer requires Python 2.7. To install run:

pip install valideer

Or for the latest version:

git clone [e-mail]:podio/valideer.git
cd valideer
python setup.py install

To run the tests you need to install nose and optionally coverage for coverage report:

$ pip install nose coverage
$ nosetests --with-coverage --cover-package=valideer
    ..............................................................................
    Name                  Stmts   Miss  Cover   Missing
    ---------------------------------------------------
    valideer                  2      0   100%
    valideer.base           101      0   100%
    valideer.validators     335      0   100%
    ---------------------------------------------------
    TOTAL                   438      0   100%
    ----------------------------------------------------------------------
    Ran 86 tests in 0.104s

Basic Use

We’ll demonstrate valideer using the following JSON schema example:

{
    "name": "Product",
    "properties": {
        "id": {
            "type": "number",
            "description": "Product identifier",
            "required": true
        },
        "name": {
            "type": "string",
            "description": "Name of the product",
            "required": true
        },
        "price": {
            "type": "number",
            "minimum": 0,
            "required": true
        },
        "tags": {
            "type": "array",
            "items": {
                "type": "string"
            }
        },
        "stock": {
            "type": "object",
            "properties": {
                "warehouse": {
                    "type": "number"
                },
                "retail": {
                    "type": "number"
                }
            }
        }
    }
}

This can be specified by passing a similar but less verbose structure to the valideer.parse function:

>>> import valideer as V
>>> product_schema = {
>>>     "+id": "number",
>>>     "+name": "string",
>>>     "+price": V.Range("number", min_value=0),
>>>     "tags": ["string"],
>>>     "stock": {
>>>         "warehouse": "number",
>>>         "retail": "number",
>>>     }
>>> }
>>> validator = V.parse(product_schema)

parse returns a Validator instance, which can be then used to validate or adapt values.

Validation

To check if an input is valid call the is_valid method:

>>> product1 = {
>>>     "id": 1,
>>>     "name": "Foo",
>>>     "price": 123,
>>>     "tags": ["Bar", "Eek"],
>>>     "stock": {
>>>         "warehouse": 300,
>>>         "retail": 20
>>>     }
>>> }
>>> validator.is_valid(product1)
True
>>> product2 = {
>>>     "id": 1,
>>>     "price": 123,
>>> }
>>> validator.is_valid(product2)
False

Another option is the validate method. If the input is invalid, it raises ValidationError:

>>> validator.validate(product2)
ValidationError: Invalid value {'price': 123, 'id': 1} (dict): missing required properties: ['name']

For the common use case of validating inputs when entering a function, the @accepts decorator provides some nice syntax sugar (shamelessly stolen from typecheck):

>>> from valideer import accepts
>>> @accepts(product=product_schema, quantity="integer")
>>> def get_total_price(product, quantity=1):
>>>     return product["price"] * quantity
>>>
>>> get_total_price(product1, 2)
246
>>> get_total_price(product1, 0.5)
ValidationError: Invalid value 0.5 (float): must be integer (at quantity)
>>> get_total_price(product2)
ValidationError: Invalid value {'price': 123, 'id': 1} (dict): missing required properties: ['name'] (at product)

Adaptation

Often input data have to be converted from their original form before they are ready to use; for example a number that may arrive as integer or string and needs to be adapted to a float. Since validation and adaptation usually happen simultaneously, validate returns the adapted version of the (valid) input by default.

An existing class can be easily used as an adaptor by being wrapped in AdaptTo:

>>> import valideer as V
>>> adapt_prices = V.parse({"prices": [V.AdaptTo(float)]}).validate
>>> adapt_prices({"prices": ["2", "3.1", 1]})
{'prices': [2.0, 3.1, 1.0]}
>>> adapt_prices({"prices": ["2", "3f"]})
ValidationError: Invalid value '3f' (str): invalid literal for float(): 3f (at prices[1])
>>> adapt_prices({"prices": ["2", 1, None]})
ValidationError: Invalid value None (NoneType): float() argument must be a string or a number (at prices[2])

Similar to @accepts, the @adapts decorator provides a convenient syntax for adapting function inputs:

>>> from valideer import adapts
>>> @adapts(json={"prices": [AdaptTo(float)]})
>>> def get_sum_price(json):
>>>     return sum(json["prices"])
>>> get_sum_price({"prices": ["2", "3.1", 1]})
6.1
>>> get_sum_price({"prices": ["2", "3f"]})
ValidationError: Invalid value '3f' (str): invalid literal for float(): 3f (at json['prices'][1])
>>> get_sum_price({"prices": ["2", 1, None]})
ValidationError: Invalid value None (NoneType): float() argument must be a string or a number (at json['prices'][2])

Required and additional properties

By default object properties are considered optional unless they start with “+”. This default can be inverted by calling parse with required_properties=True. In this case object properties are considered required by default unless they start with “?”. For example:

validator = V.parse({
    "+name": "string",
    "duration": {
        "+hours": "integer",
        "+minutes": "integer",
        "seconds": "integer"
    }
})

is equivalent to:

validator = V.parse({
    "name": "string",
    "?duration": {
        "hours": "integer",
        "minutes": "integer",
        "?seconds": "integer"
    }
}, required_properties=True)

Similarly, additional properties that are not specified as either required or optional are allowed by default. This default can be overriden by calling parse with additional_properties=False to disallow all additional properties, or with additional_properties=<extra_schema> to validate all additional properties using extra_schema:

>>> schema = {
>>>     "name": "string",
>>>     "duration": {
>>>         "hours": "integer",
>>>         "minutes": "integer",
>>>     }
>>> }
>>>
>>> data = {"name": "lap", "duration": {"hours":3, "minutes":33, "seconds": 12}}
>>> V.parse(schema).validate(data)
{'duration': {'hours': 3, 'minutes': 33, 'seconds': 12}, 'name': 'lap'}
>>> V.parse(schema, additional_properties=False).validate(data)
ValidationError: Invalid value {'hours': 3, 'seconds': 12, 'minutes': 33} (dict): additional properties: ['seconds'] (at duration)
>>> V.parse(schema, additional_properties="string").validate(data)
ValidationError: Invalid value 12 (int): must be string (at duration['seconds'])

Explicit Instantiation

The usual way to create a validator is by passing an appropriate nested structure to parse, as outlined above. This enables concise schema definitions with minimal boilerplate. In case this seems too cryptic or “unpythonic” for your taste, a validator can be also created explicitly from regular Python classes:

>>> from valideer import Object, HomogeneousSequence, Number, String, Range
>>> validator = Object(
>>>     required={
>>>         "id": Number(),
>>>         "name": String(),
>>>         "price": Range(Number(), min_value=0),
>>>     },
>>>     optional={
>>>         "tags": HomogeneousSequence(String()),
>>>         "stock": Object(
>>>             optional={
>>>                 "warehouse": Number(),
>>>                 "retail": Number(),
>>>             }
>>>         )
>>>     }
>>> )

Built-in Validators

valideer comes with several predefined validators, each implemented as a Validator subclass. As shown above, some validator classes also support a shortcut form that can be used to specify implicitly a validator instance.

Basic

• valideer.Boolean(): Accepts bool instances.

  Shortcut:

  "boolean"

• valideer.Integer(): Accepts integers (numbers.Integral instances), excluding bool.

  Shortcut:

  "integer"

• valideer.Number(): Accepts numbers (numbers.Number instances), excluding bool.

  Shortcut:

  "number"

• valideer.Date(): Accepts datetime.date instances.

  Shortcut:

  "date"

• valideer.Time(): Accepts datetime.time instances.

  Shortcut:

  "time"

• valideer.Datetime(): Accepts datetime.datetime instances.

  Shortcut:

  "datetime"

• valideer.String(min_length=None, max_length=None): Accepts strings (basestring instances).

  Shortcut:

  "string"

• valideer.Pattern(regexp): Accepts strings that match the given regular expression.

  Shortcut:

  Compiled regular expression

• valideer.Condition(predicate, traps=Exception): Accepts values for which predicate(value) is true. Any raised exception that is instance of traps is re-raised as a ValidationError.

  Shortcut:

  Python function or method.

• valideer.Type(accept_types=None, reject_types=None): Accepts instances of the given accept_types but excluding instances of reject_types.

  Shortcut:

  Python type. For example int is equivalent to valideer.Type(int).

• valideer.Enum(values): Accepts a fixed set of values.

  Shortcut:

  N/A

Containers

• valideer.HomogeneousSequence(item_schema=None, min_length=None, max_length=None): Accepts sequences (collections.Sequence instances excluding strings) with elements that are valid for item_schema (if specified) and length between min_length and max_length (if specified).

  Shortcut:

  [item_schema]

• valideer.HeterogeneousSequence(*item_schemas): Accepts fixed length sequences (collections.Sequence instances excluding strings) where the i-th element is valid for the i-th item_schema.

  Shortcut:

  (item_schema, item_schema, …, item_schema)

• valideer.Mapping(key_schema=None, value_schema=None): Accepts mappings (collections.Mapping instances) with keys that are valid for key_schema (if specified) and values that are valid for value_schema (if specified).

  Shortcut:

  N/A

• valideer.Object(optional={}, required={}, additional=True): Accepts JSON-like objects (collections.Mapping instances with string keys). Properties that are specified as optional or required are validated against the respective value schema. Any additional properties are either allowed (if additional is True), disallowed (if additional is False) or validated against the additional schema.

  Shortcut:

  {”property”: value_schema, “property”: value_schema, …, “property”: value_schema}. Properties that start with '+' are required, the rest are optional and additional properties are allowed.

Adaptors

• valideer.AdaptBy(adaptor, traps=Exception): Adapts a value by calling adaptor(value). Any raised exception that is instance of traps is wrapped into a ValidationError.

  Shortcut:

  N/A

• valideer.AdaptTo(adaptor, traps=Exception, exact=False): Similar to AdaptBy but for types. Any value that is already instance of adaptor is returned as is, otherwise it is adapted by calling adaptor(value). If exact is True, instances of adaptor subclasses are also adapted.

  Shortcut:

  N/A

Composite

• valideer.Nullable(schema, default=None): Accepts values that are valid for schema or None. default is returned as the adapted value of None. default can also be a zero-argument callable, in which case the adapted value of None is default().

  Shortcut:

  “?{validator_name}”. For example "?integer" accepts any integer or None value.

• valideer.NonNullable(schema=None): Accepts values that are valid for schema (if specified) except for None.

  Shortcut:

  “+{validator_name}”

• valideer.Range(schema, min_value=None, max_value=None): Accepts values that are valid for schema and within the given [min_value, max_value] range.

  Shortcut:

  N/A

• valideer.AnyOf(*schemas): Accepts values that are valid for at least one of the given schemas.

  Shortcut:

  N/A

• valideer.AllOf(*schemas): Accepts values that are valid for all the given schemas.

  Shortcut:

  N/A

• valideer.ChainOf(*schemas): Passes values through a chain of validator and adaptor schemas.

  Shortcut:

  N/A

User Defined Validators

The set of predefined validators listed above can be easily extended with user defined validators. All you need to do is extend Validator (or a more convenient subclass) and implement the validate method. Here is an example of a custom validator that could be used to enforce minimal password strength:

from valideer import String, ValidationError

class Password(String):

    name = "password"

    def __init__(self, min_length=6, min_lower=1, min_upper=1, min_digits=0):
        super(Password, self).__init__(min_length=min_length)
        self.min_lower = min_lower
        self.min_upper = min_upper
        self.min_digits = min_digits

    def validate(self, value, adapt=True):
        super(Password, self).validate(value)

        if len(filter(str.islower, value)) < self.min_lower:
            raise ValidationError("At least %d lowercase characters required" % self.min_lower)

        if len(filter(str.isupper, value)) < self.min_upper:
            raise ValidationError("At least %d uppercase characters required" % self.min_upper)

        if len(filter(str.isdigit, value)) < self.min_digits:
            raise ValidationError("At least %d digits required" % self.min_digits)

        return value

A few notes:

• The optional name class attribute creates a shortcut for referring to a default instance of the validator. In this example the string "password" becomes an alias to a Password() instance.

• validate takes an optional boolean adapt parameter that defaults to True. If it is False, the validator is allowed to skip adaptation and perform validation only. This is basically an optimization hint that can be useful if adaptation happens to be significantly more expensive than validation. This isn’t common though and so adapt is usually ignored.

Shortcut Registration

Setting a name class attribute is the simplest way to create a validator shortcut. A shortcut can also be created explicitly with the valideer.register function:

>>> import valideer as V
>>> V.register("strong_password", Password(min_length=8, min_digits=1))
>>> is_fair_password = V.parse("password").is_valid
>>> is_strong_password = V.parse("strong_password").is_valid
>>> for pwd in "passwd", "Passwd", "PASSWd", "Pas5word":
>>>     print (pwd, is_fair_password(pwd), is_strong_password(pwd))
('passwd', False, False)
('Passwd', True, False)
('PASSWd', True, False)
('Pas5word', True, True)

Finally it is possible to parse arbitrary Python objects as validator shortcuts. For example let’s define a Not composite validator, a validator that accepts a value if and only if it is rejected by another validator:

class Not(Validator):

    def __init__(self, schema):
        self._validator = Validator.parse(schema)

    def validate(self, value, adapt=True):
        if self._validator.is_valid(value):
            raise ValidationError("Should not be a %s" % self._validator.__class__.__name__, value)
        return value

If we’d like to parse '!foo' strings as a shortcut for Not('foo'), we can do so with the valideer.register_factory decorator:

>>> @V.register_factory
>>> def NotFactory(obj):
>>>     if isinstance(obj, basestring) and obj.startswith("!"):
>>>         return Not(obj[1:])
>>>
>>> validate = V.parse({"i": "integer", "s": "!number"}).validate
>>> validate({"i": 4, "s": ""})
{'i': 4, 's': ''}
>>> validate({"i": 4, "s": 1.2})
ValidationError: Invalid value 1.2 (float): Should not be a Number (at s)