Skip to main content

A library for validating data at the edges of data systems.

Project description

Horkos

A library for validating data at the edges of data systems.

Install

pip install horkos

Documentation

Detailed documentation can be found on Read the Docs.

Usage

First define the data schema:

# http-requests-schema.yaml
name: "http_requests"
description: >-
  This is the http request event data set, it is all about http requests we
  receive. It's important to include as much information about it
  as is reasonable. The hardest thing about data is understanding it after
  the fact.
fields:
  path:
    type: string
    description: >-
      The path of the url that was hit. This will be everything after the
      host portion of the url.
  params:
    type: string
    nullable: true
    checks:
      - json
    description: >-
      The parameters of the http request. If the method is a `GET` these come
      from the url otherwise they are the JSON from the request body.
  method:
    type: string
    checks:
      - name: choice
        args:
          options:
            - DELETE
            - GET
            - HEAD
            - OPTIONS
            - PATCH
            - POST
            - PUT
    description: >-
      The http method of the request. Must be one of:
      `DELETE`, `GET`, `HEAD`, `OPTIONS`, `PATCH`, `POST`, or `PUT`.
  response_code:
    type: integer
    description: The http response code of the request
  timestamp:
    type: string
    checks:
      - iso_timestamp
    description: The time at which the http request was received.

To use this schema to validate records:

import horkos

schema = horkos.load_schema('http-requests-schema.yaml')
schema.process({'path': '/foo/bar'})
# RecordValidationError: Casting errors - params is required, method is required, ...
schema.process({
    'path': '/foo/bar',
    'params': '{"foo": "bar"}',
    'method': 'BAD',
    'response_code': '200',
    'timestamp': '2020-06-15T12:34:56',
})
# RecordValidationError: Check errors - "BAD" in method did not pass choice check, ...
schema.process({
    'path': '/foo/bar',
    'params': '{"foo": "bar"}',
    'method': 'GET',
    'response_code': '200',
    'timestamp': '2020-06-15T12:34:56',
})
# {..., 'response_code': 200, ...}

CLI

It's also possible to use horkos via cli to validate csv and json files.

path,params,method,response_code,timestamp
/foo/bar,,GET,200,2020-06-15T12:34:56
/fizbuz,{"param":"value"},POST,200,2020-12-15T12:34:56
/fizbuz,,ERROR,500,2020-12-15T12:34:56
/fizbuz,,GET,BAD,2020-12-15T12:34:56

To validate the csv run:

$ horkos check -s http-requests-schema.yaml data.csv
[Row 2]: Check errors - value of "ERROR" for method did not pass choice check
[Row 3]: Casting errors - value of "BAD" for response_code could not be cast to integer
2 errors found

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

horkos-0.2.0.tar.gz (22.2 kB view details)

Uploaded Source

Built Distribution

horkos-0.2.0-py3-none-any.whl (29.0 kB view details)

Uploaded Python 3

File details

Details for the file horkos-0.2.0.tar.gz.

File metadata

  • Download URL: horkos-0.2.0.tar.gz
  • Upload date:
  • Size: 22.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.6.1 requests/2.25.0 setuptools/50.3.2 requests-toolbelt/0.9.1 tqdm/4.54.0 CPython/3.9.0

File hashes

Hashes for horkos-0.2.0.tar.gz
Algorithm Hash digest
SHA256 f6817bf0084ed2a9c3847d6c74d2874bc454f6cbcba21f79dd669bda48b49451
MD5 e0b21e80919f70fbb8329539a4fee3d9
BLAKE2b-256 dbdc5ad7c985bc48704369825c3ce44102b1e10740d0b658a44b0ef4b0b75fb9

See more details on using hashes here.

File details

Details for the file horkos-0.2.0-py3-none-any.whl.

File metadata

  • Download URL: horkos-0.2.0-py3-none-any.whl
  • Upload date:
  • Size: 29.0 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.6.1 requests/2.25.0 setuptools/50.3.2 requests-toolbelt/0.9.1 tqdm/4.54.0 CPython/3.9.0

File hashes

Hashes for horkos-0.2.0-py3-none-any.whl
Algorithm Hash digest
SHA256 d4016d51aee8e82a4075696bad257ea2722e518243a4c4974617c2346e11787b
MD5 6ab82efe45b7226706e3c3bbb1f4a2f7
BLAKE2b-256 08d2c90f6b702538104076fc420a839f23bdc9a3122e46094f6b42fbeb981c19

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page