Convert data structure to schema.
Project description
(Python implementation)
Tools for converting data into schema.
(Still very much pre-alpha!)
Installation
Using pip
pip install derek
Build from source
git clone https://github.com/benjaminwoods/derek.git
pip install python/requirements/build.txt
python -m build python
pip install python/dist/derek_
What is Derek?
Derek documents data structures.
Load some data into a tree of nodes:
# Import the main class
from derek import Derek
# Suppose that you have some JSON-compatible data
obj = [
{
'some': [1.0, 3, "4.5"],
'data': [3.4, 4.5]
},
{
'some': [2, "4.0", 1.5],
'data': [1.4]
}
]
# Feed this data into Derek.tree
root_node = Derek.tree(obj, name='MyDataStructure')
You can use .example() to see a simple example item of data:
>>> root_node.example()
[{'some': [1.0], 'data': [3.4]}]
You can produce an OAS2/OAS3 JSON schema from this data, too:
j = root_node.parse(format='oas3')
import json
print(json.dumps(j, indent=2))
{
"MyDataStructure": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": {
"oneOf": [
{
"type": "array",
"items": {
"oneOf": [
{
"type": "string"
},
{
"type": "integer"
},
{
"type": "number"
}
]
}
},
{
"type": "array",
"items": {
"type": "number"
}
}
]
}
},
"example": [
{
"some": [
1.0
],
"data": [
3.4
]
}
]
}
}
Install and use the yaml package to convert this structure to an OAS3-compliant data schema.
import yaml
print(yaml.dump(j))
MyDataStructure:
example:
- data:
- 3.4
some:
- 1.0
items:
additionalProperties:
oneOf:
- items:
type: number
type: array
- items:
oneOf:
- type: number
- type: integer
- type: string
type: array
type: object
type: array
Derek extracts schemas from APIs.
Quickly extract schemas from APIs, by feeding the returned JSON into Derek.
from derek import Derek
from pycoingecko import CoinGeckoAPI
cg = CoinGeckoAPI()
# Get all coins from CoinGecko
root_node = Derek.tree(cg.get_coins_list(), name='GetCoins')
Parse to get your schema:
j = root_node.parse(format='oas3')
import json
print(json.dumps(j, indent=2))
{
"GetCoins": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": {
"type": "string"
}
},
"example": [
{
"id": "01coin",
"symbol": "zoc",
"name": "01coin"
}
]
}
}
Derek is really lightweight.
No required dependencies. Always.
Derek is extensible.
Use libraries like pywhat and yaml to quickly extend Derek:
import json, yaml
from derek import Derek, Parser
from pywhat import Identifier
class PywhatDerek(Derek):
@property
def parser(self):
return PywhatParser()
def parse_to_yaml(self, *args, **kwargs):
return yaml.dump(
self.parse(*args, **kwargs)
)
class PywhatParser(Parser):
@classmethod
def oas2(cls, node):
# Call the superclass parser for the current node:
# _sup = cls.__mro__[PywhatParser.__mro__.index(int):]
# j = _sup.oas2(cls, node)
# All calls to the oas2 method in the superclass therefore re-route
# back to this class method, automatically handling all recursive calls
# here.
j = super(PywhatParser, cls).oas2(node)
# The rest of this function simply patches in results from a call
# to the pywhat API.
identifier = Identifier()
if all(map(lambda t: not isinstance(node.value, t), [list, dict])):
result = identifier.identify(str(node.value))
if result['Regexes'] is not None:
matches = [entry for entry in result['Regexes']['text']]
# Select the match as the longest string
map_func = lambda d: (d['Matched'], d['Regex Pattern']['Name'])
max_func = lambda tup: len(tup[0])
_, match = max(
map(map_func, matches),
key=max_func
)
j = {
**j,
'description': match
}
return j
Allowing for functionality like:
root_node = PywhatDerek.tree({
'data': ['17VZNX1SN5NtKa8UQFxwQbFeFc3iqRYhem']
}, name='Addresses')
root_node.get_oas3_yaml()
returning:
Addresses:
additionalProperties:
items:
description: "Bitcoin (\u20BF) Wallet Address"
type: string
type: array
example:
data:
- 17VZNX1SN5NtKa8UQFxwQbFeFc3iqRYhem
type: object
Derek is straightforward.
Derek is designed for ease of use. If you’re trying to use Derek functionality in a workflow and it feels like it should be easier to get your desired result, please make an issue.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.