Python utility for generating elasticsearch.json from schema.json for an IDS
Project description
ts-ids-es-json-generator
Introduction
This simple utility generates elasticsearch.json
using schema.json
.
How to Use
To run the generator, simply provide the program with the location of the IDS root directory:
$ pipenv run python -m ids_es_json_generator ~/path/to/ids/folder
elasticsearch.json
may be manually edited after automatic generation (e.g. to update non-searchable fields).
Output
The script saves elasticsearch.json
to the same directory you provided.
Assumptions
The generator operates under the following assumptions:
- It's the IDS designer's responsibility to make sure the schema and
elasticsearch.json
are correct. The generator will not validate the IDS for you.
Restrictions
- JSON pointers are dereferenced to create the mapping, so schemas with circular references or missing references will fail.
Tests
Run the tests using the following command:
$ pipenv run python -m pytest
How it works
Anywhere an array of objects appears in the schema, a nested
field type will be created in the output elasticsearch.json
, including arrays of objects contained anywhere within other arrays of objects.
The steps are:
- The schema is loaded into a dict with all of its JSON pointers dereferenced using
jsonref
, so that nested fields involvingdefinitions
are included in the elasticsearch mapping. find_array_object
will find all arrays of objects and create a dict representing their locations in the schema (see test casetest_array_simple_nested_case
in__tests__/unit/test_find_array_object.py
).generate_es_mapping
uses this dict to create the elasticsearch mapping (see test casetest_nested_keys_case
in__tests__/unit/test_generate_es_mapping.py
).
See the example files and test cases for more example usage.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for ts-ids-es-json-generator-1.0.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | c39882fb7a1cc4330d0f45a0db3bd5a5f56d3a86f6f75c308eb87f481e4ec02e |
|
MD5 | 58a0c9b292427cc4209779304c07e3a5 |
|
BLAKE2b-256 | d25c740f0bdf8cb7606a67281faca2151f75b2294ed09ad997c233e86891e4d5 |
Hashes for ts_ids_es_json_generator-1.0.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dd9ab37eecf4807ca63c4e456fdefe7b70cd3b576a746a93029fa834449ae031 |
|
MD5 | 9f0b161248294005b6c677474bd25de0 |
|
BLAKE2b-256 | 58b06d3212d0c37d65f842fbefdf01b5b4f88a2a03d7e9d3ca09533a3f96a1fa |