yaql 0.3

Yet Another Query Language

YAQL - Yet Another Query Language
====

At the beginning of millennium the growing trend towards data formats standardization and application integrability made
XML extremely popular. XML became lingua franca of the data. Applications tended to process lots of XML files ranging
from small config files to very large datasets. As these data often had a complex structure with many levels of
nestedness it is quickly became obvious that there is a need for specially crafted domain specific languages to query
these data sets. This is how XPath and later XQL were born.

With later popularization of REST services and Web 2.0 JSON started to take XML’s place. JSON’s main advantage (besides
being simpler than XML) is that is closely reassembles data structures found in most programming languages (arrays,
dictionaries, scalars) making it very convenient for data serialization. As JSON lacked all the brilliant XML-related
technologies like XSLT, XML Schema, XPath etc. various attempts to develop similar languages for JSON were made. One of
those efforts was JSONPath library developed in 2007 by Stefan Gössner. Initial implementation was for PHP and
JavaScript languages, but later on ports to other languages including Python were written.

JSONPath allows navigation and querying, well, JSONs.
Suppose we have JSON as in following:

{
"customers": [
{
"customer_id": 1,
"name": "John",
"orders": [{
"order_id": 1,
"item": "Guitar",
"quantity": 1
}]
},{
"customer_id": 2,
"name": "Paul",
"orders": [ {
"order_id": 2,
"item": "Banjo",
"quantity": 2
},{
"order_id": 3,
"item": "Piano",
"quantity": 1
}]
}
]
}


then

`jsonpath(data, "$.customers[0].name") -> [‘John’]`
`jsonpath(data, "$.customers[*].orders[*].order_id") -> [1, 2, 3]`

But what if we need, for example to find order having ID = 2? Here is how it done in JSONPath:

`jsonpath(data, "$.customers[*].orders[?(@.order_id == 2)") -> [{'order_id': 2, 'item': 'Banjo', 'quantity': 2}]`

The construct `[?(expression)]` allows to filter items using any Python expression in our case. `@` character is
replaced with current value and then the whole expression is evaluated. Evaluation of arbitrary Python expression
requires using `eval()` function unless one wants to develop his own complete parser and interpreter of Python
programming language. Needless to say that `eval()` is a great security breach. If JSONPath expressions are used to
simplify program logic it would not be a big deal, but what if JSONPath is written by program users?

JSONPath expression is just a plain string. There is no such concept as parameter. That is if one want to find order
having ID = some variable value he has to dynamically construct expression string using string formatting or
concatenation. And again that is might be okay for internal usage but would became difficult for external usage and also
open the doors for injection attacks (remember SQL injection?)

Another limitation of JSONPath is JSON itself. Technically speaking JSONPath operates not on the JSON itself (i.e. text
representation) but on a JSON-like object model that is mixture of arrays, dictionaries and scalar values. But what is
one want to query object model consisting of custom objects? What if some parts of this model are dynamically computed?
Or the model is a graph rather than a tree?

It seems like JSONPath is good enough to use in Python code when you can `eval()` things and have many helper function
to work with data besides JSONPath capabilities but is not enough for external use when you need to have sufficient
power to query model without manual coding and have it still secure.
This is why we designed YAQL. YAQL follows the JSONPath ideas and has very similar syntax but offers much more for data
querying.

Expressions are quite similar to JSONPath. Here is how examples above can be translated to YAQL:

`$.customers[0].name -> $.customers[0].name (no change)`
`$.customers[*].orders[*].order_id -> $.customers.orders.order_id`

the main addition to JSONPath is functions and operators. Consider the following YAQL expressions:

`$.customers.orders[$.quantity > 0].quantity.sum() -> 4`
`$.customers.orders.select($.quantity * $.quantity).sum() -> 6`
`$.customers.orders.order_id.orderDesc($) -> [3, 2, 1]`
`$.customers.orders.order_id.orderDesc($).take(2) -> [3, 2]`
`$.customers.orders.order_id.orderDesc($).first() -> 3`

Does it mean that YAQL has large built-in function and operator library?. Yes, YAQL library has a out of the box large
set of commonly used functions. But they are not built-in. All the functions and operators (which are also function:
`a + b = operator_+(a, b)` etc) are user-supplied. User is free to add other functions that could be used in expressions
and to remove standard ones.

JSONPath library needs 2 arguments - input JSON data and an a expression. YAQL library requires third
parameter - context.

Context is a repository of functions and variables that can be used in expressions. So all the functions above are just
ordinary Python functions that are registered in Context object. But because they all need to be registered in Context
user can always customize them, add his own model-specific ones and have full control over the expression evaluation.