---

title: Tellaro Query Language (TQL) class: repo-spec audience: in-repo status: current owner: tql-team last_verified: '2026-09-04' verification_note: 'CONTRADICTED on SEMANTICS. The 2026-09-04 pass corrected twelve TQL literals that did not parse and recorded that "PARSING is not EVALUATING" -- then still missed three breaking changes that leave every query parsing exactly as before, which is the failure that warning describes. Corrected here: (1) in / in_cs becoming element-wise eq_ci / eq was filed under the heading "Also user-visible, NOT BREAKING" and described only as case-insensitivity, when its own commit says f in [''5''] no longer matches a stored "05" in Python -- it is now numbered section 8; (2) the matches/regexp half of the scalar-operand fix (a non-string operand is now a translated pattern where it was raw JSON) appeared nowhere -- now section 9; the "35 of 216 operator x literal combinations" this note used to quote was removed in round 4 as unreproducible, since 18 x 6 is 108 and the pinned matrix in tests/unit/test_scalar_operand_parity.py is 8 x 6; (3) the operator-less field | <predicate> form, which used to parse to exists and match every record that merely HAD the field -- now section 10. The intro said "six operator families" above SEVEN numbered sections; it now says the count IS the section count, so the next addition cannot silently drift it. A fourth false claim was found and fixed: the Type Hints parity warning said a hint and a mutator chain "are written in opposite orders in the two runtimes, and neither accepts the other''s spelling" -- closed by cea6024 on this same branch. Executed, in BOTH engines, rather than read: the in/eq matrix over [{f:5},{f:"5"},{f:"05"},{f:5.0}] for in [''5''], in [5], in_cs [''5''], eq ''5'' and eq 5 (all five agree across engines, and reproduce the commit''s recorded before/after); the emitted DSL for matches/regexp over boolean and numeric operands; ip | is_loopback, ip | is_private and the ip | lowercase projection control; both hint/mutator orders on three mutator+hint shapes; empty-paren mutators (accepted by both, EXCEPT geoip_lookup/geo, which is now documented); and the null/exists matrix that section 1 rests on. Every TQL-shaped literal in a fenced block was re-extracted and re-parsed: 248 of 255 parse across the nine docs touched, and all seven that do not are deliberate -- five are the "WRONG" examples in TROUBLESHOOTING.md and two are Rust-only collection spellings on the crates.io page, each already carrying a parity warning and each verified to run under the Rust CLI. Round 3 of the doc audit moved the Rust-only pointer up by one; nothing else on this page changed, because the change behind it (a non-list operand to in) is Rust-only and its section lives on the crates.io page. Round 4 added a shared section for the top/bottom count Python refused inconsistently (ef65d17), which is what moved the shared count. NO COUNT IS RESTATED IN THIS NOTE, deliberately: every number on this page is derived from the headings by tests/unit/test_breaking_change_counts.py, and a count typed here is a fifth copy of the figure that test exists to compute -- this note carried a stale "ten" for exactly that reason. Round 4''s claims were re-measured by EXECUTION at ef65d17^ and at HEAD rather than read off the commit message: validate(), to_opensearch() and query() were run over five groups at both commits, reproducing the emitted {"terms": {"size": 99999999999999999999}} that passed validate(), and the four buckets returned with health_status "green" for stats count() by g top -1. Not checked here: the stats/aggregation surface, the cache and pagination examples, and the performance figures (already marked unsubstantiated).' verified_by: tql-docs-parity-campaign@2026-09-04 source_refs:

• tellaro-query-language:src/tql/init.py
• tellaro-query-language:pyproject.toml
• tellaro-query-language:src/tql/mutators/init.py
• tellaro-query-language:src/tql/parser_components/grammar.py
• tellaro-query-language:tql/src/parser/grammar.pest
• tellaro-query-language:src/tql/evaluator_components/field_access.py
• tellaro-query-language:src/tql/opensearch_components/query_converter.py
• tellaro-query-language:src/tql/opensearch_components/field_mapping.py

---

Tellaro Query Language (TQL)

A flexible, human-friendly query language for searching and filtering structured data across files, databases, and search engines.

TQL provides a unified, readable syntax for expressing complex queries that works seamlessly with:

• Files: Query JSON, JSONL, CSV files directly with CLI or Python API
• OpenSearch/Elasticsearch: Convert TQL to DSL queries automatically
• In-Memory Data: Filter Python dictionaries and lists
• Statistical Analysis: Built-in aggregations and grouping

# Query JSON files directly
results = tql.query("logs.jsonl", "status = 200 AND response_time > 500")

# Query OpenSearch with automatic DSL translation
results = tql.execute_opensearch(client, "events-*",
    "user.role = 'admin' AND timestamp > '2024-01-01'")

# Aggregate data with stats
results = tql.query("sales.json", "region = 'west' | stats sum(revenue) by product")

---

🚀 Quick Start

Installation

# Install from PyPI (Python package)
pip install tellaro-query-language

# Install with OpenSearch support
pip install tellaro-query-language[opensearch]

# Or install Rust CLI (300x faster for large files)
cargo install tellaro-query-language

Query Files with CLI

TQL includes a blazing-fast command-line interface for querying files:

# Query JSON/JSONL files
tql 'status = "active"' users.json
tql 'age > 25 AND city = "NYC"' data.jsonl

# Query CSV files (auto-detects headers)
tql 'price > 100 AND category = "electronics"' products.csv

# Statistical aggregations
tql '| stats count() by status' events.jsonl
tql 'status = 200 | stats average(response_time) by endpoint' logs.jsonl

# Process folders recursively
tql 'level = "ERROR"' logs/ --pattern "*.jsonl" --recursive

# Pipe data from stdin
cat data.jsonl | tql 'score > 90'

Performance: The Rust CLI processes 50MB files in milliseconds vs. seconds for Python implementations.

Query Files with Python API

from tql import TQL

tql = TQL()

# Query JSON files directly
results = tql.query("data.json", "user.role = 'admin' AND status = 'active'")

# Query with field transformations
results = tql.query("logs.jsonl", "email | lowercase contains '@example.com'")

# Statistical analysis
results = tql.query("sales.json", "| stats sum(revenue), avg(price) by category")

Query In-Memory Data

from tql import TQL

tql = TQL()
data = [
    {'name': 'Alice', 'age': 30, 'city': 'NYC'},
    {'name': 'Bob', 'age': 25, 'city': 'LA'},
    {'name': 'Charlie', 'age': 35, 'city': 'NYC'}
]

# Simple queries. `query()` returns a result DICT, not a list -- the matching
# records are under 'results', alongside 'total' and the query health report.
results = tql.query(data, 'age > 27')
# Returns: {'total': 2, 'post_processing_applied': False, 'health_status': 'green',
#           'health_reasons': [],
#           'results': [{'name': 'Alice', 'age': 30, 'city': 'NYC'},
#                       {'name': 'Charlie', 'age': 35, 'city': 'NYC'}]}
for record in results['results']:
    ...

# Logical operators
results = tql.query(data, 'age >= 30 AND city = "NYC"')
# results['results'] -> [{'name': 'Alice', ...}, {'name': 'Charlie', ...}]

# Field transformations
results = tql.query(data, 'name | lowercase = "alice"')

Query OpenSearch

from opensearchpy import OpenSearch
from tql import TQL

# Initialize OpenSearch client
client = OpenSearch(
    hosts=['localhost:9200'],
    http_auth=('admin', 'admin'),
    use_ssl=True,
    verify_certs=False
)

# Initialize TQL with field mappings
mappings = {
    'user.name': {'type': 'keyword'},
    'user.email': {'type': 'text'},
    'timestamp': {'type': 'date'}
}
tql = TQL(mappings)

# Execute queries with automatic DSL translation
results = tql.execute_opensearch(
    opensearch_client=client,
    index='users-*',
    query='user.name = "admin" AND status = "active"'
)

# Complex queries with mutators and post-processing
results = tql.execute_opensearch(
    opensearch_client=client,
    index='logs-*',
    query='email | lowercase contains "@example.com" AND level = "ERROR"'
)
# TQL automatically applies post-processing for mutators

---

⚠️ Upgrading: breaking changes in this release

This release makes eleven breaking changes to what a query means. Each change is deliberate, applied identically in the Python, Rust and OpenSearch backends, and motivated by the same failure: a query that returned zero hits with no error, which in a detection rule is indistinguishable from "nothing happened".

Read this section before upgrading a saved query, a detection rule, or anything that builds TQL from a template.

The count is the number of numbered sections below — nothing else, and it is now asserted rather than asked for. It said "six operator families" while seven sections followed it, because a family was added without the sentence being re-counted; the recount that fixed that asked, in prose, that every new breaking change get a new numbered section — and the next commit added one without recounting anyway. tests/unit/test_breaking_change_counts.py now derives both numbers here from the headings themselves. Seven further breaking changes affect the Rust runtime only and are not counted here; they are in tql/README.md, the crates.io page.

1. is null and not exists now mean HAS NO VALUE

What changed. is null and not exists both now mean absent or explicitly null. is not null and exists are their exact complement. The in-memory evaluators previously implemented a three-state model in which is null meant only "present and null" and not exists meant only "absent key" — so a record whose field was absent satisfied neither f is null nor f is not null, and a record whose field was null satisfied neither f exists nor f not exists.

Why. No backend could ever honour the three-state model. OpenSearch does not index JSON nulls, so must_not: {exists: {field}} — which both DSL backends already emitted for both spellings — cannot tell an absent field from a null one. The evaluators were answering a question their own translation cannot ask.

What to check. Any saved query using is null, is not null or not exists over a field that is sometimes absent. exists is unchanged.

What to write instead. Nothing, in most cases — the new answer is the one the OpenSearch path was already giving. If you specifically wanted "present and explicitly null", write f eq null, which selects only the explicitly-null records (f is null selects those and the absent ones).

See the table under Null and existence semantics for the full matrix, including the {"f": []} case (an empty array is a present value) and the one residual in-memory-vs-cluster divergence.

2. matches / regexp searches unless you anchor it

What changed. Against OpenSearch, f matches 'abc' and f matches '^abc$' emitted identical DSL, and both meant "the whole field value is exactly abc" — while the in-memory evaluator used re.search and matched abc anywhere. The same query meant two different things depending on where it ran. An unanchored pattern is now wrapped in .* so it searches; ^ and $ are still translated rather than wrapped, so a deliberately anchored pattern keeps full-match semantics.

What to check. Any OpenSearch query with an unanchored pattern. It will now match strictly more documents than it did.

What to write instead. If you relied on the old implicit anchoring, add the anchors: f matches '^abc$'.

3. A LIST operand to any / none is refused

What changed. f any ['a','b'], f none ['a','b'] and every spelling of them now raise TQLTypeError instead of answering — on translation and in the in-memory evaluator. The scalar form is unchanged, and a one-element list still unwraps to it.

Why. f any ['a','b'] does not ask "is any element of f one of these two values" — it asks whether a single element of f equals the two-element list, which no scalar element can. The clause previously produced either an HTTP 400 that failed the whole search, or a silent match_none, which made f none [...] an exclusion clause that excluded nothing.

The evaluator was raised to the translator in a second step, once the two were measured selecting different record sets rather than different kinds of the same answer. A translator refusal is not local to the clause — it fails the whole query — so code eq 1 OR f any ['a','b'] returned rows in memory and nothing through OpenSearch, and f none ['a','b'] returned every record in memory and nothing through OpenSearch. The refusal is decided from the query alone, never from the data, so short-circuiting and / or cannot make a well-typed query out of an ill-typed one for some corpora and not others.

What to write instead. f in ['a','b'] — which is what the query meant. The error message names it.

all / not_all are deliberately exempt: they are answered by a script query where a list is a valid (never-matching) operand.

4. Case-sensitive operators refuse a field that cannot answer them

What changed. contains_cs, startswith_cs, endswith_cs, in_cs and their not_ forms now raise TQLTypeError against an analyzed text field with no case-preserving subfield, instead of silently matching nothing.

Why. The analyzer lowercased every token it indexed, so the case the query asks about is not in the index at all. Measured on OpenSearch 2.19.4 against an analyzed field holding "A MiXeD Value": wildcard *MiXeD* returned 0 hits, must_not(wildcard *MiXeD*) returned every document, and the control wildcard *mixed* returned every document. The negated direction is the dangerous one — an exclusion clause that excludes nothing.

What to check. Any _cs query against a text-mapped field.

What to write instead. Either add a .keyword subfield to the mapping, or use the case-insensitive operator (contains, in, …). The error names both remedies. A .keyword or wildcard subfield, a keyword base field, and non-string types (long, ip, date, boolean) all still translate, as does an unmapped field.

5. A type hint that cannot read a value skips the record, not the query

(Migration note reproduced verbatim from the commit that made the change.)

A query that previously ERRORED on a heterogeneous field now returns results. Nothing that previously returned results changes -- with one exception, and it is the one to check: a query using a NEGATED comparator with a type hint over a field whose values do not all read as the hinted type. Those records used to abort the query; they now answer false rather than being swept in. If any saved query relied on the abort as a data-quality alarm, that alarm is gone.

An author who wanted the strict behaviour should assert the shape explicitly rather than leaning on the hint's failure mode -- f::int exists and f::int > 75 selects records readable as an int, and a separate rule over the un-hinted field finds the ones that are not. There is no flag to restore the raise, deliberately: two evaluators with two modes is how the engines drifted before.

An unknown hint name is still a hard error; only an unreadable value is a skip. See Type Hints for the boundaries, including what a skip means under negation and under the existence operators.

6. ::string renders a boolean lowercase

What changed. flag::string eq 'true' over {"flag": true} previously matched nothing, because the hint applied Python's str() and str(True) is "True". It now matches; flag::string eq 'True' no longer does.

Why. JSON writes true/false, TQL's own boolean literals are true/false, and OpenSearch stores true/false. The capitalised spelling was the only one in the stack, in the one place a value has been explicitly cast to a string in order to be compared as text.

What to check. Any query comparing a boolean field cast with ::string or ::str against the literal 'True' or 'False'.

Scope. Only a top-level boolean, and only under string/str. A boolean nested in a list or object still renders True.

7. The string operators render a boolean lowercase too

What changed. f contains_cs true, startswith_cs, endswith_cs, matches / regexp and their not_ forms compare against true / false rather than True / False — on both sides, the operand written in the query and the value read out of the record. f contains_cs true over {"f": "x true y"} now matches and over {"f": "x True y"} no longer does; f contains_cs 'ru' over {"f": true} now matches.

Why. The evaluator and the OpenSearch translator were selecting opposite documents for the same query. Measured over [{"f": "x true y"}, {"f": "x True y"}]:

	selects
evaluator (both engines)	the "x True y" record
translator → wildcard *true*	the "x true y" document

Two execution paths, different answers, no error on either. Section 6 moved ::string to the lowercase spelling and a separate change moved the translator; this is the remaining half. OpenSearch settles it outright for a boolean-mapped field — term b: "True" is rejected (only [true] or [false] are allowed) while term b: "true" matches.

What to check. Any query comparing a boolean field, or a text field holding the word, against the literal true / false with a case-sensitive string operator or with matches / regexp. The case-insensitive forms (contains, startswith, endswith) lowercase both sides and are unaffected.

Also fixed by the same change. In Python the positive and negated spellings disagreed with each other — f contains_cs true rendered True while f not contains_cs true rendered true, because the negated spelling arrives from the parser one list-level deeper and dodged the boolean coercion. Both now render true.

8. in / in_cs are element-wise eq_ci / eq

What changed. f in [a, b] is f eq_ci a or f eq_ci b, and f in_cs is the eq twin. Both engines re-implemented equality inside the membership arm instead of delegating to it, and drifted from it in opposite directions. They now delegate, so in answers exactly what the equivalent chain of eq_ci / eq clauses answers.

in and not_in remain the case-insensitive forms; in_cs / not_in_cs are the case-sensitive ones. That part is unchanged.

What to check. Any in / not_in / in_cs / not_in_cs query whose list holds a numeric-looking value, quoted or unquoted, over a field that stores numbers as text (or the reverse). Nothing else moves. Measured over [{"f": 5}, {"f": "5"}, {"f": "05"}, {"f": 5.0}]:

Query	Before	Now (both engines)
f in ['5']	Python also matched "05"	5, "5", 5.0
f in [5]	Rust did not match "05"	5, "5", "05", 5.0

So this is breaking, narrowly and in both directions: f in ['5'] no longer matches a stored "05" in Python, and f in [5] now does match it in Rust.

Why. Quoting is the author's type declaration, and eq has honoured it in both engines for some time; in is the operator that never got the rule. Python coerced every element numerically regardless of quoting — a list operand can never be a quoted literal internally, so the guard that preserves the distinction was always off for these operators. Rust went the other way and compared a string field against pre-lowercased element text, so f in [5] was false on the stored "05" while f eq_ci 5 — the same question with one element — was true.

What to write instead. Nothing, if the operand's quoting already says what you meant. If you want the numeric reading, write the element unquoted ([5]); if you want the text reading, quote it (['5']).

The OpenSearch half is inert. A terms clause is resolved by the field's mapping, not by the literal's JSON type — measured on OpenSearch 2.19.4, a long field holding 5 is selected by ["5"], [5], ["05"] and ["5.0"] alike. Only the in-memory evaluators, which have no mapping to consult, could select different records.

9. matches / regexp translate a non-string operand into a pattern

What changed. f matches true and f regexp 5 passed the operand straight through to OpenSearch as raw JSON — {"regexp": {"f": true}}. They now render it as a translated pattern, exactly as a string operand is: {"regexp": {"f": {"value": ".*true.*", "flags": "NONE"}}}.

Why. The reasoning for passing it through was that OpenSearch would report the type error. It does not. Measured on OpenSearch 2.19.4 against a field holding the word:

{"regexp": {"f": true}}        -> 0 hits, NO ERROR
{"regexp": {"f": ".*true.*"}}  -> 1 hit

The boolean is coerced to the string "true" and Lucene's regexp engine anchors implicitly, so the pattern must equal the whole value — a silent zero, which is the outcome this translator exists to prevent. {"regexp": {"f": null}} is worse: a hard HTTP 400 (value cannot be null) that fails the entire search, including every unrelated clause in it.

What to check. Any matches / regexp clause whose operand is not quoted — a bare true, false or number. Note that this is one visible symptom of a wider fix: the parser hands the negated spelling of a scalar operator its operand wrapped in a one-element list, and the Python converter unwrapped that list after type inference had already run, so the two arms of one operator disagreed about what the operand was (f contains_cs true rendered *True* while f not contains_cs true rendered *true*). It spans contains, startswith, endswith, all three _cs twins, matches, regexp, cidr, any and all, in both the positive and the negated spelling.

No combination COUNT is given, deliberately. This page carried "35 of 216 combinations over an 18-operator × 6-literal matrix" until round 4, and it does not reproduce on any axis: 18 × 6 is 108, not 216; the operators the sentence itself names are eleven, not eighteen; and the matrix the suite actually pins, SCALAR_OPERATORS × LITERALS in tests/unit/test_scalar_operand_parity.py, is 8 × 6 = 48. A product of two derived sets means nothing without both of them named — the same rule that retired the pushdown release's 373. Read the matrix off that test, which computes it.

What to write instead. Quote the operand if you meant the literal text (f matches 'true'). The rendered pattern is the same either way; quoting just makes the intent explicit.

10. field | <predicate> with no operator is a FILTER, not exists

What changed. When the last mutator in a chain returns a boolean — is_private, is_global, is_loopback, is_multicast, is_link_local — the operator-less form now means eq true. It used to parse to exists, so ip | is_loopback matched every record that merely had an ip field while reading as a filter.

What to check. Any saved query using one of the five IP predicates with no comparison operator. The explicit spellings — ip | is_loopback = true, ip | is_private eq false — are unchanged, and every example in this README already used one of those.

Why this was the worst shape. An unconditional-true clause dressed as a predicate cannot fail loudly; it quietly stops filtering and keeps returning plausible results. Both engines had it by opposite routes: Python carried the predicate names as a literal list written out in three places and only ever listed two of the five, so two predicates filtered and three did not; Rust enumerated nothing at all and emitted exists for every mutator, so all five matched everything. The set is now derived on both sides — Rust asks Mutator::returns_boolean, Python reads BOOLEAN_PREDICATE_MUTATORS — and a test pins the two equal, so a sixth predicate needs no parser edit.

A transforming mutator is unchanged. ip | lowercase is a projection: keep every record that has the field and apply the mutator on the way out. It still parses to exists.

11. A top / bottom count the engine cannot honour is refused at every surface

What changed. A bucket count that no engine can honour — negative, or past usize::MAX — is now refused by validate(), evaluate(), to_opensearch() and query() alike. Previously each surface answered differently, and the two signs failed in different directions.

Overflow was refused nowhere. stats count() by g top 99999999999999999999 passed validate() and emitted {"terms": {"field": "g", "size": 99999999999999999999}}. OpenSearch parses size into a 32-bit int, so that is not a large query but an unservable one — while the same saved query on the Rust detection engine was refused when it was written.

A negative count was refused only by the surfaces that CHECK a query. query() — the one surface that actually runs a query against records — routes through neither validate() nor to_opensearch(), so it saw no refusal at all. Measured over five groups, before the fix:

Query	validate()	query()
stats count() by g top -1	refused	4 buckets, health_status: "green"
stats sum(n) top -1 by g	refused	4 buckets, health_status: "green"

Four of five. results[:-1] does not fail and does not empty the result — it silently drops the last bucket and returns an answer nothing downstream can distinguish from a real one. A truncated aggregate reported healthy is worse than a raise, which at least surfaces.

top 0 is unaffected. Zero is representable, so it still parses, still validates, and is still refused only where it always was — in the translator, for the shapes a terms aggregation cannot express. Both engines agree on it.

What to check. Anything that computes a count — top <count - 1> against an empty count, say — or renders one from a template where the value can arrive negative or unbounded. Under query() a negative count was returning one bucket short and reporting green; it now raises TQLError. A positive or absent count is unchanged.

Also user-visible, not breaking

• cidr translates to OpenSearch rather than being refused, and now iterates array-valued fields in both evaluators — related.ip cidr '10.0.0.0/8' matches ["10.1.2.3", "8.8.8.8"]. It is refused on a keyword-mapped field, where term would compare the literal string "10.0.0.0/8"; map the field as ip.
• Word operators have boundary guards. f nonexists — the obvious typo for f not exists — used to parse as none with the value 'xists' and return a match. It is now a syntax error, as are f containszq, f eqzq, f isnull and f::intzq.
• Four mutator names are rejected at parse. count, unique, first and last parsed and silently returned nothing; they now raise TQLSyntaxError.
• Rust type hints are load-bearing. They were parsed, stored on the AST, and read by nothing, so f::int eq 'Hello' answered identically to the un-hinted query. Rust also gained the date, geo, object and ip hint names, which Python already had — the two runtimes now accept the same sixteen.
• An empty list literal is refused at parse, in every runtime. f in [] was already a TQLSyntaxError here and in the editor, and this is unchanged for Python — but the Rust runtime used to parse it, so the same query was green while you typed it, refused when you saved it, and valid in the agent's detection engine. The refusal now covers in, not_in, in_cs, not_in_cs, any, all, none and between. A non-empty list is untouched, and f in [''] is still a list containing one empty string.

---

🎯 Core Features

🔍 Unified Query Syntax

Write one query, run it anywhere - files, OpenSearch, in-memory data:

# Same query works everywhere
query = 'status = "active" AND age > 25'

# Query files
tql.query("users.json", query)

# Query OpenSearch
tql.execute_opensearch(client, "users-*", query)

# Query Python data
tql.query(python_list, query)

📁 First-Class File Support

Query files as easily as databases:

# JSON/JSONL files
tql.query("logs.jsonl", "level = 'ERROR'")

# CSV files with automatic header detection
tql.query("products.csv", "price > 100 AND stock < 10")

# Folders with glob patterns
tql.query("logs/2024/*.jsonl", "status = 500", recursive=True)

# Streaming for large files (CLI)
$ tql 'status = 200' large-file.jsonl  # Processes without loading to memory

🔄 25+ Field Mutators

Transform data inline before comparison:

# String transformations
'email | lowercase | trim = "admin@example.com"'
'name | uppercase = "JOHN DOE"'

# Encoding/decoding
'data | b64decode | lowercase = "secret"'
'password | md5 = "5f4dcc3b5aa765d61d8327deb882cf99"'

# Network operations
'ip | is_private = true'           # Check if IP is RFC 1918
'domain | defang = "hxxp://evil[.]com"'  # Security analysis

# DNS lookups
'hostname | nslookup contains "8.8.8.8"'

# GeoIP enrichment. A mutator name cannot contain a dot, so `| geoip.country_name`
# does NOT parse. `| geoip_lookup` enriches the record in place; the enriched
# `geo.*` fields are then filtered as ordinary fields.
'ip | geoip_lookup exists AND geo.country_name = "United States"'

# List operations
'scores | avg > 80'
'prices | sum between [100, 500]'

📊 Statistical Aggregations

Analyze data with built-in stats functions:

# Simple aggregations
tql.query(data, '| stats count(), sum(revenue), avg(price)')

# Grouped analysis
tql.query(data, '| stats count() by status, region')

# Top N analysis
tql.query(data, '| stats sum(sales, top 10) by product')

# Combined filtering and stats
tql.query(data, 'region = "west" | stats avg(revenue) by category')

🔧 OpenSearch Integration

Seamless OpenSearch/Elasticsearch integration:

• Automatic DSL Translation: TQL queries → OpenSearch Query DSL
• Smart Field Mapping: Handles keyword vs text fields automatically
• Post-Processing: Apply mutators that OpenSearch can't handle
• Pagination Support: Handle large result sets efficiently

# TQL handles field mapping automatically
mappings = {'user.email': {'type': 'text', 'fields': {'keyword': {'type': 'keyword'}}}}
tql = TQL(mappings)

# Exact match uses .keyword automatically
query = 'user.email = "admin@example.com"'  # Uses user.email.keyword

# Mutators trigger post-processing when needed
query = 'user.email | lowercase contains "admin"'  # Post-processes results

---

📖 Syntax Guide

Comparison Operators

# Equality
'status = "active"'           # Exact match (alias: eq)
'status != "inactive"'        # Not equal (alias: ne)

# Numeric comparisons
'age > 25'                    # Greater than
'age >= 18'                   # Greater or equal
'age < 65'                    # Less than
'age <= 100'                  # Less or equal

# String operations (case-INSENSITIVE)
'email contains "@example.com"'      # Substring
'name startswith "John"'            # Prefix
'filename endswith ".pdf"'          # Suffix

# Case-SENSITIVE twins, and their negations
'name contains_cs "John"'           # also startswith_cs, endswith_cs
'name not_contains_cs "John"'       # also not_startswith_cs, not_endswith_cs
'role in_cs ["Admin"]'              # also not_in_cs

# Regex. The pattern SEARCHES unless you anchor it -- see "Regex semantics".
'email matches "@example\\.com"'     # matches anywhere in the value
'email regexp "^\\w+@\\w+\\.\\w+$"'  # ^...$ means the WHOLE value

# Range and membership
'age between [18, 65]'              # Inclusive range
'status in ["active", "pending"]'   # Value in list (case-insensitive)
'status not in ["deleted"]'         # Value not in list (case-insensitive)

# Existence checks -- `exists` and `is not null` are the SAME question,
# and so are `not exists` and `is null`. See the matrix just below.
'field exists'                      # Field is present AND non-null
'field is not null'                 # identical to `field exists`
'field not exists'                  # Field is absent OR present-and-null
'field is null'                     # identical to `field not exists`

# Network operations
'ip cidr "192.168.0.0/16"'          # IP in CIDR range

There is no range operator in either runtime, despite older revisions of this page listing one as an alias for between. Use between.

Null and existence semantics

exists / is not null and not exists / is null are exact complements. Every record satisfies exactly one side, on every backend:

document	is null	not exists	is not null	exists
{}	match	match	--	--
{"f": null}	match	match	--	--
{"f": "x"}	--	--	match	match
{"f": []}	--	--	match	match

An empty array is a present value: the producer deliberately wrote a field with zero elements, so [] satisfies exists and is not null, not is null. [null] and ["a", null] follow the same rule.

Residual divergence, stated rather than left to be found. OpenSearch does not index a JSON null and does not index an empty array, so against a cluster {"f": []} and {"f": [null]} DO match not exists / is null. The in-memory evaluators can see _source and answer per the table above. This is the one document shape where the two execution paths differ.

Regex semantics

matches and regexp are the same operator, and an unanchored pattern searches: f matches 'abc' matches any value containing abc. An anchored pattern still means a whole-value match — ^ and $ are honoured, not wrapped:

'f matches "abc"'      # matches "xxabcxx"
'f matches "^abc$"'    # matches only "abc"

Against OpenSearch this is translated by wrapping an unanchored pattern in .*, because Lucene's regexp query anchors implicitly. Anchored patterns are passed through unwrapped.

Logical Operators

# AND (all conditions must be true)
'age > 25 AND city = "NYC"'
'status = "active" AND role in ["admin", "moderator"]'

# OR (any condition must be true)
'city = "NYC" OR city = "LA"'
'status = "admin" OR role = "superuser"'

# NOT (negates condition)
'NOT (age < 18)'
'NOT status = "deleted"'

# Complex expressions with parentheses
'(age > 25 AND city = "NYC") OR (status = "vip" AND score > 90)'

Collection Operators

Collection operators are written field-first, with the operator between the field and the value:

# any - at least one array element matches
'tags any "premium"'
'user.roles any "admin"'

# all - every array element equals the value
'status all "active"'
'scores all 80'

# none - no array element matches
'flags none "spam"'
'violations.severity none "critical"'

# Negated spellings
'tags not_any "premium"'
'status not_all "active"'
'flags not_none "spam"'

The operator-first spelling — ANY tags = "premium" — parses in the Rust runtime only. The Python parser raises TQLSyntaxError on it. Use the field-first form above for queries that must run under both.

The operand is a scalar. tags any ['a', 'b'] asks whether some single element of tags equals the two-element list, which no scalar element can, so both the OpenSearch translator and the in-memory evaluator raise TQLTypeError and name in as the operator you wanted:

'tags any ["a", "b"]'      # TQLTypeError -- ill-typed operand
'tags in ["a", "b"]'       # what that query meant
'tags any "a"'             # fine; a one-element list unwraps to this

all / not_all are the deliberate exception: they are answered by a script query that compares each element to the operand, where a list is a valid (never-matching) value.

Nested Field Access

# Dot notation for nested objects
'user.profile.email contains "@example.com"'
'metadata.tags.priority = "high"'

# Array indexing also uses dot notation -- bracket syntax is NOT accepted
'tags.0 = "urgent"'
'history.5.status = "completed"'

Field Mutators Reference

String Mutators

• lowercase, uppercase - Case conversion
• trim - Remove whitespace
• split(delimiter) - Split string into array
• length - Get string length
• replace(old, new) - Replace substring

Encoding Mutators

• b64encode, b64decode - Base64 encoding/decoding
• urldecode - URL decode
• hexencode, hexdecode - Hex encoding/decoding
• md5, sha256 - Cryptographic hashing

Network/Security Mutators

• refang - Convert defanged indicators (hxxp → http)

• defang - Defang URLs for safe display

• is_private - Check if IP is private (RFC 1918)

• is_global - Check if IP is globally routable

• is_loopback - Check if IP is loopback (127.0.0.0/8, ::1)

• is_multicast - Check if IP is multicast (224.0.0.0/4, ff00::/8)

• is_link_local - Check if IP is link-local (169.254.0.0/16, fe80::/10)

  All five return a boolean, take no arguments, and match if any element of a multi-valued field qualifies. They are post-processed rather than pushed down to OpenSearch, because the value they compute is not the value that is indexed.

  "Take no arguments" means exactly that. Both parsers now also tolerate empty parens on a mutator — ip | is_private() parses and evaluates identically to ip | is_private in both runtimes — but the bare name is the spelling to write. The GeoIP mutators are the exception, and the one place the parens are not portable: ip | geo() exists and ip | geoip_lookup() exists parse under the Rust runtime and raise TQLSyntaxError in Python.

  With no comparison operator, the bare form is a filter. ip | is_loopback means ip | is_loopback = true. It used to parse to exists and match every record that merely had an ip field — see breaking change 10.

DNS Mutators

• nslookup - Resolve hostname to IP addresses

GeoIP Mutators

• geoip_lookup (aliases geo, geoip) - Enrich IP with geolocation data. All three spellings work in every runtime. Earlier releases documented geoip as Rust-only and advised geoip_lookup "for queries that run under both"; that constraint is gone — Python accepts geoip as of this release, so a query written against either name behaves identically. The same applies to is_loopback, is_multicast and is_link_local above, and to the not_any / not_all / not_none operator spellings, all of which Rust accepted while Python did not.
• Returns: geo.country_name, geo.city_name, geo.location, geo.continent_code, etc.

List Mutators

• any, all - Boolean aggregations
• avg, average - Calculate mean
• sum - Calculate sum
• min, max - Find min/max values

A mutator name TQL does not know is rejected at parse time, not at evaluation. count, unique, first and last used to parse and then quietly return nothing; they now raise TQLSyntaxError. There is no public registration API — the permitted set is ALLOWED_MUTATORS in src/tql/mutators/__init__.py, and it is the same set in all three runtimes.

Type Hints

A ::hint suffix tells TQL how to read a field's value before the comparison runs.

'value::number > 75'          # read as a number, then compare
'flag::string eq "true"'      # read as a string, then compare
'addr::ip eq "10.1.2.3"'      # read as an IP address

A hint and a mutator chain may be written in EITHER order. f::number | trim > 75 and f | trim::number > 75 both parse, in both runtimes and in the TypeScript editor, and select the same records. This was a parity divergence — each grammar rejected the other's spelling — and it was closed in this release. If you find a page still describing one order as runtime-specific, that page is stale.

Both runtimes accept the same sixteen names, and every one of them reaches a decision:

Hint	Behaviour
string, str	render the value as a string
integer, int, number, decimal, float, double	numeric conversion
boolean, bool	boolean conversion
ip	validate as an IP address (and as a CIDR under the cidr operator)
array, list, date, geo, object	assert the field's shape; no value conversion is performed

Three rules govern what happens when a hint meets a value:

1. A value the hint cannot read skips the record — it does not abort the query. value::number > 75 over [{v: 80}, {v: "abc"}, {v: 90}] matches 80 and 90; "abc" simply does not match.
2. A skipped record matches nothing, positively or negatively. f::number != 75 over "abc" is false, not true. The same holds for not_contains, not_startswith, not_endswith, not_in and not_cidr, and for all four existence operators: f::int exists, f::int not exists, f::int is null and f::int is not null are all false over {"f": "Hello"} — f does not exist as an int, which is what the hint asked. An absent or null field is not a read failure, so f::int not exists over {} and f::int is null over {"f": null} are unchanged.
3. A query-level not (...) does still invert a skip. not is a boolean operator over the comparison's answer, the same way it already behaves for an absent field, and the same way must_not answers on the cluster path.

An unknown hint name is a hard error — f::bogus is rejected by the parser.

::string renders a boolean as true / false, not Python's True / False:

'flag::string eq "true"'   # matches {"flag": true}
'flag::string eq "True"'   # matches nothing

That divergence from str() is deliberate and applies to a top-level boolean under string/str only. A boolean nested in a list or object still renders True, so f::string eq "['a', True]" is how you match {"f": ["a", true]}.

Type hints are honoured by the in-memory evaluators. The OpenSearch execution path ignores them: the cluster reads the value as the mapping says it is stored.

---

📊 Statistical Aggregations

TQL includes a powerful stats engine for data analysis:

Available Functions

# Counting
'| stats count()'              # Count all records
'| stats count(field)'         # Count non-null values
'| stats unique_count(field)'  # Count distinct values

# Numeric aggregations
'| stats sum(revenue)'         # Calculate sum
'| stats avg(price)'           # Calculate average (aliases: average, mean)
'| stats min(age), max(age)'   # Find min/max values
'| stats median(score)'        # Calculate median

# Statistical measures
'| stats std(values)'          # Standard deviation
# Full name: '| stats standard_deviation(values)'. Note that neither `stdev`
# nor `stddev` works in either runtime, despite `stddev` appearing in the Rust
# grammar -- the PEG matches the shorter `std` first, so the longer spelling is
# unreachable. The same shadowing affects `percentiles`, `percentile_rank(s)`
# and `pct_rank(s)` in Rust.
'| stats percentile(score, 95)' # Calculate percentile

# Value extraction
'| stats values(category)'     # Get unique values

Grouping and Top N

# Group by single field
'| stats count() by status'

# Group by multiple fields
'| stats sum(revenue) by region, category'

# Top N analysis. Both spellings are portable since 2026-09-04 -- both runtimes
# accept `| stats sum(sales, top 10) by product` and
# `| stats sum(sales) top 10 by product`, and both now return the SAME ten
# buckets. This modifier ranks by the AGGREGATE value.
'| stats sum(sales, top 10) by product'

# The GROUP-BY bucket limit is a different modifier with the same keyword: it
# ranks by doc_count, not by the aggregate. `by product top 10` and
# `sum(sales) top 10 by product` will disagree on which ten unless doc_count and
# the sum happen to rank the products identically.
'| stats sum(sales) by product top 10'

# Multiple aggregations
'| stats count(), sum(revenue), avg(price) by status'

Combined Filtering and Stats

# Filter then aggregate
'status = "success" AND region = "west" | stats avg(revenue) by category'

# Complex analytics
'timestamp > "2024-01-01" | stats count(), sum(bytes), avg(response_time) by endpoint'

---

🔌 OpenSearch Integration Guide

Setup

from opensearchpy import OpenSearch
from tql import TQL

# Create OpenSearch client
client = OpenSearch(
    hosts=['localhost:9200'],
    http_auth=('admin', 'admin'),
    use_ssl=True,
    verify_certs=False
)

# Get index mappings
response = client.indices.get_mapping(index='users-*')
mappings = response['users-2024']['mappings']['properties']

# Initialize TQL with mappings
tql = TQL(mappings)

Query Translation

TQL automatically translates queries to OpenSearch DSL:

# TQL Query
query = 'age > 25 AND status = "active"'

# Translates to OpenSearch DSL:
{
    "query": {
        "bool": {
            "must": [
                {"range": {"age": {"gt": 25}}},
                {"term": {"status.keyword": "active"}}
            ]
        }
    }
}

# Execute seamlessly
results = tql.execute_opensearch(client, 'users-*', query)

Field Mapping Intelligence

TQL automatically handles field types:

# Text field with keyword subfield
mappings = {
    'email': {
        'type': 'text',
        'fields': {
            'keyword': {'type': 'keyword'}
        }
    }
}

# Exact match - uses .keyword automatically
'email = "admin@example.com"'  # → term query on email.keyword

# Substring - also uses .keyword, with case_insensitive: true
'email contains "example"'      # → wildcard on email.keyword

# Mutators trigger post-processing
'email | lowercase = "admin@example.com"'  # → fetch + filter

A field that cannot answer the question is refused, not answered with zero

Some operators need information the index does not contain. TQL raises TQLTypeError rather than emitting a query that can never match — a clause that returns zero hits with no error is indistinguishable from "nothing happened", and in the negated direction it is an exclusion that silently excludes nothing.

Query	Field mapping	Result
f contains_cs "A" (also startswith_cs, endswith_cs, in_cs, and their not_ forms)	analyzed text, no case-preserving subfield	TQLTypeError — the analyzer lowercased every indexed token, so the case being asked about is not in the index
f cidr "10.0.0.0/8"	keyword	TQLTypeError — term expands CIDR notation only on an ip field; on a keyword it compares the literal string "10.0.0.0/8"
f any ["a", "b"]	any	TQLTypeError — the operand is ill-typed; use in

Each error names the remedy: add a .keyword subfield or use the case-insensitive operator; map the field as ip or give it an ip subfield; use in.

An UNMAPPED field is still allowed through in every one of these cases. With no mappings TQL cannot tell an analyzed field from a keyword one, and refusing would break every query against an index whose mappings could not be fetched. So do not read a successful translation as proof the field can answer — read it as proof TQL knows of no reason it cannot.

Post-Processing

When OpenSearch can't handle operations, TQL applies post-processing:

# Mutators that require post-processing
'email | lowercase contains "admin"'    # Post-process: case conversion
'data | b64decode contains "secret"'    # Post-process: decode
'ip | geoip_lookup exists'             # Post-process: GeoIP lookup

# TQL automatically:
# 1. Executes base query in OpenSearch
# 2. Fetches results
# 3. Applies mutators in Python
# 4. Filters results
# 5. Returns final matches

Query Analysis

Analyze queries before execution to understand performance implications:

# Analyze query health
analysis = tql.analyze_query('email | lowercase contains "admin"', context='opensearch')

print(f"Health: {analysis['health']['status']}")  # 'fair' (post-processing)
print(f"Score: {analysis['health']['score']}")    # 85
print(f"Post-processing: {analysis['mutator_health']['requires_post_processing']}")  # True

# Recommendations for optimization
for issue in analysis['health']['issues']:
    print(f"Issue: {issue['message']}")
    print(f"Fix: {issue['recommendation']}")

---

📚 Documentation

Comprehensive documentation is available in the docs/ directory:

User-facing TQL documentation — getting started, query basics, the operator and mutator references, OpenSearch integration, stats, the API reference and the cookbook — lives in the tellaro-docs repository under public/tql/ and is published from there.

What stays in this repo is contributor material: see docs/README.md.

---

⚡ Performance

Benchmarks

Unsubstantiated. These figures predate the current code and could not be reproduced: the repo ships no benchmark harness (no benches/, no [[bench]] target, no pytest-benchmark). Treat them as rough historical claims rather than measurements.

Python Implementation:

• In-memory queries: ~10,000 records/sec
• File parsing (JSON): ~5MB/sec
• OpenSearch queries: Limited by network latency

Rust CLI (300x faster):

• In-memory queries: ~3,000,000 records/sec
• File parsing (JSON): ~150MB/sec
• Large file streaming: Process 50MB in ~200ms

Optimization Tips

# Use CLI for large files (300x faster)
$ tql 'status = 200' 50MB-file.jsonl  # ✓ Fast (Rust)
$ python -m tql 'status = 200' 50MB-file.jsonl  # ✗ Slow (Python)

# Pre-compile queries for reuse
ast = tql.parse('age > 25 AND status = "active"')
results1 = tql.evaluate(ast, dataset1)
results2 = tql.evaluate(ast, dataset2)

# Use OpenSearch for large datasets
tql.execute_opensearch(client, 'huge-index-*', query)  # Leverages OpenSearch's speed

# Minimize post-processing
'email.keyword = "admin@example.com"'  # ✓ Fast (OpenSearch only)
'email | lowercase = "admin@example.com"'  # ✗ Slower (post-processing)

---

🛠️ Development

Installation

# Clone repository
git clone https://github.com/tellaro/tellaro-query-language.git
cd tellaro-query-language

# Install dependencies (recommended)
uv sync

# Or with pip
pip install -e .

Testing

# Run all tests
uv run tql-tests

# Run specific test file
uv run pytest tests/unit/test_parser.py -v

# Run with coverage
uv run tql-cov

# Run integration tests (requires OpenSearch)
cp .env.example .env  # Configure OpenSearch connection
uv run pytest tests/integration/test_opensearch_integration.py -v

Code Quality

# Format + lint in one step
uv run tql-lint-all

# Format code (ruff replaces black, isort and flake8 -- none is a dependency)
uv run ruff format src tests

# Type checking
uv run pyright

# Linting
uv run ruff check src tests

# Security checks
uv run bandit -c bandit.yml -r src/

---

🗺️ Roadmap

✅ Implemented Features

• ✅ Core query engine with all operators
• ✅ 25+ field mutators (string, encoding, network, DNS, GeoIP, list)
• ✅ Statistical aggregations with grouping
• ✅ File support (JSON, JSONL, CSV)
• ✅ OpenSearch/Elasticsearch backend
• ✅ Intelligent post-processing
• ✅ Rust CLI for performance
• ✅ Mutator caching for GeoIP/DNS
• ✅ Query health analysis

🚧 In Progress

• 🚧 OpenSearch stats aggregation translation
• 🚧 Additional hash functions (SHA1, SHA512)
• 🚧 JSON parsing mutator
• 🚧 Timestamp conversion mutators

📋 Planned Features

• 📋 ElasticSearch backend support
• 📋 PostgreSQL/MySQL backends
• 📋 Query optimization engine
• 📋 Custom mutator plugins
• 📋 GraphQL-style field selection
• 📋 Parallel record evaluation
• 📋 Incremental file processing

🔮 Future Considerations

• 🔮 Time-series specific operators
• 🔮 Machine learning integration
• 🔮 Distributed query execution
• 🔮 Query caching layer

---

🤝 Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

How to Contribute

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Make your changes
4. Run tests (uv run tql-tests)
5. Run linters (uv run tql-lint)
6. Commit changes (git commit -m 'Add amazing feature')
7. Push to branch (git push origin feature/amazing-feature)
8. Open a Pull Request

---

📄 License

Tellaro Query Language (TQL) is source-available software with specific usage terms:

✅ Permitted Uses:

• Personal use (individual, non-commercial)
• Organizational use (within your company/organization)
• Integration into your applications and services
• Internal tools and automation

❌ Restricted Uses:

• Creating derivative query language products
• Commercial redistribution or resale
• Offering TQL-based commercial services to third parties
• Using source code to build competing products

For commercial licensing inquiries, contact: support@tellaro.io

See LICENSE for complete terms and conditions.

---

🔗 Related Projects

• TQL Rust - High-performance Rust implementation
• Tellaro Platform - Security operations platform using TQL

---

💬 Support

• Issues: GitHub Issues
• Documentation: Full Documentation
• Examples: the cookbook in tellaro-docs/public/tql/cookbook/
• Email: support@tellaro.io

---

🌟 Quick Examples

Security Log Analysis

# Find high-severity events from private IPs
query = '''
    source_ip | is_private = true AND
    severity in ["high", "critical"] AND
    (tags any "malware" OR url | defang contains "suspicious")
'''
results = tql.query("security-logs.jsonl", query)

E-commerce Analytics

# Analyze sales by region for premium products
query = '''
    product_tier = "premium" AND
    order_date > "2024-01-01" |
    stats sum(revenue), avg(order_value), count() by region
'''
results = tql.query("sales.json", query)

System Monitoring

# Find servers with high resource usage
query = '''
    hostname | nslookup exists AND
    (cpu_usage > 80 OR memory_usage > 90) AND
    status = "production"
'''
results = tql.execute_opensearch(client, "metrics-*", query)

---

Made with ❤️ by the Tellaro Team