dictionquery

Declarative aggregation queries over lists of dicts — no dataframe, no schema, no dependencies.

from dictionquery import count, fields, overall, run, values

data = [
    dict(a='x', b=1, c=None),
    dict(a='y', b=2, c=None),
    dict(a='z', b=2, c=None),
]

run(data, [overall('total', count()), fields(values(count()))])
# {'total': 3,
#  'a': {'x': 1, 'y': 1, 'z': 1},
#  'b': {1: 1, 2: 2},
#  'c': {None: 3}}

That query says: count the rows overall, and for every field, count how often each distinct value appears. Point it at a JSON dump you have never seen before and it will tell you what is in there.

Install

pip install dictionquery      # or: poetry add dictionquery

Requires Python 3.11+. The runtime has no dependencies.

Concepts

Three kinds of callable compose to make a query.

Kind	Shape	Examples
Aggregate	Iterable[Any] -> Any	count(), values(count())
Query part	list[dict] -> dict	fields(...), field(name, ...), overall(name, ...)
Predicate	item -> bool	positive, not_(none)

A query is any iterable of query parts. run hands the whole dataset to each part and merges the fragments they return:

run(data, {overall('rows', count()), fields(count())})

Aggregates nest, which is where the expressiveness comes from — values(count()) is "group by distinct value, then count each group", and fields(values(count())) applies that to every field.

Filters take an item

Every filter argument is a predicate over a single item, returning whether that item takes part. An item is whatever the surrounding construct aggregates — so the same count(...) call filters different things depending on where you put it:

sales = [
    dict(region='north', units=3, rep='ann'),
    dict(region='south', units=3, rep='bob'),
    dict(region='north', units=7, rep='cal'),
]

# under overall(), rows are the items
overall('north', count(lambda row: row['region'] == 'north'))(sales)
# {'north': 2}

# under fields(), one field's values are the items
fields(count(positive))(sales)
# {'region': 0, 'units': 3, 'rep': 0}

The filter on fields and overall themselves is the exception that proves the rule: both choose which records the query part looks at, so both take a row. The two stack — a row filter narrows the records, then a value filter narrows what is aggregated from them:

big = lambda value: isinstance(value, int) and value > 5
north = lambda row: row['region'] == 'north'

fields(count(big), north)(sales)   # {'region': 0, 'units': 1, 'rep': 0}

An aggregate used on its own filters whatever you hand it:

count(positive)([10, 0, -5])   # 1

Query parts

fields(aggregate, filter=None, default=NOT_SET)

Applies aggregate to each field across the dataset, keyed by field name in first-seen order. filter is a row predicate; rejected rows contribute no values and cannot introduce a field. Filters nested inside aggregate test one field value at a time.

Rows that lack a field are skipped. Pass default= to substitute a value instead:

data = [dict(a=1), dict(a=2, b=3)]

fields(count())(data)                  # {'a': 2, 'b': 1}   — b missing from row 1
fields(list, default=None)(data)       # {'a': [1, 2], 'b': [None, 3]}

field(name, aggregate, filter=None, default=NOT_SET)

The single-field counterpart of fields, and its override. Put both in a query and name is reported this way while every other field falls back to the general treatment:

run(data, {fields(count()), field('b', values(count()))})
# {'a': 3, 'b': {1: 1, 2: 2}, 'c': 3}   — b broken down, everything else counted

run applies the override whichever order the query is in — parts that name what they produce (field, overall) are applied after the general fields — so an unordered query like a set still expresses an override. Result keys stay in the order the query gave them; overriding a field's value does not move its key.

Because it sees one field, the aggregate can be type-specific in a way fields could not risk:

field('price', sum)(orders)      # safe; fields(sum) would hit a text field and raise

Its filter narrows only this part — the rest of the query still sees every row. A field no row carries still gets an entry, from the aggregate over nothing:

field('nope', count())(data)     # {'nope': 0}

overall(name, aggregate, filter=None)

Applies aggregate to the rows themselves and publishes the result under name.

overall('total', count())(data)                                   # {'total': 2}
overall('adults', count(), lambda row: row['age'] >= 18)(people)

run(data, query)

Runs every part over the same dataset and merges the results. data may be any iterable of rows (it is materialized once and never mutated); query may be a set, list, or tuple. Every part sees the whole dataset, so a filter on one part never narrows another.

Parts that name what they produce (field, overall) are applied after the general fields, which is what makes overriding work from an unordered query. If two equally specific parts produce the same key the later one wins — and within a set "later" is not predictable, so keep those names distinct.

Aggregates

count(filter=None)

Counts the items it is given, falsy ones included. With a filter, counts only matching items — field values under fields, rows under overall.

count()([None, 0, ''])                             # 3
fields(count(positive))(sales)                     # per field, positive values only
overall('big', count(lambda r: r['units'] > 5))    # rows with big orders

values(aggregate, filter=None)

Groups items by distinct value (first-seen order, hashable values), then applies aggregate to each group.

values(count())(['x', 'y', 'y'])  # {'x': 1, 'y': 2}
values(list)(['x', 'y', 'y'])     # {'x': ['x'], 'y': ['y', 'y']}

Note the difference between filtering the group and filtering what is counted in it: a filter on values drops items before grouping, so a value only they carried gets no entry at all, while a filter on the inner count keeps the entry and shrinks its number.

fields(values(count(), lambda v: v != 'south'))(sales)['region']  # {'north': 2}
fields(values(count(big)))(sales)['units']                        # {3: 0, 7: 1}

Bring your own

Any callable over an iterable is an aggregate, so list, set, max, sum, and statistics.mean all drop straight in:

fields(list)(people)                    # every value of every field
fields(max)(people)                     # per-field maximum

Under fields your aggregate gets that field's values; under overall it gets the rows.

Note that fields applies the aggregate to every field, so a type-specific aggregate like sum will raise on a text field. Name the field instead:

field('score', sum)(people)                              # {'score': 13}
run(people, {fields(count()), field('score', sum)})      # sum that one, count the rest

Predicates

Filters are ordinary predicates, and dictionquery re-exports a vocabulary of them so most filters need no lambda.

Group	Names
Transformers	not_(p), and_(*ps), or_(*ps)
Constants	always, never
Emptiness	none, not_none, empty, none_or_empty, none_or_empty_or_whitespace
Numbers	number, whole_number, positive, negative, zero, none_or_zero

These are plain functions, so pass them by name — count(positive), not count(positive()). Only the transformers are called.

from dictionquery import and_, count, none_or_empty_or_whitespace, not_, number, negative

count(not_(none_or_empty_or_whitespace))(notes)   # notes with actual text
count(and_(number, not_(negative)))(scores)       # 0 and up, ignoring None and strings

They drop straight into a fields query, where the items are values. Where the item is a row — an aggregate under overall, or the filter on fields/overall itself — pull the field out with a lambda first:

fields(count(not_(none_or_empty_or_whitespace)))(people)             # value-shaped
overall('scored', count(lambda row: number(row.get('score'))))(people)   # row-shaped

Details worth knowing:

• Nothing raises on an off-type value. positive('nope') is False, not a TypeError — the fields of a dict dataset are rarely uniform, and a filter that explodes on row 3 is useless.
• empty is length-based. empty(0) is False; 0 is not an empty container. That is the distinction a bare truthiness check gets wrong.
• bool is not a number. positive(True) is False despite bool subclassing int, because a True in a data field is a flag, not the quantity 1.
• whole_number means "no fractional part", orthogonal to sign: -7 and 3.0 qualify, 3.5 does not. Compose the other readings — and_(whole_number, positive) for the counting numbers, and_(whole_number, not_(negative)) for the non-negative integers.
• NaN and infinity fail positive, negative, zero, and whole_number.

NOT_SET

The sentinel meaning "this row has no such field", distinct from a stored None. It is the default for fields(..., default=) and never appears in a result.

Development

poetry install
poetry run pytest                              # test suite
poetry run pytest --doctest-modules dictionquery   # docstring examples
python examples.py

CI runs all three on Python 3.11, 3.12, and 3.13 for every push and pull request.

Releasing

Branch	Goes to	Version published
develop	TestPyPI	<version>.dev<run number> — a fresh build every push
main	PyPI	exactly what pyproject.toml says

Both publish only after the full test matrix passes. A push to main that does not bump the version is a no-op rather than a failure (skip-existing), so shipping a release means bumping the version and pushing:

poetry version patch     # or minor / major
git commit -am "Release $(poetry version --short)" && git push origin main

One-time setup

Publishing uses trusted publishing — OIDC, no API tokens stored in the repo. On each of PyPI and TestPyPI, add a pending publisher under Your projects → Publishing:

Field	Value
PyPI project name	dictionquery
Owner	beattyml1
Repository	dictionquery
Workflow	ci.yml
Environment	pypi on PyPI, testpypi on TestPyPI

The environment names must match the environment: keys in the workflow. To use API tokens instead, drop the environment: and permissions: id-token blocks and give the publish step password: ${{ secrets.PYPI_API_TOKEN }}.

License

MIT — see LICENSE.