🔍 Deepfinder

What is Deepfinder?

Deepfinder reads values out of nested data using a dot path. Instead of a ladder of if statements and .get() calls, you write the shape of what you want:

>>> from deepfinder import deep_find
>>> user = {'name': 'ash', 'links': {'pokehub': '@ash'}}
>>> deep_find(user, 'links.pokehub')
'@ash'

It has no dependencies, ships type hints, and supports Python 3.9+.

Every example in this file is executed as part of the test suite, so nothing here can drift away from what the code actually does.

Key features

• Dot paths into dictionaries, sequences and objects: 'user.profile.name'
• Indexing, including from the end: 'users.0.name', 'users.-1.name'
• Fan-out over sequences: 'users.*.name'
• Null handling: 'users.?.email' for the first hit, 'users.*?.email' for all of them
• Never raises on a lookup: a miss yields default
• Container subclasses that carry the method with them

Why use it?

Reading one optional field out of a real API response is mostly defensive plumbing. Say you want every non-empty email in a paginated payload:

>>> response = {
...     'data': {
...         'users': [
...             {'profile': {'contact': {'email': 'ash@pallet.town'}}},
...             {'profile': {'contact': {}}},
...             {'profile': {'contact': {'email': 'misty@cerulean.city'}}},
...         ],
...     },
... }

By hand, every level needs a guard, because any of them can be missing:

>>> emails = []
>>> for user in response.get('data', {}).get('users', []):
...     email = user.get('profile', {}).get('contact', {}).get('email')
...     if email is not None:
...         emails.append(email)
>>> emails
['ash@pallet.town', 'misty@cerulean.city']

The same thing as a path:

>>> deep_find(response, 'data.users.*?.profile.contact.email')
['ash@pallet.town', 'misty@cerulean.city']

Contents

• Installation
• Path syntax
• Quick start
• API reference
• Behaviour worth knowing
• Paths and untrusted input
• Deprecated: nativify()
• Development
• Contributing

Installation

pip install deepfinder

No dependencies, on any supported Python. The package ships a py.typed marker, so type checkers pick up its annotations with no stub package.

Path syntax

The separator is configurable with path_token, which is also how you reach keys that contain a dot:

>>> deep_find({'a.b': {'c': 1}}, 'a.b/c', path_token='/')
1

Quick start

Dictionaries and lists

>>> trainer = {
...     'name': 'ash',
...     'pokemons': [
...         {'name': 'pikachu', 'type': 'electric'},
...         {'name': 'charmander', 'type': 'fire'},
...     ],
... }
>>> deep_find(trainer, 'pokemons.0.name')
'pikachu'
>>> deep_find(trainer, 'pokemons.-1.name')
'charmander'
>>> deep_find(trainer, 'pokemons.*.name')
['pikachu', 'charmander']

Missing values

A lookup never raises. When it does not resolve, you get default:

>>> deep_find(trainer, 'pokemons.99.name') is None
True
>>> deep_find(trainer, 'pokemons.99.name', default='unknown')
'unknown'

First hit, and all the hits

>>> squad = {
...     'pokemons': [
...         {'name': 'pikachu'},
...         {'name': 'charmander', 'ball': 'superball'},
...         {'name': 'lucario', 'ball': 'ultraball'},
...     ],
... }
>>> deep_find(squad, 'pokemons.?.ball')
'superball'
>>> deep_find(squad, 'pokemons.*?.ball')
['superball', 'ultraball']

* keeps one slot per item, so it tells you which items missed:

>>> deep_find(squad, 'pokemons.*.ball')
[None, 'superball', 'ultraball']

Objects

Instance attributes, __slots__, class attributes and properties all resolve:

>>> class Address:
...     def __init__(self, city):
...         self.city = city
>>> class Trainer:
...     region = 'Kanto'
...     def __init__(self, name, address):
...         self.name = name
...         self.address = address
...     @property
...     def display_name(self):
...         return self.name.title()
>>> ash = Trainer('ash', Address('Pallet Town'))
>>> deep_find(ash, 'address.city')
'Pallet Town'
>>> deep_find(ash, 'display_name')
'Ash'
>>> deep_find(ash, 'region')
'Kanto'

Methods are not values, so a segment that collides with a method name misses rather than handing back a bound method:

>>> deep_find(ash, 'display_name.upper', default='not found')
'not found'

Named tuples resolve both ways:

>>> from collections import namedtuple
>>> Point = namedtuple('Point', ['x', 'y'])
>>> deep_find({'p': Point(1, 2)}, 'p.y')
2
>>> deep_find({'p': Point(1, 2)}, 'p.0')
1

Mappings

Anything that is a Mapping resolves by key, not just dict:

>>> from collections import ChainMap
>>> deep_find(ChainMap({'a': 1}, {'b': 2}), 'b')
2

Containers that carry the method

>>> from deepfinder.entity import DeepFinderDict, DeepFinderList
>>> DeepFinderDict(squad).deep_find('pokemons.?.ball')
'superball'
>>> DeepFinderList([squad]).deep_find('0.pokemons.*?.ball')
['superball', 'ultraball']

Both accept the same path_token and default arguments as deep_find.

API reference

deep_find(obj, path, path_token='.', default=None)

Returns the resolved value, or default.

Raises TypeError if path is not a string and ValueError if path_token is empty. Nothing else: a path that cannot be resolved is a miss, not an error.

DeepFinderDict.deep_find(path, path_token='.', default=None)

DeepFinderList.deep_find(path, path_token='.', default=None)

The same arguments and the same semantics, with the container itself as obj. Both classes are generic, so DeepFinderList[int] and DeepFinderDict[str, int] keep their element types through a type checker:

>>> numbers: DeepFinderList[int] = DeepFinderList([1, 2, 3])
>>> numbers.deep_find('-1')
3

Behaviour worth knowing

These are the sharp edges, all of them covered by tests.

Paths and untrusted input

deep_find walks data, not the interpreter. Dunder segments never resolve, and attributes are never read off modules, functions, frames, tracebacks, coroutines or code objects, and methods do not resolve. A path therefore cannot pivot from your data into module globals or frame locals:

>>> deep_find(ash, '__class__')  is None
True
>>> deep_find(ash, 'display_name.__globals__') is None
True

That said, deep_find will happily return any value your own object graph exposes. If paths come from users, keep deciding for yourself which roots you hand it.

Argument validation

Misuse of the API is loud, unlike a lookup that simply misses:

>>> deep_find({'a': 1}, 1)
Traceback (most recent call last):
    ...
TypeError: path must be a str, got int
>>> deep_find({'a': 1}, 'a', path_token='')
Traceback (most recent call last):
    ...
ValueError: path_token must not be empty

Deprecated: nativify()

deepfinder.entity.nativify() rebinds builtins.list and builtins.dict so that containers built through those constructors gain a deep_find method. It is deprecated as of 1.6.0: it mutates the interpreter for every library in the process, and it never affected list and dict literals, which are built by bytecode that does not consult builtins. Use DeepFinderList / DeepFinderDict, or just call deep_find.

Development

git clone https://github.com/otsobide/deepfinder.py
cd deepfinder.py
make install    # installs the package plus the dev extras
make check      # lint, format check, type check, tests with coverage

Individual targets:

make lint       # ruff check
make format     # ruff format
make typecheck  # mypy --strict
make test       # unittest
make coverage   # unittest under coverage, fails under 100%
make build      # sdist + wheel, validated with twine

To run one test module or a single test:

python -m unittest tests.unit.deep_find_in_lists_test
python -m unittest tests.unit.deep_find_in_lists_test.TestFindInLists.test_all_values_of_list

Changelog

Release notes live in CHANGELOG.md.

Contributing

Contributions are welcome. Please keep the suite green and the coverage at 100%, and add a test that fails before your fix and passes after it. CONTRIBUTING.md has the ground rules and the release procedure.

License

This project is licensed under the MIT License - see the LICENSE file for details.