Skip to main content

Datalookup - Deep nested data filtering

https://img.shields.io/badge/python-3.9-blue.svg https://github.com/pyshare/datalookup/actions/workflows/tests.yml/badge.svg Code coverage Status https://github.com/pyshare/datalookup/actions/workflows/linters.yml/badge.svg Documentation Status https://img.shields.io/badge/code%20style-black-000000.svg

The Datalookup library makes it easier to filter and manipulate your data. The module is inspired by the Django Queryset Api and it’s lookups.

Installation

$ pip install datalookup

Example

Throughout the below examples, we’ll refer to the following data, which comprise a list of authors with the books they wrote.

data = [
    {
        "id": 1,
        "author": "J. K. Rowling",
        "books": [
            {
                "name": "Harry Potter and the Chamber of Secrets",
                "genre": "Fantasy",
                "published": "1998"
            },
            {
                "name": "Harry Potter and the Prisoner of Azkaban",
                "genre": "Fantasy",
                "published": "1999"
            }
        ]
    },
    {
        "id": 2,
        "author": "Agatha Christie",
        "books": [
            {
                "name": "And Then There Were None",
                "genre": "Mystery",
                "published": "1939"
            }
        ]
    }
]

Datalookup makes it easy to find an author by calling one of the methods of the Dataset class like filter() or exclude(). There are multiple ways to retrieve an author.

Basic filtering

Use one of the field of your author dictionary to filter your data.

from datalookup import Dataset

# Use Dataset to manipulate and filter your data
books = Dataset(data)

# Retrieve an author using the author name
authors = books.filter(author="J. K. Rowling")
assert len(authors) == 1
assert authors[0].author == "J. K. Rowling"

# Retrieve an author using '__in' lookup
authors = books.filter(id__in=[2, 3])
assert len(authors) == 1
assert authors[0].author == "Agatha Christie"

# Retrieve an author using 'exclude' and '__contains' lookup
authors = books.exclude(author__contains="Christie")
assert len(authors) == 1
assert authors[0].author == "J. K. Rowling"

AND, OR - filtering

Keyword argument queries - in filter(), etc. - are “AND”ed together. If you need to execute more complex queries (for example, queries with OR statements), you can combine two filter request with “|”.

# Retrieve an author using multiple filters with a single request (AND). This
# filter use the '__icontains' lookup. Same as '__contains' but case-insensitive
authors = books.filter(books__name__icontains="and", books__genre="Fantasy")
assert len(authors) == 1
assert authors[0].author == "J. K. Rowling"

# Retrieve an author by combining filters (OR)
authors = books.filter(author="Stephane Capponi") | books.filter(
    author="J. K. Rowling"
)
assert len(authors) == 1
assert authors[0].author == "J. K. Rowling"

Cascade filtering

Sometimes you will want to filter the author but also the related books. It is possible to do that by calling the on_cascade() method before filtering.

# Filter the author but also the books of the author
authors = books.on_cascade().filter(
    books__name="Harry Potter and the Chamber of Secrets"
)
assert len(authors) == 1
assert authors[0].author == "J. K. Rowling"

# The books are also filtered
assert len(authors[0].books) == 1
assert authors[0].books[0].name == "Harry Potter and the Chamber of Secrets"

List of available lookups

Field lookups are used to specify how a the dataset should query the results it returns. They’re specified as keyword arguments to the Dataset methods filter() and exclude(). Basic lookups keyword arguments take the form “field__lookuptype=value”. (That’s a double-underscore).

As a convenience when no lookup type is provided (like in books.filter(id=1)) the lookup type is assumed to be exact.

# author is one of the field of the dictionary
# '__contains' is the lookup
books.filter(author__contains="Row")

Lookup

Case-insensitive lookup

Description

exact

iexact

Exact match

contains

icontains

Containment test

startswtih

istartswith

Starts with a specific string

endswith

iendswith

Ends with a specific string

regex

iregex

Regular expression match

in

In a given iterable; strings (being iterables) are accepted

gt

Grater than

gte

Greater that or equal

lt

Lower than

lte

Lower than or equal to

range

Range between two values. Integer only

isnull

Check that a field is null. Takes either True or False

contained_by

Check data is a subset of the passed values. ArrayField only

overlap

Data shares any results with the passed values. ArrayField only

len

Check length of the array. ArrayField only

Documentation

Datalookup does not stop here. The full documentation is in the docs directory or online at https://datalookup.readthedocs.io/en/latest/

Contribution

Anyone can contribute to Datalookup’s development. Checkout our documentation on how to get involved: https://datalookup.readthedocs.io/en/latest/internals/contributing.html

License

Copyright Stephane Capponi and others, 2023 Distributed under the terms of the MIT license, Datalookup is free and open source software.

Datalookup was inspired by Django and only the RegisterLookupMixin was copied. Everything else was inspired and re-interpreted. You can find the license of Django in the licenses folder.

Metadata

Release files for datalookup 1.0.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for datalookup 1.0.1
File Size Uploaded Provenance Actions
datalookup-1.0.1.tar.gz 41.3 kB Without provenance Details for datalookup-1.0.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for datalookup 1.0.1
File Interpreter ABI Platform Actions
datalookup-1.0.1-py3-none-any.whl Python 3 none any Details for datalookup-1.0.1-py3-none-any.whl

Total release size: 56.0 kB

Release files / datalookup-1.0.1.tar.gz

Download URL datalookup-1.0.1.tar.gz
Size 41.3 kB
Tags Source
SHA-256 checksum
How to use checksums
85119cc0bbefca1c85b33185bdd580b2d099e5e918a7e92c1df2b47d00cc94c4
BLAKE2b-256 checksum
How to use checksums
d4ef3f64222107848557885d77507d3eec2cf278b049e0610a37363abff71920
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/5.0.0 CPython/3.10.7

Release files / datalookup-1.0.1-py3-none-any.whl

Download URL datalookup-1.0.1-py3-none-any.whl
Size 14.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
6c454dc23a482947e632380c5f022574565c65059cd718fa417fa2fb9de01c80
BLAKE2b-256 checksum
How to use checksums
45f3d00588353777b9308acbb384219e8f905a48f58aa26a87ec473875757b70
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/5.0.0 CPython/3.10.7

Security for datalookup 1.0.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page