persista

Overview

persista is a lightweight Python library that provides simple, consistent building blocks for persisting and caching data. It offers a uniform key-value store interface backed by multiple storage engines (in-memory, SQLite, DuckDB, LMDB, Redis, PostgreSQL) with both sync and async APIs, plus TTL caching and HTTP fetch utilities.

Quick Links:

• Documentation
• Installation
• Features
• Contributing
• License

Why persista?

Storage backends have different APIs, and switching between them (e.g. moving from an in-memory store in tests to Redis or PostgreSQL in production) usually means rewriting code. persista solves this with a single, consistent BaseStore interface:

Store and retrieve values:

>>> from persista.store import InMemoryStore
>>> with InMemoryStore() as store:
...     store.set("user:1", {"name": "Alice"})
...     store.get("user:1")
...
{'name': 'Alice'}

Swap the backend without changing the calling code:

>>> import tempfile
>>> from pathlib import Path
>>> from persista.store import SQLiteStore
>>> with tempfile.TemporaryDirectory() as tmpdir:
...     with SQLiteStore(Path(tmpdir).joinpath("data.sqlite")) as store:
...         store.set("user:1", {"name": "Alice"})
...         store.get("user:1")
...
{'name': 'Alice'}

Cache expensive calls with a TTL:

>>> from persista.cache import cached
>>> @cached(ttl=60)
... def slow_call(x: int) -> int:
...     return x**2
...
>>> slow_call(4)
16

See the documentation for detailed examples.

Features

persista provides a comprehensive set of utilities for persisting and caching data:

🗄️ Key-Value Stores

A consistent BaseStore interface for storing dict values under string keys, with both synchronous and a-prefixed asynchronous methods on every store:

• Uniform API across backends: get/aget, get_many/aget_many, set/aset, set_many/aset_many, delete/adelete, filter/afilter, iteration
• Backends: InMemoryStore, SQLiteStore, DuckDBStore, LmdbStore, RedisStore, PostgresStore
• Typed variants (TypedSQLiteStore, TypedPostgresStore, ...) and pickle-backed variants (PickleLmdbStore, PickleRedisStore) for non-dict values
• Configurable conflict handling on writes ("raise", "skip", "overwrite", "merge")

⏱️ TTL Caching

Time-to-live caching for functions and values, with sync and async variants:

• Cache, with sync and async (a-prefixed) methods, for explicit cache instances
• cached / async_cached decorators for caching function calls
• A shared default cache via get_cache

🌐 HTTP Utilities

Helpers to fetch HTTP responses with automatic retries, built on top of requests or httpx:

• fetch_response (sync, requests); get_response/post_response/put_response/ patch_response/delete_response/send_request (sync, httpx) and their _async counterparts (async, httpx)
• HttpClient/AsyncHttpClient: class-based wrappers around httpx.Client/httpx.AsyncClient with the same retries, plus optional response caching via a Cache

🧾 Records

A Record container (a stable UUID id plus a metadata dict) with helpers to filter, sort, persist, and analyze collections of records:

• filter_by_metadata/filter_by_metadata_range/filter_by_metadata_values, sort_by_metadata: filter and sort records by metadata
• RecordStore and ready-to-use backends (InMemoryRecordStore, SQLiteRecordStore, DuckDBRecordStore, typed variants, ...) built on top of persista.store
• persista.record.analysis: exact/near-duplicate detection, id-collision detection, diffing two snapshots, orphan-reference checks, schema-shape clustering, value frequency/cardinality, and metadata stats/reporting
• generate_fake_records for synthetic test data (requires the faker extra)

>>> from persista.record import Record
>>> from persista.record.analysis import find_duplicate_record_ids
>>> records = [
...     Record(id="a", metadata={"source": "a.pdf"}),
...     Record(id="b", metadata={"source": "a.pdf"}),
...     Record(id="c", metadata={"source": "b.pdf"}),
... ]
>>> find_duplicate_record_ids(records)
[['a', 'b']]

Installation

We highly recommend installing persista in a virtual environment to avoid dependency conflicts.

Using uv (recommended)

uv is a fast Python package installer and resolver:

uv pip install persista

Install with specific optional dependencies:

uv pip install persista[redis,httpx]  # with Redis and httpx support

Using pip

Alternatively, you can use pip:

pip install persista

Install with specific optional dependencies:

pip install persista[redis,httpx]  # with Redis and httpx support

Requirements

• Python: 3.10 or higher
• Core dependencies: coola

Optional dependencies, enabled per-backend:

For detailed installation instructions, see the documentation.

Contributing

Contributions are welcome! We appreciate bug fixes, feature additions, documentation improvements, and more. Please check the contributing guidelines for details on:

• Setting up the development environment
• Code style and testing requirements
• Submitting pull requests

Whether you're fixing a bug or proposing a new feature, please open an issue first to discuss your changes.

API Stability

⚠️ Important: As persista is under active development, its API is not yet stable and may change between releases. We recommend pinning a specific version in your project’s dependencies to ensure consistent behavior.

License

persista is licensed under BSD 3-Clause "New" or "Revised" license available in LICENSE file.