Skip to main content

Docs anystore on pypi PyPI Downloads PyPI - Python Version Python test and package pre-commit Coverage Status AGPLv3+ License Pydantic v2

anystore

Store anything anywhere. anystore provides a high-level storage and retrieval interface for various supported store backends, such as redis, sql, file, http, cloud-storages and anything else supported by fsspec.

Think of it as a key -> value store, and anystore acts as a cache backend. And when keys become filenames and values become byte blobs, anystore becomes actually a file-like storage backend – but always with the same and interchangeable interface.

Why?

In our several data engineering projects we always wrote boilerplate code that handles the featureset of anystore but not in a reusable way. This library shall be a stable foundation for data wrangling related python projects.

Examples

Base cli interface:

anystore -i ./local/foo.txt -o s3://mybucket/other.txt

echo "hello" | anystore -o sftp://user:password@host:/tmp/world.txt

anystore -i https://dataresearchcenter.org > index.html

anystore --store sqlite:///db keys <prefix>

anystore --store redis://localhost put foo "bar"

anystore --store redis://localhost get foo  # -> "bar"

Use in your applications:

from anystore import smart_read, smart_write

data = smart_read("s3://mybucket/data.txt")
smart_write(".local/data", data)

Simple cache example via decorator:

Use case: @anycache is used for api view cache in ftmq-api

from anystore import get_store, anycache

cache = get_store("redis://localhost")

@anycache(store=cache, key_func=lambda q: f"api/list/{q.make_key()}", ttl=60)
def get_list_view(q: Query) -> Response:
    result = ... # complex computing will be cached
    return result

Mirror file collections:

from anystore import get_store

source = get_store("https://example.org/documents/archive1")  # directory listing
target = get_store("s3://mybucket/files", backend_config={"client_kwargs": {
    "aws_access_key_id": "my-key",
    "aws_secret_access_key": "***",
    "endpoint_url": "https://s3.local"
}})  # can be configured via ENV as well

for path in source.iterate_keys():
    # streaming copy:
    with source.open(path) as i:
        with target.open(path, "wb") as o:
            while chunk := i.read(8192):
                o.write(chunk)

Documentation

Find the docs at docs.investigraph.dev/lib/anystore

Used by

Development

This package is using poetry for packaging and dependencies management, so first install it.

Clone this repository to a local destination.

Within the repo directory, run

poetry install --with dev

This installs a few development dependencies, including pre-commit which needs to be registered:

poetry run pre-commit install

Before creating a commit, this checks for correct code formatting (isort, black) and some other useful stuff (see: .pre-commit-config.yaml)

testing

anystore uses pytest as the testing framework.

make test

anystore, (C) 2024 investigativedata.io

anystore, (C) 2025 investigativedata.io, Data and Researc Center – DARC

anystore is licensed under the AGPLv3 or later license.

Prior to version 0.3.0, anystore was released under the GPL-3.0 license.

see NOTICE and LICENSE

Metadata

Release files for anystore 1.5.6

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for anystore 1.5.6
File Size Uploaded Provenance Actions
anystore-1.5.6.tar.gz 62.4 kB Without provenance Details for anystore-1.5.6.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for anystore 1.5.6
File Interpreter ABI Platform Actions
anystore-1.5.6-py3-none-any.whl Python 3 none any Details for anystore-1.5.6-py3-none-any.whl

Total release size: 140.9 kB

Release files / anystore-1.5.6.tar.gz

Download URL anystore-1.5.6.tar.gz
Size 62.4 kB
Tags Source
SHA-256 checksum
How to use checksums
c0b787325937c78480391498eef1c4a418b5fb75bbe43fe009800ca3dc8de42e
BLAKE2b-256 checksum
How to use checksums
bb5831dd270475a2f41041c94065a8af850d445974b6cd0c7187a39a49412c6d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via poetry/2.5.1 CPython/3.14.4 Linux/7.0.0-38-generic

Release files / anystore-1.5.6-py3-none-any.whl

Download URL anystore-1.5.6-py3-none-any.whl
Size 78.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
4e57c5837de78bf2e68bc7b20a7c7a6a574002b02832a44abcb9a6790d2ed2b5
BLAKE2b-256 checksum
How to use checksums
95ea94aa946de64a8db536488fa59b5605c8e8097adaddb1876247c7e6c969af
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via poetry/2.5.1 CPython/3.14.4 Linux/7.0.0-38-generic

Security for anystore 1.5.6

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.5.6 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page