Skip to main content

Docs anystore on pypi PyPI Downloads PyPI - Python Version Python test and package pre-commit Coverage Status AGPLv3+ License Pydantic v2

anystore

Store anything anywhere. anystore provides a high-level storage and retrieval interface for various supported store backends, such as redis, sql, file, http, cloud-storages and anything else supported by fsspec.

Think of it as a key -> value store, and anystore acts as a cache backend. And when keys become filenames and values become byte blobs, anystore becomes actually a file-like storage backend – but always with the same and interchangeable interface.

Why?

In our several data engineering projects we always wrote boilerplate code that handles the featureset of anystore but not in a reusable way. This library shall be a stable foundation for data wrangling related python projects.

Examples

Base cli interface:

anystore -i ./local/foo.txt -o s3://mybucket/other.txt

echo "hello" | anystore -o sftp://user:password@host:/tmp/world.txt

anystore -i https://dataresearchcenter.org > index.html

anystore --store sqlite:///db keys <prefix>

anystore --store redis://localhost put foo "bar"

anystore --store redis://localhost get foo  # -> "bar"

Use in your applications:

from anystore import smart_read, smart_write

data = smart_read("s3://mybucket/data.txt")
smart_write(".local/data", data)

Simple cache example via decorator:

Use case: @anycache is used for api view cache in ftmq-api

from anystore import get_store, anycache

cache = get_store("redis://localhost")

@anycache(store=cache, key_func=lambda q: f"api/list/{q.make_key()}", ttl=60)
def get_list_view(q: Query) -> Response:
    result = ... # complex computing will be cached
    return result

Mirror file collections:

from anystore import get_store

source = get_store("https://example.org/documents/archive1")  # directory listing
target = get_store("s3://mybucket/files", backend_config={"client_kwargs": {
    "aws_access_key_id": "my-key",
    "aws_secret_access_key": "***",
    "endpoint_url": "https://s3.local"
}})  # can be configured via ENV as well

for path in source.iterate_keys():
    # streaming copy:
    with source.open(path) as i:
        with target.open(path, "wb") as o:
            while chunk := i.read(8192):
                o.write(chunk)

Documentation

Find the docs at docs.investigraph.dev/lib/anystore

Used by

Development

This package is using poetry for packaging and dependencies management, so first install it.

Clone this repository to a local destination.

Within the repo directory, run

poetry install --with dev

This installs a few development dependencies, including pre-commit which needs to be registered:

poetry run pre-commit install

Before creating a commit, this checks for correct code formatting (isort, black) and some other useful stuff (see: .pre-commit-config.yaml)

testing

anystore uses pytest as the testing framework.

make test

anystore, (C) 2024 investigativedata.io

anystore, (C) 2025 investigativedata.io, Data and Researc Center – DARC

anystore is licensed under the AGPLv3 or later license.

Prior to version 0.3.0, anystore was released under the GPL-3.0 license.

see NOTICE and LICENSE

Metadata

Release files for anystore 1.5.7

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for anystore 1.5.7
File Size Uploaded Provenance Actions
anystore-1.5.7.tar.gz 62.5 kB Without provenance Details for anystore-1.5.7.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for anystore 1.5.7
File Interpreter ABI Platform Actions
anystore-1.5.7-py3-none-any.whl Python 3 none any Details for anystore-1.5.7-py3-none-any.whl

Total release size: 141.1 kB

Release files / anystore-1.5.7.tar.gz

Download URL anystore-1.5.7.tar.gz
Size 62.5 kB
Tags Source
SHA-256 checksum
How to use checksums
7354e45cae3c9f165bb70ac33ea6e35a7e89ced38599653a7cc4306759fac079
BLAKE2b-256 checksum
How to use checksums
0adda04e7aee37cbdd8d8ab478af2c738ec24af1270ec84fcc3c2aeffb328634
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via poetry/2.5.1 CPython/3.14.4 Linux/7.0.0-38-generic

Release files / anystore-1.5.7-py3-none-any.whl

Download URL anystore-1.5.7-py3-none-any.whl
Size 78.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
7c05e1494856c1c1594b7567cdd61aec1ef58837246d777632a25cdba6d295aa
BLAKE2b-256 checksum
How to use checksums
87a3d7abff2b5ad52169f5dad24929daeba905309c4ccebc13817ad91fa0fac7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via poetry/2.5.1 CPython/3.14.4 Linux/7.0.0-38-generic

Security for anystore 1.5.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.5.7 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page