anystore
Store anything anywhere. anystore provides a high-level storage and retrieval interface for various supported store backends, such as redis, sql, file, http, cloud-storages and anything else supported by fsspec.
Think of it as a key -> value store, and anystore acts as a cache backend. And when keys become filenames and values become byte blobs, anystore becomes actually a file-like storage backend – but always with the same and interchangeable interface.
Why?
In our several data engineering projects we always wrote boilerplate code that handles the featureset of anystore but not in a reusable way. This library shall be a stable foundation for data wrangling related python projects.
Examples
Base cli interface:
anystore -i ./local/foo.txt -o s3://mybucket/other.txt
echo "hello" | anystore -o sftp://user:password@host:/tmp/world.txt
anystore -i https://dataresearchcenter.org > index.html
anystore --store sqlite:///db keys <prefix>
anystore --store redis://localhost put foo "bar"
anystore --store redis://localhost get foo # -> "bar"
Use in your applications:
from anystore import smart_read, smart_write
data = smart_read("s3://mybucket/data.txt")
smart_write(".local/data", data)
Simple cache example via decorator:
Use case: @anycache is used for api view cache in ftmq-api
from anystore import get_store, anycache
cache = get_store("redis://localhost")
@anycache(store=cache, key_func=lambda q: f"api/list/{q.make_key()}", ttl=60)
def get_list_view(q: Query) -> Response:
result = ... # complex computing will be cached
return result
Mirror file collections:
from anystore import get_store
source = get_store("https://example.org/documents/archive1") # directory listing
target = get_store("s3://mybucket/files", backend_config={"client_kwargs": {
"aws_access_key_id": "my-key",
"aws_secret_access_key": "***",
"endpoint_url": "https://s3.local"
}}) # can be configured via ENV as well
for path in source.iterate_keys():
# streaming copy:
with source.open(path) as i:
with target.open(path, "wb") as o:
while chunk := i.read(8192):
o.write(chunk)
Documentation
Find the docs at docs.investigraph.dev/lib/anystore
Used by
- ftmq, a query interface layer for followthemoney data
- investigraph, a framework to manage collections of structured followthemoney data
- ftmq-api, a simple api on top off
ftmqbuilt with FastApi - ftm-geocode, batch parse and geocode addresses from followthemoney entities
- ftm-lakehouse, a library to crawl, store and move around document collections and structured FollowTheMoney data (in progress)
- The OpenAleph suite in general
Development
This package is using poetry for packaging and dependencies management, so first install it.
Clone this repository to a local destination.
Within the repo directory, run
poetry install --with dev
This installs a few development dependencies, including pre-commit which needs to be registered:
poetry run pre-commit install
Before creating a commit, this checks for correct code formatting (isort, black) and some other useful stuff (see: .pre-commit-config.yaml)
testing
anystore uses pytest as the testing framework.
make test
License and Copyright
anystore, (C) 2024 investigativedata.io
anystore, (C) 2025 investigativedata.io, Data and Researc Center – DARC
anystore is licensed under the AGPLv3 or later license.
Prior to version 0.3.0, anystore was released under the GPL-3.0 license.
1 maintainer
Release files for anystore 1.3.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| anystore-1.3.1.tar.gz | 53.3 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| anystore-1.3.1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 121.4 kB
Release files / anystore-1.3.1.tar.gz
| Download URL | anystore-1.3.1.tar.gz |
|---|---|
| Size | 53.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
8531fd6310cae855f15dfad412d26686cc28023c30e7649f058676696c51c38e
|
|
BLAKE2b-256 checksum How to use checksums |
72a6b0b4ac1ad4ec4d9dac8f347db5533b407da77246e53b2cc715e9bbc825cd
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.4.1 CPython/3.14.4 Linux/7.0.0-30-generic
|
Release files / anystore-1.3.1-py3-none-any.whl
| Download URL | anystore-1.3.1-py3-none-any.whl |
|---|---|
| Size | 68.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
94372fbe107bcfea9fae6e32354fc76d51f2149f0edf7426523bd1a8983ab706
|
|
BLAKE2b-256 checksum How to use checksums |
7b63b3b3845566d0355e0f28d3af00d67cf4da0cdf6f9ad8ba7fe2c405a23621
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.4.1 CPython/3.14.4 Linux/7.0.0-30-generic
|
