Artifact Filesystem Store

Tools for storing and retrieving artifacts (files and folders) on a locally accessible filesystem path.

Features and Concepts

Artifacts are accessible for both, CIs and local developments - without needing a special blob-server available.

Artifacts have meta-data, custom key-value-pairs, like expire-time and creation-time.

Artifacts can be tagged, like a latest or a version.

Artifacts can be stored compressed or as copies. When copies are retrieved, the path inside the artifact-storeage is accessed.

File access rights inside the storage-backend are handled by the filesystem and are out of scope of this library. This said, corruption done to the storage-backend can be fatal.

There is no central database for the artifact-catalog in the store. The filesystem-structure and meta-information-files alongside the artifact-archive or -folder are used to keep track.

Identification is done through naming and revisions. Revisions can be retrieved.

The artifact-store root is communicated via an environment variable or an argument to the cli-tool.

Artifacts are assigned to/stored under a namespace. A namespace is essentially a filesystem-path.

A list of folders and an exclusions list can be provided when storing and artifact.

Scripts for CMake and GitHub/GitLab Actions are provided to use the artifact-store in these environments.

Usage

After installing the project (see below), the command line tool artifact-store is available.

A storage root must be provided via the environment variable ARTIFACT_STORE_ROOT or via the argument --storage-root. Before using a storage root, it must be initialized via

artifact-store --storage-root <path> init

or

export ARTIFACT_STORE_ROOT=<path>
artifact-store init

Once initialized, artifacts can be stored via

artifact-store store --revision <rev> --tag <tag> <namespace> <artifact-name> <paths/glob...>

and retrieved via

artifact-store retrieve --revision <rev> <namespace> <artifact-name> <target-path>

for revisions and

artifact-store retrieve --tag <tag> <namespace> <artifact-name> <target-path>

for tags.

The command

artifact-store tag --tag <tag> <namespace> <artifact-name> <new_tag>

or

artifact-store tag --revision <revision> <namespace> <artifact-name> <tag>

can be used to assign tags to revisions or create new tags from existing ones.

artifact-store meta --tag <tag>|--revision <rev> <namespace> <artifact-name> ["<key>=<value>" ...]

can be used to set meta-data key-value-pairs. The command always returns the current meta-data JSON-formatted.

The command

artifact-store list -n

lists all namespaces,

artifact-store list -a <namespace>

lists all artifacts in a namespace,

artifact-store list -r <namespace> <artifact-name>

lists all revisions of an artifact and

artifact-store list -t <namespace> <artifact-name>

lists all tags of an artifact.

Development

Create a virtual environment and install the dependencies:

python3 -m venv .venv
source .venv/bin/activate
pip install -e .[dev]

This installs the development dependencies, including pytest for running the tests.

Releases

First bump the version with bumpver: e.g.:

bumpver patch # or 'minor' or 'major'

then push the changes and tag:

git push && git push --tags

package:

python3 -m build --wheel

and upload to PyPI:

python3 -m twine upload dist/*

See also: https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/