A metadata management package based on filesystem mirroring.
Project description
fsmirror
Installation
pip install fsmirror
Functionality
Mirror project filesystems for metadata tracking. It can be useful to have a direct path mirror between code that generates data and the location in a filesystem or object store that stores the data / artifacts it generates.
Usage
- Create a configuration file like the one in
examples/example_config.yml
- Set the config path:
export FSMIRROR_CONFIG_PATH=/your/project/path/config.yml`
Use fsmirror
for managing where to store artifacts, the following pseudocode is
an example of how it should be used:
from some_data_pipeline import task
from storage_client import Client
from fsmirror.fsmirror import FSMirror, config
fm = FSMirror(config=config)
data = task()
storage_path = fm.mirror_relative(task, with_id=True)
Client.save_to_s3(storage_path)
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
fsmirror-0.1.tar.gz
(4.3 kB
view hashes)