file-golem 0.1.14

This package handles file I/O for larger projects, with the aim to simplify/standardize working with many datasets and 3rd party codebases across different naming conventions, data types, and storage systems.

To run unit tests, run

$python -m unittest discover -s tests


$for dev, it is better to run pip install -e .

Project set up

System Identity File

Create a file named .system_identity.txt at the top level project directory. Decide on a name for your local system, and add it as a single line to the the file as:

In conf/system_configs/system_conf.yaml, input the following:

System config

Create a config file in conf/system_configs/system_conf.yaml. This holds information file_golem needs to navigate your file systems:

filelock_timeout: <number of seconds to wait for filelock (default: wait indefinitely)>

system_paths: : base_data_directory: <project directory (output of pwd())>

base_conda_name: <path to conda environment when calling conda env list>
external_calls:
  <name of external codebase used in code>:
    entry_point: <absolute path of top project directory>
    conda_path: <path to external environment when calling conda env list>
    python_entry_point: <Name of the python file (from top level project directory sans .py)>


(optional) data_directory_name: <name of subdirectory where untracked data is stored > (default: data)

system_transfers:
    <non local mounted system name>:
        mount_point: <absolute path of the mount point on your local system>
        data_directory_extension: <path to the root of the data directory on the mounted system> (file_folem will look in mount_point/data_directory_extension) 

base_data_directory can be customized to an alternative location if desired.

Building a file path

FilePathEntries.CUSTOM_LOGIC : '<my_static_function>'

Will call my_static_function(data_args)

which will include data_io and data_type

Dealing with h5py files

H5 files efficiently store data at scale. file_golem standardizes i/o for these files. First, create a class that inherits from H5PYAbstractDatatype:

from file_golem import H5PYAbstractDatatype
class <my h5py datatype>(H5PYAbstractDatatype):
    RELATIVE_PATH_TUPLE = H5PYAbstractDatatype.RELATIVE_PATH_TUPLE + (
        <path extension 1>, <path extension 2>, ...
        )

    FILE_STRUCTURE = {
        <node name a>: {
            H5PYAbstractDatatype.DATASET_DATATYPE: <file_golem datatype of <node name a> or H5PYAbstractDatatype.ATTRITBUTE>
            }
        <group name 0>: {
            <node name a>: {
                <file_golem datatype of <node name a> >
                },
            },
        <group name 1>: {
            <node name b>: {
                <file_golem datatype of <node name b> >,
                }
            },
        <group name 2>: {
            <group name 3>: {
                ...
            }
            <node name c>: {
                <file_golem datatype of <node name c> >
                },
            },
        }

To perform i/o, one can call the typical file_golem calls,

data_io.load_data(, data_args={ .DATA: .DATASET_ADDRESS: [<group name 2>,<group name 3>,...] .DATASET_SIZE: .H5PY_IDX: ...

}

)