file-golem 0.1.15

This package handles file I/O for larger projects, with the aim to simplify/standardize working with many datasets and 3rd party codebases across different naming conventions, data types, and storage systems.

To run unit tests, run

$python -m unittest discover -s tests


$for dev, it is better to run pip install -e .

Project set up

System Identity File

Create a file named .system_identity.txt at the top level project directory. Decide on a name for your local system, and add it as a single line to the the file as:

In conf/system_configs/system_conf.yaml, input the following:

System config

Create a config file in conf/system_configs/system_conf.yaml. This holds information file_golem needs to navigate your file systems:

filelock_timeout: <number of seconds to wait for filelock (default: wait indefinitely)>

system_paths:
  <my local system name>:
    base_data_directory: <project directory (output of pwd())>

    base_conda_name: <path to conda environment when calling conda env list>
    external_calls:
      <name of external codebase used in code>:
        entry_point: <absolute path of top project directory>
        conda_path: <path to external environment when calling conda env list>
        python_entry_point: <Name of the python file (from top level project directory sans .py)>


    (optional) data_directory_name: <name of subdirectory where untracked data is stored > (default: data)

    system_transfers:
        <non local mounted system name>:
            mount_point: <absolute path of the mount point on your local system>
            data_directory_extension: <path to the root of the data directory on the mounted system> (file_folem will look in mount_point/data_directory_extension) 

base_data_directory can be customized to an alternative location if desired.

Intializing a FileGolem object

It is best practice to have a saved system config file. From there, the syntax for initializing a FileGolem object is

data_io = FileGolem(
    system_config_path = <absolute system config path>,
    is_debug = <True or False if debug messages are desired>
)

If you need to quickly initialize a FileGolem without a saved config file, the syntax is:

system_config = {
    (See above)
}
data_io = FileGolem(
    system_config = system_config,
    is_debug = <True or False if debug messages are desired>
)

Building a file path

FilePathEntries.CUSTOM_LOGIC : '<my_static_function>'

Will call my_static_function(data_args)

which will include data_io and data_type

Dealing with h5py files

H5 files efficiently store data at scale. file_golem standardizes i/o for these files. First, create a class that inherits from H5PYAbstractDatatype:

from file_golem import H5PYAbstractDatatype
class <my h5py datatype>(H5PYAbstractDatatype):
    RELATIVE_PATH_TUPLE = H5PYAbstractDatatype.RELATIVE_PATH_TUPLE + (
        <path extension 1>, <path extension 2>, ...
        )

    FILE_STRUCTURE = {
        <node name a>: {
            H5PYAbstractDatatype.DATASET_DATATYPE: <file_golem datatype of <node name a> or H5PYAbstractDatatype.ATTRITBUTE>
            }
        <group name 0>: {
            <node name a>: {
                <file_golem datatype of <node name a> >
                },
            },
        <group name 1>: {
            <node name b>: {
                <file_golem datatype of <node name b> >,
                }
            },
        <group name 2>: {
            <group name 3>: {
                ...
            }
            <node name c>: {
                <file_golem datatype of <node name c> >
                },
            },
        }

To perform i/o, one can call the typical file_golem calls,

data_io.load_data(<my h5py datatype>, data_args={
    <my h5py datatype>.DATA: <torch or numpy tensor to save>
    <my h5py datatype>.DATASET_ADDRESS: [<group name 2>,<group name 3>,...]
    <my h5py datatype>.DATASET_SIZE: <shape of the node tensor>
    <my h5py datatype>.H5PY_IDX: <single index or tensor of indices of the full tensor>
    ...

    }
)