ddeutil-io 0.2.0

Input/Output transport utility objects

Input/Output Data Transport

The Input/Output Data Transport utility objects was created for full managed engine of configuration file that include load from any config file format types like .yaml, .json, or .toml, and manage retention and version of config files lifecycle.

This package has 3 parts of core module; Register, Config, and File. The base module is the file module that is open file objects with specific format that want to open include open directory objects (I think the open dir object do not should to use now because I do not done with its code.)

:round_pushpin: Installation

pip install -U ddeutil-io

Python version supported:

Python Version	Installation	Support Fixed Bug
>=3.9,<3.14	pip install -U ddeutil-io	:heavy_check_mark:

[!NOTE] This package need to install ddeutil first to be core package namespace. You do not need to pip it because I include this package to the required list.

For optional dependencies that should to additional install;

Module	Additional dependencies
YamlFl	pip install PyYaml
TomlFl	pip install toml
MsgpackFl	pip install msgpack

:dart: Features

The features of this package is Input/Output data transport utility objects.

Module	Name	Description	Remark
files.base	PathSearch
	RegexConf
files.file	Fl	Open File object that use to open any normal or compression file from current local file system
	EnvFlMixin
	EnvFl
	YamlEnvFl
	YamlFl
	YamlFlResolve
	JsonEnvFl
	JsonFl
	CsvFl
	CsvPipeFl
	TomlEnvFl
	TomlFl
	PickleFl		no cover
	MarshalFl		no cover
	MsgpackFl		no cover
config	ConfFl
	ConfSQLite
register	Register	Register Object that contain configuration loading methods and metadata management.
	FullRegister

:beers: Usages

I will show some usage example of function in this package. If you want to use complex or adjust some parameter, please see doc-string or real source code (I think it do not complex and you can see how that function work).

Files

For example, I will represent YamlEnvFl object that passing environment variable to reading content before passing to the Yaml loader.

data:
  get: HELLO ${HELLO}

import os
from ddeutil.io import YamlEnvFl

os.environ["HELLO"] = "WORLD"
content = YamlEnvFl('./.pre-commit-config.yaml').read(safe=True)
assert content['data']['get'] == "HELLO WORLD"

Config

Config Object is the dir system handler object that manage any files in that input dir path like load, save, load_stage, save_stage, or files.

from pathlib import Path
from ddeutil.io.config import ConfFl

config: ConfFl = ConfFl(path=Path('./conf'), compress="gzip")

data = config.load('config_file.yaml')
config.save_stage('./stage/file.json', data)

Register

The Register Object is the metadata generator object for the config data. If you passing name and configs to this object, it will find the config name in any stage storage and generate its metadata to you.

from ddeutil.io.register import Register
from ddeutil.io.param import Params

registry: Register = Register(
    name='examples:conn_data_local_file',
    params=Params.model_validate({
        "stages": {
            "raw": {"format": "{naming:%s}.{timestamp:%Y%m%d_%H%M%S}"},
        },
    }),
)
registry.move(stage="raw")

The raw data of this config was written in conn_file.yaml file.

conf/
  examples/
    conn_file.yaml

When call move method, it will transfer data from .yaml file to json file with the data hashing algorithm.

data/
  raw/
    conn_file_20240101_000000.json