Skip to main content

Input/Output transport utility objects

Project description

Input/Output Data Transport

test codecov pypi version python support version size code style: black type check: mypy pre-commit

The Input/Output Data Transport utility objects was created for full managed engine of configuration file that include load from any config file format types like .yaml, .json, or .toml, and manage retention and version of config files lifecycle.

This package has 3 parts of core module; Register, Stores, and Files. The base module is the file module that is open file objects with specific format that want to open include open directory objects (I think the open dir object do not should to use now because I do not done with its code.)

:round_pushpin: Installation

pip install -U ddeutil-io

Python version supported:

Python Version Installation Support Fixed Bug
>=3.9,<3.14 pip install -U ddeutil-io :heavy_check_mark:

[!NOTE] This package need to install ddeutil first to be core package namespace. You do not need to pip it because I include this package to the required list.

For optional dependencies that should to additional install;

Module Additional dependencies
YamlFl pip install PyYaml
TomlFl pip install toml
MsgpackFl pip install msgpack

:dart: Features

The features of this package is Input/Output data transport utility objects.

Module Name Description Remark
files PathSearch
RegexConf
files.file Fl Open File object that use to open any normal or compression file from current local file system
EnvFlMixin
EnvFl
YamlEnvFl
YamlFl
YamlFlResolve
JsonEnvFl
JsonFl
CsvFl
CsvPipeFl CSV open file object with pipe (|) seperator charactor.
TomlEnvFl
TomlFl
PickleFl no cover
MarshalFl no cover
MsgpackFl no cover
stores StoreFl
StoreSQLite
register Register Register Object that contain configuration loading methods and metadata management.
ArchiveRegister

:beers: Usages

I will show some usage example of function in this package. If you want to use complex or adjust some parameter, please see doc-string or real source code (I think it do not complex and you can see how that function work).

Files

For example, I will represent YamlEnvFl object that passing environment variable to reading content before passing to the Yaml loader.

data:
  get: HELLO ${HELLO}
import os
from ddeutil.io import YamlEnvFl

os.environ["HELLO"] = "WORLD"
content = YamlEnvFl('./source.yaml').read(safe=True)
assert content['data']['get'] == "HELLO WORLD"

[!NOTE] This module do not implement special function on IO like the build-in open function. It also make standard read and write file objects.

Store

Store object is the storing dir system handler object that manage any files in that dir path with get, move, load, save, or ls operations.

from ddeutil.io.stores import StoreFl

store: StoreFl = StoreFl(path='./conf', compress="gzip")

data = store.get(name='config_file.yaml')
store.save('./stage/file.json', data)
conf/
  examples/
    config_file.yaml
  stage/
    file.json

Register

The Register Object is the metadata generator object for the config data. If you passing name and configs to this object, it will find the config name in any stage storage and generate its metadata to you.

from ddeutil.io.register import Register
from ddeutil.io.config import Params

registry: Register = Register(
    name='examples:conn_data_local_file',
    params=Params(**{
        "stages": {
            "raw": {"format": "{naming:%s}.{timestamp:%Y%m%d_%H%M%S}"},
        },
    }),
)
registry.move(stage="raw")

The raw data of this config was written in conn_file.yaml file.

conf/
  examples/
    conn_file.yaml

When call move method, it will transfer data from .yaml file to json file with the data hashing algorithm.

data/
  raw/
    conn_file_20240101_000000.json

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

ddeutil_io-0.2.3.tar.gz (39.7 kB view details)

Uploaded Source

Built Distribution

ddeutil_io-0.2.3-py3-none-any.whl (32.8 kB view details)

Uploaded Python 3

File details

Details for the file ddeutil_io-0.2.3.tar.gz.

File metadata

  • Download URL: ddeutil_io-0.2.3.tar.gz
  • Upload date:
  • Size: 39.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.12.7

File hashes

Hashes for ddeutil_io-0.2.3.tar.gz
Algorithm Hash digest
SHA256 702ac6398c024b1ec91b1497c28dcecf103c175ec6c3dc66cc6125ff19e99947
MD5 a533767d1616aa0f317a9d3756039a20
BLAKE2b-256 2fc1d256c71b03753cb6d071a61bdfa75dade93e8055f7f255099b43802d663c

See more details on using hashes here.

File details

Details for the file ddeutil_io-0.2.3-py3-none-any.whl.

File metadata

  • Download URL: ddeutil_io-0.2.3-py3-none-any.whl
  • Upload date:
  • Size: 32.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.12.7

File hashes

Hashes for ddeutil_io-0.2.3-py3-none-any.whl
Algorithm Hash digest
SHA256 3f2e059f68c717da630f3504fc0a4d56997d194bacaf253dc52aeb4698acf8ef
MD5 1ddc13edfa813aa7cc89d312968450f7
BLAKE2b-256 d8e32c7485335e5157139afc63ee1aee384f361499cd1b83133c5ff156fa7120

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page