ds-efficiency 0.1.1

A professional toolkit for Data Science efficiency, featuring automatic logging, runtime type checking, and DataFrame schema validation.

ds_efficiency

A professional toolkit for Data Science efficiency, featuring automatic logging, runtime type checking, and DataFrame schema validation.

Key Features

• MetaEngine: A metaclass that automatically adds logging, timing, and type validation to your classes.
• DataFrame Validation: Validate Polars DataFrames using docstring schemas.
• Granular Logging: Automatic method-level loggers for easier debugging.
• Runtime Type Checking: Powered by typeguard to ensure data integrity.

Installation

Using uv (recommended):

uv pip install ds_efficiency

Or using pip:

pip install ds_efficiency

Quick Start

Using MetaEngine

Simply set MetaEngine as the metaclass for your data processing classes.

import polars as pl
from ds_efficiency import MetaEngine

class DataProcessor(metaclass=MetaEngine):
    def process_data(self, df: pl.DataFrame) -> pl.DataFrame:
        """
        Schema:
        -------
        df: pl.DataFrame
            |-- id: int
            |-- value: float
        Meta instruction: Drop extra columns.
        """
        # The 'df' is automatically validated and casted before this line
        self.log_process_data.info("Calculating total...")
        return df.with_columns((pl.col("value") * 2).alias("doubled"))

# Usage
processor = DataProcessor()
df = pl.DataFrame({"id": [1, 2], "value": [10.5, 20.0], "extra": [True, False]})
result = processor.process_data(df)
# 'extra' column is dropped, types are checked, and timing is logged!

Logging Configuration

You can configure the log file at the class or environment level:

class MyClass(metaclass=MetaEngine):
    log_file = "logs/my_app.log"
    log_level = "DEBUG"

Or set the environment variable:

export METAENGINE_LOG_FILE="logs/global.log"

License

MIT