dmqclib 0.5.1

This package aims to offer helper functions that simplify model building and evaluation

dmqclib

dmqclib is a Python library that provides a configuration-driven workflow for machine learning, simplifying dataset preparation, model training, and data classification. It is a core component of the AIQC project.

Installation

The package is available on PyPI and conda-forge.

Using pip:

pip install dmqclib

Using conda:

conda install -c conda-forge dmqclib

Core Concepts

The library is designed around a three-stage workflow:

1. Dataset Preparation: Ingest raw data and transform it into a feature-rich dataset ready for training.
2. Training & Evaluation: Train machine learning models and evaluate their performance using cross-validation.
3. Classification: Apply a trained model to classify new, unseen data.

Each stage is controlled by a YAML configuration file, allowing you to define and reproduce your entire workflow with ease.

Usage

The general workflow for any task in dmqclib follows these steps:

1. Generate a Configuration Template: Create a starter YAML file for the task (e.g., prepare, train, classify).
2. Customize the Configuration: Edit the YAML file to specify paths, dataset names, and other parameters.
3. Run the Task: Load the configuration and execute the main function for the task.

1. Dataset Preparation

This workflow processes your input data and creates training, validation, and test sets.

Step 1: Generate a configuration template.

import dmqclib as dm

# This creates 'prepare_config.yaml' with predefined sections
dm.write_config_template(file_name="/path/to/prepare_config.yaml", stage="prepare")

Step 2: Customize prepare_config.yaml. You must edit the file to set the correct input/output paths and define your dataset. See the Configuration section for details.

Step 3: Run the preparation process.

import dmqclib as dm

config = dm.read_config("/path/to/prepare_config.yaml")
dm.create_training_dataset(config)

This generates the following output folders:

• summary: Statistics of input data used for normalization.
• select: Profiles with bad observation flags (positive samples) and good profiles (negative samples).
• locate: Observation records for both positive and negative profiles.
• extract: Features extracted from the observation records.
• training: The final training, validation, and test datasets.

2. Model Training and Evaluation

This workflow uses the prepared dataset to train a model and evaluate its performance.

Step 1: Generate a training configuration template.

import dmqclib as dm

dm.write_config_template(file_name="/path/to/training_config.yaml", stage="train")

Step 2: Customize training_config.yaml. Edit the file to point to your prepared dataset and define training parameters.

Step 3: Train and evaluate the model.

import dmqclib as dm

config = dm.read_config("/path/to/training_config.yaml")
dm.train_and_evaluate(config)

This generates the following output folders:

• validate: Results from the cross-validation process.
• build: The final trained models and their evaluation results on the test dataset.

3. Data Classification

This workflow applies a trained model to classify all observations in a dataset.

Step 1: Generate a classification configuration template.

import dmqclib as dm

dm.write_config_template(file_name="/path/to/classification_config.yaml", stage="classify")

Step 2: Customize classification_config.yaml. Edit the file to point to the input data and the trained model.

Step 3: Run classification.

import dmqclib as dm

config = dm.read_config("/path/to/classification_config.yaml")
dm.classify_dataset(config)

This workflow processes a dataset using a trained model and generates:

• classify: The final classification results and a summary report.

Configuration

Configuration is managed via YAML files. The write_config_template function provides a starting point that you must customize for each module.

1. Dataset Preparation (stage="prepare")

The preparation config requires you to modify two key sections:

• path_info_sets: Defines the location of input and output data.

  path_info_sets:
    - name: data_set_1
      common:
        base_path: /path/to/data # EDIT: Root output directory
      input:
        base_path: /path/to/input # EDIT: Directory with input files
        step_folder_name: ""
      split:
        step_folder_name: training
  

• data_sets: Defines a specific dataset to be processed.

  data_sets:
    - name: dataset_0001  # EDIT: Your data set name
      dataset_folder_name: dataset_0001  # EDIT: Your output folder
      input_file_name: nrt_cora_bo_4.parquet # EDIT: Your input filename
  

2. Training and Evaluation (stage="train")

The training config links the prepared data to the model training process.

• path_info_sets: Defines where to find the prepared dataset and where to save model artifacts.

  path_info_sets:
    - name: data_set_1
      common:
        base_path: /path/to/data # EDIT: Root output directory
      input:
        step_folder_name: training
  

• training_sets: Links to a dataset prepared in the previous workflow.

  training_sets:
    - name: training_0001  # EDIT: Your training name
      dataset_folder_name: dataset_0001  # EDIT: Your output folder
  

3. Classification (stage="classify")

The classification config uses a trained model to classify new data.

• path_info_sets: Defines paths for raw data, models, and classification results.

  path_info_sets:
    - name: data_set_1
      common:
        base_path: /path/to/data # EDIT: Root output directory
      input:
        base_path: /path/to/input # EDIT: Directory with input files
        step_folder_name: ""
      model:
        base_path: /path/to/model  # EDIT: Directory with model files
        step_folder_name: model
      concat:
        step_folder_name: classification # EDIT: Directory with classification results
  

• classification_sets: Defines a specific dataset to be classified.

  classification_sets:
    - name: classification_0001  # EDIT: Your classification name
      dataset_folder_name: dataset_0001  # EDIT: Your output folder
      input_file_name: nrt_cora_bo_4.parquet   # EDIT: Your input filename
  

Contributing & Development

We welcome contributions! Please use the following guidelines for development.

Environment Setup

We recommend using uv for managing the development environment.

1. Install python, uv, and ruff (e.g., via conda or mamba):

   # Using mamba (recommended)
   mamba create -n dmqclib-dev -c conda-forge python=3.12 uv ruff
   mamba activate dmqclib-dev
   

2. Navigate to the project root and create the virtual environment:

   cd /path/to/dmqclib
   uv sync
   

3. (Optional) Install the library in editable mode. This is sometimes needed before running tests.

   uv pip install -e .
   

Running Tests

Unit tests are run with pytest.

uv run pytest -v

Code Style (Linting & Formatting)

We use Ruff for linting and formatting.

Linting:

# Lint the library source code
uvx ruff check src

# Lint the test code
uvx ruff check tests

Formatting:

# Format the library source code
uvx ruff format src

# Format the test code
uvx ruff format tests

Documentation (for Maintainers)

Project documentation is hosted on Read the Docs.

Building Docs Locally

1. Update Docstrings (Requires Google Gemini API Key):

   # Update docstrings for source files
   python ./docs/scripts/update_docstrings.py src docs/scripts/prompt_main.txt
   
   # Update docstrings for test files
   python ./docs/scripts/update_docstrings.py tests docs/scripts/prompt_unittest.txt
   

2. Review Docstrings: Manually review all modified files. Remove generated headers/footers and correct any sections marked with "Issues:".

3. Update API Documents: From the project root, run:

   uv run sphinx-apidoc -f --remove-old --module-first -o docs/source/api src/dmqclib
   

4. Build HTML: From the project root, run:

   cd docs
   uv run make html
   cd ..
   

   You can view the generated site by opening docs/build/html/index.html in a browser.

Deployment (for Maintainers)

PyPI

The package is published to PyPI automatically via a GitHub Action whenever a new release is created on GitHub.

Anaconda.org (Manual)

Publishing to the takayasaito channel on Anaconda.org is a manual process.

1. Install build tools:

   mamba install -c conda-forge conda-build anaconda-client grayskull
   

2. Generate Recipe: From the project root, run grayskull pypi dmqclib. This creates dmqclib/meta.yaml.

3. Build Package: conda build dmqclib

4. Upload Package:

   anaconda login
   anaconda upload /path/to/your/conda-bld/noarch/dmqclib-*.conda
   

5. Cleanup: Copy dmqclib/meta.yaml to conda/meta.yaml for version control and remove the temporary dmqclib directory.

conda-forge (Manual)

Submitting or updating the package on conda-forge involves creating a pull request to the conda-forge/staged-recipes repository.

1. Fork and clone the staged-recipes repository.
2. Create a new branch (e.g., git checkout -b dmqclib-recipe).
3. Generate a strict recipe: grayskull pypi dmqclib --strict-conda-forge.
4. Review meta.yaml and ensure it meets conda-forge standards.
5. Commit, push, and open a pull request to the staged-recipes repository.