Skip to main content

aiqclib

PyPI - Version Check Package codecov CodeFactor DOI

aiqclib is a Python library that provides a configuration-driven workflow for machine learning, simplifying dataset preparation, model training, and data classification. It is a core component of the AIQC project that aims to enhance anomaly detection in CTD (Conductivity, Temperature, Depth) data.

ML Algorithms Supported by aiqclib

Category Algorithm Short Name Method
Tree-Based & Ensemble XGBoost XGB Ensemble (Boosting)
Random Forest RF Ensemble (Bagging)
Decision Tree DT Tree
Linear & Geometric Logistic Regression Logit Linear
Linear Discriminant Analysis LDA Linear / Statistical
Support Vector Machine SVM Geometric
Instance-Based (Distance-Based) K-Nearest Neighbors KNN Distance-based
Probabilistic Gaussian Naive Bayes GNB Probabilistic
Neural Network Multilayer Perceptron MLP Neural Network

Installation

The package is available on PyPI.

Using pip:

pip install aiqclib

Using conda: (not published yet: the conda-forge recipe submission is still in progress; use pip or uv for now)

conda install -c conda-forge aiqclib

Documentation

Project documentation is hosted on Read the Docs.

Core Concepts

The library is designed around a three-stage machine learning workflow:

  1. Dataset Preparation: Prepare feature datasets from raw data and generate training, validation, and test data sets.
  2. Training & Evaluation: Train machine learning models and evaluate their performance using cross-validation.
  3. Classification: Apply a trained model to classify new, unseen data.

The automated real-time QC tests (Argo/CTD RTQC tests) are built in as qc_* features, so their flags can be used as training features. The Near-Real Time Quality Control (NRT QC) pipeline that runs them on their own and writes a final NRT flag per variable is the separate nrtqclib package, which builds on aiqclib (it was aiqclib.run_nrt_qc up to 0.14.0).

All three stages run at the observation level (one row per observation) by default, or at the profile level (one labeled row per profile/cast, with binary or bad-fraction proportion labels) via the profile step classes and configuration templates (extension="profile"); see the Profile-Level Pipeline guide in the documentation.

Each module is controlled by a YAML configuration file, allowing you to define and reproduce your entire workflow with ease.

Usage

The general workflow for any task in aiqclib follows these steps:

  1. Generate a Configuration Template: Create a starter YAML file for the task (e.g., prepare, train, classify).
  2. Customize the Configuration: Edit the YAML file to specify paths, dataset names, and other parameters.
  3. Run the Task: Load the configuration and execute the main function for the task.

1. Dataset Preparation

This workflow processes your input data and creates training, validation, and test sets.

Step 1: Generate a configuration template.

import aiqclib as aq

aq.write_config_template(file_name="/path/to/prepare_config.yaml", stage="prepare")

Step 2: Customize prepare_config.yaml. You must edit the file to set the correct input/output paths and define your dataset. See the Configuration section for details.

Step 3: Run the preparation process.

import aiqclib as aq

config = aq.read_config("/path/to/prepare_config.yaml")
aq.create_training_dataset(config)

This generates the following output folders:

  • summary: Statistics of input data used for normalization.
  • select: Profiles with bad observation flags (positive samples) and good profiles (negative samples).
  • locate: Observation records for both positive and negative profiles.
  • extract: Features extracted from the observation records.
  • training: The final training, validation, and test datasets.

2. Model Training and Evaluation

This workflow uses the prepared dataset to train a model and evaluate its performance.

Step 1: Generate a training configuration template.

import aiqclib as aq

aq.write_config_template(file_name="/path/to/training_config.yaml", stage="train")

Step 2: Customize training_config.yaml. Edit the file to point to your prepared dataset and define training parameters.

Step 3: Train and evaluate the model.

import aiqclib as aq

config = aq.read_config("/path/to/training_config.yaml")
aq.train_and_evaluate(config)

This generates the following output folders:

  • validate: Results from the cross-validation process.
  • build: The final trained models and their evaluation results on the test dataset.

3. Data Classification

This workflow applies a trained model to classify all observations in a dataset.

Step 1: Generate a classification configuration template.

import aiqclib as aq

aq.write_config_template(file_name="/path/to/classification_config.yaml", stage="classify")

Step 2: Customize classification_config.yaml. Edit the file to point to the input data and the trained model.

Step 3: Run classification.

import aiqclib as aq

config = aq.read_config("/path/to/classification_config.yaml")
aq.classify_dataset(config)

This workflow processes a dataset using a trained model and generates:

  • classify: The final classification results and a summary report.

Running several datasets at once

run_batch runs one phase ("prepare", "train", "classify") or "all" of them over a table of dataset names, returning a summary of every run:

import aiqclib as aq

summary = aq.run_batch(
    "datasets.txt",                                  # table of set names per dataset
    mode="all",
    prepare_config="prepare_config.yaml",
    training_config="training_config.yaml",
    classification_config="classification_config.yaml",
    verbose=True,
)

See the batch guide in the documentation for the table format and options.

Configuration

Configuration is managed via YAML files. The write_config_template function provides a starting point that you must customize for each module.

To inspect a stage's defaults without writing a file, read_config_template takes the same arguments and returns the configuration object instead:

print(aq.read_config_template(stage="prepare"))

1. Dataset Preparation (stage="prepare")

The preparation config requires you to modify two key sections:

  • path_info_sets: Defines the location of input and output data.

    path_info_sets:
      - name: data_set_1
        common:
          base_path: /path/to/data # EDIT: Root output directory
        input:
          base_path: /path/to/input # EDIT: Directory with input files
          step_folder_name: ""
        split:
          step_folder_name: training
    
  • data_sets: Defines a specific dataset to be processed.

    data_sets:
      - name: dataset_0001  # EDIT: Your data set name
        dataset_folder_name: dataset_0001  # EDIT: Your output folder
        input_file_name: nrt_cora_bo_4.parquet # EDIT: Your input filename
    

2. Training and Evaluation (stage="train")

The training config links the prepared data to the model training process.

  • path_info_sets: Defines where to find the prepared dataset and where to save model artifacts.

    path_info_sets:
      - name: data_set_1
        common:
          base_path: /path/to/data # EDIT: Root output directory
        input:
          step_folder_name: training
    
  • training_sets: Links to a dataset prepared in the previous workflow.

    training_sets:
      - name: training_0001  # EDIT: Your training name
        dataset_folder_name: dataset_0001  # EDIT: Your output folder
    

3. Classification (stage="classify")

The classification config uses a trained model to classify new data.

  • path_info_sets: Defines paths for raw data, models, and classification results.

    path_info_sets:
      - name: data_set_1
        common:
          base_path: /path/to/data # EDIT: Root output directory
        input:
          base_path: /path/to/input # EDIT: Directory with input files
          step_folder_name: ""
        model:
          base_path: /path/to/model  # EDIT: Directory with model files
          step_folder_name: model
        concat:
          step_folder_name: classification # EDIT: Directory with classification results
    
  • classification_sets: Defines a specific dataset to be classified.

    classification_sets:
      - name: classification_0001  # EDIT: Your classification name
        dataset_folder_name: dataset_0001  # EDIT: Your output folder
        input_file_name: nrt_cora_bo_4.parquet   # EDIT: Your input filename
    

Contributing & Development

We welcome contributions! Development setup (uv environment, test data), running tests, and code style are documented in CONTRIBUTING.md.

Releasing & Deployment (for Maintainers)

The release process (versioning checklist), building the docs, and deployment to PyPI, conda-forge, and Anaconda.org are documented in RELEASING.md.

Metadata

Release files for aiqclib 0.15.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for aiqclib 0.15.2
File Size Uploaded Provenance Actions
aiqclib-0.15.2.tar.gz 553.9 kB With provenance Details for aiqclib-0.15.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for aiqclib 0.15.2
File Interpreter ABI Platform Actions
aiqclib-0.15.2-py3-none-any.whl Python 3 none any Details for aiqclib-0.15.2-py3-none-any.whl

Total release size: 839.7 kB

Release files / aiqclib-0.15.2.tar.gz

Download URL aiqclib-0.15.2.tar.gz
Size 553.9 kB
Tags Source
SHA-256 checksum
How to use checksums
0c89c27e4e7d20d786f53a619dcc1895edf3590f072c86a84d98f91b0e38010b
BLAKE2b-256 checksum
How to use checksums
a061ac30a1e05d0e6308c609a55770d63279d7cffcd1f9352c3014855f06b0d9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 5, 2026.

Transparency log

Release files / aiqclib-0.15.2-py3-none-any.whl

Download URL aiqclib-0.15.2-py3-none-any.whl
Size 285.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
0f8866e1cdb886e4c781b30d37b987609889be144d5f8e8307a35139e4d1b543
BLAKE2b-256 checksum
How to use checksums
17c556b2c7cccdcb7bdd7de41f4b06d7611193f9922c2aff3b1a1d92460ee63a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 5, 2026.

Transparency log

Security for aiqclib 0.15.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for aiqclib-0.15.2.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 5, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page