Skip to main content

aiqclib

PyPI - Version Check Package codecov CodeFactor DOI

aiqclib is a Python library that provides a configuration-driven workflow for machine learning, simplifying dataset preparation, model training, and data classification. It is a core component of the AIQC project that aims to enhance anomaly detection in CTD (Conductivity, Temperature, Depth) data.

ML Algorithms Supported by aiqclib

Category Algorithm Short Name Method
Tree-Based & Ensemble XGBoost XGB Ensemble (Boosting)
Random Forest RF Ensemble (Bagging)
Decision Tree DT Tree
Linear & Geometric Logistic Regression Logit Linear
Linear Discriminant Analysis LDA Linear / Statistical
Support Vector Machine SVM Geometric
Instance-Based (Distance-Based) K-Nearest Neighbors KNN Distance-based
Probabilistic Gaussian Naive Bayes GNB Probabilistic
Neural Network Multilayer Perceptron MLP Neural Network

Installation

The package is available on PyPI.

Using pip:

pip install aiqclib

Using conda: (not published yet: the conda-forge recipe submission is still in progress; use pip or uv for now)

conda install -c conda-forge aiqclib

Documentation

Project documentation is hosted on Read the Docs.

Core Concepts

The library is designed around a three-stage machine learning workflow:

  1. Dataset Preparation: Prepare feature datasets from raw data and generate training, validation, and test data sets.
  2. Training & Evaluation: Train machine learning models and evaluate their performance using cross-validation.
  3. Classification: Apply a trained model to classify new, unseen data.

In addition, a standalone Near-Real Time Quality Control (NRT QC) module applies automated real-time QC tests (Argo/CTD RTQC tests) to temperature and salinity profiles, producing per-item flag columns and a final NRT flag per variable.

All three stages run at the observation level (one row per observation) by default, or at the profile level (one labeled row per profile/cast, with binary or bad-fraction proportion labels) via the profile step classes and configuration templates (extension="profile"); see the Profile-Level Pipeline guide in the documentation.

Each module is controlled by a YAML configuration file, allowing you to define and reproduce your entire workflow with ease.

Usage

The general workflow for any task in aiqclib follows these steps:

  1. Generate a Configuration Template: Create a starter YAML file for the task (e.g., prepare, train, classify).
  2. Customize the Configuration: Edit the YAML file to specify paths, dataset names, and other parameters.
  3. Run the Task: Load the configuration and execute the main function for the task.

1. Dataset Preparation

This workflow processes your input data and creates training, validation, and test sets.

Step 1: Generate a configuration template.

import aiqclib as aq

aq.write_config_template(file_name="/path/to/prepare_config.yaml", stage="prepare")

Step 2: Customize prepare_config.yaml. You must edit the file to set the correct input/output paths and define your dataset. See the Configuration section for details.

Step 3: Run the preparation process.

import aiqclib as aq

config = aq.read_config("/path/to/prepare_config.yaml")
aq.create_training_dataset(config)

This generates the following output folders:

  • summary: Statistics of input data used for normalization.
  • select: Profiles with bad observation flags (positive samples) and good profiles (negative samples).
  • locate: Observation records for both positive and negative profiles.
  • extract: Features extracted from the observation records.
  • training: The final training, validation, and test datasets.

2. Model Training and Evaluation

This workflow uses the prepared dataset to train a model and evaluate its performance.

Step 1: Generate a training configuration template.

import aiqclib as aq

aq.write_config_template(file_name="/path/to/training_config.yaml", stage="train")

Step 2: Customize training_config.yaml. Edit the file to point to your prepared dataset and define training parameters.

Step 3: Train and evaluate the model.

import aiqclib as aq

config = aq.read_config("/path/to/training_config.yaml")
aq.train_and_evaluate(config)

This generates the following output folders:

  • validate: Results from the cross-validation process.
  • build: The final trained models and their evaluation results on the test dataset.

3. Data Classification

This workflow applies a trained model to classify all observations in a dataset.

Step 1: Generate a classification configuration template.

import aiqclib as aq

aq.write_config_template(file_name="/path/to/classification_config.yaml", stage="classify")

Step 2: Customize classification_config.yaml. Edit the file to point to the input data and the trained model.

Step 3: Run classification.

import aiqclib as aq

config = aq.read_config("/path/to/classification_config.yaml")
aq.classify_dataset(config)

This workflow processes a dataset using a trained model and generates:

  • classify: The final classification results and a summary report.

Running several datasets at once

run_batch runs one phase ("prepare", "train", "classify", "nrt_qc") or "all" of the first three over a table of dataset names, returning a summary of every run:

import aiqclib as aq

summary = aq.run_batch(
    "datasets.txt",                                  # table of set names per dataset
    mode="all",
    prepare_config="prepare_config.yaml",
    training_config="training_config.yaml",
    classification_config="classification_config.yaml",
    verbose=True,
)

NRT QC is run the same way, with mode="nrt_qc" and nrt_qc_config=.... It is deliberately not part of "all", because its flags are an input to the prepare phase rather than a step of it.

See the batch guide in the documentation for the table format and options.

4. Near-Real Time Quality Control (NRT QC)

This workflow applies automated real-time QC tests to a dataset without needing a trained model.

Step 1: Generate an NRT QC configuration template.

import aiqclib as aq

aq.write_config_template(file_name="/path/to/nrt_qc_config.yaml", stage="nrt_qc")

Step 2: Customize nrt_qc_config.yaml. Edit the file to point to the input data and select the QC items and their thresholds. Prepare one configuration file per region (the regional ranges differ).

Step 3: Run the NRT QC process.

import aiqclib as aq

config = aq.read_config("/path/to/nrt_qc_config.yaml")
aq.run_nrt_qc(config)

This generates:

  • qc: The input data with one flag column per QC item.
  • nrt_qc: The final output parquet (input columns + item columns + temp_nrt_flag / psal_nrt_flag).
  • compare: Optional per-variable reports comparing existing NRT QC flags with the newly computed ones.

Configuration

Configuration is managed via YAML files. The write_config_template function provides a starting point that you must customize for each module.

To inspect a stage's defaults without writing a file, read_config_template takes the same arguments and returns the configuration object instead:

print(aq.read_config_template(stage="prepare"))

1. Dataset Preparation (stage="prepare")

The preparation config requires you to modify two key sections:

  • path_info_sets: Defines the location of input and output data.

    path_info_sets:
      - name: data_set_1
        common:
          base_path: /path/to/data # EDIT: Root output directory
        input:
          base_path: /path/to/input # EDIT: Directory with input files
          step_folder_name: ""
        split:
          step_folder_name: training
    
  • data_sets: Defines a specific dataset to be processed.

    data_sets:
      - name: dataset_0001  # EDIT: Your data set name
        dataset_folder_name: dataset_0001  # EDIT: Your output folder
        input_file_name: nrt_cora_bo_4.parquet # EDIT: Your input filename
    

2. Training and Evaluation (stage="train")

The training config links the prepared data to the model training process.

  • path_info_sets: Defines where to find the prepared dataset and where to save model artifacts.

    path_info_sets:
      - name: data_set_1
        common:
          base_path: /path/to/data # EDIT: Root output directory
        input:
          step_folder_name: training
    
  • training_sets: Links to a dataset prepared in the previous workflow.

    training_sets:
      - name: training_0001  # EDIT: Your training name
        dataset_folder_name: dataset_0001  # EDIT: Your output folder
    

3. Classification (stage="classify")

The classification config uses a trained model to classify new data.

  • path_info_sets: Defines paths for raw data, models, and classification results.

    path_info_sets:
      - name: data_set_1
        common:
          base_path: /path/to/data # EDIT: Root output directory
        input:
          base_path: /path/to/input # EDIT: Directory with input files
          step_folder_name: ""
        model:
          base_path: /path/to/model  # EDIT: Directory with model files
          step_folder_name: model
        concat:
          step_folder_name: classification # EDIT: Directory with classification results
    
  • classification_sets: Defines a specific dataset to be classified.

    classification_sets:
      - name: classification_0001  # EDIT: Your classification name
        dataset_folder_name: dataset_0001  # EDIT: Your output folder
        input_file_name: nrt_cora_bo_4.parquet   # EDIT: Your input filename
    

Contributing & Development

We welcome contributions! Development setup (uv environment, test data), running tests, and code style are documented in CONTRIBUTING.md.

Releasing & Deployment (for Maintainers)

The release process (versioning checklist), building the docs, and deployment to PyPI, conda-forge, and Anaconda.org are documented in RELEASING.md.

Metadata

Release files for aiqclib 0.12.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for aiqclib 0.12.2
File Size Uploaded Provenance Actions
aiqclib-0.12.2.tar.gz 528.5 kB With provenance Details for aiqclib-0.12.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for aiqclib 0.12.2
File Interpreter ABI Platform Actions
aiqclib-0.12.2-py3-none-any.whl Python 3 none any Details for aiqclib-0.12.2-py3-none-any.whl

Total release size: 799.6 kB

Release files / aiqclib-0.12.2.tar.gz

Download URL aiqclib-0.12.2.tar.gz
Size 528.5 kB
Tags Source
SHA-256 checksum
How to use checksums
48779419bd6abbf1f30bebc885d7a2585ec44c78d5cb5ad71862954f5a74a1df
BLAKE2b-256 checksum
How to use checksums
62862335f99f51bf0126b49b8fef1ec8a25deadfb3226f176034b4a70701785d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 21, 2026.

Transparency log

Release files / aiqclib-0.12.2-py3-none-any.whl

Download URL aiqclib-0.12.2-py3-none-any.whl
Size 271.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
9528d010a7f1da7e17589036a04607336d5c9d87a50b9e0115af0341825ea272
BLAKE2b-256 checksum
How to use checksums
297a5ac324aaad07f549aa38610901c2b4094fd01529240318e295441b1a5f3c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 21, 2026.

Transparency log

Security for aiqclib 0.12.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for aiqclib-0.12.2.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 21, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page