Skip to main content

aiqclib

PyPI - Version Check Package codecov CodeFactor DOI

aiqclib is a Python library that provides a configuration-driven workflow for machine learning, simplifying dataset preparation, model training, and data classification. It is a core component of the AIQC project that aims to enhance anomaly detection in CTD (Conductivity, Temperature, Depth) data.

ML Algorithms Supported by aiqclib

Category Algorithm Short Name Method
Tree-Based & Ensemble XGBoost XGB Ensemble (Boosting)
Random Forest RF Ensemble (Bagging)
Decision Tree DT Tree
Linear & Geometric Logistic Regression Logit Linear
Linear Discriminant Analysis LDA Linear / Statistical
Support Vector Machine SVM Geometric
Instance-Based (Distance-Based) K-Nearest Neighbors KNN Distance-based
Probabilistic Gaussian Naive Bayes GNB Probabilistic
Neural Network Multilayer Perceptron MLP Neural Network

Installation

The package is available on PyPI.

Using pip:

pip install aiqclib

Using conda: (not published yet: the conda-forge recipe submission is still in progress; use pip or uv for now)

conda install -c conda-forge aiqclib

Documentation

Project documentation is hosted on Read the Docs.

Core Concepts

The library is designed around a three-stage machine learning workflow:

  1. Dataset Preparation: Prepare feature datasets from raw data and generate training, validation, and test data sets.
  2. Training & Evaluation: Train machine learning models and evaluate their performance using cross-validation.
  3. Classification: Apply a trained model to classify new, unseen data.

In addition, a standalone Near-Real Time Quality Control (NRT QC) module applies automated real-time QC tests (Argo/CTD RTQC tests) to temperature and salinity profiles, producing per-item flag columns and a final NRT flag per variable.

All three stages run at the observation level (one row per observation) by default, or at the profile level (one labeled row per profile/cast, with binary or bad-fraction proportion labels) via the profile step classes and configuration templates (extension="profile"); see the Profile-Level Pipeline guide in the documentation.

Each module is controlled by a YAML configuration file, allowing you to define and reproduce your entire workflow with ease.

Usage

The general workflow for any task in aiqclib follows these steps:

  1. Generate a Configuration Template: Create a starter YAML file for the task (e.g., prepare, train, classify).
  2. Customize the Configuration: Edit the YAML file to specify paths, dataset names, and other parameters.
  3. Run the Task: Load the configuration and execute the main function for the task.

1. Dataset Preparation

This workflow processes your input data and creates training, validation, and test sets.

Step 1: Generate a configuration template.

import aiqclib as aq

aq.write_config_template(file_name="/path/to/prepare_config.yaml", stage="prepare")

Step 2: Customize prepare_config.yaml. You must edit the file to set the correct input/output paths and define your dataset. See the Configuration section for details.

Step 3: Run the preparation process.

import aiqclib as aq

config = aq.read_config("/path/to/prepare_config.yaml")
aq.create_training_dataset(config)

This generates the following output folders:

  • summary: Statistics of input data used for normalization.
  • select: Profiles with bad observation flags (positive samples) and good profiles (negative samples).
  • locate: Observation records for both positive and negative profiles.
  • extract: Features extracted from the observation records.
  • training: The final training, validation, and test datasets.

2. Model Training and Evaluation

This workflow uses the prepared dataset to train a model and evaluate its performance.

Step 1: Generate a training configuration template.

import aiqclib as aq

aq.write_config_template(file_name="/path/to/training_config.yaml", stage="train")

Step 2: Customize training_config.yaml. Edit the file to point to your prepared dataset and define training parameters.

Step 3: Train and evaluate the model.

import aiqclib as aq

config = aq.read_config("/path/to/training_config.yaml")
aq.train_and_evaluate(config)

This generates the following output folders:

  • validate: Results from the cross-validation process.
  • build: The final trained models and their evaluation results on the test dataset.

3. Data Classification

This workflow applies a trained model to classify all observations in a dataset.

Step 1: Generate a classification configuration template.

import aiqclib as aq

aq.write_config_template(file_name="/path/to/classification_config.yaml", stage="classify")

Step 2: Customize classification_config.yaml. Edit the file to point to the input data and the trained model.

Step 3: Run classification.

import aiqclib as aq

config = aq.read_config("/path/to/classification_config.yaml")
aq.classify_dataset(config)

This workflow processes a dataset using a trained model and generates:

  • classify: The final classification results and a summary report.

Running several datasets at once

run_batch runs one phase ("prepare", "train", "classify", "nrt_qc") or "all" of the first three over a table of dataset names, returning a summary of every run:

import aiqclib as aq

summary = aq.run_batch(
    "datasets.txt",                                  # table of set names per dataset
    mode="all",
    prepare_config="prepare_config.yaml",
    training_config="training_config.yaml",
    classification_config="classification_config.yaml",
    verbose=True,
)

NRT QC is run the same way, with mode="nrt_qc" and nrt_qc_config=.... It is deliberately not part of "all", because its flags are an input to the prepare phase rather than a step of it.

See the batch guide in the documentation for the table format and options.

4. Near-Real Time Quality Control (NRT QC)

This workflow applies automated real-time QC tests to a dataset without needing a trained model.

Step 1: Generate an NRT QC configuration template.

import aiqclib as aq

aq.write_config_template(file_name="/path/to/nrt_qc_config.yaml", stage="nrt_qc")

Step 2: Customize nrt_qc_config.yaml. Edit the file to point to the input data and select the QC items and their thresholds. Prepare one configuration file per region (the regional ranges differ).

Step 3: Run the NRT QC process.

import aiqclib as aq

config = aq.read_config("/path/to/nrt_qc_config.yaml")
aq.run_nrt_qc(config)

This generates:

  • qc: The input data with one flag column per QC item.
  • nrt_qc: The final output parquet (input columns + item columns + temp_nrt_flag / psal_nrt_flag).
  • compare: Optional per-variable reports comparing existing NRT QC flags with the newly computed ones.

Configuration

Configuration is managed via YAML files. The write_config_template function provides a starting point that you must customize for each module.

To inspect a stage's defaults without writing a file, read_config_template takes the same arguments and returns the configuration object instead:

print(aq.read_config_template(stage="prepare"))

1. Dataset Preparation (stage="prepare")

The preparation config requires you to modify two key sections:

  • path_info_sets: Defines the location of input and output data.

    path_info_sets:
      - name: data_set_1
        common:
          base_path: /path/to/data # EDIT: Root output directory
        input:
          base_path: /path/to/input # EDIT: Directory with input files
          step_folder_name: ""
        split:
          step_folder_name: training
    
  • data_sets: Defines a specific dataset to be processed.

    data_sets:
      - name: dataset_0001  # EDIT: Your data set name
        dataset_folder_name: dataset_0001  # EDIT: Your output folder
        input_file_name: nrt_cora_bo_4.parquet # EDIT: Your input filename
    

2. Training and Evaluation (stage="train")

The training config links the prepared data to the model training process.

  • path_info_sets: Defines where to find the prepared dataset and where to save model artifacts.

    path_info_sets:
      - name: data_set_1
        common:
          base_path: /path/to/data # EDIT: Root output directory
        input:
          step_folder_name: training
    
  • training_sets: Links to a dataset prepared in the previous workflow.

    training_sets:
      - name: training_0001  # EDIT: Your training name
        dataset_folder_name: dataset_0001  # EDIT: Your output folder
    

3. Classification (stage="classify")

The classification config uses a trained model to classify new data.

  • path_info_sets: Defines paths for raw data, models, and classification results.

    path_info_sets:
      - name: data_set_1
        common:
          base_path: /path/to/data # EDIT: Root output directory
        input:
          base_path: /path/to/input # EDIT: Directory with input files
          step_folder_name: ""
        model:
          base_path: /path/to/model  # EDIT: Directory with model files
          step_folder_name: model
        concat:
          step_folder_name: classification # EDIT: Directory with classification results
    
  • classification_sets: Defines a specific dataset to be classified.

    classification_sets:
      - name: classification_0001  # EDIT: Your classification name
        dataset_folder_name: dataset_0001  # EDIT: Your output folder
        input_file_name: nrt_cora_bo_4.parquet   # EDIT: Your input filename
    

Contributing & Development

We welcome contributions! Development setup (uv environment, test data), running tests, and code style are documented in CONTRIBUTING.md.

Releasing & Deployment (for Maintainers)

The release process (versioning checklist), building the docs, and deployment to PyPI, conda-forge, and Anaconda.org are documented in RELEASING.md.

Metadata

Release files for aiqclib 0.14.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for aiqclib 0.14.0
File Size Uploaded Provenance Actions
aiqclib-0.14.0.tar.gz 598.0 kB With provenance Details for aiqclib-0.14.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for aiqclib 0.14.0
File Interpreter ABI Platform Actions
aiqclib-0.14.0-py3-none-any.whl Python 3 none any Details for aiqclib-0.14.0-py3-none-any.whl

Total release size: 905.9 kB

Release files / aiqclib-0.14.0.tar.gz

Download URL aiqclib-0.14.0.tar.gz
Size 598.0 kB
Tags Source
SHA-256 checksum
How to use checksums
aa01aceaed50df48892c6b19c9ff056177ace29b6c0b5fdfd6f08155da440f51
BLAKE2b-256 checksum
How to use checksums
2296ad284e74434276aaa35de3ee7ba12d0a5dab65e06b8931e243c18db51a5c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 5, 2026.

Transparency log

Release files / aiqclib-0.14.0-py3-none-any.whl

Download URL aiqclib-0.14.0-py3-none-any.whl
Size 307.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
e57c4033e48ffc5ad72099c218caa5bccd8c80fd6faf7a67666e072a069e206f
BLAKE2b-256 checksum
How to use checksums
1acc81225626b97a9e6fd35e2719888e684c5706e7fed725191282e707b80e96
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 5, 2026.

Transparency log

Security for aiqclib 0.14.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for aiqclib-0.14.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 5, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page