aiqclib
aiqclib is a Python library that provides a configuration-driven workflow for machine learning, simplifying dataset preparation, model training, and data classification. It is a core component of the AIQC project that aims to enhance anomaly detection in CTD (Conductivity, Temperature, Depth) data.
ML Algorithms Supported by aiqclib
| Category | Algorithm | Short Name | Method |
|---|---|---|---|
| Tree-Based & Ensemble | XGBoost | XGB | Ensemble (Boosting) |
| Random Forest | RF | Ensemble (Bagging) | |
| Decision Tree | DT | Tree | |
| Linear & Geometric | Logistic Regression | Logit | Linear |
| Linear Discriminant Analysis | LDA | Linear / Statistical | |
| Support Vector Machine | SVM | Geometric | |
| Instance-Based (Distance-Based) | K-Nearest Neighbors | KNN | Distance-based |
| Probabilistic | Gaussian Naive Bayes | GNB | Probabilistic |
| Neural Network | Multilayer Perceptron | MLP | Neural Network |
Installation
The package is available on PyPI.
Using pip:
pip install aiqclib
Using conda: (not published yet: the conda-forge recipe submission is still in progress; use pip or uv for now)
conda install -c conda-forge aiqclib
Documentation
Project documentation is hosted on Read the Docs.
Core Concepts
The library is designed around a three-stage machine learning workflow:
- Dataset Preparation: Prepare feature datasets from raw data and generate training, validation, and test data sets.
- Training & Evaluation: Train machine learning models and evaluate their performance using cross-validation.
- Classification: Apply a trained model to classify new, unseen data.
The automated real-time QC tests (Argo/CTD RTQC tests) are built in as qc_* features, so their flags can be used as training features. The Near-Real Time Quality Control (NRT QC) pipeline that runs them on their own and writes a final NRT flag per variable is the separate nrtqclib package, which builds on aiqclib (it was aiqclib.run_nrt_qc up to 0.14.0).
All three stages run at the observation level (one row per observation) by default, or at the profile level (one labeled row per profile/cast, with binary or bad-fraction proportion labels) via the profile step classes and configuration templates (extension="profile"); see the Profile-Level Pipeline guide in the documentation.
Each module is controlled by a YAML configuration file, allowing you to define and reproduce your entire workflow with ease.
Usage
The general workflow for any task in aiqclib follows these steps:
- Generate a Configuration Template: Create a starter YAML file for the task (e.g.,
prepare,train,classify). - Customize the Configuration: Edit the YAML file to specify paths, dataset names, and other parameters.
- Run the Task: Load the configuration and execute the main function for the task.
1. Dataset Preparation
This workflow processes your input data and creates training, validation, and test sets.
Step 1: Generate a configuration template.
import aiqclib as aq
aq.write_config_template(file_name="/path/to/prepare_config.yaml", stage="prepare")
Step 2: Customize prepare_config.yaml.
You must edit the file to set the correct input/output paths and define your dataset. See the Configuration section for details.
Step 3: Run the preparation process.
import aiqclib as aq
config = aq.read_config("/path/to/prepare_config.yaml")
aq.create_training_dataset(config)
This generates the following output folders:
- summary: Statistics of input data used for normalization.
- select: Profiles with bad observation flags (positive samples) and good profiles (negative samples).
- locate: Observation records for both positive and negative profiles.
- extract: Features extracted from the observation records.
- training: The final training, validation, and test datasets.
2. Model Training and Evaluation
This workflow uses the prepared dataset to train a model and evaluate its performance.
Step 1: Generate a training configuration template.
import aiqclib as aq
aq.write_config_template(file_name="/path/to/training_config.yaml", stage="train")
Step 2: Customize training_config.yaml.
Edit the file to point to your prepared dataset and define training parameters.
Step 3: Train and evaluate the model.
import aiqclib as aq
config = aq.read_config("/path/to/training_config.yaml")
aq.train_and_evaluate(config)
This generates the following output folders:
- validate: Results from the cross-validation process.
- build: The final trained models and their evaluation results on the test dataset.
3. Data Classification
This workflow applies a trained model to classify all observations in a dataset.
Step 1: Generate a classification configuration template.
import aiqclib as aq
aq.write_config_template(file_name="/path/to/classification_config.yaml", stage="classify")
Step 2: Customize classification_config.yaml.
Edit the file to point to the input data and the trained model.
Step 3: Run classification.
import aiqclib as aq
config = aq.read_config("/path/to/classification_config.yaml")
aq.classify_dataset(config)
This workflow processes a dataset using a trained model and generates:
- classify: The final classification results and a summary report.
Running several datasets at once
run_batch runs one phase ("prepare", "train", "classify") or "all" of
them over a table of dataset names, returning a summary of every run:
import aiqclib as aq
summary = aq.run_batch(
"datasets.txt", # table of set names per dataset
mode="all",
prepare_config="prepare_config.yaml",
training_config="training_config.yaml",
classification_config="classification_config.yaml",
verbose=True,
)
See the batch guide in the documentation for the table format and options.
Configuration
Configuration is managed via YAML files. The write_config_template function provides a starting point that you must customize for each module.
To inspect a stage's defaults without writing a file, read_config_template takes the same arguments and returns the configuration object instead:
print(aq.read_config_template(stage="prepare"))
1. Dataset Preparation (stage="prepare")
The preparation config requires you to modify two key sections:
-
path_info_sets: Defines the location of input and output data.path_info_sets: - name: data_set_1 common: base_path: /path/to/data # EDIT: Root output directory input: base_path: /path/to/input # EDIT: Directory with input files step_folder_name: "" split: step_folder_name: training
-
data_sets: Defines a specific dataset to be processed.data_sets: - name: dataset_0001 # EDIT: Your data set name dataset_folder_name: dataset_0001 # EDIT: Your output folder input_file_name: nrt_cora_bo_4.parquet # EDIT: Your input filename
2. Training and Evaluation (stage="train")
The training config links the prepared data to the model training process.
-
path_info_sets: Defines where to find the prepared dataset and where to save model artifacts.path_info_sets: - name: data_set_1 common: base_path: /path/to/data # EDIT: Root output directory input: step_folder_name: training
-
training_sets: Links to a dataset prepared in the previous workflow.training_sets: - name: training_0001 # EDIT: Your training name dataset_folder_name: dataset_0001 # EDIT: Your output folder
3. Classification (stage="classify")
The classification config uses a trained model to classify new data.
-
path_info_sets: Defines paths for raw data, models, and classification results.path_info_sets: - name: data_set_1 common: base_path: /path/to/data # EDIT: Root output directory input: base_path: /path/to/input # EDIT: Directory with input files step_folder_name: "" model: base_path: /path/to/model # EDIT: Directory with model files step_folder_name: model concat: step_folder_name: classification # EDIT: Directory with classification results
-
classification_sets: Defines a specific dataset to be classified.classification_sets: - name: classification_0001 # EDIT: Your classification name dataset_folder_name: dataset_0001 # EDIT: Your output folder input_file_name: nrt_cora_bo_4.parquet # EDIT: Your input filename
Contributing & Development
We welcome contributions! Development setup (uv environment, test data), running tests, and code style are documented in CONTRIBUTING.md.
Releasing & Deployment (for Maintainers)
The release process (versioning checklist), building the docs, and deployment to PyPI, conda-forge, and Anaconda.org are documented in RELEASING.md.
Metadata
Release files for aiqclib 0.15.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| aiqclib-0.15.1.tar.gz | 552.1 kB | With provenance | Details for aiqclib-0.15.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| aiqclib-0.15.1-py3-none-any.whl | Python 3 | none | any | Details for aiqclib-0.15.1-py3-none-any.whl |
Total release size: 837.6 kB
Release files / aiqclib-0.15.1.tar.gz
| Download URL | aiqclib-0.15.1.tar.gz |
|---|---|
| Size | 552.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
26eb5dc76e31061fe0b0f7b2394b52765bb20d5e5caa752cde4e7f4b328b61eb
|
|
BLAKE2b-256 checksum How to use checksums |
69356c87ac6945dccd6039b36ce726de759359c1996b36fbbaed3ab8421de1f8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 5, 2026.
Transparency logRelease files / aiqclib-0.15.1-py3-none-any.whl
| Download URL | aiqclib-0.15.1-py3-none-any.whl |
|---|---|
| Size | 285.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
ce9fd6a8bcb9d40302049dc0e05f399581d487f98e261a504e520d2b71296e11
|
|
BLAKE2b-256 checksum How to use checksums |
c912eae02d68d4b2f426733dcb6e4606bbc72bc6fa55310a807e5d0adb11cd26
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 5, 2026.
Transparency logSecurity for aiqclib 0.15.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for aiqclib-0.15.1.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 5, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.