qqtools 1.2.11

A small tool package for qq

✨qqtools✨

A lightweight library, crafted and battle-tested daily by qq, to make PyTorch life a little easier.

I’ve gathered the repetitive parts of my day-to-day work and refined them into this slim utility library. It serves as my personal toolkit for handling data, training, and experiments, designed to keep projects moving fast with cleaner code and smoother workflows (and hopefully yours too!).

Built for me, shared for you.

Requirements

• torch>=2.0 for full functionality
  • Some components maintain backward compatibility with torch==1.x
  • Recommended: torch>=2.4
• pyyaml>=6.0
  • Recommended to use YAML format for all configuration files.

This provides a unified approach to drive and manage all workflow operations.

To get started quickly, install it via pip:

pip install qqtools

Install with full features:

pip install qqtools[full]

Data Format Support

Non-torch formats:

qDict : Enhanced of basic Dict.
qScalaDict : Dict[str, num]. A dict that maps str to scala;
qListData : List[dict]. A list of dicts.

Torch-related data formats

qData
qBatchList

Simple Training Loop

For jupyter users

import qqtools as qt
qt.import_common(globals())

x = np.random.rand(100, 5)
y = np.random.rand(100)

# dataset wrap
xs = [ x[i] for i in range(len(x))]
ys = [ y[i] for i in range(len(y))]
data_list = [ qt.qData({'x': x[i], 'y':y[i]})  for i in range(len(x))] 
dataset = qt.qDictDataset(data_list=data_list)
dataloader = qt.qDictDataloader()

# model
model = qt.nn.qMLP([5,5,1], activation="relu")
loss_fn = torch.nn.MSELoss()
optimizer = torch.optim.AdamW(model.parameters(), lr=1.0e-4, weight_decay=0.01)

# device
device = torch.device("cuda")
model.to(device)

# loop
for epoch in range(100):
    for batch in dataloader:
        batch.to(device)
        out = model(batch.x)
        loss = loss_fn(out, batch.y)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    print(f"{epoch} {loss.item():4.6f}")

Individual Modules

The following modules are consumers of the core functionality provided by this package. Each is designed to be independent, allowing for sole import.

under plugins/

• qchem
• qpipeline
• qhyperconnect
• qexp (Linux only)

qexp Quick Start

qexp is a shared-root experiment queue for Linux GPU hosts.

Install optional experiment dependencies:

pip install qqtools[exp]

Basic commands:

qexp init --shared-root /mnt/share/myproject/.qexp --machine gpu-a
qexp submit --shared-root /mnt/share/myproject/.qexp --machine gpu-a --name demo -- python train.py --epochs 10
qexp agent start --shared-root /mnt/share/myproject/.qexp --machine gpu-a --background
qexp list --shared-root /mnt/share/myproject/.qexp --machine gpu-a
qexp machines --shared-root /mnt/share/myproject/.qexp --machine gpu-a
qexp top --shared-root /mnt/share/myproject/.qexp --machine gpu-a
qexp logs <task_id> --follow
qexp cancel <task_id>
qexp clean --dry-run
qexp clean --older-than-seconds 604800

Python API:

from qqtools.plugins import qexp

task = qexp.submit(
    qexp.load_root_config("/mnt/share/myproject/.qexp", "gpu-a"),
    command=["python", "train.py", "--epochs", "10"],
    name="demo",
)
print(task.task_id)

Release validation notes:

• packaged Python API surface: from qqtools.plugins import qexp
• packaged CLI surface: qexp
• actual execution is supported on local Linux GPU hosts with tmux installed
• non-Linux development is limited to parsing, rendering, and test validation flows

Test

tox