PetTrace ML

Python tools for research on visual re-identification of individual dogs and cats. The distribution is pettrace-ml; the Python import is pettrace. Dataset preparation, model inference, metric learning and evaluation live in the package. Notebooks describe and run the experiments.

Installation

Python 3.11–3.13 is supported. Install a published version in your Python environment:

python -m pip install pettrace-ml==0.2.11

The default install includes plots. Colab already supplies IPython and widgets; the [notebooks] extra adds them for standalone Jupyter environments. Use [training] to also install PyTorch, ONNX export and pandas. For development, use pip install -e ..

Colab is the primary notebook target. The first cell of datasets/data_lookup.ipynb is:

%pip install -q "pettrace-ml==0.2.11"

Use a fresh Colab CPU runtime. CI tests this ordinary installation against the reported Linux x86_64 / Python 3.13.18 environment and rejects changes to its existing dependencies. The hash-checked requirements/data-lookup-colab.txt profile reconstructs that CI baseline; it is not installed over Colab. It records the supplied runtime constraints, not Google's entire VM image. The hosted frontend and authenticated Drive mount need validation in Colab. A plain pip install resolves compatible dependencies; it does not apply a full environment lock.

Model weights and datasets are separate artifacts and are not included in the package. Data lookup uses CPU. GPU screening requires the separately validated E3 environment; installing the training extra alone does not configure CUDA.

Public methods

lookup_duplicates(..., near_cache="/content/data-lookup/signatures", drive_root=...) adds conservative resized/recompressed-copy proposals before the distinct-photo minimum. It compares unique image hashes across labels, quarantines involved labels and records each dHash/RGB link for quarantine_groups / quarantine_gallery. Labels are never merged. Omit near_cache for the existing CSV-only audit. Drive restores signatures first and only recovers originals without signatures. New signatures are saved in immutable incremental chunks, shared across source volumes. This CPU pass still needs source bytes the first time; it does not run YOLO, infer animal identity or guarantee detection of every edited copy.

Load a refined dataset for an experiment (PetTrace 0.2.8+):

from pettrace import data

dataset, provenance = data.load(
    "v2.50",
    drive_root="/content/drive/MyDrive/PetTrace",
)
photos = dataset.rows

The loader verifies a local version first, then restores the requested snapshot from the mounted Drive preparation backup, or downloads its published alessandro54/pettrace-training release from Hugging Face. Mount/authorize Drive beforehand; HF_TOKEN is needed only for private Hub access. The default cache is datasets/build/pettrace-training; use cache_dir to choose another local location. The returned dataset has root, rows, summary and fingerprint; keep provenance with the experiment report.

The first Hub commit is saved locally and reused even if a remote tag moves. Missing files can be recovered; corrupt hashes or unsafe paths fail validation. Drive is read-only here; Hub downloads are cached locally. A lost Colab VM loses that local cache, while existing Drive backups remain available. The loader never downloads raw ZIPs, runs YOLO, cleans data or trains. An unavailable version must be prepared/published separately. Frozen experiment notebooks retain their recorded loaders and pins; the immutable benchmark API remains in data.versions.

These are the supported import paths for the research workflows. Helpers whose names start with _ are internal. Existing experiment imports continue to work.

Duplicate audit without model inference

Given a raw review.csv containing source paths, labels and recorded SHA-256 hashes:

import json
from pettrace.data.lookup import lookup_duplicates

report = lookup_duplicates("review.csv", "reports/data-lookup")
summary = json.loads((report / "summary.json").read_text())
print(summary["files"], summary["unique_images"], summary["duplicate_copies"])

This exports one representative per hash, every original label, a mapping of duplicate copies and identity conflicts for review. It does not delete photos or merge identities. Counts cover the supplied inventory, not unobserved photos. Recorded hashes are used without rereading images; exact-byte duplicates do not identify every visually similar photo.

candidates.csv separately selects one image per hash after manual drops, whole-identity quarantine and the distinct-photo minimum (default four). excluded.csv records every noncandidate and its reason. Pending source labels remain unverified; this is not yet a training dataset. load_candidates(report, review_csv) checks integrity and the matching source inventory before returning candidates and exclusions.

Pass candidate_index=report to PrepareConfig or TrainingPreparation to apply this index before YOLO. The latter requires an explicit new version, for example version="v3.100". The current review must match exactly; later decisions require a new index. Preparation still protects the frozen benchmark, checks image bytes, runs screening and near-duplicate curation and reapplies the photo minimum. Existing v2 versions are not changed. Raw image restoration remains separate; the index reduces screening work, not raw transfer volume.

In Colab, after mounting Drive, recover the confirmed raw inventory of the selected volume:

from pathlib import Path
from pettrace.data.lookup import restore_lookup_review

drive = Path("/content/drive/MyDrive/PetTrace")
review = restore_lookup_review(drive, "/content", percent=100)
report = lookup_duplicates(review, drive / "data-lookup")

This uses PetTrace preparation backups in checkpoints/preparation. It copies and verifies the TAR block containing the CSV and restores only that file. Missing backups fail explicitly; it does not silently change the percentage or run preparation. Reports persist in Drive. The optional quarantine_gallery needs the original photos at their recorded paths.

Experiments and reproducibility

Use the versioned experiment notebooks for the complete public-test and private-PetTrace protocols. Keep their galleries separate. Retrieval is closed-set and same-species, with photo-level leave-one-out evaluation. EER fitted and reported on the same pairs is descriptive, not an independently calibrated threshold.

All PetTrace experiment runs must publish results to MLflow. Low-level metric functions alone do not complete an experiment. Model registration and serving promotion are separate from publication. Keep saved evidence when publication fails so it can be retried. Data preparation is a separate workflow.

pip install resolves the declared dependency ranges; it does not apply uv.lock. For a locked environment use the matching Git tag and uv sync --locked, or the experiment's published runtime profile. Reproduction also requires the recorded data/model hashes, configuration, seed and compatible hardware. A version pin does not guarantee identical floating-point results across hardware.

The package code is MIT-licensed. Original datasets and model weights retain their own licenses. See release instructions and dataset preparation.