simet 1.0.0

Synthetic image metrics - An evaluation framework for your synthetic images

Simet

Simet is a lightweight, extensible framework for evaluating synthetic image datasets against real datasets. It gives you:

• A unified pipeline to load images (from local folders or built-in datasets), apply transforms, extract features (Inception-v3 out of the box), and run metrics such as FID, Precision/Recall (FAISS), and ROC-AUC.
• Restraints (threshold checks) to enforce quality bars (e.g., FID ≤ 40, ROC-AUC ≥ 0.85) and return a single pass/fail.
• A CLI for simple or fully-configurable runs, and a clean Python API for scripting.
• Practical scalability features: feature caching, subsampling, FAISS IVF indexes, optional AMP, and deterministic seeding.

---

Installation

pip install simet

Python 3.12 is needed. CUDA is optional (for faster feature extraction and FAISS GPU support).

---

Quick start

1) Fast check with the CLI (simple two-paths mode)

simet simple /path/to/real_images /path/to/synth_images

Runs a minimal pipeline (no restraints) using default transforms and Inception-v3 feature extraction, printing metric (FID, Precision/Recall, ROC AUC) results.

2) Full pipeline from YAML

simet pipeline pipeline.yaml

pipeline.yaml example:

pipeline:
  loader:
    real_provider:
      type: LocalProviderWithoutClass
      path: data/real
    synth_provider:
      type: LocalProviderWithoutClass
      path: data/synth
    provider_transform:
      type: InceptionTransform
    feature_extractor:
      type: InceptionFeatureExtractor

  restraints:
    - type: FIDRestraint
      upper_bound: 40.0
    - type: PrecisionRecallRestraint
      lower_bound: [0.70, 0.60]
    - type: RocAucRestraint
      lower_bound: 0.85

CLI returns exit code 0 if all restraints pass.

---

Python API

from pathlib import Path
from simet.dataset_loaders import DatasetLoader
from simet.feature_extractor import InceptionFeatureExtractor
from simet.pipeline import Pipeline
from simet.providers import LocalProviderWithoutClass
from simet.restraints import FIDRestraint, PrecisionRecallRestraint, RocAucRestraint
from simet.services import LoggingService, SeedingService
from simet.transforms import InceptionTransform

LoggingService.setup_logging()
SeedingService.set_global_seed(42)

loader = DatasetLoader(
    real_provider=LocalProviderWithoutClass(Path("data/real")),
    synth_provider=LocalProviderWithoutClass(Path("data/synth")),
    provider_transform=InceptionTransform(),
    feature_extractor=InceptionFeatureExtractor(),
)

pipeline = Pipeline(
    loader=loader,
    restraints=[
        FIDRestraint(upper_bound=40.0),
        PrecisionRecallRestraint(lower_bound=[0.70, 0.60]),
        RocAucRestraint(lower_bound=0.85),
    ],
)

ok = pipeline.run()
print("PASS" if ok else "FAIL")

---

CLI reference

simet simple REAL_PATH SYNTH_PATH [--log-path LOG_DIR]

Run a simple pipeline with default settings.

simet simple-pipeline CONFIG_PATH [--log-path LOG_DIR]

Run YAML mode without restraints.

Example simple.yaml:

pipeline:
  real_path: data/real
  synth_path: data/synth
  metrics: [FID, PrecisionRecall, RocAuc]

Specifying no metrics will run all available (FID, Precision/Recall, ROC AUC).

simet pipeline CONFIG_PATH [--log-path LOG_DIR]

Run the full configurable pipeline (providers, transforms, feature extractor, restraints).

---

Concepts & Extensibility

• Providers → simet.providers: local or built-in datasets.
• Transforms → simet.transforms: image preprocessing (e.g. InceptionTransform).
• Feature Extractors → simet.feature_extractor: cached Inception-v3 features.
• Metrics → simet.metrics: FID, Precision/Recall (FAISS), ROC-AUC.
• Restraints → simet.restraints: wrap metrics with thresholds.
• Services → logging, seeding, caching, subsampling.

Add custom components by subclassing:

Provider → providers.base.Provider
Transform → transforms.base.Transform
Feature Extractor → feature_extractor.FeatureExtractor
Metric → metrics.base.Metric
Restraint → restraints.base.Restraint[T]

and register them via the corresponding *Parser.

---

Scalability notes

• FAISS IVF & batched search for large sets
• GPU / AMP support
• Feature caching per dataset hash
• Subsampling to balance dataset sizes
• Deterministic seeding via SeedingService

---

Examples

See main for a complete demo.

---

License

MIT (or your chosen license)