euler-preprocess 3.6.0

Physics-based preprocessing (fog, etc.) for RGB+depth datasets

euler-preprocess

Physics-based preprocessing transforms for multi-modal RGB+depth datasets. Built on top of euler-loading and ds-crawler.

Available transforms:

Command	Description
euler-preprocess fog	Synthetic fog via the Koschmieder atmospheric scattering model
euler-preprocess sky-depth	Override depth values in sky regions with a constant
euler-preprocess radial	Convert planar (z-buffer) depth to radial (Euclidean) depth

Installation

uv pip install "euler-preprocess[gpu,progress] @ git+https://github.com/d-rothen/euler-fog"

Usage

euler-preprocess fog       -c configs/example_dataset_config.json
euler-preprocess sky-depth -c configs/sky_depth_dataset_config.json
euler-preprocess radial    -c configs/radial_dataset_config.json

Configuration

Every subcommand takes a dataset config JSON that points to the input data and a transform config. Each modality path must be a directory indexed by ds-crawler with an euler_loading property that specifies the loader and function. This allows euler-loading to auto-select the correct dataset-specific loader.

Dataset Config

{
  "transform_config_path": "configs/run1.json",
  "output_path": "/path/to/output",
  "output_slot": "rgb",
  "sample": 42,
  "modalities": {
    "rgb": {"path": "/path/to/rgb", "split": "train"},
    "depth": "/path/to/depth",
    "semantic_segmentation": "/path/to/classSegmentation"
  },
  "hierarchical_modalities": {
    "intrinsics": {"path": "/path/to/intrinsics"}
  },
  "pipeline": {
    "output_root": "/pipeline/output",
    "outputs_manifest_path": "/pipeline/output/.euler_pipeline/pipeline_outputs.json",
    "output_targets": [
      {
        "slot": "rgb",
        "datasetType": "rgb",
        "relativePath": "foggy_rgb",
        "path": "/pipeline/output/foggy_rgb",
        "storage": "directory"
      }
    ]
  }
}

Field	Description
transform_config_path	Path to the transform-specific config (see below). fog_config_path is also accepted for backward compatibility.
output_path	Output root used when no pipeline target overrides it. Optional if pipeline.output_root or pipeline.output_targets[].path supplies the destination.
output_slot	Optional slot selector when pipeline.output_targets contains multiple entries. Defaults to rgb for fog, depth for sky-depth, and depth for radial.
sample	Optional 0-based euler-loading dataset index. When set, only dataset[sample] is transformed, which is useful for small augmented benchmark slices from large datasets.
samples	Optional multi-sample selector. Use a list of 0-based indices ([0, 10, 20]) or a slice object such as {"start": 0, "stop": 1000, "step": 2, "count": 100}. stop is exclusive; count caps the selected indices after slicing. Do not set both sample and samples.
modalities	Regular modalities that participate in sample-ID intersection. Each value is either a plain path string or an object with a path key and an optional split key (see below). Which modalities are required depends on the transform (see table below).
hierarchical_modalities	Per-scene data (e.g. intrinsics). Same format as modalities. Loaded once per scene and cached.
pipeline	Optional runtime routing block compatible with euler-inference (output_root, outputs_manifest_path, output_targets).

Inline splits

When a modality directory contains ds-crawler split files (.ds_crawler/split_<name>.json), you can select a subset of the data by setting the split key on that modality. Sample IDs are matched by intersection across all modalities, so specifying a split on a single modality is sufficient to restrict the entire dataset.

For quick slices after euler-loading has matched modalities, set samples. For example, {"samples": {"step": 2}} processes every second matched sample, and {"samples": {"start": 10, "step": 5, "count": 20}} processes 20 samples starting at index 10 with stride 5.

Required modalities per transform:

Transform	modalities	hierarchical_modalities
fog	rgb, depth, semantic_segmentation	intrinsics when available; used for radial depth conversion and camera-profile optics
sky-depth	depth, semantic_segmentation	—
radial	depth	intrinsics

Pipeline Runtime Block

pipeline follows the same shape as euler-inference:

{
  "pipeline": {
    "output_root": "/pipeline/output",
    "outputs_manifest_path": "/pipeline/output/.euler_pipeline/pipeline_outputs.json",
    "output_targets": [
      {
        "slot": "depth",
        "datasetType": "depth",
        "relativePath": "radial_depth.zip",
        "path": "/pipeline/output/radial_depth.zip",
        "storage": "zip"
      }
    ]
  }
}

Notes:

• output_root is only a fallback when output_path is omitted.
• A matching output_targets[].slot overrides the write root for that run.
• output_targets[].modelModalityId is optional. When provided it is copied into the pipeline manifest; when omitted it is left out there as well.
• storage: "directory" writes a dataset directory and storage: "zip" writes a zip dataset.
• storage: "file" is parsed but rejected at runtime.
• When outputs_manifest_path is set and a pipeline target is matched, finalization writes .euler_pipeline/pipeline_outputs.json with the same manifest shape used by euler-inference.

---

Fog Transform

Fog Config

Controls the fog simulation.

{
  "airlight": "from_sky",
  "seed": 1337,
  "depth_scale": 1.0,
  "resize_depth": true,
  "contrast_threshold": 0.05,
  "render_input_space": "srgb",
  "mode": "sample",
  "device": "cpu",
  "gpu_batch_size": 4,
  "capture": { "preset": "camera" },
  "camera_profile": "dashcam",
  "augmentations": { ... },
  "selection": { ... },
  "models": { ... }
}

Field	Description
airlight	Required. Airlight estimation method: "from_sky" (mean sky colour), "dcp" (dark channel prior), or "dcp_heuristic" (robust DCP with sky-guided colouring when sky pixels exist).
seed	Random seed for reproducibility. null for non-deterministic.
depth_scale	Multiplier applied to depth values after loading.
resize_depth	Resize the depth map to match the RGB resolution (bilinear).
contrast_threshold	Threshold C_t used in the visibility-to-attenuation conversion (default 0.05).
render_input_space	Colour space of the RGB supplied to the fog renderer. Use "srgb" for display-encoded dataset images so fog and airlight are mixed in scene-linear RGB; use "linear" for legacy configs or already-linear radiance.
mode	Optional scenario mode. Omit it or use "sample" for current one-scenario-per-image behavior; use "progressive" to render every scenario step for every image.
device	"cpu", "cuda", "mps", or "gpu" (alias for cuda).
gpu_batch_size	Batch size when running on GPU. Uniform-model samples are batched; heterogeneous samples are processed individually.
capture / capture_artifacts	Optional post-fog camera artifact pipeline. Omit it or set {"stages": []} for the legacy no-op path. Set true, {"preset": "camera"}, or a custom stages list to enable optics, raw sensor, ISP, and compression artifacts.
camera_profile	Optional named or inline camera profile whose stage defaults are merged into the capture stack before per-stage overrides. Built-ins are "default", "generic", "dashcam", and "low_light_fog".
camera_profiles	Optional map of project-specific named profiles. Use this for calibrated lens, sensor, ISP, and transport settings.
scenario_profiles	Optional top-level correlated condition sampler. Each sampled scenario can choose the fog model, override model parameters, select the airlight method, switch camera profile/settings, and force named capture-stage condition profiles.
augmentations	Optional stepped augmentation set. When present, every input sample produces every configured augmentation and uses the file-id hierarchy output layout described below.

Processing Pipeline

Fog generation is split into two phases:

1. Ideal scene rendering: physics-based fog and auxiliary scattering_coefficient / atmospheric_light maps are computed.
2. Capture artifacts: camera-specific effects are applied to the rendered RGB only. Physical fog maps stay stable while the RGB output can receive exposure shifts, lens blur, vignetting, raw sensor noise, Bayer/demosaicing artifacts, ISP tone/gamma/sharpening, and JPEG/resize/quantization effects.

This keeps physical fog maps stable while making the RGB output extensible for real-camera simulation.

Capture Artifact Stack

Enable the recommended camera stack with:

"capture": { "preset": "camera" }

or equivalently:

"capture": true

For tighter control, provide explicit stages in camera order:

"camera_profiles": {
  "real_drive_front": {
    "optics": {
      "lens_distortion": -0.012,
      "vignetting_strength": 0.18,
      "windshield_haze": {"enabled": true, "probability": 0.55}
    },
    "sensor": {
      "bayer_pattern": "RGGB",
      "iso": {"dist": "choice", "values": [200, 400, 800]},
      "base_iso": 100,
      "auto_exposure": {
        "enabled": true,
        "metering": "center_weighted",
        "target_luminance": {"dist": "uniform", "min": 0.16, "max": 0.24},
        "highlight_protection": 0.7,
        "resolve_iso": true,
        "max_iso": 1600
      },
      "full_well_electrons": [14000, 12000, 13000],
      "read_noise_electrons": {"dist": "uniform", "min": 2.0, "max": 6.0},
      "shadow_recovery_noise": {
        "enabled": true,
        "luminance_threshold": {"dist": "uniform", "min": 0.16, "max": 0.24},
        "luma_sigma": {"dist": "uniform", "min": 0.0, "max": 0.001},
        "chroma_sigma": {"dist": "uniform", "min": 0.012, "max": 0.028},
        "chroma_mode": "balanced",
        "red_chroma_gain": {"dist": "uniform", "min": 0.7, "max": 1.0},
        "blue_chroma_gain": {"dist": "uniform", "min": 1.7, "max": 2.7},
        "chroma_axis_correlation": {"dist": "uniform", "min": 0.05, "max": 0.25},
        "chroma_spatial_sigma": 0.0,
        "chroma_fine_fraction": 1.0,
        "chroma_luminance_preservation": 1.0,
        "black_noise_floor": 0.25,
        "black_suppression_luminance": 0.03,
        "black_suppression_softness": 0.08
      },
      "black_level": [0.003, 0.0035, 0.003],
      "white_level": [1.0, 0.995, 1.0],
      "adc_bit_depth": 12,
      "post_demosaic_bit_depth": 12
    },
    "isp": {
      "tone_map": "reinhard",
      "gamma": "srgb",
      "denoise_sigma": 0.2,
      "sharpen_amount": 0.2,
      "saturation": 0.9
    },
    "transport": {
      "jpeg": {"enabled": true, "quality": {"dist": "uniform", "min": 65, "max": 92}},
      "bit_depth": 8
    }
  }
},
"camera_profile": "real_drive_front",
"capture": {
  "stages": [
    {
      "type": "optics",
      "blur_sigma": {"dist": "uniform", "min": 0.2, "max": 0.8},
      "vignetting_strength": 0.15,
      "windshield_haze": {"enabled": true, "probability": 0.4},
      "droplets": {"enabled": false}
    },
    {
      "type": "sensor",
      "input_space": "srgb",
      "exposure_gain": {"dist": "uniform", "min": 0.85, "max": 1.2},
      "row_noise_sigma": 0.003
    },
    {
      "type": "isp",
      "tone_map": "reinhard",
      "gamma": "srgb",
      "denoise_sigma": 0.2,
      "sharpen_amount": 0.2,
      "saturation": 0.9
    },
    {
      "type": "transport",
      "jpeg": {"enabled": true, "quality": {"dist": "uniform", "min": 65, "max": 92}},
      "bit_depth": 8
    }
  ]
}

Supported stage types:

Stage	Main effects
optics	Defocus/MTF blur, motion blur, bloom, veiling glare, vignetting, chromatic aberration, lens distortion, windshield haze, optional droplets.
sensor	Image-driven or sampled exposure, white balance, camera matrix, Bayer mosaic, shot/read noise, fixed-pattern noise, row/column banding, shadow-local recovery noise, hot/dead pixels, bilinear demosaic.
isp	Denoising, color correction, tone mapping, sRGB/gamma, local contrast, sharpening halos, saturation shifts.
transport	Crop/resize, bit-depth quantization, JPEG round-trip compression.
exposure	Lightweight standalone exposure and white-balance stage for simple custom chains.

Set sensor.auto_exposure.enabled to meter the rendered image before raw sensor sampling. target_luminance, metering, highlight_*, and gain bounds choose the exposure; resolve_iso can raise ISO from the metering pressure, dark pixel fraction, and fog opacity. When auto exposure is enabled, exposure_gain still applies as scenario-specific exposure compensation.

Fog-aware metering modes use CaptureContext.depth_m, k_map, fog opacity, and attributes.sky_mask when available. Use "metering": "fog_aware_center_weighted" or "sky_aware_center_weighted" and tune sky_suppression, fog_meter_suppression, depth_meter_decay_m, and min_meter_weight to keep bright sky or dense far-field airlight from dominating the exposure meter. Legacy metering modes are unchanged unless these suppression keys are present.

Set sensor.sensor_identity.enabled for persistent sensor structure across frames. The identity cache is deterministic for the same sensor_id, seed, image shape, and Bayer pattern. prnu_sigma adds multiplicative pixel-response non-uniformity before shot noise; dsnu_sigma, persistent_row_sigma, and persistent_column_sigma add fixed raw-domain offsets; persistent hot/dead pixel probabilities create stable bad-pixel masks that combine with the existing per-image bad-pixel probabilities.

Set sensor.shadow_recovery_noise.enabled to add extra post-demosaic luma and chroma corruption only where the pre-exposure rendered luminance was low. This is useful for reducing broad global grain while keeping lifted shadows visibly noisy. For dark high-ISO scenes, keep luma_sigma much lower than chroma_sigma, use chroma_mode: "balanced", and leave chroma_luminance_preservation near 1.0; this makes the local corruption read as color noise instead of black luma speckles. red_chroma_gain, blue_chroma_gain, and chroma_axis_correlation can match camera-specific color noise, including blue/purple-biased high-ISO noise. Keep chroma_spatial_sigma near 0 and avoid chroma subsampling when the target noise is fine-grained rather than blocky. black_noise_floor with black_suppression_luminance/black_suppression_softness reduces the extra noise in near-clipped black regions, so the strongest visible noise sits in dim-but-readable shadows.

Set sensor.noise_adjustment for relative, scenario-level noise controls on top of the selected camera/condition profile. level: 1.0 leaves the authored profile unchanged; lower values suppress read/static/chroma noise and higher values amplify it. static_chroma_bias ranges from -1.0 for more fixed pattern, row/column, banding, and bad-pixel noise to 1.0 for more chromatic/high-ISO-looking shadow noise:

{
  "capture_overrides": {
    "sensor": {
      "condition_profile": "nominal_gloom",
      "noise_adjustment": {
        "enabled": true,
        "level": 1.25,
        "static_chroma_bias": 0.35
      }
    }
  }
}

Any stage can define condition_profiles to sample coherent per-image settings before the stage runs. This is useful for exposure states where ISO, exposure gain, read noise, banding, and dark/fog noise modulation should move together:

{
  "type": "sensor",
  "condition_profiles": [
    {"name": "clean_daylight", "weight": 0.25, "exposure_gain": 1.0, "iso": 100},
    {"name": "underexposed_noisy", "weight": 0.25, "exposure_gain": 0.65, "iso": 1600}
  ]
}

isp.tone_map supports "reinhard", "aces", "clip", and "lut". The LUT mode uses a cheap interpolated 1D camera-response curve:

{
  "type": "isp",
  "tone_map": "lut",
  "tone_map_strength": 1.0,
  "tone_map_lut": [0.0, 0.006, 0.014, 0.028, 0.052, 0.090, 0.145, 0.220, 0.320, 0.450, 0.610, 0.780, 0.900, 0.965, 0.995, 1.0],
  "tone_map_lut_domain": "linear",
  "gamma": "srgb"
}

Top-level scenario_profiles sample one latent scene/camera condition before rendering. The selected scenario is merged over the root config, so it can drive fog density, atmospheric light, camera profile, capture-stage overrides, ISP, and compression together:

"scenario_profiles": [
  {
    "name": "clean_low_noise_haze",
    "weight": 0.22,
    "model": "heterogeneous_k_ls",
    "airlight_method": "dcp_heuristic",
    "models": {
      "heterogeneous_k_ls": {
        "visibility_m": {"dist": "uniform", "min": 60.0, "max": 130.0}
      }
    },
    "capture_overrides": {
      "sensor": {"condition_profile": "clean_daylight"},
      "isp": {"denoise_sigma": {"dist": "uniform", "min": 0.08, "max": 0.32}}
    }
  },
  {
    "name": "underexposed_dense_gloom",
    "weight": 0.25,
    "model": "heterogeneous_k_ls",
    "airlight_method": "dcp_heuristic",
    "models": {
      "heterogeneous_k_ls": {
        "visibility_m": {"dist": "uniform", "min": 18.0, "max": 55.0},
        "scene_illumination": {
          "enabled": true,
          "global_ev": {"dist": "uniform", "min": 0.25, "max": 0.85},
          "near_ev": {"dist": "uniform", "min": 0.35, "max": 1.20},
          "near_decay_depth_m": {"dist": "uniform", "min": 10.0, "max": 22.0},
          "fog_coupled_ev": {"dist": "uniform", "min": 0.10, "max": 0.45},
          "sky_weight": 0.0
        }
      }
    },
    "capture_overrides": {
      "sensor": {
        "condition_profile": "underexposed_noisy",
        "auto_exposure": {
          "enabled": true,
          "metering": "fog_aware_center_weighted",
          "target_luminance": {"dist": "uniform", "min": 0.13, "max": 0.20},
          "highlight_protection": 0.78,
          "manual_gain_weight": 0.0,
          "sky_suppression": 0.85,
          "fog_meter_suppression": 0.65
        }
      },
      "transport": {"jpeg": {"quality": {"dist": "uniform", "min": 54, "max": 78}}}
    }
  }
]

capture_overrides is merged after camera-profile and stage settings. Use condition_profile to force one named profile from a stage's condition_profiles; if omitted, the stage continues sampling its own profile weights locally.

Set top-level "mode": "progressive" to emit every configured scenario for every input image instead of sampling one scenario. Each scenario accepts "steps" and "progressive_weight" (or "max_weight" / "weight" as aliases); the transform writes steps from weight 0 through the scenario's configured weight, and weight 1 matches the original scenario. Fog density is progressed in scattering-coefficient space, while numeric camera/config values blend from the base config toward the scenario config. Progressive blends clamp probability-like values and non-negative physical factors back into valid mathematical domains so extrapolated weights above 1 do not create invalid render parameters. Source-backed outputs are written as fog_progression variants under each source file id.

Fog Model

The core equation is the Koschmieder model (atmospheric scattering):

I_fog(x) = I(x) * t(x)  +  L_s * (1 - t(x))

where:

• I(x) is the original RGB colour at pixel x
• t(x) = exp(-k * d(x)) is the transmittance, which falls exponentially with depth d and attenuation coefficient k
• L_s is the atmospheric light (airlight), i.e. the colour of the fog/sky light scattered towards the camera
• k is derived from a meteorological visibility distance V: k = -ln(C_t) / V

Distant objects are attenuated more (t approaches 0) and replaced by airlight, just as in real fog.

For gloomy conditions, add scene_illumination inside a fog model config. This darkens pre-fog scene radiance I(x) before the atmospheric scattering equation, so near objects can become plausibly overcast or storm-lit instead of passing through unchanged. global_ev applies to the whole non-sky scene, near_ev adds extra near-field darkening with near_decay_depth_m, fog_coupled_ev adds a term proportional to local fog opacity, and sky_weight: 0.0 preserves sky pixels when a sky mask is available.

How Each Modality is Used

RGB — The clean scene image. Normalised to float32 in [0, 1]. This is the I(x) term in the fog equation -- it gets blended with the airlight according to transmittance.

Depth — A per-pixel depth map in metres. Provides the d(x) term in the transmittance calculation t(x) = exp(-k * d(x)). Pixels with greater depth receive more fog. Invalid values (NaN, inf, negative) are clamped to zero (treated as infinitely close, receiving no fog).

Semantic Segmentation — A per-pixel semantic segmentation map from which a boolean sky mask is derived, loaded via euler-loading's dataset-specific semantic_segmentation loader. The sky mask is used for airlight estimation when the airlight method is "from_sky": the mean RGB of all sky pixels in the clean image is used as the airlight colour L_s.

Intrinsics (optional) — When present, planar (z-buffer) depth is converted to radial (Euclidean) depth before fog is applied.

Airlight Estimation

The airlight config key selects how the atmospheric light L_s is estimated:

Method	Description
from_sky	Mean RGB of sky pixels in the clean image. Falls back to white [1, 1, 1] when no sky pixels exist.
dcp	Dark Channel Prior — selects the brightest pixel (by channel sum) among the top 0.1% darkest-channel pixels.
dcp_heuristic	Robust DCP heuristic — pools the brighter half of the top 0.1% darkest-channel pixels, and when sky pixels exist it uses the brightest sky colours as the chromaticity prior while preserving DCP-derived luminance. Optional bias controls can nudge the result toward white or a cool fog tint.

GPU-native implementations (DCPAirlightTorch, DCPHeuristicAirlightTorch) are used automatically when running on GPU.

When airlight is "dcp_heuristic", you can optionally add:

"dcp_heuristic": {
  "patch_size": 15,
  "top_percent": 0.001,
  "white_bias": 0.1,
  "cool_bias": 0.15,
  "cool_target": [0.93, 0.97, 1.0]
}

• white_bias mixes the final airlight toward neutral white.
• cool_bias mixes the final airlight toward a sky-relative cool target.
• cool_target is the cool-white anchor used to derive that sky-relative target. When sky pixels exist, the effective cool target is a blend of the estimated sky colour and cool_target; without sky pixels, it falls back to the airlight estimate and cool_target.
• white_bias + cool_bias must be <= 1.
• The tint bias preserves the estimated airlight luminance, so it shifts colour without silently changing fog density.

Airlight Intensity Dampening

Estimated airlight is dampened by default as fog density increases. This keeps strong fog closer to the low, grey lighting seen in real in-car fog footage instead of letting DCP-style estimates wash dense fog toward white.

Each fog model can override the dampening curve:

"airlight_dampening": {
  "enabled": true,
  "apply_to": "estimated",
  "reference_visibility_m": 80.0,
  "min_factor": 0.45,
  "max_factor": 1.0,
  "strength": 1.0
}

The factor is: min_factor + (max_factor - min_factor) / (1 + strength * beta / reference_beta). min_factor and max_factor must be finite and non-negative. Values above 1.0 are allowed when you intentionally want to brighten estimated airlight; the final RGB output is still clamped to the valid image range. strength must remain finite and non-negative. reference_beta is either reference_scattering_coefficient / reference_beta, or it is derived from reference_visibility_m using the model's contrast threshold. The default applies only when atmospheric_light uses an estimated airlight method ("from_sky", "dcp", or "dcp_heuristic"); literal RGB atmospheric_light values stay exact unless apply_to is set to "all". Set "enabled": false or apply_to: "none" to preserve the previous undampened behavior.

For heterogeneous_ls and heterogeneous_k_ls, the Perlin atmospheric-light field is sampled around the dampened base airlight, so the spatial variation does not reintroduce the old over-illuminated look.

Model Selection

Each image is assigned a fog model via the selection block:

"selection": {
  "mode": "weighted",
  "weights": {
    "uniform": 0.25,
    "heterogeneous_k": 0.35,
    "heterogeneous_ls": 0.25,
    "heterogeneous_k_ls": 0.15
  }
}

• fixed mode: always use a single named model.
• weighted mode: randomly select a model per image according to normalised weights.

Four models are available:

Model	Description
uniform	Constant k and L_s. Standard homogeneous fog.
heterogeneous_k	Spatially-varying k, constant L_s. Simulates patchy fog / fog banks.
heterogeneous_ls	Constant k, spatially-varying L_s. Simulates scattered-light colour variation.
heterogeneous_k_ls	Both k and L_s vary spatially. Most expressive model.

Visibility Distribution

Each model specifies a visibility_m distribution from which a visibility distance (in metres) is sampled per image:

dist	Parameters	Description
constant	value	Fixed value.
uniform	min, max	Uniform random in range.
normal	mean, std, optional min/max	Gaussian, optionally clamped.
lognormal	mean, sigma, optional min/max	Log-normal.
choice	values, optional weights	Discrete weighted choice.

The sampled visibility V is converted to the attenuation coefficient: k = -ln(C_t) / V. This happens once per output image. For heterogeneous_k and heterogeneous_k_ls, that sampled value is the base coefficient that is then spatially modulated by the noise field.

Stepped Augmentations

For benchmark generation, set augmentations in the fog config. This switches the fog transform from one sampled output per input to one output per configured variant:

{
  "airlight": "from_sky",
  "seed": 1337,
  "contrast_threshold": 0.05,
  "augmentations": {
    "file_id_hierarchy_name": "file_id",
    "attribute_key": "fog_augmentation",
    "models": ["uniform"],
    "visibility_m": [10, 20, 40, 70, 100],
    "airlight_methods": ["from_sky"]
  }
}

The matrix form above expands as the Cartesian product of models, visibility_m (MOR in metres), optional scattering_coefficients / beta, and airlight choices. file_id_hierarchy_name names the inserted hierarchy level when the underlying ds-crawler writer has a hierarchy separator; the directory name is the source file id in either case. For tighter control, use explicit variants:

"augmentations": {
  "variants": [
    {
      "id": "mor_010m_sky",
      "model": "uniform",
      "visibility_m": 10,
      "airlight_method": "from_sky"
    },
    {
      "id": "beta_0.15_white",
      "model": "heterogeneous_k",
      "scattering_coefficient": 0.15,
      "atmospheric_light": [1.0, 1.0, 1.0],
      "k_hetero": {
        "scales": "smooth_auto",
        "correlation_length_fraction": 0.25,
        "octaves": 3,
        "min_factor": 0.65,
        "max_factor": 1.45,
        "contrast": 0.65,
        "normalize_to_mean": true
      }
    }
  ]
}

Each output entry receives per-file ds-crawler attributes under fog_augmentation, including the augmentation id, source id, source full id, model, actual scattering coefficient, actual atmospheric light, and configured MOR/beta descriptors when available. euler-loading exposes these as sample["attributes"]["rgb"]["fog_augmentation"].

Heterogeneous Noise Fields

Both k_hetero and ls_hetero use Perlin FBM (fractional Brownian motion) to generate spatially-varying factor fields. For realistic fog, prefer the smooth mode: it keeps Perlin wavelengths tied to the image size, then optionally reduces noise contrast and applies a final blur before mapping the noise to physical factors.

"k_hetero": {
  "scales": "smooth_auto",
  "correlation_length_fraction": 0.25,
  "octaves": 3,
  "max_scale": null,
  "min_factor": 0.65,
  "max_factor": 1.45,
  "contrast": 0.65,
  "smooth_sigma_fraction": 0.0,
  "normalize_to_mean": true
}

The noise field (values in [0, 1]) is mapped to a factor field: factor(x) = min_factor + (max_factor - min_factor) * noise(x). contrast < 1 compresses the noise around 0.5 before this mapping, avoiding extreme local fog density. When normalize_to_mean is true, the factor field is rescaled so its spatial mean equals 1.0, preserving the overall fog density while introducing spatial variation. In other words, with heterogeneous k: k(x) = k_sampled * factor(x). If visibility_m / MOR was sampled from a distribution, k_sampled is the coefficient derived from that one sampled MOR. With normalize_to_mean: true, the arithmetic mean of the per-pixel k map equals k_sampled; the median is not forced to match. With normalize_to_mean: false, the map mean shifts by the mean of the factor field.

For heterogeneous_ls / heterogeneous_k_ls, ls_hetero can also include a weak view-direction illumination prior. This modulates the atmospheric-light field itself, so the rendered effect is still gated by fog transmittance:

"ls_hetero": {
  "ls_gradient": {
    "enabled": true,
    "probability": 0.65,
    "axis": "vertical",
    "top_factor": {"dist": "uniform", "min": 1.03, "max": 1.14},
    "bottom_factor": {"dist": "uniform", "min": 0.88, "max": 0.99},
    "gamma": {"dist": "uniform", "min": 0.85, "max": 1.6},
    "normalize_to_mean": true,
    "fog_opacity_weight": 0.65
  }
}

Parameter	Effect
min_factor / max_factor	Range of the multiplicative factor.
normalize_to_mean	Rescale factors so the image-wide mean equals the base value. Recommended for k_hetero.
scales: "smooth_auto"	Build low-frequency Perlin scales from the image size.
correlation_length_fraction	Approximate smallest fog feature size as a fraction of the shorter image side. Larger values create smoother gradients.
octaves / lacunarity / max_scale	Control how many increasingly broad Perlin components are mixed.
contrast	Compress or expand the Perlin range before mapping to factors. Values below 1 are recommended.
smooth_sigma / smooth_sigma_fraction	Optional final Gaussian blur in pixels or as a fraction of the shorter image side.
ls_gradient	Optional L_s top-to-bottom or left-to-right factor field. Keep it weak and probabilistic to avoid a deterministic image-position shortcut.

Fog Output

CLI runs write a source-backed RGB dataset. The output keeps the source RGB dataset's relative paths, basenames, extensions, and ds-crawler metadata so the result stays loadable by euler-loading:

<output_path>/
  .ds_crawler/dataset-head.json
  .ds_crawler/ds-crawler.json
  .ds_crawler/index.json
  Scene01/
    Camera_0/
      00000.png

When a pipeline target is present, pipeline.output_targets[].path replaces output_path entirely. Standalone/direct FogTransform(...) usage without the CLI still uses the legacy per-model layout with config.json sidecars.

With augmentations enabled, source-backed outputs are written one level below the source file id instead:

<output_path>/
  .ds_crawler/dataset-head.json
  .ds_crawler/ds-crawler.json
  .ds_crawler/index.json
  Scene01/
    Camera_0/
      00000/
        mor_10m_airlight_from_sky.png
        mor_20m_airlight_from_sky.png

Auxiliary scattering_coefficient and atmospheric_light pipeline targets use the same file-id hierarchy and write matching .npy augmentation files.

---

Sky-Depth Transform

Overrides depth values in sky regions with a configurable constant. Useful for datasets where sky depth is encoded as zero or infinity and needs to be normalised to a large finite value.

Sky-Depth Config

{
  "sky_depth_value": 1000.0
}

Field	Description
sky_depth_value	Depth value assigned to all sky pixels. Defaults to 1000.0.

Sky-Depth Output

CLI runs write a source-backed depth dataset mirroring the input depth modality's paths, filenames, extensions, and metadata. Standalone/direct SkyDepthTransform(...) usage keeps the legacy .npy output behavior.

---

Radial Transform

Converts planar (z-buffer) depth to radial (Euclidean) depth using camera intrinsics. For each pixel (u, v):

d_radial(u, v) = d_planar(u, v) * sqrt(((u - cx)/fx)^2 + ((v - cy)/fy)^2 + 1)

Radial Config

{}

No special parameters are required. The transform reads intrinsics from the intrinsics hierarchical modality.

Radial Output

CLI runs write a source-backed depth dataset mirroring the input depth modality's layout and writer metadata. The emitted index.json also flips meta.radial_depth to true. Standalone/direct RadialTransform(...) usage keeps the legacy .npy output behavior.