objective-metrics 1.2.0

Command line tool for Image and Video Quality Assessment including MDTVSFA, FAST-VQA, VMAF, MUSIQ and more...

How it works

General info:

When you install this package you will have acces to objective_metrics_run command line tool.

It's designed for running some of image/video quality metrics on a dataset.

Results are given in .csv files (format: filename, metric value) for each dataset and metric.

Name of resulting .csv file for each dataset and metric: {dataset_name}_{metric}.csv

About configs:

To use it, you have to make a config (.yaml file) file for each dataset that you want to calculate.

Config files should be placed in ./datasets_configs directory.

Script takes dataset names as positional arguments. Each name should exactly match one config file from ./datasets_configs directory.

About worklists:

Also, for each dataset you should have a {config_filename}_list.json file to compute NR metrics and {config_filename}_pairs.json for FR IQA metrics.

Both files also should be placed in ./datasets_configs directory and exactly match one config file (.yaml) and script positional argument (if you want to compute that dataset).

Note: name of dataset in script positional argument (and in the names of files with configs) may differ from dataset_name field in config.yaml (which only affects on the name of files with results (.csv) and logs).

{config_filename}_list.json is simply a list of images/videos to compute.

{config_filename}_pairs.json is a dictionary with reference-distorted pairs.

Example files:

You can see examples of configfs in dataset_configs folder.

Also you can see examples of building {name}_list.json and {name}_pairs.json files in get_configs.ipynb notebook.

More details about all config files are provided below.

Example structure of current work space:

dataset_configs/
├─ ...
├─ live-wcvqd_list.json  
├─ live-wcvqd_pairs.json  
├─ live-wcvqd.yaml  
├─ ...
├─ tid2013_list.json  
├─ tid2013_pairs.json  
├─ tid2013.yaml
├─ ...  
├─ live-vqc_list.json
├─ live-vqc.yaml  
├─ ...

$ objective_metrics_run live-wcvqd tid2013

This command will run pipeline on two datasets: live-wcvqd and tid2013.

Structure of config files:

Common for image and video datasets part

• type - img/vid
• dataset_pth - path to dataset folder
• dataset_name - dataset name for resulting .csv files ({dataset_name}_{metric}.csv)
• output - name of directory with resulting .csv files
• logs_output - name of file, where logs will be written (also will be created log file with process number in name)

Image Dataset

Example tid2013.yaml:

type: img
dataset_pth: {PTH}/TID2013
dataset_name: TID2013
output: image_metric_values
logs_output: objective_metrics_run.log
fr_metrics:
  - psnr
  - pieapp
nr_metrics:
  - koncept512
  - unique

• fr_metrics - list of FR IQA metrics to compute
• nr_metrics - list of NR IQA metrics to compute

{config_filename}_list.json (tid2013_list.json)

It is mandatory if you want to use NR metrics. File should be in .json format with the following structure:

['img_pth_1', ...,'img_pth_n']

{config_filename}_pairs.json (tid2013_pairs.json)

It is mandatory if you want to use FR metrics. File should be in .json format with the following structure:

{'ref_img_pth_1': ['dist_img_pth_1', ...,'dist_img_pth_n'], ..., 'ref_img_pth_n': [...]}

One record corresponds to one reference image and the list of its distortions.

Video Dataset

Note: IQA methods are computed frame-wise with further averaging.

Example live-wcvqd.yaml:

type: vid
dataset_pth: {pth}/LIVE_WCVQD
dataset_name: LIVE-WCVQD
output: video_metric_values
logs_output: objective_metrics_run.log
fr_iqa_metrics:
  - ahiq
  - lpips
nr_iqa_metrics:
  - niqe
  - koncept512
fr_vqa_metrics:
  - vmaf
nr_vqa_metrics:
  - mdtvsfa
  - dover

• fr_iqa_metrics - list of FR IQA metrics to compute

• nr_iqa_metrics - list of NR IQA metrics to compute

• fr_vqa_metrics - list of FR VQA metrics to compute

• nr_vqa_metrics - list of NR VQA metrics to compute

{config_filename}_list.json (live-wcvqd_list.json)

It is mandatory if you want to use NR metrics. File should be in .json format with the following structure:

['video_pth_1', ...,'video_pth_n']

{config_filename}_pairs.json (live-wcvqd_pairs.json)

It is mandatory if you want to use FR IQA metrics. File should be in .json format with the following structure:

Dict[str, List[Tuple[List[str], Literal["same", "repeat_each", "repeat_last"]]]]

{
	ref_1:
		[
			([dist_1, dist_2, ...], mode_1),
			...,
			([dist_1, dist_2, ...], mode_k)
		], 
	...,
	ref_k:
		[
			([...], mode_1),
			...,
			([...], mode_k)
		]
}

One record corresponds to one reference video and the list of lists of its distortions.

Algortihm will iterate along videos from one tuple simultaneously untill the shortest of them will end.

---

Example:

{
	ref: 
		[
			([dist_1, dist_2, dist_long], "same"),
			([dist_3, dist_4], "repeat_each")
		]
}

Here dist_1 and dist_2 have the same length that is also equal to ref length.

dist_long is longer than dist_1 and dist_2, so the excess frames of dist_long will be dropped and you will see warning in log file.

dist_3 and dist_4 have the same length, but it's n times shorter than the ref video length. So, to match the ref length, each frame of dist_3 and dist_4 will be taken n times. If n isn't integer then frames to repeat will be evenly spaced.

---

Algorithm will compute frame to frame scores untill one of the videos will end. If one of the source or distorted videos is shorter, then warning will be thrown in the log file.

If number of frames in the reference video is greater then in distorted videos from one record (may be beacuse of frame dropping distortion), then you can provide mode for handling this situation.

mode:

• same - iterate untill the shortest will end.

• repeat_each - if the shortest distorted video is n times shorter than reference, then each frame of the shortest distorted video and corresponding frames of other distorted videos will be repeated n times. If n isn't an integer then frames to repeat will be evenly spaced. It works almost like ffmpeg framerate filter, except that after repeat_each you will always have the same lengths for ref and dist.

• repeat_last - in all distorted videos last frame will be repeated as many times as needed to the shortest distorted video match the reference

If source video is shorter, then use mode == "same" to simply drop excess frames of distorted videos. That's may be suitable, for example, when distorted videos have stalls.

Recommendation: this config has been designed in such a way that it is convenient to group the videos with same length in one tuple and apply one processing method to them. It also speeds up the calculation process.

About VMAF

For the sake of generality, VMAF in this package is using the same config file and same modes, as needed for FR IQA: {config_filename}_pairs.json .

There is no need to group the videos with the same length. For one reference you can just make one or two groups (tuples) with different mode.

Each pair of videos will be processed individually.

For proper work, when computing VMAF, both videos will be brought to one fps (reference fps) with ffmpeg framerate filter.

After that, if videos still have different length - two ffmpeg framesync options are available: shortest and repeatlast.

Modes same and repeat_each will be treated as shortest.

Mode repeat_last will be treated as repeatlast.

And videos will be always brought to one framerate!

To specify number of threads, set enviroment variable VMAF_N_THREADS. (Default: 1)

Note: all paths in files img_list, img_pairs, video_list, video_pairs should be relative to dataset_pth!

Some technical details

0. Application firstly check if CUDA is available (and if CuPy is installed for SI and CF). If true, then gpu is used. If you don't want to use gpu, set enviroment variable CUDA_VISIBLE_DEVICES=-1.

1. All necessary checkpoints for models will be automatically downloaded in $HOME/.cache/{...}. All real paths will be written in log files.

2. Application is always trying to not to stop because of errors and process them on the go with logging to log file

3. All files with values and logs are written interactively line by line

4. Images and Frames of Video are rescaled to one size before computing FR IQA metrics (using bicubic interpolation)

5. IQA metrics for videos are computed frame-wise with further averaging

   1. If some of frames can't be computed during loop - they will be thrown away with warning in log file

6. Application is trying to minimize the number of memory accesses

7. All one type (NR of FR) IQA metrics are computed in one pass

   1. For images and videos, this means that they will be read only once for all listed in config one type metrics

   2. Also, all distorted images/videos from one set in {name}_pairs.json will be computed simultaneously, which reduces the reading of reference to one time

   3. Because of this, it is necessary to keep all initialized models in memory, so be careful with cuda when writing configs !

8. If you have build Decord with gpu and want to use NVDEC, set enviroment variable DECORD_GPU=1

9. The application first tries to read videos using the Decord with gpu acceleration (if DECORD_GPU=1). Next, if it didn't work out (if Decord threw an exception), with Decord on cpu . If this also failed (if Decord threw an exception), then with the help of OpenCV on cpu.

   1. Build Decord with gpu acceleration (with NVDEC) to make reading videos from memory even faster !

10. If you don't want to use Decord at all and only want OpenCV, set enviroment variable USE_CV2=1

11. If you want to see the time of counting each frame in the video with NR or FR IQA metrics in logs, set enviroment variable DEBUG_VIDEO=1. It will also make ffmpeg vmaf output visible in logs.

Dependencies:

• Python: >=3.10,<3.11
• ffmpeg (build with libvmaf for VMAF)
• decord (for NR VQA) (you can build decord with gpu to use NVDEC)
• CUDA: >= 10.2 (OPTIONAL if use GPU)
• CuPy (OPTIONAL if use SI, CF with GPU)

How to install:

Via pip from GitLab:

pip install objective-metrics --index-url https://__token__:<your_personal_token>@vg-code.gml-team.ru/api/v4/projects/716/packages/pypi/simple

You need to request token and replace <your_personal_token> with it.

Docker

Read here about Docker support. Dockerfiles are in ./Docker/ folder.

List of available metrics

Name of metrics for config files are in brackets !

IQA

NR

Paper Link	Method	Code	Used in AD	Greater-Better
pdf	SPAQ Baseline (spaq-bl)	PyTorch	No	Yes
pdf	SPAQ MT-A (spaq-mta)	PyTorch	No	Yes
pdf	SPAQ MT-S (spaq-mts)	PyTorch	Yes	Yes
pdf	HyperIQA (hyperiqa)	PyTorch	Yes	Yes
pdf	CNNIQA (cnniqa)	PyTorch	Yes	Yes
arXiv	Linearity (linearity)	PyTorch	Yes	Yes
arXiv	PaQ2PiQ (paq2piq)	PyTorch	Yes	Yes
arXiv	CLIPIQA (clipiqa)	PyTorch	No	Yes
arXiv	CLIPIQA+ (clipiqa+)	PyTorch	Yes	Yes
arXiv	KonCept512 (koncept512)	PyTorch	Yes	Yes
arXiv	MANIQA (maniqa)	PyTorch	No	Yes
arXiv	TReS (tres)	PyTorch	No	Yes
arXiv	MUSIQ (musiq)	PyTorch	Yes	Yes
arXiv	PI (pi)	PyTorch	No	No
arXiv	DBCNN (dbcnn)	PyTorch	Yes	Yes
arXiv	NIMA (nima)	PyTorch	Yes	Yes
arXiv	NRQM (nrqm)	PyTorch	No	Yes
pdf	ILNIQE (ilniqe)	PyTorch	No	No
pdf	BRISQUE (brisque)	PyTorch	No	No
pdf	NIQE (niqe)	PyTorch	No	No
arXiv	UNIQUE (unique)	PyTorch	Yes	Yes
arXiv	TOPIQ (topiq_nr)	PyTorch	Yes	Yes
ITU	Spatial Information (si)	self-made	No	-
ResearchGate	Colourfulness (cf)	self-made	No	-

FR

PSNR, SSIM, MS-SSIM, CW-SSIM are computed on Y channel in YUV (YCbCr) color space.

Paper Link	Method	Code	Used in AD	Greater-Better
arXiv	TOPIQ (topiq_fr)	PyTorch		Yes
arXiv	AHIQ (ahiq)	PyTorch		Yes
arXiv	PieAPP (pieapp)	PyTorch		Yes
arXiv	LPIPS (lpips)	PyTorch		Yes
arXiv	DISTS (dists)	PyTorch		Yes
arXiv	CKDN1 (ckdn)	PyTorch		Yes
pdf	FSIM (fsim)	PyTorch		Yes
wiki	SSIM (ssim)	PyTorch		Yes
pdf	MS-SSIM (ms_ssim)	PyTorch		Yes
pdf	CW-SSIM (cw_ssim)	PyTorch		Yes
arXiv	PSNR (psnr)	PyTorch		Yes
pdf	VIF (vif)	PyTorch		Yes
arXiv	GMSD (gmsd)	PyTorch		Yes
pdf	NLPD (nlpd)	PyTorch		Yes
IEEE Xplore	VSI (vsi)	PyTorch		Yes
pdf	MAD (mad)	PyTorch		Yes
IEEE Xplore	SR-SIM (srsim)	PyTorch		Yes
IEEE Xplore	DSS (dss)	PyTorch		Yes
arXiv	HaarPSI (haarpsi)	PyTorch		Yes
arXiv	MDSI (mdsi)	PyTorch		Yes
pdf	MS-GMSD (msgmsd)	PyTorch		Yes

[1] This method use distorted image as reference. Please refer to the paper for details.

VQA

NR

Paper Link	Method	Code	Used in AD	Greater-Better
arXiv	MDTVSFA (mdtvsfa)	PyTorch		Yes
arXiv	FAST-VQA (FAST-VQA)	PyTorch		Yes
arXiv	FasterVQA (FasterVQA)	PyTorch		Yes
arXiv	DOVER (dover)	PyTorch		Yes
ITU	Temporal Information (ti)	self-made	No	-

FR

Paper Link	Method	Code	Used in AD	Greater-Better
wiki	VMAF (vmaf)	FFMPEG VMAF	Yes	Yes

Useful resources

VQMT - Tool with objective metrics

PyIQA - Python Library

PIQ - Python Library

Table of Metrics

MSU Metric Benchmark

Metric calculation pipeline

0. Transmuxing

Transmux your videos from .yuv format to some container with metadata (mp4, webm, mkv, y4m, ...). You can use lossles encoding to save space.

1. Make configs for your datasets

a. Worklists

If you want to calculate NR metrics, then make {config_filename}_list.json.

If you want to calculate FR IQA metrics, then make {config_filename}_pairs.json.

That is the most complicated part, where you shold specify all videos that you want to be computed.

You can find some examples of worklists in datasets_configs/ folder and ways of parsing datasets filenames in get_configs.ipynb.

b. Config

Make config file {dataset_name}.yaml, where write all necessary information, like path to dataset.

There you can write a list of metrics, that you want to compute over the dataset.

You can find examples of configs in datasets_configs/ folder.

2. Run application

Run objective_metrics_run script from command line.

You also can build a docker image and run a container. Read here about Docker support. Dockerfiles are in ./Docker/ folder.

3. Watch the logs and values

a. Logs

Application will make two log files and will be writing to them interactively.

One log file will be called the same as the number of process in which application was started (in case of docker image it will be just 1.log). This log file will contain general information about queue of datasets.

Second log file will be named as a name of a dataset, that is now being calculated. This log file will contain information about all steps of calculation and about all errors.

b. Values

Application will make a .csv file for each metric and dataset and will be writing to them interactively.

You can watch metric values in the corresponding .csv files. If due to errors, for this concrete image or video the metric could not be calculated, then the value in the file will be Nan.

4. Error handling

Most common error is CUDA out of memory.

You can parse log and csv files to define what images and videos and what metrics failed to be calculated.

You can look examples of log parsers in log_parsers.py file.

After you defined all failed images/videos and metrics - you can make new worklists and configs to try again, when you will have more free cuda memory available.

For example, if due to CUDA out of memory error for some metric more then 10 frames of video failed to be calculated, then this is an excuse to make another attmept for this video for this metric.