Report PlasEval evaluations

Installation

With uv (recommended)

See also uv.

uv tool install plaseval-report

With virtualenv

python3.13 -m virtualenv .venv
source .venv/bin/activate
python3.13 -m pip install plaseval-report

Usage

# If installed with uv tool
uvx plaseval-report --help
# If installed in virual environment
plaseval-report --help

Input data

The main file is a TSV file with the following base content (independent of comp or eval commands):

Column ID	Type	Description
species_id	str	Species code
sample_uid	str	Sample UID
method_code	str	Method code

The following columns are specific to comp command:

Column ID	Type	Description	Measure code
Cuts	float	Normalized cut cost	cuts
Joins	float	Normalized join cost	joins
Extra_ctgs	float	Extra contigs cost	extra
Missing_ctgs	float	Missing contigs cost	miss
Dissimilarity	float	Dissimilarity	diss

The following columns are specific to eval command:

Column ID	Type	Description	Measure code
unw_precision	float	Unweighted precision	unw_prec
unw_recall	float	Unweighted recall	unw_recall
unw_f1	float	Unweighted F1	unw_f1
w_precision	float	Weighted precision	w_prec
w_recall	float	Weighted recall	w_recall
w_f1	float	Weighted F1	w_f1

The configuration of the figures/stats is detailed in the config.yaml file (optional):

#
# (Optional) Method codes options
#
methods:
  #
  # (Optional) List of method codes to consider.
  # If not set, all the method codes are considered, and the order is given by the TSV file.
  # If the key to_show is not set, the method order is given by the `methods` list.
  #
  to_consider:
    - <method_code>
    - ...
  #
  # (Optional) List of method codes to show among the ones in `to_consider`.
  # If the key to_show is set, the method order is given by the `to_show` list.
  #
  to_show:
    - <method_code>
    - ...
  #
  # (Optional) Map method code to labels
  #
  labels:
    #
    # One line labels.
    # If not set, take the wrap labels otherwise the method codes.
    #
    one_line:
      <method_code>: <str>
      ...
    #
    # Labels potentially on several lines.
    # If not set, take the one_line labels otherwise the method codes.
    #
    wrap:
      <method_code>: <str>
      ...
  #
  # (Optional) List of pairs of methods to annotate with stats
  #
  statannotate:
    - - <method_code>
      - <method_code>
    - ...
  #
  # Method figure aesthetics
  #
  fig_aes:
    palette: <str> # default: Set3, see https://matplotlib.org/stable/users/explain/colors/colormaps.html#qualitative
    #
    # Map method to palette index
    # By default follow the order of the methods to show.
    # If one method is missing in the map, automatically set the index to unused ones, then cycle.
    #
    color_indices:
      <method_code>: <int> # The index of the color in the palette
      ...
    #
    # (Optional) Highlight specific methods in the figures (e.g. your own tool).
    #
    label_highlight:
      #
      # How to render the highlighted labels/marks.
      # - bold:    fontweight="bold" on the label/tick text
      # - color:   recolor the label/tick text and/or the bar/point using `color`
      # - outline: add a colored edge/border around the bar or marker using `color`
      # Several modes can be combined in a list, e.g. [bold, color].
      #
      mode: bold | color | outline | [bold, color, ...] # default: bold
      #
      # (Optional) Color used when mode includes `color` or `outline`.
      # Any matplotlib-compatible color string (name, hex, etc.).
      # If not set, the color is taken from the palette.
      #
      color: <str> # default: null
      #
      # List of method codes to highlight.
      #
      items:
        - <method_code>
        - ...
#
# (Optional) Species options
# It follows the same structure as for `methods`
#
species:
  ...

#
# (Optional) Measures options
#
measures:
  #
  # (Optional) List of measures to consider.
  # If not set, all the measures are considered, and the order is given by the TSV file.
  #
  to_consider:
    - <measure_code>
    - ...
  #
  # (Optional) List of measures to show among the ones in `to_consider`.
  #
  to_show:
    - <measure_code>
    - ...
#
# (Optional) Samples removal strategy (for the methods listed in `methods`).
# The option is valid for al but result-presence figures and stats.
#
remove_samples: fails | nothing # default: fails
#
# Figure aesthetics (Optional, everything is optional)
#
fig_aes:
  context: notebook | paper | talk | poster # default: notebook
  focus: true | false                       # default: false

PlasEval comp/eval command results

The next section illustrates how to generate the figures. Generating the statistics tables is following the same process.

plaseval-report fig --help
plaseval-report stats --help

In the next sections, we must tell which methods we want to consider.

Result presence figures

Know for each tool how many samples have been evaluated by PlasEval:

plaseval-report fig res-presence "$merge_evals_tsv" "$figs_dir/res-presence" --config "$config_yaml"

Distribution figures

plaseval-report fig distribution "$merge_evals_tsv" "$figs_dir/distribution" --config "$config_yaml"

Versus figures

Generate a versus figure:

x_axis="pbhmf_rfpl"
y_axis="gpcc_rfpl"

plaseval-report comp fig versus "$merge_evals_tsv" "$figs_dir/versus" "$x_axis" "$y_axis" --config "$config_yaml"

[!NOTE] Keys to_show are ignored in the versus figure.

[!TIP] If you are not setting the option remove_samples to fails, you can simply list the two methods of the axes in the to_consider key.

Repeat stats figures

The above figures use an additional TSV file, repeat_stats.tsv:

Column ID	Description
sample_uid	Sample ID
species_id	Species ID
num_contigs	Sum over the bins of the number of contigs
num_unique_contigs	Size of set of contigs present in at least one bin
repeat_ratio	Repeat ratio. Defined as num_unique_contigs / num_contigs. If defined (i.e. num_contigs > 0), it is a positive float $> 1$. If not defined (i.e. $0/0$), the cell is empty.

An overview:

# To get a count of defined and undefined repeat ratios
plaseval-report fig repeat-stats overview count "$repeat_stats_tsv" "$figs_dir/repeat-stats/overview" --config "$config_yaml"
# To get a distribution of the (defined) repeat ratios
plaseval-report fig repeat-stats overview distribution "$repeat_stats_tsv" "$figs_dir/repeat-stats/overview" --config "$config_yaml"

Evaluation measures according to the repeat ratio:

# To get the evolution of evaluation measures according to (defined) repeat ratio
plaseval-report fig repeat-stats evals evolution "$merge_evals_tsv" "$repeat_stats_tsv" "$figs_dir/repeat_stats/eval" --config "$config_yaml"

Joining the PlasEval comp and eval evaluations

In order to have the comp and eval figures on the same samples, we can join the two TSV files:

plaseval-report utils join-measures --help

join_tsv=comp_eval_merge_evals.tsv
plaseval-report utils join-measures "$merge_comp_tsv" "$merge_eval_tsv" "$join_tsv"

Joining the two set of measures is relevant when remove_samples option is set to fails because filtering on the joined TSV ensures all the samples have a comp and a eval evaluation.