Report PlasEval evaluations

Installation

git clone

With uv (recommended)

See also uv.

uv sync
# For Bash:
source .venv/bin/activate
# For Fish:
source .venv/bin/activate.fish

With virtualenv

python3.13 -m virtualenv .venv
source .venv/bin/activate
python3.13 -m pip install .

Usage

In a test directory:

tests_dir=tests

mkdir $tests_dir

The next section is about PlasEval comp command. For PlasEval eval command, the processes are the same.

For more details, use --help option:

plaseval-report --help
# or with uv
uv run plaseval-report --help

Input data

The main file is a TSV file with the following base content (independent of comp or eval commands):

Column ID	Type	Description
species_id	str	Species code
sample_uid	str	Sample UID
method_code	str	Method code

The following columns are specific to comp command:

Column ID	Type	Description	Measure code
Cuts	float	Normalized cut cost	cuts
Joins	float	Normalized join cost	joins
Extra_ctgs	float	Extra contigs cost	extra
Missing_ctgs	float	Missing contigs cost	miss
Dissimilarity	float	Dissimilarity	diss

The following columns are specific to eval command:

Column ID	Type	Description	Measure code
unw_precision	float	Unweighted precision	unw_prec
unw_recall	float	Unweighted recall	unw_recall
unw_f1	float	Unweighted F1	unw_f1
w_precision	float	Weighted precision	w_prec
w_recall	float	Weighted recall	w_recall
w_f1	float	Weighted F1	w_f1

The configuration of the figures/stats is detailed in the config.yaml file (optional):

#
# (Optional) Method codes options
#
methods:
  #
  # (Optional) List of method codes to consider.
  # If not set, all the method codes are considered, and the order is given by the TSV file.
  # If the key to_show is not set, the method order is given by the `methods` list.
  #
  to_consider:
    - <method_code>
    - ...
  #
  # (Optional) List of method codes to show among the ones in `to_consider`.
  # If the key to_show is set, the method order is given by the `to_show` list.
  #
  to_show:
    - <method_code>
    - ...
  #
  # (Optional) Map method code to labels
  #
  labels:
    #
    # One line labels.
    # If not set, take the wrap labels otherwise the method codes.
    #
    one_line:
      <method_code>: <str>
      ...
    #
    # Labels potentially on several lines.
    # If not set, take the one_line labels otherwise the method codes.
    #
    wrap:
      <method_code>: <str>
      ...
  #
  # (Optional) List of pairs of methods to annotate with stats
  #
  statannotate:
    - - <method_code>
      - <method_code>
    - ...
  #
  # Method figure aesthetics
  #
  fig_aes:
    palette: <str> # default: Set3, see https://matplotlib.org/stable/users/explain/colors/colormaps.html#qualitative
    #
    # Map method to palette index
    # By default follow the order of the methods to show.
    # If one method is missing in the map, automatically set the index to unused ones, then cycle.
    #
    color_indices:
      <method_code>: <int> # The index of the color in the palette
      ...
#
# (Optional) Species options
# It follows the same structure as for `methods`
#
species:
  ...

#
# (Optional) Measures options
#
measures:
  #
  # (Optional) List of measures to consider.
  # If not set, all the measures are considered, and the order is given by the TSV file.
  #
  to_consider:
    - <measure_code>
    - ...
  #
  # (Optional) List of measures to show among the ones in `to_consider`.
  #
  to_show:
    - <measure_code>
    - ...
#
# (Optional) Samples removal strategy (for the methods listed in `methods`).
# The option is valid for al but result-presence figures and stats.
#
remove_samples: fails | nothing # default: fails
#
# Figure aesthetics (Optional, everything is optional)
#
fig_aes:
  context: notebook | paper | talk | poster # default: notebook
  focus: true | false                       # default: false

PlasEval comp/eval command results

In what follow, the same subcommands are available for both comp and eval commands. The only difference is in the name of the measures (specific to comp and eval). In what follow, we focus on the comp command.

The next section illustrates how to generate the figures. Generating the statistics tables is following the same process.

plaseval-report comp fig --help

In the next sections, we must tell which methods we want to consider.

Result presence figures

Know for each tool how many samples have been evaluated by PlasEval:

plaseval-report comp fig res-presence "$merge_evals_tsv" "$figs_dir/res-presence" --config "$config_yaml"

Distribution figures

plaseval-report comp fig distribution "$merge_evals_tsv" "$figs_dir/distribution" --config "$config_yaml"

Versus figures

Generate a versus figure:

x_axis="pbhmf_rfpl"
y_axis="gpcc_rfpl"

plaseval-report comp fig versus "$merge_evals_tsv" "$figs_dir/versus" "$x_axis" "$y_axis" --config "$config_yaml"

[!NOTE] Keys to_show are ignored in the versus figure.

[!TIP] If you are not setting the option remove_samples to fails, you can simply list the two methods of the axes in the to_consider key.

Repeat stats figures

The above figures use an additional TSV file, repeat_stats.tsv:

Column ID	Description
sample_uid	Sample ID
species_id	Species ID
repeat_ratio	Repeat ratio

An overview:

plaseval-report fig repeat-stats overview "$repeat_stats_tsv" "$figs_dir/repeat-stats/overview" --config "$config_yaml"

Evaluation measures according to the repeat ratio:

plaseval-report fig repeat-stats eval "$merge_evals_tsv" "$repeat_stats_tsv" "$figs_dir/repeat_stats/eval" --config "$config_yaml"

Joining the PlasEval comp and eval evaluations

In order to have the comp and eval figures on the same samples, we can join the two TSV files:

plaseval-report utils join-measures --help

join_tsv=comp_eval_merge_evals.tsv
plaseval-report utils join-measures "$merge_comp_tsv" "$merge_eval_tsv" "$join_tsv"

Joining the two set of measures is relevant when remove_samples option is set to fails because filtering on the joined TSV ensures all the samples have a comp and a eval evaluation.