SLR Decision Merger

SLR Decision Merger is a Python toolkit to assist researchers in conducting Systematic Literature Reviews (SLRs) by dividing BibTeX files among reviewers and then later merging their decisions. It streamlines the process of managing reviewer assignments and consolidating review outcomes.

Features

• Paper Division: Randomly divide papers from BibTeX files among multiple reviewers
• Configurable Assignment: Specify the reviewers and how many reviewers should review each paper

Installation

1. Using pip (Recommended)

Ensure you have Python installed (version 3.7 or higher is recommended).

pip install slrdecisionmerger

2. Installing from Git

If you want the latest development version:

pip install git+https://gitlab.rakshitmittal.net/rmittal/slrdecisionmerger.git

3. Setting Up a Virtual Environment (Optional but Recommended)

Creating a virtual environment helps manage dependencies and avoid conflicts.

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

Then proceed with the installation steps above.

Configuration

SLR Decision Merger uses a configuration file (config.json) to manage settings. The idea is that each SLR project has a unique configuration file. Ensure that you create and properly configure this file before running the tool. Example config.json

{
  "reviewers": ["alice", "bob", "james", "john"],
  "reviewers_per": 2, 
  "cache_directory": "/absolute/path/to/directory",
  "division_directory": "/absolute/path/to/directory",
  "review_directory": "/absolute/path/to/directory",
  "report_directory": "/absolute/path/to/directory"
}

{
  "reviewers": 4,
  "reviewers_per": 2, 
  "cache_directory": "/absolute/path/to/directory",
  "division_directory": "/absolute/path/to/directory",
  "review_directory": "/absolute/path/to/directory",
  "report_directory": "/absolute/path/to/directory"
}

Configuration Fields

• reviewers: Union[ int, List[str] ] : Either the total number of anonymous reviewers, or a list of names of reviewers.
• reviewers_per: int : Number of reviewers that should review each paper.
• cache_directory : str : Absolute path to directory containing the bib files to be distributed among reviewers.
• division_directory : str : Absolute path to directory where reviewer assignment bib-files will be created.
• review_directory : str : Absolute path to directory containing bib-files with reviewer evaluations.
• report_directory : str : Absolute path to directory where generated reports will be saved.

Note: reviewers_per cannot be greater than number of reviewers.

Usage

Basic Command

python -m slrdecisionmerger --config config.json [options]

Available Options

• --config: Path to the configuration JSON file (default: config.json)
• --divide: Divide papers among reviewers
• --merge: Merge reviewer decisions and generate reports
• --debug: Enable debug-level logging
• --merge_strat: Merge strategy. One of "conservative", "liberal", "consensus"

Examples

1. Divide Papers Among Reviewers

python -m slrdecisionmerger --config config.json --divide

2. Divide Papers with Debug Logging

python -m slrdecisionmerger --config config.json --divide --debug

3. Merge Reviewer Decisions with Consensus strategy

python -m slrdecisionmerger --config config.json --merge --merge_strat consensus

Decision Merging

After reviewers have evaluated their assigned papers, the --merge command consolidates their decisions. The tool identifies a reviewer's decision based on the groups field of a BibTeX entry. If the group is relevant, the paper is considered accepted; otherwise, it is considered rejected. It is assumed that the reviwers used (JabRef)[https://www.jabref.org/] to put their decisions (each reviewer-specific file generated by divide has automatically added JabRef annotations at the end).

Merge Strategies (--merge_strat)

You can define how the tool handles agreements and disagreements using one of three strategies:

• conservative: A paper is accepted if at least one reviewer marked it as relevant. This strategy maximizes recall, ensuring no potentially relevant paper is missed.
• liberal: A paper is accepted only if no reviewers marked it as non-relevant (e.g., OFF_TOPIC, NO_PEER_REV). This ensures that only papers with unanimous or partial positive consensus are included.
• consensus (Default): This is the most balanced strategy.
  • A paper is accepted if it has at least one relevant vote and no non-relevant votes.
  • A paper is rejected if it has only non-relevant votes.
  • If a paper receives both relevant and non-relevant votes, it is marked as a conflict.

Generated Files

Based on the chosen strategy, the merge process generates the following files in your report_directory:

• accepted.bib: A BibTeX file containing all papers that were accepted according to the merge strategy.
• conflicted.bib (Consensus strategy only): A BibTeX file containing all papers that had conflicting reviews.
• conflicts.csv (Consensus strategy only): A detailed CSV report of all conflicts. It includes the paper's metadata (title, author, year, doi, url) and a column for each reviewer showing their specific decision (accept or reject).
• conflict_matrix.csv (Consensus strategy only): A CSV file showing a symmetric matrix of the number of disagreements between each pair of reviewers. This is useful for analyzing inter-rater reliability.

License

MIT License