🕵️ RCAEval: A Benchmark for Root Cause Analysis

FSE 2026 WWW 2025 ASE 2024 FSE 2024

RCAEval is an open-source benchmark that offers nine datasets with 735 real failure cases, and an evaluation framework for root cause analysis (RCA) in microservice systems. It includes 15 reproducible baselines covering metric-based, trace-based, and multi-source RCA methods.

Table of Contents

• Prerequisites
• Installation
• How-to-use
  • Data format
  • Basic usage example
• Available Datasets
• Available Baselines
• Benchmark
• For TORAI Paper
• For EventADL Paper
• Licensing
• Acknowledgments
• Change Logs
• Citation
• Contact

Prerequisites

We recommend using machines equipped with at least 8 cores, 16GB RAM, and ~50GB available disk space with Ubuntu 22.04 or Ubuntu 20.04, and Python3.12 (Python 3.14 is also supported).

Installation

The default environment, which is used for most methods, can be easily installed as follows. Detailed installation instructions for all methods are in SETUP.md.

Open your terminal and run the following commands

sudo apt update -y
sudo apt install -y build-essential \
  libxml2 libxml2-dev zlib1g-dev \
  python3-tk graphviz

Clone RCAEval from GitHub

git clone https://github.com/phamquiluan/RCAEval.git && cd RCAEval

Create virtual environment with Python 3.12 (refer SETUP.md to see how to install Python3.12 on Linux)

python3.12 -m venv env
. env/bin/activate

Install RCAEval using pip

pip install -e .[default]

Or, install RCAEval from PyPI

# Install RCAEval from PyPI
pip install RCAEval[default]

Test the installation

python -m pytest tests/test.py::test_basic

Expected output after running the above command (it takes less than 1 minute)

$ pytest tests/test.py::test_basic
============================== test session starts ===============================
platform linux -- Python 3.12.12, pytest-7.3.1, pluggy-1.0.0
rootdir: /home/ubuntu/RCAEval
plugins: dvc-2.57.3, hydra-core-1.3.2
collected 1 item                                                                 

tests/test.py .                                                            [100%]

=============================== 1 passed in 3.16s ================================

How-to-use

Data format

The telemetry data must be presented as pandas.DataFrame. We require the data to have a column named time that stores the timestep. A sample of valid data could be downloaded using the download_data() or download_multi_source_data() method that we will demonstrate shortly below.

Basic usage example

A basic example to use BARO, a metric-based RCA baseline, to perform RCA are presented as follows,

# You can put the code here to a file named test.py
from RCAEval.e2e import baro
from RCAEval.utility import download_data, read_data

# download a sample data to data.csv
download_data()

# read data from data.csv
data = read_data("data.csv")
anomaly_detected_timestamp = 1692569339

# perform root cause analysis
root_causes = baro(data, anomaly_detected_timestamp)["ranks"]

# print the top 5 root causes
print("Top 5 root causes:", root_causes[:5])

Expected output after running the above code (it takes around 1 minute)

$ python test.py
Downloading data.csv..: 100%|████████████████████| 570k/570k [00:00<00:00, 19.8MiB/s]
Top 5 root causes: ['emailservice_mem', 'recommendationservice_mem', 'cartservice_mem', 'checkoutservice_latency', 'cartservice_latency']

A tutorial of using Multi-source BARO to diagnose failure using multi-source telemetry data (metrics, logs, and traces) is presented in docs/multi-source-rca-demo.ipynb.

A tutorial of using BARO to diagnose code-level faults is presented in docs/code-level-rca.ipynb.

Available Datasets

RCAEval benchmark includes nine datasets organized into three benchmark suites (RE1, RE2, RE3), each covering three microservice systems (Online Boutique, Sock Shop, Train Ticket). Together, these datasets feature 735 failure cases with 11 fault types. Each failure case includes annotated root cause service and root cause indicator (e.g., specific metric or log indicating the root cause).

RE1 Datasets (375 cases): Metric-only data supporting metric-based RCA methods. Five fault types (CPU, MEM, DISK, DELAY, LOSS) across five services per system, with five repetitions per fault-service pair.

RE2 Datasets (270 cases): Multi-source data (metrics, logs, traces) supporting multi-source RCA methods. Six fault types (adds SOCKET) across five services per system, with three repetitions per fault-service pair.

RE3 Datasets (90 cases): Multi-source data focusing on code-level faults (F1-F5). Supports diagnosing code-level faults through telemetry data, e.g., leveraging stack traces in logs or response codes in traces.

File Structure

Each dataset directory follows the naming convention: {benchmark}_{service}_{fault}_{instance}

• metrics.json: Time-series metrics data
• inject_time.txt: Fault injection timestamp (Unix timestamp)
• logs.csv: Log data (RE2 and RE3 only)
• traces.csv: Trace data (RE2 and RE3 only)

Our datasets and their description are publicly available with the following information:

• Hugging Face (recommended): https://huggingface.co/datasets/phamquiluan/RCAEval (Parquet, 3.4GB on disk, downloadable one suite or one case at a time)
• Figshare: https://figshare.com/articles/dataset/RCAEval_A_Benchmark_for_Root_Cause_Analysis_of_Microservice_Systems/31048672 (more structured format, with additional preprocessed series)
• Zenodo: https://zenodo.org/records/14590730 (DOI for citation: https://doi.org/10.5281/zenodo.14590730)

The Hugging Face copy stores the same 735 cases as Parquet instead of metrics.json / logs.csv / traces.csv. It is 3.4GB to download against 5.2GB of zip archives on Zenodo, and about eleven times smaller once extracted, since those archives expand to roughly 38GB. Unlike a zip archive it can also be fetched a suite or a single case at a time, so you never download telemetry you will not use:

from huggingface_hub import snapshot_download

# one suite (use "re2*" or "re3*" for the others, or omit to get all 735 cases)
snapshot_download(
    repo_id="phamquiluan/RCAEval",
    repo_type="dataset",
    allow_patterns="re1*",
    local_dir="data",
)

cases.parquet in that repository indexes all 735 cases with their ground-truth root cause service, fault type, injection time, and telemetry sizes, so you can select a subset before downloading any telemetry:

import pandas as pd

idx = pd.read_parquet("hf://datasets/phamquiluan/RCAEval/cases.parquet")
print(idx[idx.dataset == "RE2-TT"][["case", "root_cause_service", "fault", "n_metrics"]])

RCAEval.utility.read_metrics, read_logs and read_traces read either layout, so code written against the original files works unchanged with the Parquet copy:

from RCAEval.utility import read_metrics

df = read_metrics("data/re1ob_adservice_cpu_1")  # metrics.parquet or metrics.json

We also provide utility functions to download our datasets using Python. The downloaded datasets will be available at directory data.

from RCAEval.utility import (
    download_re1_dataset,
    download_re2_dataset,
    download_re3_dataset,
)

download_re1_dataset()
download_re2_dataset()
download_re3_dataset()

$ python test.py
Downloading RE1.zip..: 100%|█████████████████████| 390M/390M [01:02<00:00, 6.22MiB/s]
Downloading RE2.zip..: 100%|███████████████████| 4.21G/4.21G [11:23<00:00, 6.17MiB/s]
Downloading RE3.zip..: 100%|█████████████████████| 534M/534M [01:29<00:00, 5.97MiB/s]

Available Baselines

RCAEval stores all the RCA methods in the e2e module (implemented in RCAEval.e2e). There are many RCA baselines available: RUN, CausalRCA, CIRCA, RCD, MicroCause, EasyRCA, MSCRED, BARO, 𝜖-Diagnosis, TraceRCA, MicroRank, PDiagnose, Multi-source BARO, Multi-source RCD, Multi-source CIRCA, TORAI, EventADL.

Benchmark

We provide a script named main.py to assist in reproducing the results from our RCAEval paper. This script can be executed using Python with the following syntax:

python main.py [-h] [--dataset DATASET] [--method METHOD]

The available options and their descriptions are as follows:

options:
  -h, --help            Show this help message and exit
  --dataset DATASET     Choose a dataset. Valid options:
                        [re2-ob, re2-ss, re2-tt, etc.]
  --method METHOD       Choose a method (`causalrca`, `microcause`, `e_diagnosis`, `baro`, `rcd`, `circa`, etc.)

For example, in Table 6, BARO achieves Avg@5 of 0.72, 0.99, 1, 0.83, 0.64, and 0.8 for CPU, MEM, DISK, SOCKET, DELAY, LOSS, and AVERAGE on the Train Ticket dataset. To reproduce these results, you can run the following commands:

python  main.py --method baro --dataset re2-tt

The expected output should be exactly as presented in the paper (it takes less than 1 minute to run the code)

$ python  main.py --method baro --dataset re2-tt --length 20
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 90/90 [00:45<00:00,  1.98it/s]
--- Evaluation results ---
Avg@5-CPU:   0.72
Avg@5-MEM:   0.99
Avg@5-DISK:  1.0
Avg@5-SOCKET: 0.83
Avg@5-DELAY: 0.63
Avg@5-LOSS:  0.64
---
Avg speed: 0.51

We can replace the baro method with other methods (e.g., circa) and substitute re2-tt with other datasets to replicate the corresponding results shown in Table 6. This reproduction process is also integrated into our Continuous Integration (CI) setup. For more details, refer to the .circleci/config.yml file.

Multi-source RCA benchmark

The multi-source methods (Multi-source BARO, Multi-source RCD, and Multi-source CIRCA) consume metrics, logs, and traces together. They run on the RE2/RE3 datasets, which include the required logts.csv, tracets_err.csv, and tracets_lat.csv files in each failure case:

# Multi-source BARO (Python 3.12, default environment)
python main.py --method mmbaro --dataset re2-ob

# Multi-source CIRCA (Python 3.12, default environment)
python main.py --method mmcirca --dataset re2-ob

# Multi-source RCD (requires the RCD environment: Python 3.8, `pip install -e .[rcd]`, see docs/SETUP.md)
python main.py --method mmrcd --dataset re2-ob

A function-level tutorial using Multi-source BARO on a single failure case is available in docs/multi-source-rca-demo.ipynb.

For TORAI Paper

TORAI Datasets (270 cases): A processed version of the RE2 datasets, containing pre-aggregated time-series for metrics, logs, and traces (where available). These datasets are used by the TORAI method and are available separately on Figshare: https://doi.org/10.6084/m9.figshare.31925976.

TORAI requires a separate Python 3.8 environment and its own datasets. For full instructions, see docs/TORAI.md for installation and reproducibility instructions.

python main.py --method torai --dataset torai-ob --length 10

--- Evaluation results ---
Avg@5-CPU:   0.96
Avg@5-MEM:   0.93
Avg@5-DISK:  1.0
Avg@5-SOCKET: 0.93
Avg@5-DELAY: 0.8
Avg@5-LOSS:  0.84

For EventADL Paper

EventADL localizes root-cause actors/resources over CloudTrail-style API-call event logs, rather than the metrics/traces/logs used by other RCAEval methods. It requires a separate Python 3.12 environment. For full instructions, see docs/EVENTADL.md for installation and usage instructions.

python main.py --method eventadl --dataset eventadl-falcon

Licensing

This repository includes code from various sources with different licenses. We have included their corresponding LICENSE into the LICENSES directory:

• BARO: Licensed under the MIT License. Original source: BARO GitHub Repository.
• CausalRCA: No License. Original source: CausalRCA GitHub Repository.
• CIRCA: Licensed under the BSD 3-Clause License. Original source: CIRCA GitHub Repository.
• E-Diagnosis: Licensed under the BSD 3-Clause License. Original source: PyRCA GitHub Repository.
• MicroCause: Licensed under the Apache License 2.0. Original source: MicroCause GitHub Repository.
• RCD: Licensed under the MIT License. Original source: RCD GitHub Repository.
• RUN: No License. Original source: RUN GitHub Repository.

For the code implemented by us and for our datasets, we distribute them under the MIT LICENSE.

Acknowledgments

We would like to express our sincere gratitude to the researchers and developers who created the baselines used in our study. Their work has been instrumental in making this project possible. We deeply appreciate the time, effort, and expertise that have gone into developing and maintaining these resources. This project would not have been feasible without their contributions.

Change Logs

• [Aug 2026] Published all nine datasets (735 cases) on Hugging Face as Parquet, 3.4GB on disk and downloadable one suite or one case at a time, with a cases.parquet index of every case and its ground-truth root cause. Added read_metrics, read_logs and read_traces, which read either the original or the Parquet layout.
• [Jul 2026] Made the multi-source RCA benchmark runnable via main.py (Multi-source BARO/RCD/CIRCA) and fixed PyPI packaging so wheels include all subpackages.
• [Jul 2026] Added EventADL, an event-log root cause localization method accepted at FSE'26.
• [Jun 2026] The version of RCAEval with TORAI used in our FSE'26 paper is available in the fse26 branch.
• [Apr 2026] Added TORAI, a multi-source RCA method accepted at FSE'26.
• [Mar 2025] The version of RCAEval used in our WWW'25 paper are available in the www25 branch.
• [Dec 2024] The prior version of RCAEval used in our ASE'24 paper are available in the ase24 branch.

Citation

@article{pham2026eventadl,
  title={EventADL: Open-Box Anomaly Detection and Localization Framework for Events in Cloud-Based Service Systems},
  author={Pham, Luan and Nicolet, Victor and Dodds, Joey and Guan, Hui and Kroening, Daniel},
  journal={Proceedings of the ACM on Software Engineering},
  volume={3},
  number={FSE},
  pages={4070--4093},
  year={2026}
}

@article{pham2026torai,
  title={TORAI: Multi-source Root Cause Analysis for Blind Spots in Microservice Service Call Graph},
  author={Pham, Luan and Ha, Huong and Zhang, Xiuzhen and Zhang, Hongyu},
  journal={Proceedings of the ACM on Software Engineering},
  volume={3},
  number={FSE},
  pages={2928--2951},
  year={2026}
}

@inproceedings{pham2025rcaeval,
  title={RCAEval: A Benchmark for Root Cause Analysis of Microservice Systems with Telemetry Data},
  author={Pham, Luan and Zhang, Hongyu and Ha, Huong and Salim, Flora and Zhang, Xiuzhen},
  booktitle={Companion Proceedings of the ACM on Web Conference 2025},
  pages={777--780},
  year={2025}
}

@inproceedings{pham2024root,
  title={Root Cause Analysis for Microservice System based on Causal Inference: How Far Are We?},
  author={Pham, Luan and Ha, Huong and Zhang, Hongyu},
  booktitle={Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering},
  pages={706--715},
  year={2024}
}

@inproceedings{pham2024baro,
  title={BARO: Robust root cause analysis for microservices via multivariate bayesian online change point detection},
  author={Pham, Luan and Ha, Huong and Zhang, Hongyu},
  journal={Proceedings of the ACM on Software Engineering},
  volume={1},
  number={FSE},
  pages={2214--2237},
  year={2024},
}

Contact

phamquiluan@gmail.com