RCAEval 0.0.8

No project description provided

🕵️ Root Cause Analysis for Microservices based on Causal Inference: How Far Are We?

This repository includes artifacts for reuse and reproduction of experimental results presented in our ASE'24 paper titled "Root Cause Analysis for Microservices based on Causal Inference: How Far Are We?".

Table of Contents

• Installation
  • Prerequisites
  • Pre-installation
  • Install the Dev environment
  • Install the RCD environment
  • Install the fGES environment
• How-to-use
  • Data format
  • Basic usage example
• Reproducibility
  • Reproduce RQ1 - Causal Discovery Performance
  • Reproduce RQ2 - Root Cause Analysis Performance
  • Reproduce RQ3 - Efficiency
  • Reproduce RQ4 - Data lengths
• Download Experimental Data - Supplementary Material
• Licensing
• Acknowledgments

Installation

We maintain 3 environments for different methods that could be installed easily as follows.

Prerequisites

• Ubuntu 22.04
• 8 CPU, 16GB RAM, 30GB free disk.

Pre-installation

Open your terminal and run commands

sudo apt update -y
sudo apt install -y build-essential \
  libxml2 libxml2-dev zlib1g-dev \
  python3-tk graphviz

sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt update -y
sudo apt-get install -y python3.8 python3.8-dev python3.8-venv
sudo apt-get install -y python3.10 python3.10-dev python3.10-venv

Install the Dev environment

By installing this environment, you can run the following:

• PC-based, FCI-based, LiNGAM-based, GES-based
• CausalRCA
• CIRCA, Nsigma, Dummy

python3.10 -m venv env-dev
. env-dev/bin/activate
pip install pip==20.0.2
pip install -e .[dev]

Install the RCD environment

By installing this environment, you can run the RCD algorithm

python3.8 -m venv env-rcd
. env-rcd/bin/activate
pip install pip==20.0.2
pip install -e .[rcd]
bash script/link.sh

Install the fGES environment

By installing this environment, you can run the fGES-based algorithm

python3.8 -m venv env-fges
. env-fges/bin/activate

pip install pip==20.0.2
pip install -e .[rcd]

cd LIB
pip install -e .
cd ..

# it MUST be performed in this order
sudo apt-get install -y gcc graphviz libgraphviz-dev pkg-config
pip install dill pygobnilp
pip install -U numba
pip install category_encoders sortedcontainers fcit
pip install pgmpy
pip install feature_engine

How-to-use

Data format

TBD

Basic usage example

TBD

Reproducibility

Reproduce RQ1 - Causal Discovery Performance

To reproduce the causal discovery performance, as presented in Table 3. You can download the corresponding dataset and extracted to folder ./data. Then, you can run the file graph_eval.py to obtain the results for one iteration. For example:

As presented in Table 3, PC achieves F1, F1-S, and SHD of 0.49, 0.65, and 16 on the CIRCA 10 dataset. To reproduce this results as presented in the Table 3. You can run the following commands:

python graph_eval.py -i data/syn_circa/10 -m pc -w 5

The expected output should be exactly as presented in the paper (it takes around 1 minute to run the code)

F1:   0.49
F1-S: 0.65
SHD:  16

We can replace the method pc and dataset syn_circa/10 to replicate corresponding results.

Reproduce RQ2 - Root Cause Analysis Performance

To reproduce the root cause analysis performance, as presented in Table 5. You can download the corresponding dataset and extracted to folder ./data. Then, you can run the file eval.py to reproduce the results. For example:

As presented in Table 5, NSigma [ $t_\Delta = 0$ ] achieves Avg@5 of 0.94, 1, 0.9, 0.98, and 0.67 for CPU, MEM, DISK, DELAY, and LOSS fault types on the Online Boutique dataset. To reproduce the RCA performance of NSigma [ $t_\Delta = 0$ ] as presented in the Table 5. You can run the following commands:

python eval.py -i data/online-boutique -o output-tmp -m nsigma --iter-num 10 -w 10 --length 10

Expected output after running the above code (it takes around 1 minute)

The results are exactly as presented in the paper (Table 5).

Evaluation results
s_cpu: 0.94
s_mem: 1.0
s_disk: 0.9
s_delay: 0.98
s_loss: 0.67

As presented in Table 5, NSigma [ $t_\Delta = 60$ ] achieves Avg@5 of 0.16, 0.24, 0.43, 0.55, and 0.38 for CPU, MEM, DISK, DELAY, and LOSS fault types on the Online Boutique dataset. To reproduce the RCA performance of NSigma [ $t_\Delta = 60$ ] as presented in the Table 5. You can run the following commands:

python eval.py -i data/online-boutique -o output-tmp -m nsigma --iter-num 10 -w 10 --length 10 --ad-delay 60

Expected output after running the above code (it takes around 1 minute)

The results are exactly as presented in the paper (Table 5).

Evaluation results
s_cpu: 0.16
s_mem: 0.24
s_disk: 0.43
s_delay: 0.55
s_loss: 0.38

We can replace the method nsigma by baro, pc_pagerank, fci_pagerank, rcd, e_diagnosis, etc. to replicate corresponding results.

Reproduce RQ3 - Efficiency

The efficiency is captured in our evaluation script and saved in the corresponding output directory.

Reproduce RQ4 - Data lengths

Our RQ4 relies on the scripts of RQ1 and RQ2, which we presented above.

Download Experimental Data - Supplementary Material

You can download our supplementary material and all the data we used for this research from this Google Drive.

Licensing

This repository includes code from various sources with different licenses:

• CIRCA: Licensed under the BSD 3-Clause License. Original source: CIRCA GitHub Repository.
• RCD: Licensed under the MIT License. Original source: RCD GitHub Repository.
• E-Diagnosis: Licensed under the BSD 3-Clause License. Original source: PyRCA GitHub Repository.
• CausalAI: Licensed under the BSD 3-Clause License. Original source: CausalAI GitHub Repository.
• MicroCause: Licensed under the BSD 3-Clause License. Original source: MicroCause GitHub Repository.

We have included their corresponding LICENSE into the LICENSES directory. For the code implemented by us, we distribute them under the MIT LICENSE.

Acknowledgments

We would like to express our sincere gratitude to the researchers and developers who created the baselines used in our study. Their work has been instrumental in making this project possible. We deeply appreciate the time, effort, and expertise that have gone into developing and maintaining these resources. Your commitment to advancing the field of root cause analysis and making your work accessible to the research community has been invaluable. This project would not have been feasible without your contributions.