Desbordante: high-performance data profiler

What is it?

Desbordante is a high-performance data profiler oriented towards exploratory data analysis. It is capable of discovering and validating many different patterns in data using various algorithms.

Table of Contents

• Tool Positioning
• Main Features
• Usage Examples
• I still don't understand how to use Desbordante and patterns :(
• Installation
• Installation from sources
• Troubleshooting
• Cite
• Contacts and Q&A

Tool Positioning

Why Desbordante? Why not just use an LLM or other black-box machine learning approaches?

Unlike these, Desbordante takes a symbolic, explainable approach to data analysis, producing explicit rules or rule-shaped artifacts that are verifiable, such as data dependencies, association rules, or other patterns. This is a distinct paradigm for data analysis that complements black-box learning. In contrast to free-form model outputs, discovered patterns have precisely defined semantics and can be algorithmically validated against the underlying data.

These patterns add data-grounded and rule-based structure, which provides useful properties for systems built around machine learning and large language models: consistency, provenance, explainability, and auditability. Further, the patterns can serve as inputs to downstream data-quality checks, validation procedures, integrity constraints, training-data analysis, and other rule-based safeguards. They can also be used to choose a direction of an ablation study, to generate examples that violate rules for use in machine learning model training and testing, to assist fact completion, and to facilitate acceptance checks and guardrails for pipelines involving retrieval-augmented generation. This is roughly the motivation used for the work in [1], but it applies more broadly to the patterns Desbordante works with.

[1] François Amat. Mining Rules on Tabular Data. Institut Polytechnique de Paris, 2025.

From other angles, the discovered patterns can be used in many ways:

• For scientific data, especially those obtained experimentally, an interesting pattern allows to formulate a hypothesis that could lead to a scientific discovery. In some cases it even allows one to draw conclusions immediately, if there is enough data. At the very least, the found pattern can be used to provide a direction for further study.

• For business data, it is also possible to obtain a hypothesis based on found patterns. However, there are more down-to-earth and more in-demand applications in this case: clearing errors in data, finding and removing inexact duplicates, performing schema matching, and many more.

• For database data, found patterns can help with recovering/defining primary and foreign keys, checking/setting up all kinds of integrity constraints.

Main features

Desbordante supports three types of tasks.

The Discovery task is designed to identify all instances of a specified pattern type of a given dataset.

The Validation task is different: it is designed to check whether a specified pattern instance is present in a given dataset. This task not only returns True or False, but it also explains why the instance does not hold (e.g. it can list table rows with conflicting values).

For some patterns Desbordante supports a dynamic task variant. The distinguishing feature of dynamic algorithms compared to classic (static) algorithms is that after a result is obtained, the table can be changed and a dynamic algorithm will update the result based just on those changes instead of processing the whole table again. As a result, they can be up to several orders of magnitude faster than classic (static) ones in some situations.

Desbordante support four data types:

• Tabular
• Graph
• Transactional
• Event sequences

Tabular data patterns

• Exact functional dependencies (discovery and validation)
• Approximate functional dependencies, with
  • $g_1$ metric — classic AFDs (discovery and validation)
  • $\mu+$ metric (discovery)
  • $\tau$ metric (discovery)
  • $pdep$ metric (discovery)
  • $\rho$ metric (discovery)
• Probabilistic functional dependencies, with PerTuple and PerValue metrics (discovery and validation)
• Classic soft functional dependencies (with correlations), with $\rho$ metric (discovery and validation)
• Dynamic validation of exact and approximate ($g_1$) functional dependencies
• Relaxed functional dependencies (discovery)
• Numerical dependencies (validation)
• Conditional functional dependencies (discovery and validation)
• Inclusion dependencies
  • Exact inclusion dependencies (discovery and validation)
  • Approximate inclusion dependencies, with $g^{'}_{3}$ metric (discovery and validation)
• Conditional inclusion dependencies (discovery and validation)
• Order dependencies:
  • set-based axiomatization (discovery and validation)
  • list-based axiomatization (discovery)
• Approximate order dependencies:
  • set-based axiomatization (discovery and validation)
• Sequential dependencies (validation)
• Metric functional dependencies (validation)
• Fuzzy algebraic constraints (discovery)
• Domain Probabilistic and Approximate Constraints (validation)
• Differential Dependencies (discovery and validation)
• Unique column combinations:
  • Exact unique column combination (discovery and validation)
  • Approximate unique column combination, with $g_1$ metric (discovery and validation)
• Numerical association rules (discovery)
• Matching dependencies (discovery and validation)
• Denial constraints
  • Exact denial constraints (discovery and validation)
  • Approximate denial constraints, with $g_1$ metric (discovery)

Graph data patterns

• Graph functional dependencies (discovery and validation)
• Graph differential dependencies (validation)
• Frequent subgraphs (discovery)

Transactional data patterns

• Association rules (discovery and validation)

Event sequence data patterns

• Frequent episode mining episode (discovery)
• Maximal frequent episode (discovery)
• Top-k frequent episode (discovery) This package uses the library of the Desbordante platform, which is written in C++. This means that depending on the algorithm and dataset, the runtimes may be cut by 2-10 times compared to the alternatives.

Usage examples

Desbordante features can be accessed from within Python programs by employing the Desbordante Python library. The library is implemented in the form of Python bindings to the interface of the Desbordante C++ core library, using pybind11. Apart from discovery and validation of patterns, this interface is capable of providing valuable additional information which can, for example, describe why a given pattern does not hold.

We want to demonstrate the power of Desbordante through examples where some patterns are extracted from tabular data, providing non-trivial insights. The patterns are quite complex and require detailed explanations, as well as a significant amount of code. This takes up quite a bit of space. Therefore, we do not include the actual code here; instead, we provide a clear (albeit simplified) definition and a link to a Colab notebook with interactive examples. The examples themselves are very detailed and allow users to understand the pattern and how to extract it using Desbordante.

1. Differential Dependencies (DD). DD is a statement of the form X -> Y, where X and Y are sets of attributes. It indicates that for any two rows, $t$ and $s$, if the attributes in $X$ are similar, then the attributes in $Y$ will also be similar. The similarity for each attribute is defined as: $diff(t[X_i], s[X_i]) \in [val_1, val_2]$, where $t[X_i]$ is the value of attribute $X_i$ in row $t$, $val$ is a constant, and $diff$ is a function that typically calculates the difference, often through simple subtraction. A live Python example that provides insight into the definition and demonstrates how to use this pattern in Desbordante is available here.
2. Numeric Association Rules (NAR). NAR is a statement of the form X -> Y, where X and Y are conditions, specified on disjoint sets of attributes. Each condition takes a form of $A_1 \wedge A_2 \wedge \ldots \wedge A_n$, where $A_i$ is either $Attribute_i \in$ $[constant_{i}^{1}; constant_{i}^{2}]$ or $Attribute_i$ = $constant_i^3$. Furthermore, the statement includes the support (sup) and confidence (conf) values, which lie in $[0; 1]$. The rule can be interpreted as follows: 1) the supp share of rows in the dataset satisfies both the X and Y conditions, and 2) the conf share of rows that satisfy the X also satisfies Y. A live Python example that provides insight into the definition and demonstrates how to use this pattern in Desbordante is available here.
3. Matching Dependencies (MD). MD is a statement of the form X -> Y, where X and Y are sets of so-called column matches. Each column match includes: 1) a metric (e.g., Levenshtein distance, Jaccard similarity, etc.), 2) a left column, and 3) a right column. Note that this pattern may involve two tables in its column matches. Finally, each match has its own threshold, which is applied to the corresponding metric and lies in the $[0; 1]$ range. The dependency can be interpreted as follows: any two records that satisfy X will also satisfy Y. A live Python example that provides insight into the definition and demonstrates how to use this pattern in Desbordante is available here.
4. Denial Constraints (DC). A denial constraint is a statement that says: "For all pairs of rows in a table, it should never happen that some condition is true". Formally, DC $\varphi$ is a conjunction of predicates of the following form: $\forall s,t \in R, s \neq t: \textlnot (p_1 \wedge \ldots \wedge p_m)$. Each $p_k$ has the form $column_i$ $op$ $column_j$, where $op \in {>, <, \leq, \geq, =, \neq}$. A live Python example that provides insight into the definition and demonstrates how to use this pattern in Desbordante is available here

Desbordante offers examples for each supported pattern, sometimes several if the pattern is complex or needs to highlight its unique characteristics compared to others in the same family. We have mentioned only a small portion here, which is available in Colab. The rest can be found in our example folder.

Finally, Desbordante allows end users to solve various data quality problems by constructing ad-hoc Python programs, incorporating different Python libraries, and utilizing the search and validation of various patterns. To demonstrate the power of this approach, we have implemented several demo scenarios:

1. Typo detection
2. Data deduplication
3. Anomaly detection

There is also an interactive demo for all of them, and all of these python scripts are here. The ideas behind them are briefly discussed in this preprint (Section 3).

I still don't understand how to use Desbordante and patterns :(

No worries! Desbordante offers a novel type of data profiling, which may require that you first familiarize yourself with its concepts and usage. The most challenging part of Desbordante are the primitives: their definitions and applications in practice. To help you get started, here’s a step-by-step guide:

1. First of all, we provide several examples for each supported pattern. These examples illustrate both the pattern itself and how to use it in Python. You can check them out here.
2. Each of our patterns was introduced in a research paper. These papers typically provide a formal definition of the pattern, examples of use, and its application scope. We recommend at least skimming through them. Don't be discouraged by the complexity of the papers! To effectively use the patterns, you only need to read the more accessible parts, such as the introduction and the example sections.
3. Finally, do not hesitate to ask questions in the mailing list (link below) or create an issue.

Papers about patterns

Here is a list of papers about patterns, organized in the recommended reading order in each item.

Tabular data patterns

• Exact functional dependencies
  • Thorsten Papenbrock et al. 2015. Functional dependency discovery: an experimental evaluation of seven algorithms. Proc. VLDB Endow. 8, 10 (June 2015), 1082–1093.
  • Thorsten Papenbrock and Felix Naumann. 2016. A Hybrid Approach to Functional Dependency Discovery. In Proceedings of the 2016 International Conference on Management of Data (SIGMOD '16). Association for Computing Machinery, New York, NY, USA, 821–833.
• Approximate functional dependencies ($g_{1}, \mu+, \tau, pdep, \rho$ metrics)
  • Marcel Parciak et al. 2024. Measuring Approximate Functional Dependencies: A Comparative Study. In Proceedings 2024 IEEE 40th International Conference on Data Engineering (ICDE), Utrecht, Netherlands, 2024, pp. 3505-3518
  • Sebastian Kruse and Felix Naumann. 2018. Efficient discovery of approximate dependencies. Proc. VLDB Endow. 11, 7 (March 2018), 759–772.
  • Yka Huhtala et al. 1999. TANE: An Efficient Algorithm for Discovering Functional and Approximate Dependencies. Comput. J. 42(2): 100-111
• Probabilistic functional dependencies ($PerTuple$ and $PerValue$ metrics)
  • Daisy Zhe Wang et al. Functional Dependency Generation and Applications in Pay-As-You-Go Data Integration Systems. WebDB 2009
  • Daisy Zhe Wang et al. Discovering Functional Dependencies in Pay-As-You-Go Data Integration Systems. Tech Rep. UCB/EECS-2009-119.
• Classic soft functional dependencies ($\rho$ metric)
  • Ihab F. Ilyas et al. 2004. CORDS: automatic discovery of correlations and soft functional dependencies. In Proceedings of the 2004 ACM SIGMOD international conference on Management of data (SIGMOD '04). Association for Computing Machinery, New York, NY, USA, 647–658.
• Relaxed functional dependencies
  • Loredana Caruccio, Vincenzo Deufemia, Giuseppe Polese: A genetic algorithm to discover relaxed functional dependencies from data. SEBD 2017: 146
• Numerical Dependencies
  • Paolo Ciaccia et al. 2013. Efficient derivation of numerical dependencies. Information Systems, Volume 38, Issue 3. Pages 410-429.
• Conditional functional dependencies
  • Rammelaere, J., Geerts, F. (2019). Revisiting Conditional Functional Dependency Discovery: Splitting the “C” from the “FD”. Machine Learning and Knowledge Discovery in Databases. ECML PKDD 2018.
• Exact and approximate inclusion dependencies
  • Falco Dürsch et al. 2019. Inclusion Dependency Discovery: An Experimental Evaluation of Thirteen Algorithms. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (CIKM '19). Association for Computing Machinery, New York, NY, USA, 219–228.
  • Sebastian Kruse, et al. Fast Approximate Discovery of Inclusion Dependencies. BTW 2017: 207-226
  • Marchi, F.D., Lopes, S. & Petit, JM. Unary and n-ary inclusion dependency discovery in relational databases. J Intell Inf Syst 32, 53–73 (2009)
• Conditional Inclusion Dependencies
  • Jana Bauckmann, Ziawasch Abedjan, Ulf Leser, Heiko Müller, and Felix Naumann. 2012. Discovering conditional inclusion dependencies. In Proceedings of the 21st ACM international conference on Information and knowledge management (CIKM '12). Association for Computing Machinery, New York, NY, USA, 2094–2098.
• Order dependencies:
  • Jaroslaw Szlichta et al. 2017. Effective and complete discovery of order dependencies via set-based axiomatization. Proc. VLDB Endow. 10, 7 (March 2017), 721–732.
  • Langer, P., Naumann, F. Efficient order dependency detection. The VLDB Journal 25, 223–241 (2016)
• Approximate order dependencies
  • R. Karegar, P. Godfrey, L. Golab, M. Kargar, D. Srivastava, J Szlichta Efficient Discovery of Approximate Order Dependencies. EDBT 2021: 427-432
• Sequential dependencies:
  • Lukasz Golab, Howard Karloff, Flip Korn, Avishek Saha, and Divesh Srivastava. 2009. Sequential dependencies. Proc. VLDB Endow. 2, 1 (August 2009), 574–585.
• Metric functional dependencies
  • N. Koudas et al. "Metric Functional Dependencies," 2009 IEEE 25th International Conference on Data Engineering, Shanghai, China, 2009, pp. 1275-1278.
• Fuzzy algebraic constraints
  • Paul G. Brown and Peter J. Hass. 2003. BHUNT: automatic discovery of Fuzzy algebraic constraints in relational data. In Proceedings of the 29th international conference on Very large data bases - Volume 29 (VLDB '03), Vol. 29. VLDB Endowment, 668–679.
• Domain Probabilistic and Approximate Constraints
  • Flip Korn, S. Muthukrishnan, and Yunyue Zhu. Checks and balances: monitoring data quality problems in network traffic databases. VLDB '03, Vol. 29. VLDB Endowment, 536–547.
• Differential dependencies
  • Shaoxu Song and Lei Chen. 2011. Differential dependencies: Reasoning and discovery. ACM Trans. Database Syst. 36, 3, Article 16 (August 2011), 41 pages.
• Exact and approximate unique column combinations
  • Sebastian Kruse and Felix Naumann. 2018. Efficient discovery of approximate dependencies. Proc. VLDB Endow. 11, 7 (March 2018), 759–772.
• Numerical association rules
  • Minakshi Kaushik, Rahul Sharma, Iztok Fister Jr., and Dirk Draheim. 2023. Numerical Association Rule Mining: A Systematic Literature Review. 1, 1 (July 2023), 50 pages.
  • Fister, Iztok & Fister jr, Iztok. 2020. uARMSolver: A framework for Association Rule Mining. 10.48550/arXiv.2010.10884.
• Matching dependencies
  • Philipp Schirmer, Thorsten Papenbrock, Ioannis Koumarelas, and Felix Naumann. 2020. Efficient Discovery of Matching Dependencies. ACM Trans. Database Syst. 45, 3, Article 13 (September 2020), 33 pages. https://doi.org/10.1145/3392778
• Denial constraints
  • X. Chu, I. F. Ilyas and P. Papotti. Holistic data cleaning: Putting violations into context. 2013. IEEE 29th International Conference on Data Engineering (ICDE), Brisbane, QLD, Australia, 2013, pp. 458-469,
  • Zifan Liu, Shaleen Deep, Anna Fariha, Fotis Psallidas, Ashish Tiwari, and Avrilia Floratou. 2024. Rapidash: Efficient Detection of Constraint Violations. Proc. VLDB Endow. 17, 8 (April 2024), 2009–2021.
  • Renjie Xiao, Zijing Tan, Haojin Wang, and Shuai Ma. 2022. Fast approximate denial constraint discovery. Proc. VLDB Endow. 16, 2 (October 2022), 269–281.
  • Meifan Zhang, Hongzhi Wang, Jianzhong Li, and Hong Gao, "One-Pass Inconsistency Detection Algorithms for Big Data," in IEEE Access, vol. 7, pp. 22377-22394, 2019

Graph data patterns

• Graph functional dependencies
  • Wenfei Fan, Yinghui Wu, and Jingbo Xu. 2016. Functional Dependencies for Graphs. In Proceedings of the 2016 International Conference on Management of Data (SIGMOD '16). Association for Computing Machinery, New York, NY, USA, 1843–1857.
  • Wenfei Fan, Chunming Hu, Xueli Liu, and Ping Lu. 2020. Discovering Graph Functional Dependencies. ACM Trans. Database Syst. 45, 3, Article 15 (September 2020), 42 pages.
• Graph differential dependencies
  • Zhang, Y., Kwashie, S., Bewong, M., Hu, J., Mahboubi, A., Guo, X., & Feng, Z. Discovering graph differential dependencies. Australasian Database Conference (ADC), 2023.
• Frequent sub-graphs
  • Xifeng Yan and Jiawei Han, "gSpan: graph-based substructure pattern mining," 2002 IEEE International Conference on Data Mining, 2002. Proceedings., Maebashi City, Japan, 2002, pp. 721-724.

Transactional data patterns

• Association rules
  • Charu C. Aggarwal, Jiawei Han. 2014. Frequent Pattern Mining. Springer Cham. pp 471.

Event sequence data patterns

• Frequent episode mining (all, maximal, and top-k)
  • H. Mannila, H. Toivonen, A. I. Verkamo. 1997. Discovery of Frequent Episodes in Event Sequences. Data Mining and Knowledge Discovery 1(3), 259-289.
  • P. Fournier-Viger, M. S. Nawaz, Y. He, Y. Wu, F. Nouioua, U. Yun. 2022. MaxFEM: Mining Maximal Frequent Episodes in Complex Event Sequences. MIWAI 2022, pp. 86-98.
  • P. Fournier-Viger, Y. Yang, P. Yang, J. C.-W. Lin, U. Yun. 2020. TKE: Mining Top-K Frequent Episodes. IEA/AIE 2020, pp. 832-845.

Installation

The source code is currently hosted on GitHub at https://github.com/Desbordante/desbordante-core

Wheels for all released version are available at the Python Package Index (PyPI) for manylinux_2_28 (Ubuntu 21.04+) and macOS 15.0+.

$ pip install desbordante

Installation from sources

Install all dependencies listed in README.md.

Then, in the Desbordante directory (the same one that contains this file), execute:

python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install .

Troubleshooting

No type hints in IDE

If type hints don't work for you in Visual Studio Code, for example, then install stubs using the command:

pip install desbordante-stubs

NOTE: Stubs may not fully support current version of desbordante package, as they are updated independently.

Cite

If you use this software for research, please cite our core paper:

@inproceedings{10.1145/3703323.3703725,
   author = {Chernishev, George and Polyntsov, Michael and Chizhov, Anton and Stupakov, Kirill and Shchuckin, Ilya and Smirnov, Alexander and Strutovsky, Maxim and Shlyonskikh, Alexey and Firsov, Mikhail and Manannikov, Stepan and Bobrov, Nikita and Goncharov, Daniil and Barutkin, Ilia and Yakshigulov, Vadim and Shalnev, Vladislav and Muraviev, Kirill and Rakhmukova, Anna and Shcheka, Dmitriy and Chernikov, Anton and Kuzin, Yakov and Sinelnikov, Michael and Abrosimov, Grigorii and Popov, Dmitriy and Demchenko, Artem and Belokonny, Sergey and Soloveva, Liana-Iuliia and Kurbatov, Yaroslav and Vyrodov, Mikhail and Saliou, Arthur and Gaisin, Eduard and Smirnov, Kirill},
   title = {Desbordante: from benchmarking suite to high-performance science-intensive data profiler},
   year = {2025},
   isbn = {9798400711244},
   publisher = {Association for Computing Machinery},
   address = {New York, NY, USA},
   url = {https://doi.org/10.1145/3703323.3703725},
   doi = {10.1145/3703323.3703725},
   booktitle = {Proceedings of the 8th International Conference on Data Science and Management of Data (12th ACM IKDD CODS and 30th COMAD)},
   pages = {234--243},
   numpages = {10},
   keywords = {Data Mining, Data Profiling, Pattern Extraction, Data Analysis, Knowledge Discovery, Data Exploration, Anomaly Detection, Data Wrangling},
   location = {},
   series = {CODS-COMAD '24}
}

or cite one of our papers, if you use a particular part:

1. George Chernishev, et al. Solving Data Quality Problems with Desbordante: a Demo. CoRR abs/2307.14935 (2023).
2. M. Strutovskiy, N. Bobrov, K. Smirnov and G. Chernishev, "Desbordante: a Framework for Exploring Limits of Dependency Discovery Algorithms," 2021 29th Conference of Open Innovations Association (FRUCT), 2021, pp. 344-354.
3. A. Smirnov, A. Chizhov, I. Shchuckin, N. Bobrov and G. Chernishev, "Fast Discovery of Inclusion Dependencies with Desbordante," 2023 33rd Conference of Open Innovations Association (FRUCT), Zilina, Slovakia, 2023, pp. 264-275.
4. A. Chernikov, Y. Litvinov, K. Smirnov, and G. Chernishev, "FastGFDs: Efficient Validation of Graph Functional Dependencies with Desbordante," 2023 33rd Conference of Open Innovations Association (FRUCT), Zilina, Slovakia, 2023, Issue 2 (Works in Progress), pp. 346-352.
5. Y. Kuzin, D. Shcheka, M. Polyntsov, K. Stupakov, M. Firsov and G. Chernishev, "Order in Desbordante: Techniques for Efficient Implementation of Order Dependency Discovery Algorithms," 2024 35th Conference of Open Innovations Association (FRUCT), Tampere, Finland, 2024, pp. 413-424.
6. I. Barutkin, M. Fofanov, S. Belokonny, V. Makeev and G. Chernishev, "Extending Desbordante with Probabilistic Functional Dependency Discovery Support," 2024 35th Conference of Open Innovations Association (FRUCT), Tampere, Finland, 2024, pp. 158-169.
7. A. Shlyonskikh, M. Sinelnikov, D. Nikolaev, Y. Litvinov and G. Chernishev, "Lightning Fast Matching Dependency Discovery with Desbordante," 2024 36th Conference of Open Innovations Association (FRUCT), Lappeenranta, Finland, 2024, pp. 729-740.
8. M. Ivanov, M. Smirnov, A. Strazdina and G. Chernishev, "Scalable Maximal Frequent Episode Mining with Desbordante," 2026 39th Conference of Open Innovations Association (FRUCT), Helsinki, Finland, 2026, pp. 102-113.
9. I. Kozhukov et al., "Efficient Discovery of Conditional Dependencies with Desbordante," 2026 39th Conference of Open Innovations Association (FRUCT), Helsinki, Finland, 2026, pp. 130-141.

Contacts and Q&A

If you have any questions regarding the tool usage you can ask it in our google group. To contact dev team email George Chernishev, Alexey Shlyonskikh or Michael Polyntsov.