pyjedai 0.0.7

An open-source library that builds powerful end-to-end Entity Resolution workflows.

An open-source library that leverages Python’s data science ecosystem to build
powerful end-to-end Entity Resolution workflows.

---

Overview

pyJedAI is a python framework, aiming to offer experts and novice users, robust and fast solutions for multiple types of Entity Resolution problems. It is builded using state-of-the-art python frameworks. pyJedAI constitutes the sole open-source Link Discovery tool that is capable of exploiting the latest breakthroughs in Deep Learning and NLP techniques, which are publicly available through the Python data science ecosystem. This applies to both blocking and matching, thus ensuring high time efficiency, high scalability as well as high effectiveness, without requiring any labelled instances from the user.

Key-Features

• Input data-type independent. Both structured and semi-structured data can be processed.
• Various implemented algorithms.
• Easy-to-use.
• Utilizes some of the famous and cutting-edge machine learning packages.
• Offers supervised and un-supervised ML techniques.

Open demos are available in:

       

Google Colab Hands-on demo:

Details on the Architecture

The purpose of this framework is to demonstrate how ER can be accomplished by expert and novice users in an intuitive, yet efficient and effective way. pyJedai addresses the following task: Given a source and a target dataset, S and T, respectively, discover the set of links L = {(s,owl:sameAS, t)|s ∈ S ∧ t ∈ T}. Its architecture appears in the bellow figure. The first module is the data reader, which specifies the user input. pyJedAI supports both semi-structured and structured data as input. The former, which include SPARQL endpoints and RDF/OWL dumps, are read by RDFLib. The latter, which include relational databases as well as CSV and JSON files, are read by pandas. In this way, pyJedAI is able to interlink any combination of semi-structured and structured data sources, which is a unique feature. The second step in pyJedAI’s pipeline performs block building, a coarsegrained process that clusters together similar entities. The end result consists of a set of candidate pairs, which are examined analytically by the subsequent steps. pyJedAI implements the same established methods for similarity joins and blocking as JedAI, such as Standard Blocking and Sorted Neighborhood, but goes beyond all Link Discovery tools by incorporating recent, state-of-the-art libraries for nearest neighbor search like FALCONN and FAISS.

The entity matching step estimates the actual similarity between the candidate pairs. Unlike all other Link Discovery tools, which rely exclusively on string similarity measures like edit distance and Jaccard coefficient, pyJedAI leverages the latest advanced NLP techniques, like pre-trained embeddings (e.g., word2vect, fastText and Glove) and transformer language models (i.e., BERT and its variants). More specifically, pyJedAI supports packages like strsimpy, Gensimand Hugging Face. This unique feature boosts pyJedAI’s accuracy to a significant extent, without requiring any labelled instances from the user. The last step performs entity clustering to further increase the accuracy. The relevant techniques consider the global information provided by the similarity scores of all candidate pairs in order to take local decisions for each pair of entity descriptions. pyJedAI implements and offers the same established algorithms as JedAI, using NetworkX to ensure high time efficiency. Finally, users are able to evaluate, visualize and store the results of the selected pipeline through the intuitive interface of Jupyter notebooks. In this way, pyJedAI facilitates its use by researchers and practitioners that are familiar with the data science ecosystem, regardless of their familiarity with ER and Link Discovery, in general.

Install

Install the latest version of pyjedai [requires python >= 3.8]:

pip install pyjedai

More on PyPI.

Tutorials

Tutorial	Notebook
Clean-Clean Entity Resolution.	CleanCleanER.ipynb
Dirty Entity Resolution.	DirtyER.ipynb
Fine-Tuning using Optuna.	Optuna.ipynb
User-Friendly Approach. WorkFlow module.	WorkFlow.ipynb
Raw data to pandas DataFrame.	Readers.ipynb

Dependencies

         


           

See the full list of dependencies and all versions used, in this file.

Status

Bugs, Discussions & News

GitHub Discussions is the discussion forum for general questions and discussions and our recommended starting point. Please report any bugs that you find here.

Java - Wed Application

For Java users checkout the initial JedAI. There you can find Java based code and a Web Application for interactive creation of ER workflows.

Team & Authors

• Konstantinos Nikoletos
• Jakub Maciejewski
• George Papadakis
• Manolis Koubarakis

Research and development is made under the supervision of Pr. Manolis Koubarakis. This is a research project by the AI-Team of the Department of Informatics and Telecommunications at the University of Athens.

License

Released under the Apache-2.0 license (see LICENSE.txt).

Copyright © 2023 AI-Team, University of Athens

---

       

This project is being funded in the context of STELAR that is an HORIZON-Europe project.