csv-reconcile-fingerprint

A scoring plugin for csv-reconcile using fingerprint clustering. It generates a fingerprint of the input string by normalizing, removing punctuation, and sorting unique tokens. Based on the OpenRefine clustering implementation https://openrefine.org/docs/technical-reference/clustering-in-depth and code from this gist by @pietz.

The resulting strings are compared with Jaccard distance to output a score between 0 and 100.

Installation and Usage

Install this library using pip:

pip install csv-reconcile

This a plugin to the csv reconciliation plugin. So you just have to install csv reconcile package and specify the scorer with '--scorer fingerprint' when initiating the reconciliation service.

Development

To contribute to this library, first checkout the code. Then create a new virtual environment:

cd csv-reconcile-fingerprint
python -m venv venv
source venv/bin/activate

Now install the dependencies and test dependencies:

python -m pip install -e '.[test]'

To run the tests:

python -m pytest