mergic 0.0.1

With the mergic.Blender:

The distance calculation, cutoff evaluation, and partition creation are currently all in mergic.py make:

# see all the possible partitions by their statistics
./mergic.py make names_all.txt

# make a partition using a cutoff of 0.303
./mergic.py make 0.303 > partition.json

Edit the partition until it’s good. Save it as partition_edited.json.

You can check that your partition is valid and see a cute summary:

./mergic.py check partition_edited.json
# 669 items in 354 groups

You could proceed directly, but there are also diffing tools! Generate a diff:

./mergic.py diff partition.json partition_edited.json > partition_diff.json

You can apply a diff to reconstruct an edited version:

./mergic.py apply partition.json partition_diff.json > partition_rebuilt.json

Now if you mergic.py diff the files partition_edited.json and partition_rebuilt.json the result should just be {} (no difference).

To generate a CSV merge table that you’ll be able to use with any other tool:

./mergic.py table partition_edited.json > partition.csv

Now the file partition.csv has two columns, original and mergic, where original contains all the values that appeared in the original data and mergic contains the deduplicated keys. You can join this on to your original data and go to town.

# pip install python-Levenshtein
import Levenshtein
Levenshtein.distance("fuzzy", "wuzzy")
# 1

# pip install fuzzywuzzy
from fuzzywuzzy import fuzz
100 - fuzz.ratio("Levensthein", "Leviathan")
# 50