Skip to main content

CLI for semantic string similarity

Project description

Semantic String Similarity CLI

simil is a CLI interface to spacy's string similarity engine. It uses the en_vectors_web_lg dataset to compare strings for their English semantic similarity. Given two words, phrases, or sentences, simil will tell you how similar their meanings are.

Installation

First install simil itself:

$ pip3 install --user -U simil

Now install one of spacy's web_vector models:

$ python3 -m spacy download en_vectors_web_lg

You can choose between en_vectors_web_lg, en_core_web_lg, and en_core_web_md, (en_core_web_sm don't include word vectors at all, and can't be used with simil.) simil will use the largest model that you have installed, with preference for the vectors model over a core model.

I suggest using the large vectors model (en_vectors_web_lg), but you might want to use a smaller model in order to save on disk space or memory usage.

Usage:

$ sim first_file.txt second_file.txt # compare two files
$ sim -s "first string" "second string" # compare two strings

The output is a number between 0 and 1, representing how similar the two strings are.

Details:

simil uses Spacy's word vector models trained with GLoVe, such as en_vectors_web_lg.

This can be a large dataset, which makes for long startup times. So simil spins off a process in the background to hold the model, and works under a client-server model with it. This means that if you run simil a number of times in a row, only the first run is slow.

This background process does take up a fair bit of memory, typically around 2GB (for the en_vectors_web_lg model). After 10 minutes of inactivity it will automatically be killed, in order not to take up memory indefinitely. You can change the length of this timeout with the --timeout flag.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

simil-0.0.2.tar.gz (4.4 kB view hashes)

Uploaded Source

Built Distribution

simil-0.0.2-py3-none-any.whl (17.9 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page