PySentence-Similarity 😊

Information

PySentence-Similarity is a tool designed to identify and find similarities between sentences and a base sentence, expressed as a percentage 📊. It compares the semantic value of each input sentence to the base sentence, providing a score that reflects how related or similar they are. This tool is useful for various natural language processing tasks such as clustering similar texts 📚, paraphrase detection 🔍 and textual consequence measurement 📈.

The models were converted to ONNX format to optimize and speed up inference. Converting models to ONNX enables cross-platform compatibility and optimized hardware acceleration, making it more efficient for large-scale or real-world applications 🚀.

• High accuracy: Utilizes a robust Transformer-based architecture, providing high accuracy in semantic similarity calculations 🔬.
• Cross-platform support: The ONNX format provides seamless integration across platforms, making it easy to deploy across environments 🌐.
• Scalability: Efficient processing can handle large datasets, making it suitable for enterprise-level applications 📈.
• Real-time processing: Optimized for fast output, it can be used in real-world applications without significant latency ⏱️.
• Flexible: Easily adaptable to specific use cases through customization or integration with additional models or features 🛠️.
• Low resource consumption: The model is designed to operate efficiently, reducing memory and CPU/GPU requirements, making it ideal for resource-constrained environments ⚡.
• Fast and user-friendly: The library offers high performance and an intuitive interface, allowing users to quickly and easily integrate it into their projects 🚀.

Installation 📦

• Requirements: Python 3.8 or higher.

# install from PyPI
pip install pysentence-similarity

# install from GitHub
pip install git+https://github.com/goldpulpy/pysentence-similarity.git

Support models 🤝

You don't need to download anything; the package itself will download the model and its tokenizer from a special HF repository.

Below are the models currently added to the special repository, including their file size and a link to the source.

PySentence-Similarity supports FP32, FP16, and INT8 dtypes.

• FP32: 32-bit floating-point format that provides high precision and a wide range of values.
• FP16: 16-bit floating-point format, reducing memory consumption and computation time, with minimal loss of precision (typically less than 1%).
• INT8: 8-bit integer quantized format that greatly reduces model size and speeds up output, ideal for resource-constrained environments, with little loss of precision.

Usage examples 📖

Compute similarity score 📊

Let's define the similarity score as the percentage of how similar the sentences are to the original sentence (0.75 = 75%), default compute function is cosine

You can use CUDA 12.X by passing the device='cuda' parameter to the Model object; the default is cpu. If the device is not available, it will automatically be set to cpu.

from pysentence_similarity import Model
from pysentence_similarity.utils import compute_score

# Create an instance of the model all-MiniLM-L6-v2; the default dtype is `fp32`
model = Model("all-MiniLM-L6-v2", dtype="fp16")

sentences = [
    "This is another test.",
    "This is yet another test.",
    "We are testing sentence similarity."
]

# Convert sentences to embeddings
# The default is to use mean_pooling as a pooling function
source_embedding = model.encode("This is a test.")
embeddings = model.encode(sentences, progress_bar=True)

# Compute similarity scores
# The rounding parameter allows us to round our float values
# with a default of 2, which means 2 decimal places.
compute_score(source_embedding, embeddings)
# Return: [0.86, 0.77, 0.48]

compute_score returns in the same index order in which the embedding was encoded.

Let's see the sentence and its evaluation from a computational function

# Compute similarity scores
scores = compute_score(source_embedding, embeddings)

for sentence, score in zip(sentences, scores):
    print(f"{sentence} ({score})")

# Output prints:
# This is another test. (0.86)
# This is yet another test. (0.77)
# We are testing sentence similarity. (0.48)

You can use the computational functions: cosine, euclidean, manhattan, jaccard, pearson, minkowski, hamming, kl_divergence, chebyshev, bregman or your custom function

from pysentence_similarity.compute import euclidean

compute_score(source_embedding, embeddings, compute_function=euclidean)
# Return: [2.52, 3.28, 5.62]

You can use max_pooling, mean_pooling, min_pooling or your custom function

from pysentence_similarity.pooling import max_pooling

source_embedding = model.encode("This is a test.", pooling_function=max_pooling)
embeddings = model.encode(sentences, pooling_function=max_pooling)
...

Search similar sentences 🔍

from pysentence_similarity import Model
from pysentence_similarity.utils import search_similar

# Create an instance of the model
model = Model("all-MiniLM-L6-v2", dtype="fp16")

# Test text
sentences = [
    "Hello my name is Bob.",
    "I love to eat pizza.",
    "We are testing sentence similarity."
    "Today is a sunny day.",
    "London is the capital of England.",
    "I am a student at Stanford University."
]

# Convert query sentence to embedding
query_embedding = model.encode("What's the capital of England?")

# Convert sentences to embeddings
embeddings = model.encode(sentences)

# Search similar sentences
similar = search_similar(
    query_embedding=query_embedding,
    sentences=sentences,
    embeddings=embeddings,
    top_k=3  # number of similar sentences to return
)

# Print similar sentences
for idx, (sentence, score) in enumerate(similar, start=1):
    print(f"{idx}: {sentence} ({score})")

# Output prints:
# 1: London is the capital of England. (0.81)
# 2: Hello my name is Bob. (0.06)
# 3: I love to eat pizza. (0.05)

With use storage

from pysentence_similarity import Model, Storage
from pysentence_similarity.utils import search_similar

model = Model("all-MiniLM-L6-v2", dtype="fp16")
query_embedding = model.encode("What's the capital of England?")

storage = Storage.load("my_storage.h5")

similar = search_similar(
    query_embedding=query_embedding,
    storage=storage,
    top_k=3
)
...

Splitting ✂️

from pysentence_similarity import Splitter

# Default split markers: '\n'
splitter = Splitter()

# If you want to separate by specific characters.
splitter = Splitter(markers_to_split=["!", "?", "."], preserve_markers=True)

# Test text
text = "Hello world! How are you? I'm fine."

# Split from text
splitter.split_from_text(text)
# Return: ['Hello world!', 'How are you?', "I'm fine."]

At this point, sources for the splitting are available: text, file, URL, CSV, and JSON.

Storage 💾

The storage allows you to save and link sentences and their embeddings for easy access, so you don't need to encode a large corpus of text every time. The storage also enables similarity searching.

The storage must store the sentences themselves and their embeddings.

from pysentence_similarity import Model, Storage

# Create an instance of the model
model = Model("all-MiniLM-L6-v2", dtype="fp16")

# Create an instance of the storage
storage = Storage()
sentences = [
    "This is another test.",
    "This is yet another test.",
    "We are testing sentence similarity."
]

# Convert sentences to embeddings
embeddings = model.encode(sentences)

# Add sentences and their embeddings
storage.add(sentences, embeddings)

# Save the storage
storage.save("my_storage.h5")

Load from the storage

from pysentence_similarity import Model, Storage
from pysentence_similarity.utils import compute_score

# Create an instance of the model and storage
model = Model("all-MiniLM-L6-v2", dtype="fp16")
storage = Storage.load("my_storage.h5")

# Convert sentence to embedding
source_embedding = model.encode("This is a test.")

# Compute similarity scores with the storage
compute_score(source_embedding, storage)
# Return: [0.86, 0.77, 0.48]

License 📜

This project is licensed under the MIT License. See the LICENSE file for details

Created by goldpulpy with ❤️