Corpus-Show
Corpus-Show helps to understand the corpus data distribution through various values generated from Sentence Transformer. (It's not such a great package, but It simply helps you visualize comfortably.)
- Corpus-Show performs sentence embedding via Sentence Transformers, a Python framework for state-of-the-art sentence, text and image embeddings. [Paper] [Document] [Huggingface model]
- You can visualize the embedded sentences of each document generated from SentenceTransformers.
- Corpus-Show can also generate clusters with sentences embedded array through Scikit-Learn KMeans.
- The sentence transformer model is downloaded through the hugging face interface, and the default model is set to
paraphrase-xlm-r-multilingual-v1, which supports multiple languages. However, you can easily input your custom model as a sentence transformer model through the hugging face interface. It is also easy to fine-tune via SBERT. For more models, please see this page.
Installation
pip install corpusshow
- This package may not work properly in m1/m2 mac environment. If you are using Mac m1/m2, please use the git repository as a submodule because it has minimal encapsulation. [issue#1]
Tutorial
We provide tutorial notebooks for all the features we offer. We plan to provide additional docstrings or documentation from the official distribution version (major version 1 or higher).
- Main-tutorials: https://github.com/DSDanielPark/corpus-show/blob/main/tutorials/corpusshow_tutorial.ipynb
- Sub-tutorial-folder: https://github.com/DSDanielPark/corpus-show/blob/main/tutorials
Main Feature
It helps to create a simple but useful plot as shown below with a simple dataframe and column names as input, such as the following BBC sample dataset in ./data/bbc_news_dataset.csv.
| news | topic | |
|---|---|---|
| 0 | Oil rebounds from weather effect (...) | business |
| 1 | Indonesia 'declines debt freeze' (...) | business |
| ... | ... | ... |
| 601 | EU software patent law faces axe (...) | tech |
1. CorpusClster
Contains 1 static method. You can create great pictures with:
from corpusshow import CorpusCluster
# Class arguments
csv_file_path = '../data/bbc_news_dataset.csv'
sentence_transformer_model_name = 'paraphrase-xlm-r-multilingual-v1'
target_col = 'news'
num_cluster = 4
# Get class object
cc = CorpusCluster(csv_file_path, sentence_transformer_model_name, target_col, num_cluster)
# 1. quick_corpus_show method:
# Show figures without k-means clustering
cc.quick_corpus_show('topic', 'tsne2d', False, 'fig1.png')
cc.quick_corpus_show('topic', 'tsne3d', False, 'fig2.png')
cc.quick_corpus_show('topic', 'pca2d', False, 'fig3.png')
cc.quick_corpus_show('topic', 'pca3d', False, 'fig4.png')
# 2. quick_cluster_show method:
# Show figures with k-means clustering
df_returned = cc.quick_cluster_show('tsne2d', False, 'fig5.png')
df_returned = cc.quick_cluster_show('tsne3d', False, 'fig6.png')
df_returned = cc.quick_cluster_show('pcda2d', False, 'fig7.png')
df_returned = cc.quick_cluster_show('pcda2d', False, 'fig8.png')
- If you want to change the design of the plot, use matplotlib's RcParams method or the returned dataframe.
References
[1] Scikit-Learn https://scikit-learn.org
[2] Matplotlib https://matplotlib.org/
[3] Huggingface Sentence Transformer https://huggingface.co/sentence-transformers
[4] SBERT https://www.sbert.net/
Use Case
[1] Korean-news-topic-classification-using-KO-BERT: all plots were created through Corpus-Show and Quick-Show.
Contacts
Maintainer: Daniel Park, South Korea e-mail parkminwoo1991@gmail.com
Metadata
Release files for corpusshow 0.1.8
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| corpusshow-0.1.8.tar.gz | 10.0 kB | Without provenance | Details for corpusshow-0.1.8.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| corpusshow-0.1.8-py3-none-any.whl | Python 3 | none | any | Details for corpusshow-0.1.8-py3-none-any.whl |
Total release size: 20.7 kB
Release files / corpusshow-0.1.8.tar.gz
| Download URL | corpusshow-0.1.8.tar.gz |
|---|---|
| Size | 10.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
22b67263c75fc37489da1a6947da8c10a9d55a3b2c642a6fa1cec8ff6c75b79c
|
|
BLAKE2b-256 checksum How to use checksums |
202e09c03f9d64b4a62e7f55f0a08410032b1e5604e3d3e527a8eef2d99f426a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.9.12
|
Release files / corpusshow-0.1.8-py3-none-any.whl
| Download URL | corpusshow-0.1.8-py3-none-any.whl |
|---|---|
| Size | 10.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
86cff99dc7eaf095a66af247dc8a839d1c4c665377a22175007036e39ccc4fd4
|
|
BLAKE2b-256 checksum How to use checksums |
4cf6e1bbe6f983946a20a42d260314df62fcc10090af5a5cae32a9e8246301cc
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.9.12
|
Security for corpusshow 0.1.8
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.