Latent Semantic Analysis
Pipeline for training LSA models using Scikit-Learn.
Usage
Instead of writing custom code for latent semantic analysis, you just need:
- install pipeline:
pip install latent-semantic-analysis
- run pipeline:
- either in terminal:
lsa-train --path_to_config config.yaml
- or in python:
import latent_semantic_analysis
latent_semantic_analysis.train(path_to_config="config.yaml")
NOTE: more about config file here.
No data preparation is needed, only a csv file with raw text column (with arbitrary name).
Config
The user interface consists of only one files:
- config.yaml - general configuration with sklearn TF-IDF and SVD parameters
Change config.yaml to create the desired configuration and train LSA model with the following command:
- terminal:
lsa-train --path_to_config config.yaml
- python:
import latent_semantic_analysis
latent_semantic_analysis.train(path_to_config="config.yaml")
Default config.yaml:
seed: 42
path_to_save_folder: models
# data
data:
data_path: data/data.csv
sep: ','
text_column: text
# tf-idf
tf-idf:
lowercase: true
ngram_range: (1, 1)
max_df: 1.0
min_df: 1
# svd
svd:
n_components: 10
algorithm: arpack
NOTE: tf-idf and svd are sklearn TfidfVectorizer and TruncatedSVD parameters correspondingly, so you can parameterize instances of these classes however you want.
Output
After training the model, the pipeline will return the following files:
model.joblib- sklearn pipeline with LSA (TF-IDF and SVD steps)config.yaml- config that was used to train the modellogging.txt- logging filedoc2topic.json- document embeddingsterm2topic.json- term embeddings
Requirements
Python >= 3.6
Citation
If you use latent-semantic-analysis in a scientific publication, we would appreciate references to the following BibTex entry:
@misc{dayyass2021lsa,
author = {El-Ayyass, Dani},
title = {Pipeline for training LSA models},
howpublished = {\url{https://github.com/dayyass/latent-semantic-analysis}},
year = {2021}
}
Metadata
Release files for latent-semantic-analysis 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| latent-semantic-analysis-0.1.0.tar.gz | 6.9 kB | Without provenance | Details for latent-semantic-analysis-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| latent_semantic_analysis-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for latent_semantic_analysis-0.1.0-py3-none-any.whl |
Total release size: 17.0 kB
Release files / latent-semantic-analysis-0.1.0.tar.gz
| Download URL | latent-semantic-analysis-0.1.0.tar.gz |
|---|---|
| Size | 6.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c7fda93d959bb33fe6ef1814da4f33ff898687a603218617370a35adae6373e4
|
|
BLAKE2b-256 checksum How to use checksums |
24060548a0ebae5d68812e3dba2396aa26967ae400e9a8dd123dd87db5fd245d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.6.3
|
Release files / latent_semantic_analysis-0.1.0-py3-none-any.whl
| Download URL | latent_semantic_analysis-0.1.0-py3-none-any.whl |
|---|---|
| Size | 10.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
6951542da4b005ced22dde0f8611fbdb59a9ebe4cc773a73bf7ed322f401c64c
|
|
BLAKE2b-256 checksum How to use checksums |
28fb6bc524b2d2abc2ba57039e8ce02fd8f40845c0f242c9ef67591b0b67c1bc
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.6.3
|
Security for latent-semantic-analysis 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.