With AuDoLab you can perform Latend Direchlet Allocation on highly imbalanced datasets.
Project description
AuDoLab
With AuDoLab you can perform Latend Direchlet Allocation on highly imbalanced datasets.
Installation
Stable release
To install AuDoLab, run this command in your terminal:
$ pip install AuDoLab
This is the preferred method to install AuDoLab, as it will always install the most recent stable release.
If you don’t have pip installed, this Python installation guide can guide you through the process.
From sources
The sources for AuDoLab can be downloaded from the Github repo.
You can either clone the public repository:
$ git clone git://github.com/ArneTillmann/AuDoLab
Or download the tarball:
$ curl -OJL https://github.com/ArneTillmann/AuDoLab/tarball/master
Once you have a copy of the source, you can install it with:
$ python setup.py install
Usage
Before the actuall usage you want to download the stopwords for nltk by running:
import nltk nltk.download('stopwords')
inside a python console. To use AuDoLab in a project:
from AuDoLab import AuDoLab import asyncio import nest_asyncio nest_asyncio.apply() from numpy import round as np_round from numpy import arange as np_arange
Then you want to create an instance of the AuDoLab class
audo = AuDoLab.AuDoLab()
In this example we used publicly available data from the nltk package:
from nltk.corpus import reuters import numpy as np import pandas as pd data = [] for fileid in reuters.fileids(): tag, filename = fileid.split("/") data.append( (filename, ", ".join( reuters.categories(fileid)), reuters.raw(fileid))) data = pd.DataFrame(data, columns=["filename", "categories", "text"])
Then you want to scrape abstracts, e.g. from IEEE with the abstract scraper:
async def scrape(): return await audo.scrape_abstracts( url=None, keywords=["cotton"], in_data="all_meta", pages=5 ) scraped_documents = asyncio.get_event_loop().run_until_complete(scrape())
The data as well as the scraped papers need to be preprocessed before use in the classifier:
preprocessed_target = audo.preprocessing(data=data, column="text") preprocessed_paper = audo.preprocessing( data=scraped_documents, column="text") target_tfidf, training_tfidf = audo.tf_idf( data=preprocessed_target, papers=preprocessed_paper, data_column="lemma", papers_column="lemma", features=100000, )
Afterwards we can train and use the classifiers and choose the desired one:
classifier = audo.one_class_svm( training=training_tfidf, predicting=target_tfidf, nus=np.round(np.arange(0.01, 0.5, 0.01), 7), quality_train=0.9, min_pred=0.001, max_pred=0.05, ) df_data = audo.choose_classifier(preprocessed_target, classifier, 2)
And finally you can estimate the topics of the data:
audo.lda_modeling(df_data, num_topics=2) a = audo.lda_visualize_topics() html = a.data with open('html_file.html', 'w') as f: f.write(html)
Free software: GNU General Public License v3
Documentation: https://AuDoLab.readthedocs.io.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for AuDoLab-0.1.13-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 787111e7fd643233a383b69322b1697c6c7b2fa18c8042c2ec02e0fa76e1d0cd |
|
MD5 | afb2cb0bd30ae822d9f2d175069669b8 |
|
BLAKE2b-256 | db76b7939dbb9f6fdb88e57f6a661ebfe7d0b6f0380c00f238f8eb88d1ca0d3b |