hyperreal 0.5.0

Hyperreal is a library and tool for intepretive topic modelling.

Hyperreal

Hyperreal is a Python tool for interactive qualitative analysis of large collections of documents.

Requirements

Hyperreal requires the installation of the Python programming language.

Installation

Hyperreal can be installed using Pip from the command line ( Windows, Mac) by running the following commands:

python -m pip install hyperreal

Usage

Hyperreal can be used in three different ways to flexibly support different use cases:

• as a command line application
• as a Python library
• as a local web application

All of hyperreal's functionality is available from the Python library, but you will need to write Python code to use it directly. The command line interface allows for quick and repeatable experimentation and automation for standard data types - for example if you often work with Twitter data the command line will allow you to rapidly work with many different Twitter data collections. The web application is currently focused solely on creating and interactive editing of models.

Command Line

The following script gives a basic example of using the command line interface for hyperreal. This will work for cases where you have a plain text file (here called corpus.txt), with each document in the collection on its own line.

If you haven't worked with the command line before, you might find the following resources useful:

• Software Carpentry resources for Mac
• Open Water Foundation resources for Windows

# Create a corpus database from a plaintext file
hyperreal plaintext-corpus create corpus.txt corpus.db

# Create an index from the corpus
hyperreal plaintext-corpus index corpus.db corpus_index.db

# Create a model from that index, in this case with 128 clusters and
# only include features present in 10 or more documents.
hyperreal model corpus_index.db --min-docs 10 --clusters 128

# Use the web interface to serve the results of that modelling
# After running this command point your web browser to http://localhost:8080
hyperreal plaintext-corpus serve corpus.db corpus_index.db

Library

This example script performs the same steps as the command line example.

from hyperreal import corpus, index

# create and populate the corpus with some documents
c = corpus.PlainTextSqliteCorpus('corpus.db')

with open('corpus.txt', 'r') as f:
  # This will drop any line that has no text (such as a paragraph break)
  docs = (line for line in f if line.strip())
  c.replace_docs(docs)


# Index that corpus - note that we need to pass the corpus object for
# initialisation.
idx = index.Index('corpus_index.db', corpus=c)
# This only needs to be done once, unless the corpus changes.
idx.index()

# Create a model on this index, with 128 clusters and only including features
# that match at least 10 documents.
idx.initialise_clusters(n_clusters=128, min_docs=10)
# Refine the model for 10 iterations. Note that you can continue to refine
# the model without initialising the clusters.
idx.refine_clusters(iterations=10)

# Inspect the output of the model using the index instance (currently quite
# limited). This will print the top 10 most frequent features in each
# cluster.
for cluster_id in idx.cluster_ids:
    cluster_features = idx.cluster_features(cluster_id)
    for feature in cluster_features[:10]:
        print(feature)

# Perform a boolean query on the corpus, looking for documents that contain
# both apples AND oranges in the text field.
q = i[('text', 'apples')] & i[('text', 'oranges')]
# Lookup all of the documents in the corpus that match this query.
docs = idx.get_docs(q)

# 'Pivot' the features in the index with respect to all cluster in the model.
#  This will show the top 10 features in each cluster that are similar to the
#  query.
for cluster_detail in idx.pivot_clusters_by_query(query, top_k=10):
    print(cluster_detail)

# This will show the top 10 features for a selected set of cluster_ids.
for cluster_detail in idx.pivot_clusters_by_query(query, cluster_ids=[3,5,7], top_k=10):
    print(cluster_detail)

Development

Installation

To install the development version:

1. Clone the repository using git.

2. From the cloned repository, use pip for an editable install:

   pip install -e .

Running Tests

The full test suite and other checks are orchestrated via tox:

python -m pip install -e .[test]

# To run just the testsuite
pytest

# To run everything, including code formatting via black and check coverage
tox