Skip to main content

Explicit Semantic Analysis

Project description

ESA-Wiki

Explicit Semantic Analysis based on Wikipedia

This is a python library which contains code to 1) construct a semantic interpreter based on data from Wikipedia and 2) apply this to various kinds of texts.

To construct an interpreter, first obtain a Wikipedia XML dump from http://dumps.wikimedia.org/enwiki/

  1. Then run python3 -m esa_wiki.xml_parse <file> with the downloaded file as its argument. This outputs some temporary files containing information on the words, links and articles encountered.

  2. Next, run python3 -m esa_wiki.generate_indices to generate lists of indices corresponding to unique words and articles encountered

  3. Finally, run python3 -m esa_wiki.matrix_builder to construct a very large sparse interpretation matrix. Each row corresponds to a unique word, each column to a 'concept', i.e. a Wikipedia article, and each entry is the TF-IDF score for word i in article j. The Matrix is saved in separate chunks to conserve memory.

medium_wiki.xml can be used as an example file for demonstration/testing purposes, as it contains only the first 100 or so Wikipedia articles.

cunning_linguistics.py then contains classes to perform text analysis and harvest tweets for analysis.

Project details


Release history Release notifications

This version

0.0.1

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for esa-wiki, version 0.0.1
Filename, size File type Python version Upload date Hashes
Filename, size esa_wiki-0.0.1-py3-none-any.whl (17.3 kB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size esa_wiki-0.0.1.tar.gz (15.0 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page