recluse 0.1.12-dirty

Reproducible Experimentation for Computational Linguistics Use

Recluse

Author: L. Amber Wilcox-O’Hearn

Contact: [e-mail]

Released under the GNU AFFERO GENERAL PUBLIC LICENSE, see COPYING file for details.

Introduction

Recluse (Reproducible Experimentation for Computational Linguistics Use) is a set of tools for running computational linguistics experiments reproducibly.

This version contains

• utils, which has a function for reading and writing unicode with regular or compressed text, and a function for splitting a file into smaller pieces. The latter is needed for some tools that load everything into RAM, or train on all the data when we would be satisfied with training on partial data.

• article_randomiser, which reproducibly randomly divides a corpus into training, development, and test sets.

• nltk_based_segmenter_tokeniser, which does sentence segmentation and word tokenisation. It is optimised for Wikipedia type text, and it has a mode that preserves the untokenised text (modulo extra whitespace).

• vocabulary_generator and the helper class vocabulary_cutter. This wraps srilm as it makes unigram counts, and then selects the most frequent.