General-purpose text classifier
Project description
GPTC
General-purpose text classifier in Python
GPTC provides both a CLI tool and a Python library.
Installation
pip install gptc
CLI Tool
Classifying text
gptc classify [-n <max_ngram_length>] <compiled model file>
This will prompt for a string and classify it, then print (in JSON) a dict of
the format {category: probability, category:probability, ...}
to stdout. (For
information about -n <max_ngram_length>
, see section "Ngrams.")
Checking individual words or ngrams
gptc check <compiled model file> <token or ngram>
This is very similar to gptc classify
, except it takes the input as an
argument, and it treats the input as a single token or ngram.
Compiling models
gptc compile [-n <max_ngram_length>] [-c <min_count>] <raw model file> <compiled model file>
This will write the compiled model encoded in binary format to <compiled model file>
.
If -c
is specified, words and ngrams used less than min_count
times will be
excluded from the compiled model.
Packing models
gptc pack <dir>
This will print the raw model in JSON to stdout. See models/unpacked/
for an
example of the format. Any exceptions will be printed to stderr.
Library
Model.serialize(file)
Write binary data representing the model to file
.
gptc.deserialize(encoded_model)
Deserialize a Model
from a file containing data from Model.serialize()
.
Model.confidence(text, max_ngram_length)
Classify text
. Returns a dict of the format {category: probability, category:probability, ...}
Note that this may not include values for all categories. If there are no common words between the input and the training data (likely, for example, with input in a different language from the training data), an empty dict will be returned.
For information about max_ngram_length
, see section "Ngrams."
Model.get(token)
Return a confidence dict for the given token or ngram. This function is very
similar to Model.confidence()
, except it treats the input as a single token
or ngram.
gptc.compile(raw_model, max_ngram_length=1, min_count=1, hash_algorithm="sha256")
Compile a raw model (as a list, not JSON) and return the compiled model (as a
gptc.Model
object).
For information about max_ngram_length
, see section "Ngrams."
Words or ngrams used less than min_count
times throughout the input text are
excluded from the model.
The hash algorithm should be left as the default, which may change with a minor version update, but it can be changed by the application if needed. It is stored in the model, so changing the algorithm does not affect compatibility. The following algorithms are supported:
md5
sha1
sha224
sha256
sha384
sha512
sha3_224
sha3_384
sha3_256
sha3_512
shake_128
shake_256
blake2b
blake2s
gptc.pack(directory, print_exceptions=False)
Pack the model in directory
and return a tuple of the format:
(raw_model, [(exception,),(exception,)...])
Note that the exceptions are contained in single-item tuples. This is to allow more information to be provided without breaking the API in future versions of GPTC.
See models/unpacked/
for an example of the format.
gptc.Classifier(model, max_ngram_length=1)
Classifier
objects are deprecated starting with GPTC 3.1.0, and will be
removed in 4.0.0. See the README from
3.0.2 if you need
documentation.
Ngrams
GPTC optionally supports using ngrams to improve classification accuracy. They are disabled by default (maximum length set to 1) for performance reasons. Enabling them significantly increases the time required both for compilation and classification. The effect seems more significant for compilation than for classification. Compiled models are also much larger when ngrams are enabled. Larger maximum ngram lengths will result in slower performance and larger files. It is a good idea to experiment with different values and use the highest one at which GPTC is fast enough and models are small enough for your needs.
Once a model is compiled at a certain maximum ngram length, it cannot be used
for classification with a higher value. If you instantiate a Classifier
with
a model compiled with a lower max_ngram_length
, the value will be silently
reduced to the one used when compiling the model.
Model format
This section explains the raw model format, which is how models are created and edited.
Raw models are formatted as a list of dicts. See below for the format:
[
{
"text": "<text in the category>",
"category": "<the category>"
}
]
GPTC handles raw models as list
s of dict
s of str
s (List[Dict[str, str]]
), and they can be stored in any way these Python objects can be.
However, it is recommended to store them in JSON format for compatibility with
the command-line tool.
Emoji
GPTC treats individual emoji as words.
Example model
An example model, which is designed to distinguish between texts written by
Mark Twain and those written by William Shakespeare, is available in models
.
The raw model is in models/raw.json
; the compiled model is in
models/compiled.json
.
The example model was compiled with max_ngram_length=10
.
Benchmark
A benchmark script is available for comparing performance of GPTC between
different Python versions. To use it, run benchmark.py
with all of the Python
installations you want to test. It tests both compilation and classification.
It uses the default Twain/Shakespeare model for both, and for classification it
uses Mark Antony's "Friends, Romans, countrymen"
speech
from Shakespeare's Julius Caesar.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.