Method to get a words probability with fixes from How to Compute the Probability of a Word.
Project description
probability-of-a-word
Code to compute a word's probability using the fixes from "How to Compute the Probability of a Word"
Installation
You can install WordsProbability directly from PyPI:
pip install wordsprobability
Or from source:
git clone git@github.com:tpimentelms/probability-of-a-word.git
cd probability-of-a-word
pip install -e .
Dependencies
WordsProbability has the following requirements:
Usage
Basic Usage
Install this repository. Then run:
$ wordsprobability --model pythia-70m --input examples/abstract.txt --output temp.tsv
The output will be a tsv file with a word per row with its respective computed surprisal
and surprisal_fixed
values.
Currently, supported models are: pythia-70m
, pythia-160m
, pythia-410m
, pythia-14b
, pythia-28b
, pythia-69b
, pythia-120b
, gpt-small
, gpt-medium
, gpt-large
, gpt-xl
.
Extra Information
Citation
If this code or the paper were usefull to you, consider citing it:
@article{pimentel-etal-2024-howto,
title = "How to Compute the Probability of a Word",
author = "Pimentel, Tiago and
Meister, Clara",
year = "2024",
eprint = {2406.14561},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
url = {https://arxiv.org/abs/2406.14561},
journal = "arXiv preprint arXiv:2406.14561",
}
Contact
To ask questions or report problems, please open an issue.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for wordsprobability-0.14-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1294e6a0f79e1bf72736b936c0fc9d530efe828f99afb454b991e5968c2c0230 |
|
MD5 | be13e83e05df32c9ea483c19fa705210 |
|
BLAKE2b-256 | fb347fc7539e19b6468040b2f75b7f8629add40a22aa52d38d9c0bde83452869 |