wordsprobability 0.17

Method to get a words probability with fixes from How to Compute the Probability of a Word.

probability-of-a-word

Code to compute a word's probability using the fixes from "How to Compute the Probability of a Word"

Installation

You can install WordsProbability directly from PyPI:

pip install wordsprobability

Or from source:

git clone [e-mail]:tpimentelms/probability-of-a-word.git
cd probability-of-a-word
pip install -e .

Dependencies

WordsProbability has the following requirements:

• Pandas
• PyTorch
• Transformers

Usage

Basic Usage

Install this repository. Then run:

$ wordsprobability --model pythia-70m --input examples/abstract.txt --output temp.tsv

The input must be a txt file, with one sequence per line. The output will be a tsv file with a word per row with its respective computed surprisal values. To also get computed surprisal_buggy values (without our paper's correction) use the optional flag --return-buggy-surprisals. Currently, supported models are: pythia-70m, pythia-160m, pythia-410m, pythia-14b, pythia-28b, pythia-69b, pythia-120b, gpt2-small, gpt2-medium, gpt2-large, gpt2-xl. The code

Using in other Applications

Import wordsprobability in your application and get surprisals with:

    from wordsprobability import get_surprisal_per_word
    df = get_surprisal_per_word(text='Hello world! Who are you???\nWho am I?', model_name='pythia-70m')

Extra Information

Citation

If this code or the paper were usefull to you, consider citing it:

@article{pimentel-etal-2024-howto,
    title = "How to Compute the Probability of a Word",
    author = "Pimentel, Tiago and
    Meister, Clara",
    year = "2024",
    eprint = {2406.14561},
    archivePrefix = {arXiv},
    primaryClass = {cs.CL},
    url = {https://arxiv.org/abs/2406.14561},
    journal = "arXiv preprint arXiv:2406.14561",
}

Contact

To ask questions or report problems, please open an issue.