certainty-estimator 1.6

This package is used to estimate certainty for scientific findings

certainty-estimator

Intro

certainty-estimator is a package used to estimate the certainty of scientfic findings. It is released with EMNLP 2021 paper Measuring Sentence-level and Aspect-level (Un)Certainty in Science Communications by Jiaxin Pei and David Jurgens.

Install

Use pip

If pip is installed, certainty-estimator could be installed directly from it:

pip3 install certainty-estimator

Dependencies

python>=3.6.0
torch>=1.6.0
transformers >= 3.1.0
numpy
math
tqdm

Example:

clone this repo to your machine: git clone https://github.com/Jiaxin-Pei/certainty-estimator.git

run the example code: python3 play.py

Estimating sentence-level certainty

Notes: During your first usage, the package will download a model file automatically, which is about 500MB.

Construct the Predictor Object

>>> from certainty_estimator.predict_certainty import CertaintyEstimator
>>> estimator = CertaintyEstimator('sentence-level')

Cuda is disabled by default, to allow GPU calculation, please use

>>> from certainty_estimator.predict_certainty import CertaintyEstimator
>>> estimator = CertaintyEstimator('sentence-level',cuda=True)

predict

predict is the core method of this package, which takes a single text or a list of texts, and returns a list of raw values in [1,6] (higher means more certain, while lower means less).

# Predict certainty for a single scientific finding
>>> text = 'The reason for this might be that the fetal central nervous system, which controls movements in general and facial movements in particular did not develop at the same rate and in the same manner as in fetuses of mothers who did not smoke during pregnancy.'
>>> estimator.predict(text)
>>> [2.6891987]

# Predict certainty for a list of scientific finding
>>> text = ['The reason for this might be that the fetal central nervous system, which controls movements in general and facial movements in particular did not develop at the same rate and in the same manner as in fetuses of mothers who did not smoke during pregnancy.', 'Mice lacking the tet1 gene were able to learn to navigate a water maze, but were unable to extinguish the memory.']
>>> estimator.predict(text)
>>> [2.6891987, 5.01066]

# when calculating certainty for a long list of findings, use the following code to display the progress
>>> from tqdm import tqdm
>>> text = [a long list of findings]
>>> estimator.predict(text,tqdm=tqdm)
>>> [2.6891987, 5.01066, ... ,4.28066, 5.77066]

Estimating aspect-level certainty

Notes: During your first usage, the package will download a model file automatically, which is about 500MB.

Construct the Predictor Object

>>> from certainty_estimator.predict_certainty import CertaintyEstimator
>>> estimator = CertaintyEstimator('aspect-level')

Cuda is disabled by default, to allow GPU calculation, please use

>>> from certainty_estimator.predict_certainty import CertaintyEstimator
>>> estimator = CertaintyEstimator('aspect-level',cuda=True)

predict

predict is the core method of this package, which takes a single text or a list of texts, and returns a list of tuples for aspect-level certainty.

# Predict certainty for a single scientific finding
>>> text = 'Mice lacking tet1 had much lower levels of hydroxymethylation -- an intermediate step in the removal of methylation -- in the hippocampus and the cortex, which are both key to learning and memory.'
>>> result = estimator.predict(text)
>>> print(result)
>>> [[('Extent', 'Uncertain'), ('Probability', 'Certain')]]
#By default, only returns the presented aspects and their certainty.

# Predict certainty for a list of scientific finding
>>> text = ['Mice lacking tet1 had much lower levels of hydroxymethylation -- an intermediate step in the removal of methylation -- in the hippocampus and the cortex, which are both key to learning and memory.', 'Dopamine and serotonin are important for different forms of flexibility associated with receiving reward and punishment.']
>>> result = estimator.predict(text)
>>> print(result)
>>> [[('Extent', 'Uncertain'), ('Probability', 'Certain')], [('Probability', 'Certain')]]


# when calculating certainty for a long list of findings, use the tqdm to display the progress
>>> from tqdm import tqdm
>>> text = [a long list of findings]
>>> estimator.predict(text,tqdm=tqdm)

Contact

Jiaxin Pei ([e-mail])