costra 1.1

No project description provided

Costra

This is a tool for automatic evaluation of Czech sentence embeddings using Costra 1.1 dataset.

More information can be found in the following paper:

• Petra Barančíková and Ondřej Bojar: Costra 1.1: An Inquiry into Geometric Properties of Sentence Spaces. In: TSD 2020. Lecture Notes in Computer Science, vol 12284. Springer, Cham.

The presentation of the paper with the accompanying video can be found here.

Installation

$ pip install costra

Usage

1. Get sentences from Costra:

from costra import costra
CostraEvaluator = costra.CostraEvaluator()
sentences = CostraEvaluator.get_sentences()

2. Generate embeddings (example with SentenceTransformers):

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("Seznam/retromae-small-cs")
embeddings = model.encode(sentences)

3. Evaluating the embeddings:

results = CostraEvaluator.evaluate(embeddings)

4. Results have the following format, with costra being the overall score.

{
    'basic': 0.063,
    'modality': 0.079,
    'time': 0.692,
    'style': 0.634,
    'generalization': 0.695,
    'opposite_meaning': 0.751,
    'costra': 0.486
}

For more detail about Costra categories, refer to the original paper.

Citation

If you use the tool, please consider citing the following paper:

@inproceedings{Costra,
  author    = {Petra Baran{\v{\c}}{\'{\i}}kov{\'{a}} and Ond{\v{\r}}ej Bojar},
  editor    = {Petr Sojka and Ivan Kope{\v{\c}}ek and Karel Pala and Ales Hor{\'{a}}k},
  title     = {Costra 1.1: An Inquiry into Geometric Properties of Sentence Spaces},
  booktitle = {Text, Speech, and Dialogue - 23rd International Conference, {TSD}
               2020, Brno, Czech Republic, September 8-11, 2020, Proceedings},
  series    = {Lecture Notes in Computer Science},
  volume    = {12284},
  pages     = {135--143},
  publisher = {Springer},
  year      = {2020},
  url       = {https://doi.org/10.1007/978-3-030-58323-1\_14},
  doi       = {10.1007/978-3-030-58323-1\_14},
}

License

The data is distributed under the Creative Commons 4.0 BY.