(Unofficial) PERIN: Permutation-invariant Semantic Parsing

Unofficial package for the PERIN parser: https://github.com/ufal/perin

Intended for easy installation and faster batching.

Installation

pip install perin_parser

Optional AMR evaluation

Install SMATCH++ and its evaluation dependencies separately:

pip install 'perin_parser[eval]'
# From this checkout:
pip install -e '.[eval]'

SMATCH++ provides AMR evaluation with ILP alignment. Its dependencies include NumPy, SciPy, and Python-MIP with a native solver. SciPy is already a base parser dependency; SMATCH++ and its solver are only requested by this extra. MCES remains the default evaluation backend.

Evaluate PERIN's MRP JSON output through the Python API:

from perin_parser.utility.evaluate import evaluate

scores = evaluate(
    'output', 'amr', 'eng', 'gold.mrp', backend='smatchpp', max_seconds=240)
# Reads output/prediction_amr_eng.json. Returns precision, recall and all (F1),
# each on the 0–1 scale. Graphs are matched by ID, independent of file order.

The full results, including per-graph ILP bounds and an exact flag, are saved to output/full_results_amr_eng.json. The files must contain the same nonempty set of AMR graph IDs for the selected language. To get these details directly, call perin_parser.utility.smatchpp_eval.evaluate_amr(pred_file, gold_file).

For model evaluation, pass amr_eval_backend='smatchpp' and optionally amr_eval_max_seconds=240 to parser.evaluate(...). Other frameworks continue to use MCES. File evaluation is also available without loading the model:

python -m perin_parser.utility.evaluate --input_dir output --framework amr --language eng --gold_file gold.mrp --backend smatchpp

For example, evaluate PENMAN prediction and reference files with:

python -m smatchpp -a predictions.txt -b gold.txt -solver ilp -graph_type amr -score_type micro

AMR standardization can produce scores that differ from historical Smatch results. Check the reported optimization status/bounds when a solver times out.

License

Copyright 2020 Institute of Formal and Applied Linguistics, Faculty of
Mathematics and Physics, Charles University, Czech Republic.

This Source Code Form is subject to the terms of the Mozilla Public
License, v. 2.0. If a copy of the MPL was not distributed with this
file, You can obtain one at http://mozilla.org/MPL/2.0/.