SemBleu: A Robust Metric for AMR Parsing Evaluation

The repository corresponds to our ACL 2019 paper entitled "SemBleu: A Robust Metric for AMR Parsing Evaluation".

• SemBleu is fast, taking less than a second to evaluate a thousand AMR pairs.
• SemBleu is accuracy without any search errors.
• SemBleu considers high-order correspondences. From our experiments, it is mostly consistent with Smatch, but SemBleu can better capture performance variations.

Usage

chmod a+x eval.sh
./eval.sh output-file-path reference-file-path

Same as Smatch, AMRs in each file are separated by one empty line, such as:

(a / ask-01 :ARG0 (b / boy) :ARG1 (q / question))

(a / answer-01 :ARG0 (g / girl) :ARG1 (q / question))

AMR data

If you're developing a new metric and would like to have a comparison. Here is the 100 AMR graphs and the corresponding system outputs.

Results

The table below lists the SemBleu scores of recent SOTA work. The numbers are obtained by running our script on their provided outputs.

| Model | SemBleu | |---|---| | LDC2015E86 || | Lyu and Titov, (ACL 2018) | 58.7 | | Groschwitz et al., (ACL 2018) | 51.8 | | Guo and Lu, (EMNLP 2018) | 50.4 | | LDC2016E25 || | Lyu and Titov, (ACL 2018) | 60.3 | | van Noord and Bos, (CLIN 2017) | 49.5 | | LDC2017T10 || | Zhang et al., (ACL 2019) | 59.9 | | Cai and Lam (EMNLP 2019) | 56.9 | | Groschwitz et al., (ACL 2018) | 52.5 | | Guo and Lu, (EMNLP 2018) | 52.4 |