AxCell: Automatic Extraction of Results from Machine Learning Papers

This repository is the official implementation of AxCell: Automatic Extraction of Results from Machine Learning Papers.

Requirements

To create a conda environment named axcell and install requirements run:

conda env create -f environment.yml

Additionally, axcell requires docker (that can be run without sudo). Run scripts/pull_docker_images.sh to download necessary images.

Datasets

We publish the following datasets:

• ArxivPapers
• SegmentedTables & LinkedResults
• PWCLeaderboards

See datasets notebook for an example of how to load the datasets provided below. The extraction notebook shows how to use axcell to extract text and tables from papers.

Evaluation

See the evaluation notebook for the full example on how to evaluate AxCell on the PWCLeaderboards dataset.

Training

• pre-training language model on the ArxivPapers dataset
• table type classifier and table segmentation on the SegmentedResults dataset

Pre-trained Models

You can download pretrained models here:

• axcell — an archive containing the taxonomy, abbreviations, table type classifier and table segmentation model. See the results-extraction notebook for an example of how to load and run the models
• language model — ULMFiT language model pretrained on the ArxivPapers dataset

Results

AxCell achieves the following performance:

| Dataset | Macro F1 | Micro F1 | | ---------- |---------------- | -------------- | | PWC Leaderboards | 21.1 | 28.7 | | NLP-TDMS | 19.7 | 25.8 |

License

AxCell is released under the Apache 2.0 license.

Citation

The pipeline is described in the following paper:

@inproceedings{axcell,
    title={AxCell: Automatic Extraction of Results from Machine Learning Papers},
    author={Marcin Kardas and Piotr Czapla and Pontus Stenetorp and Sebastian Ruder and Sebastian Riedel and Ross Taylor and Robert Stojnic},
    year={2020},
    booktitle={2004.14356}
}