RuTaBERT

Model for solving the problem of Column Type Annotation with BERT, trained on RWT-RuTaBERT dataset.

RWT-RuTaBERT dataset contains 1 441 349 columns from Russian language Wikipedia tables. With headers matching 170 DBpedia semantic types. It has fixed train / test split: | Split | Columns | Tables | Avg. columns per table | |-------|-----------| ------- | ---------------------- | | Test | 115 448 | 55 080 | 2.096 | | Train | 1 325 901 | 633 426 | 2.093 |

Table of contents

• RuTaBERT
  • Benchmark
  • Project structure
  • Configuration
  • Dataset files
  • Training
    • Locally
    • Docker
    • Slurm
  • Testing
  • Inference

Benchmark

We trained RuTaBERT with two table serialization strategies:

1. Neighboring column serialization;
2. Multi-column serialization (based on Doduo's approach);

Benchmark results on RWT-RuTaBERT dataset: | Serialization strategy | micro-F1 | macro-F1 | weighted-F1 | | ---------------------- | -------- | -------- | ----------- | | Multi-column | 0.962 | 0.891 | 0.9621 | | Neighboring column | 0.964 | 0.904 | 0.9639 |

Training parameters: | Parameter | Value | | --------------- | ----- | | batch size | 32 | | epochs | 30 | | Loss function | Cross-entropy | | GD Optimizer | AdamW(lr=5e-5, eps=1e-8) | | GPU's | 4 NVIDIA A100 (80 GB) | | random seed | 2024 | | validation split | 5% |

Project structure

📦RuTaBERT
 ┣ 📂checkpoints
 ┃ ┗ Saved PyTorch models `.pt` 
 ┣ 📂data
 ┃ ┣ 📂inference
 ┃ ┃ ┗ Tabels to inference `.csv`
 ┃ ┣ 📂test
 ┃ ┃ ┗ Test dataset files `.csv`
 ┃ ┣ 📂train
 ┃ ┃ ┗ Train dataset files `.csv`
 ┃ ┗  Directory for storing dataset files.
 ┣ 📂dataset
 ┃ ┗  Dataset wrapper classes, dataloaders
 ┣ 📂logs
 ┃ ┗ Log files (train / test / error)
 ┣ 📂model
 ┃ ┗ Model and metrics
 ┣ 📂trainer
 ┃ ┗ Trainer
 ┣ 📂utils
 ┃ ┗ Helper functions
 ┗ Entry points (train.py, test.py, inference.py), configuration, etc.

Configuration

The model configuration can be found in the file config.json.

The configuratoin argument parameters are listed below:

| argument | description | |-------------|-------------| | num_labels | Number of labels used for classification | | num_gpu | Number of GPUs to use | | save_period_in_epochs | Number characterizing with what periodicity the checkpoint is saved (in epochs) | | metrics | The classification metrics used are | | pretrained_model_name | BERT shortcut name from HuggingFace | | table_serialization_type | Method of serializing a table into a sequence | | batch_size | Batch size | | num_epochs | Number of training epochs | | random_seed | Random seed | | logs_dir | Directory for logging | | train_log_filename | File name for train logging | | test_log_filename | File name for test logging | | start_from_checkpoint | Flag to start training from checkpoint | | checkpoint_dir | Directory for storing checkpoints of model | | checkpoint_name | File name of a checkpoint (model state) | | inference_model_name | File name of a model for inference | | inference_dir | Directory for storing inference tables .csv | | dataloader.valid_split | Amount of validation subset split | | dataloader.num_workers | Number of dataloader workers | | dataset.num_rows | Number of readable rows in the dataset, if null read all rows in files | | dataset.data_dir | Directory for storing train/test/inference files | | dataset.train_path | Directory for storing train dataset files .csv | | dataset.test_path | Direcotry for storing test dataset files .csv |

We recomend to change ONLY theese parameters:

• num_gpu - Any positive ingeter number + {0}. 0 stand for training / testing on CPU.
• save_period_in_epochs - Any positive integer number, measures in epochs.
• table_serialization_type - "column_wise" or "table_wise".
• pretrained_model_name - BERT shorcut names from Huggingface PyTorch pretrained models.
• batch_size - Any positive integer number.
• num_epochs - Any positive integer number.
• random_seed - Any integer number.
• start_from_checkpoint - "true" or "false".
• checkpoint_name - Any name of model, saved in checkpoint directory.
• inference_model_name - Any name of model, saved in checkpoint directory. But we recommend to use the best models: [model_best_f1_weighted.pt, model_best_f1_macro.pt, model_best_f1_micro.pt].
• dataloader.valid_split - Real number within range [0.0, 1.0] (0.0 stands for 0 % of train subset, 0.5 stands for 50 % of train subset). Or positive integer number (Denoting a fixed number of a validation subset).
• dataset.num_rows - "null" stands for read all lines in dataset files. Positive integer means the number of lines to read in the files of the dataset.

Dataset files

Before training / testing the model you need to:

1. Download dataset repository in the same directory as RuTaBERT, example source directory strucutre:

├── src
│  ├── RuTaBERT
│  ├── RuTaBERT-Dataset
│  │  ├── move_dataset.sh

3. Run script move_dataset.sh from dataset repository, to move dataset files into RuTaBERT data directory:

RuTaBERT-Dataset$ ./move_dataset.sh

3. configure config.json file before training.

---

Training

RuTaBERT supports training / testing locally and inside Docker container. Also supports slurm workload manager.

Locally

1. Create virtual environment:

RuTaBERT$ virtualenv venv

or

RuTaBERT$ python -m virtualenv venv

2. Install requirements and start train and test.

RuTaBERT$ source venv/bin/activate &&\
    pip install -r requirements.txt &&\
    python3 train.py 2> logs/error_train.log &&\
    python3 test.py 2> logs/error_test.log

3. Models will be saved in checkpoint directory.
4. Output will be in logs/ directory (training_results.csv, train.log, test.log, error_train.log, error_test.log).

Docker

Requirements:

• Docker installation guide (ubuntu);
• NVIDIA driver;
• NVIDIA Container Toolkit installation guide (ubuntu);

1. Make sure all dependencies are installed.
2. Build image:

RuTaBERT$ sudo docker build -t rutabert .

3. Run image

RuTaBERT$ sudo docker run -d --runtime=nvidia --gpus=all \
    --mount source=rutabert_logs,target=/app/rutabert/logs \
    --mount source=rutabert_checkpoints,target=/app/rutabert/checkpoints \
    rutabert

4. Move models and logs from container after training / testing.

RuTaBERT$ sudo cp -r /var/lib/docker/volumes/rutabert_checkpoints/_data ./checkpoints

RuTaBERT$ sudo cp -r /var/lib/docker/volumes/rutabert_logs/_data ./logs

5. Don't forget to remove volumes after training! Docker wont do it for you.
6. Models will be saved in checkpoint directory.
7. Output will be in logs/ directory (training_results.csv, train.log, test.log, error_train.log, error_test.log).

Slurm

1. Create virtual environment:

RuTaBERT$ virtualenv venv

or

RuTaBERT$ python -m virtualenv venv

2. Run slurm script:

RuTaBERT$ sbatch run.slurm

3. Check job status:

RuTaBERT$ squeue

4. Models will be saved in checkpoint directory.
5. Output will be in logs/ directory (train.log, test.log, error_train.log, error_test.log).

Testing

1. Make sure data placed in data/test directory.
2. (Optional) Download pre-trained models:

RuTaBERT$ ./download.sh table_wise

or

RuTaBERT$ ./download.sh column_wise

3. Configure which model to test in config.json.
4. Run:

RuTaBERT$ source venv/bin/activate &&\
    pip install -r requirements.txt &&\
    python3 test.py 2> logs/error_test.log

5. Output will be in logs/ directory (test.log, error_test.log).

Inference

1. Make sure data placed in data/inference directory.
2. (Optional) Download pre-trained models:

RuTaBERT$ ./download.sh table_wise

or

RuTaBERT$ ./download.sh column_wise

3. Configure which model to inference in config.json
4. Run:

RuTaBERT$ source venv/bin/activate &&\
    pip install -r requirements.txt &&\
    python3 inference.py

4. Labels will be in data/inference/result.csv