Extracting Parallel Sentences with Bidirectional Recurrent Neural Networks to Improve Machine Translation

A TensorFlow implementation of the bidirectional RNN model described in the paper Extracting Parallel Sentences with Bidirectional Recurrent Neural Networks to Improve Machine Translation to extract parallel sentences from aligned comparable corpora.

Required packages

• TensorFlow (instructions)
• NumPy (instructions)
• scikit-learn (instructions)

Prepare the training data

We have provided a script to tokenize and clean your datasets using Moses.

./scripts/preprocessing.sh ~/moses/mosesdecoder ../data/train en fr 3 80
mv ../data/train.clean.en ../data/train.en
mv ../data/train.clean.fr ../data/train.fr

Training

Run the training script.

python train.py --source_train_path ../data/train.en --target_train_path ../data/train.fr --source_valid_path ../data/valid.en --target_valid_path ../data/valid.fr --checkpoint_dir ../tflogs

The models are written in checkpoint_dir.

Testing

Run the evaluation script.

python eval.py --checkpoint_dir ../tflogs --source_test_path ../data/test.en --target_test_path ../data/test.fr --reference_test_path ../data/test.ref --source_vocab_path ../data/vocabulary.source --target_vocab_path ../data/vocabulary.target

The evaluation is done on the last model saved in checkpoint_dir.

Extracting sentence pairs

Run the sentence extraction script.

python extract.py --checkpoint_dir ../tflogs --extract_dir ./samples --source_vocab_path ../data/vocabulary.source --target_vocab_path ../data/vocabulary.target --source_output_path ../data/extracted.source --target_output_path ../data/extracted.target --score_output_path ../data/extracted.score --source_language en --target_language fr --decision_threshold 0.99