A Contrastive Framework for Neural Text Generation

Authors: Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier

This repository contains code, models, and other related resources of our paper "A Contrastive Framework for Neural Text Generation".

:star2: Check out this great [blog] as well as this awesome [demo] that are generously supported by Huggingface (@huggingface :hugs:) which compares contrastive search with other popular decoding methods. Many thanks to Huggingface :hugs:!

[Use contrastive search in Huggingface transformers] In this <a href='#contrastive_in_transformers'>tutorial</a>, we demonstrate how to use contrastive search in Huggingface transformers.

If you find our paper and resources useful, please kindly leave a star and cite our papers. Thanks!

@inproceedings{su2022a,
  title={A Contrastive Framework for Neural Text Generation},
  author={Yixuan Su and Tian Lan and Yan Wang and Dani Yogatama and Lingpeng Kong and Nigel Collier},
  booktitle={Advances in Neural Information Processing Systems},
  editor={Alice H. Oh and Alekh Agarwal and Danielle Belgrave and Kyunghyun Cho},
  year={2022},
  url={https://openreview.net/forum?id=V88BafmH9Pj}
}

@article{su2023contrastive,
  title={Contrastive Search Is What You Need For Neural Text Generation},
  author={Yixuan Su and Nigel Collier},
  journal={Transactions on Machine Learning Research},
  issn={2835-8856},
  year={2023},
  url={https://openreview.net/forum?id=GbkWw3jwL9}
}

News:

[2022/11/22] Released a technical report that compares Contrastive Search with another recently proposed method, i.e. Contrastive Decoding. Check out our [paper] and [code].
[2022/11/08] :fire: Contrastive Search has now officially supported by HuggingFace for both PyTorch and TensorFlow platforms! Check out this great [blog] as well as this awesome [demo] that are generously supported by Huggingface (@huggingface :hugs:).
[2022/10/26] :fire: We have released a new manuscript "Contrastive Search Is What You Need For Neural Text Generation" which has two takeaways: (1) Autoregressive language models are naturally isotropic, therefore SimCTG training may not be necessary; (2) Contrastive search works exceptionally well on off-the-shelf language models across 16 languages. On 12 out of the 16 evaluated languages, it even performs comparably with human-written text! Paper and code are all released. Check it out!
[2022/10/13] We have added a concise explanation on the implementations of contrastive search. Please find it [here].
[2022/09/14] :blush: SimCTG is accepted to NeurIPS 2022!
[2022/09/06] :fire: We have added supports for the newly released OPT models (see "OPT: Open Pre-trained Transformer Language Models") by Meta. To see how to apply contrastive search on OPT models, check it [here]!
[2022/06/03] :fire: We have released an easy-to-use library (i.e., simctg) which allows you to use SimCTG with a simple pip install simctg and a few lines of code. Check the comprehensive and huggingface-style tutorials <a href='#tutorial'>[here]</a> and [here]!
[2022/05/06] :star: We have released MAGIC, a follow up work of SimCTG, that is the SOTA method in zero-shot multi-modal text generation tasks (e.g., zero-shot image captioning and visually grounded story generation). Check it out! [paper] [code]
[2022/04/16] We have updated instructions on how to apply contrastive search on encoder-decoder models (e.g. BART and T5). More details can be found [here].
[2022/04/07] SimCTG has been publicly deployed in the AI sentence completion module of Tencent's Effidit platform (腾讯智能创作助手). Check it out and have fun!
[2022/04/02] Add support on another benchmark (ROCStories) for the task of open-ended story generation.
[2022/03/06] Example of how to adapt our approach to open-ended story generation is released.
[2022/02/15] SimCTG is publicly released!

Catalogue:

<a href='#contrastive_in_transformers'>Apply Contrastive Search in Huggingface Transformers</a>
<a href='#introduction'>1. Introduction</a>
<a href='#contrastive_search_with_LMs'>2. Contrastive Search with GPT-2 and OPT :fire:</a>
- <a href='#install_simctg_package'>2.1. Environment Setup</a>
- <a href='#contrastive_search_with_gpt2'>2.2. Contrastive Search with GPT-2</a>
- <a href='#contrastive_search_with_opt'>2.3. Contrastive Search with OPT</a>
<a href='#models'>3. Huggingface Models</a>
<a href='#tutorial'>4. Huggingface-Style Tutorials</a> :star:
- <a href='#install_simctg'>4.1. Install and Load SimCTG</a>
- <a href='#example_train_with_simctg'>4.2. Example of Training Language Model with SimCTG</a>
  - <a href='#init_simctg'>4.2.1. Initialize Language Model</a>
  - <a href='#init_loss_class'>4.2.2. Initialize Loss Class</a>
  - <a href='#init_training_data'>4.2.3. Create Example Training Data</a>
  - <a href='#compute_loss'>4.2.4. Compute Loss</a>
- <a href='#contrastive_search_examples'>4.3. Examples of Performing Generation with Contrastive Search</a>
  - <a href='#example_document_generation'>4.3.1. Open-Ended Document Generation</a>
  - <a href='#example_dialogue_generation'>4.3.2. Open-Domain Dialogue Generation</a>
- <a href='#example_off_the_shelf_generation'>4.4. Contrastive Search with Off-the-shelf Language Models from Different Languages</a>
  - <a href='#chinese_example_off_the_shelf_generation'>4.4.1. Chinese Language Model</a>
  - <a href='#japanese_example_off_the_shelf_generation'>4.4.2. Japanese Language Model</a>
  - <a href='#korean_example_off_the_shelf_generation'>4.4.3. Korean Language Model</a>
- <a href='#training_tutorial'>4.5. Detailed Tutorial of Training SimCTG on Wikitext-103</a> :star:
- <a href='#T5_tutorial'>4.6. Apply SimCTG on T5</a> :star:
<a href='#environment_setup'>5. Environment Setup</a>
<a href='#example_usage'>6. Example Usage of Contrastive Search</a>
- <a href='#example_usage_english_simctg'>6.1. Use SimCTG Pretrained on Wikipedia Corpus</a>
- <a href='#example_usage_different_language_model'>6.2. Use Off-the-shelf Language Models from Different Languages</a>
  - <a href='#example_usage_chinese_gpt'>6.2.1. Chinese Language Model</a>
  - <a href='#example_usage_japanese_gpt'>6.2.2. Japanese Language Model</a>
  - <a href='#example_usage_korean_gpt'>6.2.3. Korean Language Model</a>
<a href='#wikitext103_tutorial'>7. Document Generation</a>
<a href='#dialogue_tutorial'>8. Open-domain Dialogue Generation</a>
<a href='#pretraining'>9. Large-Scale Pre-training with SimCTG</a>
<a href='#story_generation'>10. Open-Ended Story Generation</a>
<a href='#contrastive_for_encoder_decoder'>11. Contrastive Search on Encoder-Decoder Models</a>
<a href='#contact'>12. Contact</a>
<a href='#simctg_elsewhere'>13. SimCTG Elsewhere</a>

Apply Contrastive Search in Huggingface Transformers:

Here, we demonstrate how to use contrastive search in Huggingface transformers.

(1) Install Environment:

First, to install the required packages, please run the following commands:

pip install torch
pip install "transformers>=4.24.0"

(2) Generate Text with Contrastive Search:

Next, we show how to reproduce the result as in <a href='#contrastive_search_with_gpt2'>Section 2.2</a>

# load the LMs
import torch
from transformers import GPT2Tokenizer, GPT2LMHeadModel
model_name = 'gpt2-large'
tokenizer = GPT2Tokenizer.from_pretrained(model_name)
model = GPT2LMHeadModel.from_pretrained(model_name)
model.eval()

# prepare the prefix
prefix_text = r"DeepMind Company is"
input_ids = tokenizer(prefix_text, return_tensors='pt').input_ids

# generate the result wit

SimCTG

Install / Use

README