OpenAI Week 11-12 // NLP+CL Literature Review

In this post, I will discuss some prior work in applying contrastive learning to language.

CLEAR (Wu et al. 2020)

This paper presents a language representation model, pre-trained on the same dataset as BERT with a combined word-level MLM objective and sentence-level contrastive objective. The model architecture is based on SimCLR (thus using both positive and negative pairs to calculate the contrastive loss, and requiring a large batch size of 8192), with a nonlinear projection head over a Transformer encoder. Several data augmentation methods are tested and compared: random word deletion, random spans deletion, random reordering, and synonym substitution, as well as the various permutations of stacked combinations. Notably, any data augmentation pipeline that does not include deletion fails catastrophically. This suggests to me that the MLM objective may be dominating. For the other augmentations, good results are obtained on GLUE and SentEval, but comparisons to true state-of-the-art using e.g. SuperGLUE are not presented. The authors attempt to disentangle the effects of MLM + large batch from MLM + large batch + CL by running an ablation study, which finds that MLM + large batch leads to maller improvement to the baseline than MLM + large batch + CL; however, the former is run on a significantly smaller pre-training corpus than the latter, making the comparison inconclusive in my opinion. 

ContraCode (Jain et al. 2020)

This paper presents a contrastive framework for learning code representations. Taking un-annotated code mined from GitHub repositories, the authors use existing transpilers to generate semantically equivalent augmentations. These transformations alter syntactic structure, method/variable names, etc. but preserve functionality. Of course, the graph structure of code makes it well suited to automatically creating multiple invariant representations - natural language is harder! ContraCode uses the Momentum Contrast framework and the usual InfoNCE contrastive loss function, pairing each element with a positive and many queued negatives. They experiment with both Transformer and BiLSTM as the encoder architecture, with a query encoder trained via gradient descent and a key encoder trained via an exponential moving average of the query parameters. A task-specific MLP or Transformer decoder is used for downstream finetuning. They evaluate the performance on two main tasks: type inference and code summarization. They generally find that contrastive pre-training boosts performance by a few percent compared to supervised or MLM pre-training. The authors also experiment with using augmentations during supervised training instead and find that it degrades performance, suggesting contrastive pre-training is an important element.

DeCLUTR (Giorgi  et al. 2020)

DeCLUTR continues training a pre-trained Transformer-based language model using a contrastive objective - the usual InfoNCE contrastive loss function - alongside an MLM objective. Unlike the other papers discussed here, instead of a projection head, it uses a “mean pooler” after the encoder. The augmentation method is randomly sampling sentences from nearby within the same document. They allow multiple types of positive pairs (with sentences adjacent, overlapping, or subsuming) and multiple types of negative pairs (with sentences far apart in the same document or else from different documents entirely). They show favorable results on SentEval evaluations comparisons to true state-of-the-art using e.g. SuperGLUE are not presented.

CERT (Fang et al. 2020)

CERT is not a pre-trained language model but rather a method for fine-tuning pre-trained language models such as BERT using sentence-level contrastive learning. The motivation is to capture more sentence-level semantics. CERT uses the Momentum Contrast framework and the usual InfoNCE contrastive loss function, pairing each element with a positive and many queued negatives. The main text augmentation used is back-translation, wherein a sentence is translated from language A to language B, and then back to language A, injecting some noise. They also test a different text augmentation method for comparison; given a sentence in the training set, randomly perform one of the following: synonym replacement, random insertion, random swap, and random deletion. They find that back-translation generally performs better, and suggest a few hypotheses as to why. They show results on GLUE tasks from fine-tuning BERT, XLNet, RoBERTa, ERNIE 2.0, and ALBERT but comparisons to true state-of-the-art using e.g. SuperGLUE are not presented. In their conclusions, they discuss an interesting idea: ranking-based loss, where instead of the binary task of identifying positive pairs, the task is to predict the order of a list of augmented sentences ranked by similarity to the original sentence.