fetaphysicist

In this post, I will discuss the direction of my project and give an update on its progress. Choosing a Project One major axis of my research interests is natural language understanding. It's very clear - indeed, perhaps obvious - that language is a crucial mechanism for abstraction, synthesis, and complex reasoning in humans. OpenAI has placed big bets on natural language as a pathway to artificial general intelligence, a strategy I support wholeheartedly. Recently, the field of NLP has undergone a revolution, with pre-trained attention-based models smashing old evaluation benchmarks on downstream tasks and achieving downright spooky results.  Nevertheless, human-level language performance eludes us. Moreover, there are reservations about the exploding cost (both computational and financial) of these models. State-of-the-art Transformers with hundreds of billions of parameters cost tens of millions of dollars to train. Empirical scaling laws show that performance is driven by pow...

In this post, I will discuss some prior work in applying contrastive learning to language. CLEAR ( Wu et al. 2020 ) This paper presents a language representation model, pre-trained on the same dataset as BERT with a combined word-level MLM objective and sentence-level contrastive objective. The model architecture is based on SimCLR (thus using both positive and negative pairs to calculate the contrastive loss, and requiring a large batch size of 8192), with a nonlinear projection head over a Transformer encoder. Several data augmentation methods are tested and compared: random word deletion, random spans deletion, random reordering, and synonym substitution, as well as the various permutations of stacked combinations. Notably, any data augmentation pipeline that does not include deletion fails catastrophically. This suggests to me that the MLM objective may be dominating. For the other augmentations, good results are obtained on GLUE and SentEval, but comparisons to true state-of-the-a...