OpenAI, Week 13-14 // My Project

In this post, I will discuss the direction of my project and give an update on its progress.

Choosing a Project

One major axis of my research interests is natural language understanding. It's very clear - indeed, perhaps obvious - that language is a crucial mechanism for abstraction, synthesis, and complex reasoning in humans. OpenAI has placed big bets on natural language as a pathway to artificial general intelligence, a strategy I support wholeheartedly. Recently, the field of NLP has undergone a revolution, with pre-trained attention-based models smashing old evaluation benchmarks on downstream tasks and achieving downright spooky results. 

Nevertheless, human-level language performance eludes us. Moreover, there are reservations about the exploding cost (both computational and financial) of these models. State-of-the-art Transformers with hundreds of billions of parameters cost tens of millions of dollars to train. Empirical scaling laws show that performance is driven by power-law relationships with model size and compute. To continue progressing, we must continue to scale; yet there are serious issues with this approach, ranging from ethical concerns about resource accessibility to practical considerations about what happens if and when the demands of the model outpace available compute power.

As such, one of the "important problems" I identified early on in the Scholars program was looking for other approaches to language representation learning. Aside from the aforementioned pain point, there was a second reason I was interested in this topic. The conventional objectives such as masked language modeling (MLM) always struck me as strange and alien. If you randomly deleted 15% of a sentence whilst talking to me, I don't know that I would be able to fill in the blanks! Of course, such objectives clearly work exceptionally well and are very useful, because they allow unsupervised training over vast unlabelled datasets. But might there be other unsupervised methods that are more akin to how humans learn to extract meaning from language? In other words, I was curious about ways of using the success of the Transformer but paired with a more intuitive loss function, and whether the resulting model might also provide some ancillary efficiency gains.

With all of this in mind, I originally started chatting with my mentor about a vague thought I had: could we somehow leverage image-caption datasets to learn sentence representations by maximizing the similarity of the corresponding images? At the time, I lacked the vocabulary to cogently articulate my idea (later on, reading about OpenAI's somewhat related CLIP project helped me get a better handle on it). Nonetheless, this train of thought prompted my mentor to suggest a project in which we could attempt to learn language representations using contrastive learning, an unsupervised technique that has recently become a major player in the computer vision world. After studying up on contrastive learning, I judged that this was a very worthwhile frontier to attempt to push forward.

TL;DR Project Proposal

Contrastive learning has enjoyed renewed interest lately (see e.g. Le-Khac et al. 2020 for a recent review), though the idea of contrastive loss dates back to Hadsell et al. 2006. Briefly, contrastive learning is a self-supervised method wherein data representations are learned by comparing and contrasting data samples with one another. Each sample is copied and then augmented in some semantics-preserving way, creating a "positive pair." All other tuples of data are "negative pairs." A similarity metric is computed between pairs in the latent space, and the objective function quantifies the quality of the representations by how well the model groups positive pairs close together (and, in some versions, how well it simultaneously segregates negative pairs). 

Contrastive learning has been incredibly effective at learning visual representations for computer vision problems. However, applications to language have been few and limited thus far. We propose training a sentence representation model from scratch with contrastive learning, then comparing its performance to the state-of-the-art (e.g. BERT) using SuperGLUE benchmarks. We hope to either prove that contrastive learning, which is highly successful in computer vision, can also work for language - or else provide insight as to why it does not. Significant attention will be paid to comparing different text augmentation methods, as this is one of the key conceptual hurdles of applying contrastive frameworks to language.

I will refrain from discussing too many technical details of our proposal here, as this is still a work-in-progress and we hope to eventually publish, but I would be happy to discuss the specifics with any OpenAIer over a "donut" chat. :)

Progress

Several weeks were spent re-implementing a recent contrastive learning paper and matching its performance on CIFAR-10. This paper was brand new and no code, official or otherwise, had yet been released to accompany it. Writing my own implementation involved figuring out a few not-explicitly-mentioned tricks (such as special initialization schemes and different learning rate schedules for different parameter groups) by trial and error. Since the output of the model was representations rather than predictions, it was difficult at first to gauge whether something was or was not working based purely on the contrastive loss. Writing a kNN classifier that used the cosine similarity as its distance metric allowed me to monitor the progress every few epochs and debug issues.

Once the main framework was in place and validated, we switched gears to thinking about the base encoder we would use - a Transformer encoder. My mentor directed me to OpenAI's Sparse Transformer, and I spent several days studying the paper. I won't discuss that here, but will save it for a future blog post. However, I found myself anxious to see some results. As a scientist, I'm used to iterating quickly on small experiments. One of my Ph.D. advisor's favorite mottos was "Something pretty good today is better than something perfect next month." As a perfectionist, that philosophy went against many of my instincts, but it's also true that the main driving motivator for me was often the anticipation of seeing a result - an answer to a burning question. I talked to my mentor, who totally understood how I felt, and we figured out a way to shuffle things around. Before implementing the Sparse Transformer, I will use a simple 1D CNN-based model and work on getting a preliminary result with that. I coded up the model and this week I've been working on setting up my data pipeline. I can't wait to take this model out for a test drive!