fetaphysicist

In this post, I will discuss the direction of my project and give an update on its progress. Choosing a Project One major axis of my research interests is natural language understanding. It's very clear - indeed, perhaps obvious - that language is a crucial mechanism for abstraction, synthesis, and complex reasoning in humans. OpenAI has placed big bets on natural language as a pathway to artificial general intelligence, a strategy I support wholeheartedly. Recently, the field of NLP has undergone a revolution, with pre-trained attention-based models smashing old evaluation benchmarks on downstream tasks and achieving downright spooky results.  Nevertheless, human-level language performance eludes us. Moreover, there are reservations about the exploding cost (both computational and financial) of these models. State-of-the-art Transformers with hundreds of billions of parameters cost tens of millions of dollars to train. Empirical scaling laws show that performance is driven by pow...

OpenAI rang in the new year by unveiling two exciting projects, DALL-E and CLIP . Both are awesome and deserving of separate blog posts, so I'll focus on DALL-E for today. The page about DALL-E on the OpenAI blog does an amazing job showing off its capabilities and I highly encourage everyone to check it out. It's a bit light on the technical details - probably because the paper has yet to be released. It does briefly discuss the general idea of the model, which I've sketched below: My scribbled high-level diagram of DALL-E. Basically, DALL-E is a giant Transformer decoder that takes a combined text-image input and outputs a generated image. It's trained on image-caption datasets and tokenizes each separately; the captions are tokenized using byte-pair encoding (subword data compression for text), while the images are fed into the encoder of something like a VQ-VAE (which discretizes the latent space). Some or potentially all of the image is masked, starting from the ...

These past two weeks, I've been studying Transformers and coding up my own implementation in PyTorch. There are quite a few excellent step-by-step guides to understanding the Transformer architecture, with gorgeous visualizations, so I'm not going to try to compete! Instead, I want to achieve two things with this post: 1) share my curated curriculum recommendations, and 2) describe a few subtleties that were not addressed in any of the tutorials I came across. 1) Curriculum As I said, there are a lot of options when it comes to learning material - a potentially overwhelming number of them. Here are my favorites, in the order I recommend accessing them: The Illustrated Transformer   by Jay Alammar Attention Is All You Need: The Transformer   by Lennart Van der Goten Walkthrough: The Transformer Architecture by Matthew Barnett Positional Encoding   by Amirhossein Kazemnejad The Annotated Transformer   by Alexander Rush 2) Subtleties Here are a couple of things I ...