OpenAI, Week 5-6 // Implementing Transformer in PyTorch
These past two weeks, I've been studying Transformers and coding up my own implementation in PyTorch. There are quite a few excellent step-by-step guides to understanding the Transformer architecture, with gorgeous visualizations, so I'm not going to try to compete! Instead, I want to achieve two things with this post: 1) share my curated curriculum recommendations, and 2) describe a few subtleties that were not addressed in any of the tutorials I came across. 1) Curriculum As I said, there are a lot of options when it comes to learning material - a potentially overwhelming number of them. Here are my favorites, in the order I recommend accessing them: The Illustrated Transformer by Jay Alammar Attention Is All You Need: The Transformer by Lennart Van der Goten Walkthrough: The Transformer Architecture by Matthew Barnett Positional Encoding by Amirhossein Kazemnejad The Annotated Transformer by Alexander Rush 2) Subtleties Here are a couple of things I ...