OpenAI, Week 3-4 // Implementing ResNet in PyTorch

Last week, I studied CNNs (mostly through Andrej Karpathy's CS231n lectures and notes). This week, I implemented ResNet end-to-end in PyTorch and trained it on the CIFAR-10 dataset. I was initially a bit confused, because the ResNet paper describes two slightly different sets of architectures: the main one optimized for ImageNet, and a narrower one optimized for CIFAR-10. I ended up writing my code such that it's flexible enough to implement both versions, depending on how it's called. 

While I found many helpful guides and discussions on ResNet online, none quite laid out the details of these architectures in a "cheat sheet" way. In case it's useful to anyone else, I'm posting my handwritten notes. Below is a summary of the architecture of ResNet34. I use the notation where $K$ is the number of filters, $F$ is the size of the filter, $S$ is the stride, and $P$ is the padding.

 

And here is an explicit work-through of the dimensionality of this problem:

In addition to ResNet34, there are several other variants of the ImageNet-based architecture, named ResNetX where X refers to the total number of layers. Each variant retains the structure of 4 residual stacks but has a different distribution of blocks per stack. Some of the variants use a fancier type of block called a Bottleneck Block that has three convolution layers instead of two. 

As I mentioned, there is also the modified architecture for working with CIFAR-10, with its own variants. Below is a list of all the different variants, a description of the modified CIFAR-10 architecture, and a detailed work-through of the dimensionality of ResNet20 applied to CIFAR-10.

I hope this is helpful to others working through ResNet for the first time!