OpenAI, Week 7-8 // Timeline of AI Milestones

To round out my two month curriculum period, I thought I would share a timeline I've been assembling (with help from a fellow Scholar - thanks, Shola!) of the major deep learning research milestones from 2000 to 2020. This was a necessary step for me, as I came in having zero perspective on the field and knowing almost none of the jargon - I didn't know what ResNet, BERT, or even Transformer meant! Writing the timeline not only helped me fill in the gaps in my vocabulary, it also gave me a startling and visceral understanding of just how new all this stuff is. There's at least one major paradigm shift, breakthrough, or algorithmic revolution every year, it seems! I'm incredibly excited at the idea of contributing to this ever-growing body of research, and can't wait to start my project.

2003 - Term "word embeddings" coined
2006 - Deep Belief Networks introduced; contrastive loss introduced
2008 - GPU revolution begins
2009 - ImageNet goes live
2011 - ReLU activation function proposed
2012 - AlexNet; dropout proposed by Hinton
2013 - ZF Net; Word2Vec algorithm introduced
2014 - GAN models introduced; Seq2Seq models introduced;VGG Net
2014/15 - GoogleNet; Attention models starting to gain traction
2015 - ResNet; batch normalization introduced; skip connections introduced; TensorFlow released
2016 - PyTorch released; SqueezeNet
2017 - Transformer models introduced
2018 - BERT; ELMo; GPT
2019 - GPT-2
2020 - GPT-3