fetaphysicist

OpenAI rang in the new year by unveiling two exciting projects, DALL-E and CLIP . Both are awesome and deserving of separate blog posts, so I'll focus on DALL-E for today. The page about DALL-E on the OpenAI blog does an amazing job showing off its capabilities and I highly encourage everyone to check it out. It's a bit light on the technical details - probably because the paper has yet to be released. It does briefly discuss the general idea of the model, which I've sketched below: My scribbled high-level diagram of DALL-E. Basically, DALL-E is a giant Transformer decoder that takes a combined text-image input and outputs a generated image. It's trained on image-caption datasets and tokenizes each separately; the captions are tokenized using byte-pair encoding (subword data compression for text), while the images are fed into the encoder of something like a VQ-VAE (which discretizes the latent space). Some or potentially all of the image is masked, starting from the ...