fetaphysicist

For my project, I'm using  The Pile , an 800 GB unstructured text dataset scraped from the web, representing a diverse range of modalities from medical papers to code. Such a large dataset is best stored on the cloud. I've written out a guide for how to get set up with The Pile hosted on Azure. First, create an Azure Blob Storage account. I just followed the instructions  and used the default configurations. Meanwhile, on your local or virtual machine, install the  az CLI , blobfile , and boostedblob . This will help you easily interact with the files in your Azure storage account. Now sign in with   az login and open the link in a web browser to complete the sign in. When you try to read one of your files from blob storage using blobfile, you'll get an error that you need to set the following environment variables first: AZURE_STORAGE_KEY, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, and AZURE_TENANT_ID. You can access these by typing az ad sp create-for-rbac --name ...

In this post, I will discuss the direction of my project and give an update on its progress. Choosing a Project One major axis of my research interests is natural language understanding. It's very clear - indeed, perhaps obvious - that language is a crucial mechanism for abstraction, synthesis, and complex reasoning in humans. OpenAI has placed big bets on natural language as a pathway to artificial general intelligence, a strategy I support wholeheartedly. Recently, the field of NLP has undergone a revolution, with pre-trained attention-based models smashing old evaluation benchmarks on downstream tasks and achieving downright spooky results.  Nevertheless, human-level language performance eludes us. Moreover, there are reservations about the exploding cost (both computational and financial) of these models. State-of-the-art Transformers with hundreds of billions of parameters cost tens of millions of dollars to train. Empirical scaling laws show that performance is driven by pow...

In this post, I will discuss some prior work in applying contrastive learning to language. CLEAR ( Wu et al. 2020 ) This paper presents a language representation model, pre-trained on the same dataset as BERT with a combined word-level MLM objective and sentence-level contrastive objective. The model architecture is based on SimCLR (thus using both positive and negative pairs to calculate the contrastive loss, and requiring a large batch size of 8192), with a nonlinear projection head over a Transformer encoder. Several data augmentation methods are tested and compared: random word deletion, random spans deletion, random reordering, and synonym substitution, as well as the various permutations of stacked combinations. Notably, any data augmentation pipeline that does not include deletion fails catastrophically. This suggests to me that the MLM objective may be dominating. For the other augmentations, good results are obtained on GLUE and SentEval, but comparisons to true state-of-the-a...

OpenAI rang in the new year by unveiling two exciting projects, DALL-E and CLIP . Both are awesome and deserving of separate blog posts, so I'll focus on DALL-E for today. The page about DALL-E on the OpenAI blog does an amazing job showing off its capabilities and I highly encourage everyone to check it out. It's a bit light on the technical details - probably because the paper has yet to be released. It does briefly discuss the general idea of the model, which I've sketched below: My scribbled high-level diagram of DALL-E. Basically, DALL-E is a giant Transformer decoder that takes a combined text-image input and outputs a generated image. It's trained on image-caption datasets and tokenizes each separately; the captions are tokenized using byte-pair encoding (subword data compression for text), while the images are fed into the encoder of something like a VQ-VAE (which discretizes the latent space). Some or potentially all of the image is masked, starting from the ...