OpenAI, Week 19-20 // Empirically Comparing Two Models
When writing my own version of a workhorse model that has been implemented in a large, well-tested public codebase like PyTorch, I like to test mine via direct comparison. I've found it really helpful for giving myself peace of mind that e.g. my ResNet and Transformer implementations don't have any silly bugs in them. Here, I'll briefly discuss my process. The first thing I like to do is test that the two models have the same number of parameters (given identical architectural configurations, of course). Here is a little script: def print_num_params(model): model_parameters = filter(lambda p: p.requires_grad, model.parameters()) params = sum([np.prod(p.size()) for p in model_parameters]) print(params) print_num_params(my_model) print_num_params(torch_model) After that basic test is passed, I want to verify that the models are doing the same thing. I force the parameters in both models to take the same values, then pass the same random input tensor through each a...