July 05, 2020
David's Tips on How to Read Pytorch
Pytorch has a great design: easy and powerful. Easy enough that it is definitely possible to use pytorch without understanding what it is doing or why. But it also gets better the more you understand.
As part of summer school at MIT, next week I'm doing a lecture to introduce students to pytorch. I have written a few code examples that I hope will give students a head start on understanding the design of pytorch. Each concept is illustrated visually with a cute minimal hackable example. All the examples are notebooks that are hosted on Google Colab.
It covers tensor indexing conventions, benchmarks gpu versus cpu speeds, explains autograd with simple systems, and plots what optimizers are doing using 2d problems. Then I put the pieces together with a detailed discussion of network modules and data loaders, training toy networks where the whole space can be visualized as well as a simple but realistic five-minute ResNet training example.
Here are David's Tips on How to Read Pytorch.