Sets the gradients of all optimized
torch.Tensors to zero.
zero_grad(set_to_none=True)followed by a backward pass,
.grads are guaranteed to be None for params that did not receive a gradient. 3.
torch.optimoptimizers have a different behavior if the gradient is 0 or None (in one case it does the step with a gradient of 0 and in the other it skips the step altogether).