Initialization Schemes and Gradient Clipping

Shield:

These images were originally published in the book “Deep Learning with PyTorch Step-by-Step: A Beginner’s Guide”.

They are also available at the book’s official repository: https://github.com/dvgodoy/PyTorchStepByStep.

Index

• Back
• Vanishing Gradients
• Initialization Schemes
  • Comparing against BatchNorm
• Gradient Clipping
  • Value Clipping
  • Norm Clipping
  • Using Hooks

** CLICK ON THE IMAGES FOR FULL SIZE **

Papers

• Xavier/Glorot Initialization: Understanding the difficulty of training deep feedforward neural networks by Glorot, X., Bengion, Y. (2010)
• Kaiming/He Initialization: Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification by He, K. et al. (2015)

Vanishing Gradients

Source: Chapter Extra

Initialization Schemes

Source: Chapter Extra

Comparing against BatchNorm

Source: Chapter Extra

Gradient Clipping

Value Clipping

Source: Chapter Extra

Norm Clipping

Source: Chapter Extra

Using Hooks

Source: Chapter Extra

This work is licensed under a Creative Commons Attribution 4.0 International License.