BERT

Shield:

These images were originally published in the book “Deep Learning with PyTorch Step-by-Step: A Beginner’s Guide”.

They are also available at the book’s official repository: https://github.com/dvgodoy/PyTorchStepByStep.

Index

• Back
• Input Embeddings
• Contextual Word Embeddings
• Pretraining Tasks
  • Masked Language Modeling (MLM)
  • Next Sentence Predictions (NSP)

** CLICK ON THE IMAGES FOR FULL SIZE **

Papers

• BERT: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding by Devlin, J. et al. (2018)

Input Embeddings

Source: Chapter 11

Contextual Word Embeddings

Source: Chapter 11

Pretraining Tasks

Masked Language Modeling

Source: Chapter 11

Next Sentence Prediction

Source: Chapter 11

This work is licensed under a Creative Commons Attribution 4.0 International License.