CS60010: Deep Learning

Spring 2023

Sudeshna Sarkar

Transformer- Part 3
Sudeshna Sarkar
16 Mar 2023
Pre-training
CS60010: Deep Learning
Spring 2022

Sudeshna Sarkar

BERT etc
Sudeshna Sarkar
11 May 2022
Transfer Learning
• Leverage unlabeled data to cut
down on the number of labeled
examples needed.
• Take a network trained on a task for
which it is easy to generate labels,
and adapt it to a different task for
which it is harder.
• Train a really big language model on
billions of words, transfer to every
NLP task!
Contextualized Word Embedding

Contextualized
Word Embedding

Contextualized … the river bank …

Word Embedding

… money in the bank …

Contextualized
Word Embedding
Train contextual representations on text corpus

… own blood bank …

Uni-directional language models

• Why are LMs unidirectional?

• Reason 1: Directionality is needed to generate a well-formed probability
distribution.
• Reason 2: Words can “see themselves” in a bidirectional encoder.

• Language models only use left context or right context, but language
understanding is bidirectional.

Slide from Jacob Delvin

Word structure and subword models

• We assume a fixed vocab of tens of thousands of words, built from

the training set.
• All novel words seen at test time are mapped to a single UNK.

• taaaasty
• laern
• tarnsformerify
The byte-pair encoding algorithm
1. Start with a vocabulary containing only characters and an “end-of-word” symbol.
2. Using a corpus of text, find the most common adjacent characters “a,b”; add “ab” as a subword.
3. Replace instances of the character pair with the new subword; repeat until desired vocab size.

Common words end up being a part of the subword vocabulary, while rarer words are split into
(sometimes intuitive, sometimes not) components.
In the worst case, words are split into as many subwords as they have characters.
• taa## aaa## sty
• la## ern##
• Transformer## ify
pretraining
1. pretrained word embeddings
2. Pretraining whole models
• In modern NLP:
• All (or almost all) parameters in NLP networks are initialized via pretraining.
• Pretraining methods hide parts of the input from the model, and train the model to
reconstruct those parts.
• This has been exceptionally effective at building strong:
• representations of language
• parameter initializations for strong NLP
• models.
• Probability distributions over language that we can sample from
Pretraining through language modeling
• Recall the language modeling task:
• Model 𝑝𝜃 𝑤𝑡 |𝑤1:𝑡−1 , the probability distribution over words given their past
contexts.

• Pretraining through language modeling:

• Train a neural network to perform language modeling on a large amount of
text.
• Save the network parameters.
 The Pretraining / Finetuning Paradigm
Pretraining can improve NLP applications by serving as parameter initialization.

Step 1: Pretrain (on language modeling) Step 2: Finetune (on your task)
Lots of text; learn general things! Not many labels; adapt to the task!
goes to make tasty tea END ☺/

(Transformer, LSTM, ++ ) (Transformer, LSTM, ++ )

Iroh goes to make tasty tea … the movie was …

 Pretraining for three types of architectures
The neural architecture influences the type of pretraining, and natural use cases.

• Gets bidirectional context – can condition on future!

Encoders
• How do we train them to build strong representations?

Encoder- • Good parts of decoders and encoders?

Decoders • What’s the best way to pretrain them?

• Language models! What we’ve seen so far.

Decoders
• Nice to generate from; can’t condition on future words
 Pretraining encoders: what pretraining objective
to use?
So far, we’ve looked at language model pretraining. But encoders get bidirectional
context, so we can’t do language modeling!

Idea: replace some fraction of words in the went store

input with a special [MASK] token; predict
these words.
ℎ1 , … , ℎ𝑇
ℎ1 , … , ℎ𝑇 = Encoder 𝑤1, 𝑤2, … , 𝑤𝑇

Only add loss terms from words that are

“masked out.” If 𝑥෤ is the masked version of 𝑥, I [M] to the [M]
we’re learning 𝑝𝜃 𝑥|𝑥෤
Called Masked LM. [Devlin et al., 2018]
 BERT: Bidirectional Encoder Representations
from Transformers
Devlin et al., 2018 proposed the “Masked LM” objective and released the weights of a
pretrained Transformer, a model they labeled BERT.

Some more details about Masked LM for BERT:

• Predict a random 15% of (sub)word tokens. [Predict these!] went to store
• Replace input word with [MASK] 80% of the time
• Replace input word with a random token 10% of Transformer
the time Encoder
• Leave input word unchanged 10% of the time (but
still predict it!)
• Why? Doesn’t let the model get complacent and not I pizza to the [M]
build strong representations of non-masked words.
(No masks are seen at fine-tuning time!)
[Replaced] [Not replaced] [Masked]

[Devlin et al., 2018]

Bidirectional transformer LMs
BERT is essentially the “encoder” part of a transformer
with 15% of inputs replaced with [MASK]
Input: I [MASK] therefore I [MASK]

position-wise softmax
Output: I think therefore I am

Main idea: needing to predict missing words forces the

model to “work hard” to learn a good representation
position-wise nonlinear
repeated Nx

network Without the need for masked self-attention!

This makes it bidirectional
masked self-attention

position-wise encoder randomly mask out

some input tokens

mask = replace with [MASK]

 Training of BERT
vocabulary size
Approach 1: Predicting the
masked word
Masked LM
Linear Multi-class
Classifier
Randomly replace 15% of the
tokens with [MASK] ……

BERT

潮水 [MASK]
退了 就 知道 ……
 Training of BERT
Approach 2: Next Sentence Prediction
yes
[CLS]: the position that outputs classification results

Linear Binary [SEP]: the boundary of two sentences

Classifier Approaches 1 and 2 are used at the same time.

Randomly
swap the
order of
the
sentences BERT
50% of the
time
[CLS] 醒醒 吧 [SEP] 你 沒有 妹妹
Bidirectional Encoder Representations from
Transformers (BERT)
• BERT = Encoder of Transformer
Learned from a large amount of text
without annotation
Encoder
……

BERT

潮水 退了 就 知道 ……
BERT

• Multi-layer self-attention (Transformer)

• Input: a sentence or a pair of sentences with a separator and subword
representation
Input Representation
Hidden state corresponding to [CLS] will be used as the
sentence representation

• Use 30,000 WordPiece vocabulary on input.

• Each token is sum of three embeddings
• Single sequence is much more efficient.

Slide from Jacob Delvin

How to use BERT – Case 1

class

Linear Trained from

Classifier Input: single sentence,
Scratch
output: class

Example:
Sentiment analysis
BERT Fine-tune Document Classification

[CLS] w1 w2 w3
sentence
Sentence Classification

CLS token is used to provide classification decision

How to use BERT – Case 2

class class class

Linear Linear Linear Input: single sentence,

Cls Cls Cls output: class of each word

Example: Slot filling

BERT

[CLS] w1 w2 w3
sentence
Tagging with BERT

• Can do for a single sentence or a pair

• Tag each word piece
• Example tasks: span-based question answering, name-entity recognition,
POS tagging
How to use BERT – Case 3

Class Input: two sentences, output: class

Example: Natural Language Inference
Linear Given a “premise”, determining whether a
Classifier “hypothesis” is T/F/ unknown.

BERT

[CLS] w1 w2 [SEP] w3 w4 w5
Sentence 1 Sentence 2
Sentence-pair Classification with BERT

• Feed both sentences, and CLS token

used for classification
• Example tasks:
• Textual entailment
• Question paraphrase detection
• Question-answering pair classification
• Semantic textual similarity
• Multiple choice question answering
Model Architecture

⚫ BERT BASE
⚫ 12 layers, 768-dim per word-piece token
⚫ 12 heads.
⚫ Total parameters = 110M

⚫ BERT LARGE
⚫ 24 layers, 1024-dim per word-piece token
⚫ 16 heads.
⚫ Total parameters = 340M

BERT is basically a trained Transformer Encoder stack.

Model Details

• Data: Wikipedia (2.5B words) + BookCorpus (800M words)

• Batch Size: 131,072 words (1024 sequences * 128 length or 256
sequences * 512 length)
• Training Time: 1M steps (~40 epochs)
• Optimizer: AdamW, 1e-4 learning rate, linear decay
• BERT-Base: 12-layer, 768-hidden, 12-head
• BERT-Large: 24-layer, 1024-hidden, 16-head
• Trained on 4x4 or 8x8 TPU slice for 4 days

Slide from Jacob Delvin

 Evaluation of BERT
General Language Understanding Evaluation (GLUE) benchmark: Standard split of data to
train, validation, test, where labels for the test set is only held in the server.
• Sentence pair tasks
• MNLI, Multi-Genre Natural Language Inference
• QQP, Quora Question Pairs
• QNLI, Question Natural Language Inference
• STS-B The Semantic Textual Similarity Benchmark
• MRPC Microsoft Research Paraphrase Corpus
• RTE Recognizing Textual Entailment
• WNLI Winograd NLI is a small natural language inference dataset
• Single sentence classification
• SST-2 The Stanford Sentiment Treebank
• CoLA The Corpus of Linguistic Acceptability
 GLUE Results

MultiNLI CoLa
Premise: Hills and mountains are especially Sentence: The wagon rumbled down the
sanctified in Jainism. road. Label: Acceptable
Hypothesis: Jainism hates nature. Sentence: The car honked down the road.
Label: Contradiction Label: Unacceptable

Slide from Jacob Delvin

Results

• Fine-tuned BERT outperformed previous state of the art on 11 NLP tasks

• Since then was applied to many more tasks with similar results
• The larger models perform better, but even the small BERT performs better
than prior methods
• Variants quickly outperformed human performance on several tasks,
including span-based question answering — but what does this mean is
less clear
• Started an arms race (between industry labs) on bigger and bigger models
 Limitations of pretrained encoders
Those results looked great! Why not used pretrained encoders for everything?

If your task involves generating sequences, consider using a pretrained decoder; BERT and other
pretrained encoders don’t naturally lead to nice autoregressive (1-word-at-a-time) generation
methods.

make/brew/craft goes to make tasty tea END

Pretrained Encoder Pretrained Decoder

Iroh goes to [MASK] tasty tea Iroh goes to make tasty tea

29
 Extensions of BERT
You’ll see a lot of BERT variants like RoBERTa, SpanBERT, +++
Some generally accepted improvements to the BERT pretraining formula:
• RoBERTa: mainly just train BERT for longer and remove next sentence prediction!
• SpanBERT: masking contiguous spans of words makes a harder, more useful pretraining task

It’s bly irr## esi## sti## bly

BERT SpanBERT

It’ [MASK] [MASK] [MASK] [MASK] good

[MASK] irr## esi## sti## [MASK] good

[Liu et al., 2019; Joshi et al., 2020]

 Extensions of BERT
A takeaway from the RoBERTa paper: more compute, more data can improve pretraining
even when not changing the underlying Transformer encoder.

[Liu et al., 2019; Joshi et al., 2020]

 Full Finetuning vs. Parameter-Efficient Finetuning
Finetuning every parameter in a pretrained model works well, but is memory-intensive.
But lightweight finetuning methods adapt pretrained models in a constrained way.
Leads to less overfitting and/or more efficient finetuning and inference.

Full Finetuning Lightweight Finetuning

Adapt all parameters Train a few existing or new parameters

☺/ ☺/

(Transformer, LSTM, ++ ) (Transformer, LSTM, ++ )

… the movie was … … the movie was …

[Liu et al., 2019; Joshi et al., 2020]
 Parameter-Efficient Finetuning: Prefix-Tuning,
Prompt tuning
Prefix-Tuning adds a prefix of parameters, and freezes all pretrained parameters.
The prefix is processed by the model just like real words would be.
Advantage: each element of a batch at inference could run a different tuned model.

☺/

(Transformer, LSTM, ++ )

… the movie was …

Learnable prefix
parameters
33 [Li and Liang, 2021; Lester et al., 2021]
 Parameter-Efficient Finetuning: Low-Rank
Adaptation
Low-Rank Adaptation Learns a low-rank “diff” between the pretrained and finetuned
weight matrices.
Easier to learn than prefix-tuning.

𝑊 ∈ ℝ𝑑 ×𝑑 B ∈ ℝ𝑘 ×𝑑
☺/

(Transformer, LSTM, ++ )
A ∈ ℝ𝑑 ×𝑘

𝑊 + AB
… the movie was …

34 [Hu et al., 2021]

 Pretraining encoder-decoders: what pretraining
objective to use?
For encoder-decoders, we could do something like language modeling, but where a
prefix of every input is provided to the encoder and is not predicted.
w 𝑇+2 , … ,
ℎ1 , … , ℎ𝑇 = Encoder (w1 , … , w T )
ℎ𝑇+1 , … , ℎ2 =Decoder( w1, … , w𝑇, ℎ1, … , ℎ𝑇)

The encoder portion benefits from

bidirectional context; the decoder portion is w 𝑇+1 , … , w 2𝑇
used to train the whole model through
language modeling.
w 1, … , w 𝑇
[Raffel et al., 2018]
 Pretraining encoder-decoders: what pretraining
objective to use?
What Raffel et al., 2018 found to work best was span corruption. Their model: T5.

Replace different-length spans from the input

with unique placeholders; decode out the
spans that were removed!

This is implemented in text

preprocessing: it’s still an objective
that looks like language modeling at
the decoder side.
 Pretraining encoder-decoders: what pretraining
objective to use?
A fascinating property
of T5: it can be
finetuned to answer a
wide range of
questions, retrieving
knowledge from its
parameters.

NQ: Natural Questions

220 million params
WQ: WebQuestions
770 million params
TQA: Trivia QA 3 billion params
11 billion params
All “open-domain”
versions [Raffel et al., 2018]
 Pretraining for three types of architectures
The neural architecture influences the type of pretraining, and natural use cases.

• Language models! What we’ve seen so far.

Decoders
• Nice to generate from; can’t condition on future words
• All the biggest pretrained models are Decoders.
 Pretraining decoders
When using language model pretrained decoders, we can ignore
that they were trained to model 𝑝 𝑤𝑡 |𝑤1:𝑡−1 . ☺/
We can finetune them by training a classifier Linear A, b
on the last word’s hidden state.

ℎ1 , … , ℎ𝑇
ℎ1, … , ℎ𝑇 = Decoder (w1, … , w𝑇)
𝑦~𝐴ℎ𝑇 + 𝑏
Where A and 𝑏 are randomly initialized and
specified by the downstream task. w 1, … , w 𝑇

Gradients backpropagate through the whole the linear layer hasn’t been pretrained
network. and must be learned from scratch.]
 Pretraining decoders
It’s natural to pretrain decoders as language models and then
use them as generators, finetuning their 𝑝𝜃 (𝑤𝑡 |𝑤1:𝑡−1)
w2 w3 w4 w5 w6
This is helpful in tasks where the output is a
sequence with a vocabulary like that at A, b
pretraining time!
• Dialogue (context=dialogue history) ℎ1 , … , ℎ𝑇
• Summarization (context=document)

w1 w2 w3 w4 w5

[Note how the linear layer has been pretrained.]

 Generative Pretrained Transformer (GPT)
[Radford et al., 2018]
2018’s GPT was a big success in pretraining a decoder!
• Transformer decoder with 12 layers, 117M parameters.
• 768-dimensional hidden states, 3072-dimensional feed-forward hidden layers.
• Byte-pair encoding with 40,000 merges
• Trained on BooksCorpus: over 7000 unique books.
• Contains long spans of contiguous text, for learning long-distance dependencies.

43 [Devlin et al., 2018]

 Generative Pretrained Transformer (GPT)
[Radford et al., 2018]
How do we format inputs to our decoder for finetuning tasks?

Natural Language Inference: Label pairs of sentences as entailing/contradictory/neutral

Premise: The man is in the doorway
entailment
Hypothesis: The person is near the door

Radford et al., 2018 evaluate on natural language inference.

Here’s roughly how the input was formatted, as a sequence of tokens for the decoder.

[START] The man is in the doorway [DELIM] The person is near the door [EXTRACT]

The linear classifier is applied to the representation of the [EXTRACT] token.

 GPT-3, In-context learning, and very large models
So far, we’ve interacted with pretrained models in two ways:
• Sample from the distributions they define (maybe providing a prompt)
• Fine-tune them on a task we care about, and take their predictions.

Very large language models seem to perform some kind of learning without gradient
steps simply from examples you provide within their contexts.

GPT-3 is the canonical example of this. The largest T5 model had 11 billion parameters.
GPT-3 has 175 billion parameters.
 GPT-3, In-context learning, and very large models
Very large language models seem to perform some kind of learning without gradient
steps simply from examples you provide within their contexts.

The in-context examples seem to specify the task to be performed, and the conditional
distribution mocks performing the task to a certain extent.
Input (prefix within a single Transformer decoder context):
“ thanks -> merci
hello -> bonjour
mint -> menthe
otter -> ”
Output (conditional generations):
loutre…”
The prefix as task specification and scratch pad:
chain-of-thought

[Wei et al., 2023]

 What kinds of things does pretraining teach?
There’s increasing evidence that pretrained models learn a wide variety of things about
the statistical properties of language. Taking our examples from the start of class:
• Stanford University is located in , California. [Trivia]
• I put fork down on the table. [syntax]
• The woman walked across the street, checking for traffic over shoulder. [coreference]
• I went to the ocean to see the fish, turtles, seals, and . [lexical semantics/topic]
• Overall, the value I got from the two hours watching it was the sum total of the popcorn
and the drink. The movie was _. [sentiment]
• Iroh went into the kitchen to make some tea. Standing next to Iroh, Zuko pondered his
destiny. Zuko left the . [some reasoning – this is harder]
• I was thinking about the sequence that goes 1, 1, 2, 3, 5, 8, 13, 21, [some basic
arithmetic; they don’t learn the Fibonnaci sequence]
• Models also learn – and can exacerbate racism, sexism, all manner of bad biases.
• More on all this in the interpretability lecture!
Hard to do with BERT

• BERT cannot generate text (at least not in an obvious way)

• Masked language models are intended to be used primarily for

“analysis” tasks
Effects of Model Size

• Big models help a lot

• Going from 110M -> 340M params helps even on datasets with 3,600 labelled
examples
• Improvements have not asymptoted

Slide from Jacob Delvin

 GPT et al.
• One-directional (forward)
transformer models do have one big
position-wise softmax advantage over BERT.

• Generation is not really possible with

position-wise nonlinear BERT, but a forward (masked
repeated Nx

network
attention) model can do it!

masked self-attention
• GPT (GPT-2, GPT-3, etc.) is a classic
example of this

position-wise encoder
 GPT et al.

position-wise softmax

position-wise nonlinear
repeated Nx

network

masked self-attention

position-wise encoder
Pretrained language models summary

bidirectional transformer one-directional transformer bidirectional LSTM

+ great representations + can generate text - OK representations
- can’t generate text - OK representations (largely supplanted by BERT)
 Pretrained language models summary
• Language models can be trained on very large and unlabeled datasets of text.
• Internal learned representations depend on context: the meaning of a word is informed
by the whole sentence!
• Can even get us representations of entire sentences (e.g., the first output token for BERT)
• Can be used to either extract representations to replace standard word embeddings…
• …or directly finetuned on downstream tasks (which means we modify all the weights in the
whole language model, rather than just using pretrained model hidden states)
Improved variants of BERT
RoBERTa XLNet
• Bigger batch size is better! Key Ideas
• Autoregressive: use context to predict
• Next sentence prediction the next word
doesn’t help • Bidirectional context from
• Pretrain on more data for as permutation language modeling
long as possible! • Self-attention mechanisms, uses
Transformer-XL backbone
ALBERT
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (Lan et al, Google
and TTI Chicago, 2019)
• Innovation #1: Factorized embedding parameterization
• Use small embedding size (e.g., 128) and then project it to Transformer hidden size (e.g., 1024) with parameter
matrix

• Innovation #2: Cross-layer parameter sharing

• Share all parameters between Transformer layers

1024 1024 128

x vs x x x
100k 128 100k

Slide from J. Delvin

GPT-2, BERT

• Transformers, GPT-2, and BERT

1. A transformer uses Encoder stack to model input, and uses Decoder stack
to model output (using input information from encoder side).
2. But if we do not have input, we just want to model the “next word”, we can
get rid of the Encoder side of a transformer and output “next word” one by
one. This gives us GPT.
3. If we are only interested in training a language model for the input for
some other tasks, then we do not need the Decoder of the transformer,
that gives us BERT.