Multiscale Vision Transformers

Haoqi Fan *, 1 Bo Xiong *, 1 Karttikeya Mangalam *, 1, 2

Yanghao Li *, 1 Zhicheng Yan 1 Jitendra Malik 1, 2 Christoph Feichtenhofer *, 1
1 2
Facebook AI Research UC Berkeley

Abstract
We present Multiscale Vision Transformers (MViT) for
video and image recognition, by connecting the seminal idea
of multiscale feature hierarchies with transformer models.
Multiscale Transformers have several channel-resolution W
C
scale stages. Starting from the input resolution and a small H
channel dimension, the stages hierarchically expand the Input scale1 scale2 scale3
channel capacity while reducing the spatial resolution. This Figure 1. Multiscale Vision Transformers learn a hierarchy from
creates a multiscale pyramid of features with early lay- dense (in space) and simple (in channels) to coarse and complex
ers operating at high spatial resolution to model simple features. Several resolution-channel scale stages progressively
low-level visual information, and deeper layers at spatially increase the channel capacity of the intermediate latent sequence
coarse, but complex, high-dimensional features. We eval- while reducing its length and thereby spatial resolution.
uate this fundamental architectural prior for modeling the
dense nature of visual signals for a variety of video recog- channel corresponding to ever more specialized features.
nition tasks where it outperforms concurrent vision trans- In a parallel development, the computer vision com-
formers that rely on large scale external pre-training and munity developed multiscale processing, sometimes called
are 5-10× more costly in computation and parameters. We “pyramid” strategies, with Rosenfeld and Thurston [91], Burt
further remove the temporal dimension and apply our model and Adelson [10], Koenderink [66], among the key papers.
for image classification where it outperforms prior work There were two motivations (i) To decrease the computing re-
on vision transformers. Code is available at: https: quirements by working at lower resolutions and (ii) A better
//github.com/facebookresearch/SlowFast. sense of “context” at the lower resolutions, which could then
guide the processing at higher resolutions (this is a precursor
to the benefit of “depth” in today’s neural networks.)
1. Introduction
The Transformer [104] architecture allows learning ar-
We begin with the intellectual history of neural network
bitrary functions defined over sets and has been scalably
models for computer vision. Based on their studies of cat
successful in sequence tasks such as language comprehen-
and monkey visual cortex, Hubel and Wiesel [60] developed
sion [29] and machine translation [9]. Fundamentally, a
a hierarchical model of the visual pathway with neurons
transformer uses blocks with two basic operations. First,
in lower areas such as V1 responding to features such as
is an attention operation [4] for modeling inter-element re-
oriented edges and bars, and in higher areas to more spe-
lations. Second, is a multi-layer perceptron (MLP), which
cific stimuli. Fukushima proposed the Neocognitron [37], a
models relations within an element. Intertwining these oper-
neural network architecture for pattern recognition explic-
ations with normalization [2] and residual connections [49]
itly motivated by Hubel and Wiesel’s hierarchy. His model
allows transformers to generalize to a wide variety of tasks.
had alternating layers of simple cells and complex cells, thus
incorporating downsampling, and shift invariance, thus incor- Recently, transformers have been applied to key com-
porating convolutional structure. LeCun et al. [70] took the puter vision tasks such as image classification. In the spirit
additional step of using backpropagation to train the weights of architectural universalism, vision transformers [28, 101]
of this network. But already the main aspects of hierarchy of approach performance of convolutional models across a va-
visual processing had been established: (i) Reduction in spa- riety of data and compute regimes. By only having a first
tial resolution as one goes up the processing hierarchy and layer that ‘patchifies’ the input in spirit of a 2D convolu-
(ii) Increase in the number of different “channels”, with each tion, followed by a stack of transformer blocks, the vision
transformer aims to showcase the power of the transformer
* Equal technical contribution. architecture using little inductive bias.

6824
 Kinetics top-1 val accuracy (%)
In this paper, our intention is to connect the seminal idea IN-21K
IN-21K IN-21K
of multiscale feature hierarchies with the transformer model.
+4.6% acc
We posit that the fundamental vision principle of resolution at 1/5 FLOPs
at 1/3 Params IN-1K
and channel scaling, can be beneficial for transformer models without ImageNet
across a variety of visual recognition tasks.
MViT-B 16x4
We present Multiscale Vision Transformers (MViT), a MViT-B 32x2
transformer architecture for modeling visual data such as im- [1] ViViT-L ImageNet-21K
ages and videos. Consider an input image as shown in Fig. 1. [6] TimeSformer ImageNet-21K
Unlike conventional transformers, which maintain a constant [78] VTN ImageNet-1K / 21K

channel capacity and resolution throughout the network,

Multiscale Transformers have several channel-resolution Inference cost per video in TFLOPs (# of multiply-adds x 1012)
‘scale’ stages. Starting from the image resolution and a small Figure 2. Accuracy/complexity trade-off on Kinetics-400 for
channel dimension, the stages hierarchically expand the varying # of inference clips per video shown in MViT curves.
channel capacity while reducing the spatial resolution. This Concurrent vision-transformer based methods [84,8,1] require over
creates a multiscale pyramid of feature activations inside the 5× more computation and large-scale external pre-training on
ImageNet-21K (IN-21K), to achieve equivalent MViT accuracy.
transformer network, effectively connecting the principles
of transformers with multi scale feature hierarchies.
We further apply MViT to ImageNet [24] classification,
Our conceptual idea provides an effective design advan-
by simply removing the temporal dimension of the video
tage for vision transformer models. The early layers of our
architecture, and show significant gains over single-scale vi-
architecture can operate at high spatial resolution to model
sion transformers for image recognition. Our code and mod-
simple low-level visual information, due to the lightweight
els are available in PySlowFast [31] and PyTorchVideo [32].
channel capacity. In turn, the deeper layers can effectively
focus on spatially coarse but complex high-level features
to model visual semantics. The fundamental advantage of 2. Related Work
our multiscale transformer arises from the extremely dense Convolutional networks (ConvNets). Incorporating down-
nature of visual signals, a phenomenon that is even more sampling, shift invariance, and shared weights, ConvNets
pronounced for space-time visual signals captured in video. are de-facto standard backbones for computer vision tasks
A noteworthy benefit of our design is the presence of for image [70, 67, 94, 96, 51, 15, 18, 39, 99, 87, 46] and
strong implicit temporal bias. We show that vision trans- video [93, 36, 14, 85, 74, 112, 102, 34, 111, 40, 33, 123, 62].
former models [28] trained on natural video suffer no per- Self-attention in ConvNets. Self-attention mechanisms has
formance decay when tested on videos with shuffled frames. been used for image understanding [88, 120, 57, 7], unsu-
This indicates that these models are not effectively using pervised object recognition [80] as well as vision and lan-
the temporal information and instead rely heavily on appear- guage [83, 71]. Hybrids of self-attention operations and
ance. In contrast, when testing our MViT models on shuffled convolutional networks have also been applied to image
frames, we observe significant accuracy decay, suggesting understanding [56] and video recognition [107].
reliance on temporal information. Vision Transformers. Much of current enthusiasm in ap-
Our focus in this paper is video recognition, and we de- plication of Transformers [104] to vision tasks commences
sign and evaluate MViT for video tasks (Kinetics [64, 12], with the Vision Transformer (ViT) [28] and Detection Trans-
Charades [92], SSv2 [43] and AVA [44]). MViT provides former [11]. We build directly upon [28] with a staged model
a significant performance gain over concurrent video trans- allowing channel expansion and resolution downsampling.
formers [84, 8, 1], without any external pre-training data. DeiT [101] proposes a data efficient approach to training ViT.
In Fig. A.4 we show the computation/accuracy trade-off Our training recipe builds on, and we compare our image
for video-level inference, when varying the number of tem- classification models to, DeiT under identical settings.
poral clips used in MViT. The vertical axis shows accuracy An emerging thread of work aims at applying transform-
on Kinetics-400 and the horizontal axis the overall infer- ers to vision tasks such as object detection [5], semantic
ence cost in FLOPs for different models, MViT and concur- segmentation [121, 105], 3D reconstruction [77], pose esti-
rent ViT [28] video variants: VTN [84], TimeSformer [8], mation [113], generative modeling [17], image retrieval [30],
ViViT [1]. To achieve similar accuracy level as MViT, these medical image segmentation [16,103,117], point clouds [45],
models require significant more computation and parameters video instance segmentation [109], object re-identification
(e.g. ViViT-L [1] has 6.8× higher FLOPs and 8.5× more pa- [52], video retrieval [38], video dialogue [69], video object
rameters at equal accuracy, more analysis in §A.2) and need detection [116] and multi-modal tasks [78, 26, 86, 58, 114].
large-scale external pre-training on ImageNet-21K (which A separate line of works attempts at modeling visual data
contains around 60× more labels than Kinetics-400). with learnt discretized token sequences [110, 89, 17, 115, 21].

6825
Efficient Transformers. Recent works [106, 65, 20, 100, 23,
Add & Norm
19, 72, 6] reduce the quadratic attention complexity to make
^^^ × D
THW
transformers more efficient for natural language processing
applications, which is complementary to our approach. MatMul
Three concurrent works propose a ViT-based architecture
for video [84, 8, 1]. However, these methods rely on pre- Softmax
^^^ × THW
THW ~~~ ~~~ × D
training on vast amount of external data such as ImageNet- THW
V
21K [24], and thus use the vanilla ViT [28] with minimal MatMul & Scale
adaptations. In contrast, our MViT introduces multiscale ^^^ × D
THW ~~~ × D
THW
Q K
feature hierarchies for transformers, allowing effective mod-
PoolQ PoolQ PoolK PoolV
eling of dense visual input without large-scale external data.
THW × D THW × D THW × D
^
Q ^
K ^
V
3. Multiscale Vision Transformer (MViT) Linear Linear Linear
Our generic Multiscale Transformer architecture builds
on the core concept of stages. Each stage consists of multiple THW × D
X
transformer blocks with specific space-time resolution and Figure 3. Pooling Attention is a flexible attention mechanism that
channel dimension. The main idea of Multiscale Transform- (i) allows obtaining the reduced space-time resolution (T̂ Ĥ Ŵ ) of
ers is to progressively expand the channel capacity, while the input (T HW ) by pooling the query, Q = P(Q̂; ΘQ ), and/or
pooling the resolution from input to output of the network. (ii) computes attention on a reduced length (T̃ H̃ W̃ ) by pooling the
key, K = P(K̂; ΘK ), and value, V = P(V̂ ; ΘV ), sequences.
3.1. Multi Head Pooling Attention
We first describe Multi Head Pooling Attention (MHPA), input tensor of dimensions L = T × H × W to L̃ given by,
a self attention operator that enables flexible resolution mod-  
eling in a transformer block allowing Multiscale Transform- L + 2p − k
L̃ = +1
ers to operate at progressively changing spatiotemporal reso- s
lution. In contrast to original Multi Head Attention (MHA)
with the equation applying coordinate-wise. The pooled
operators [104], where the channel dimension and the spa-
tensor is flattened again yielding the output of P(Y ; Θ) ∈
tiotemporal resolution remains fixed, MHPA pools the se-
RL̃×D with reduced sequence length, L̃ = T̃ × H̃ × W̃ .
quence of latent tensors to reduce the sequence length (reso-
By default we use overlapping kernels k with shape-
lution) of the attended input. Fig. 3 shows the concept.
preserving padding p in our pooling attention operators, so
Concretely, consider a D dimensional input tensor X
that L̃ , the sequence length of the output tensor P(Y ; Θ),
of sequence length L, X ∈ RL×D . Following MHA [28],
experiences an overall reduction by a factor of sT sH sW .
MHPA projects the input X into intermediate query tensor
Q̂ ∈ RL×D , key tensor K̂ ∈ RL×D and value tensor V̂ ∈ Pooling Attention. The pooling operator P (·; Θ) is applied
RL×D with linear operations to all the intermediate tensors Q̂, K̂ and V̂ independently
with chosen pooling kernels k, stride s and padding p. De-
Q̂ = XWQ K̂ = XWK V̂ = XWV noting θ yielding the pre-attention vectors Q = P(Q̂; ΘQ ),
K = P(K̂; ΘK ) and V = P(V̂ ; ΘV ) with reduced se-
/ with weights WQ , WK , WV of dimensions D × D. The quence lengths. Attention is now computed on these short-
obtained intermediate tensors are then pooled in sequence ened vectors, with the operation,
length, with a pooling operator P as described below. √
Attention(Q, K, V ) = Softmax(QK T / D)V.
Pooling Operator. Before attending the input, the interme-
Naturally, the operation induces the constraints sK ≡ sV
diate tensors Q̂, K̂, V̂ are pooled with the pooling operator
on the pooling operators. In summary, pooling attention is
P(·; Θ) which is the cornerstone of our MHPA and, by ex-
computed as,
tension, of our Multiscale Transformer architecture.
√
The operator P(·; Θ) performs a pooling kernel com- PA(·) = Softmax(P(Q; ΘQ )P(K; ΘK )T / d)P(V ; ΘV ),
putation on the input tensor along each of the dimensions. √
Unpacking Θ as Θ := (k, s, p), the operator employs a where d is normalizing the inner product matrix row-wise.
pooling kernel k of dimensions kT × kH × kW , a stride s The output of the Pooling attention operation thus has its
of corresponding dimensions sT × sH × sW and a padding sequence length reduced by a stride factor of sQ Q Q
T sH sW fol-
p of corresponding dimensions pT × pH × pW to reduce an lowing the shortening of the query vector Q in P(·).

6826
 stage operators output sizes stages operators output sizes
data layer stride τ ×1×1 T ×H×W data layer stride τ ×1×1 D×T ×H×W
1×16×16, D cT ×cH ×cW , D
patch1 H
D×T × 16 ×W cube1 D× sT × H
4
×W
4
stride 1×16×16 16 stride sT ×4×4 T
   
MHA(D) MHPA(D)
scale2 ×N H
D×T × 16 ×W scale2 ×N2 D× sT ×H
4
×W
4
MLP(4D) 16 MLP(4D) T
 
MHPA(2D)
Table 1. Vision Transformers (ViT) base model starts from a scale3 ×N3 2D× sT × H
8
×W
8
MLP(8D) T
data layer that samples visual input with rate τ ×1×1 to T ×H×W  
MHPA(4D)
resolution, where T is the number of frames H height and W width. scale4 ×N4 4D× sT × 16
H
×W
16
MLP(16D) T
The first layer, patch1 projects patches (of shape 1×16×16) to form  
MHPA(8D)
a sequence, processed by a stack of N transformer blocks (stage2 ) scale5 ×N5 8D× sT × 32
H
×W
32
MLP(32D) T
H
at uniform channel dimension (D) and resolution (T × 16 ×W16
).
Table 2. Multiscale Vision Transformers (MViT) base model.
Layer cube1 , projects dense space-time cubes (of shape ct ×cy ×cw )
Multiple heads. As in [104] the computation can be paral- to D channels to reduce spatiotemporal resolution to sTT × H 4
×W4
.
lelized by considering h heads where each head is perform- The subsequent stages progressively down-sample this resolution
(at beginning of a stage) with MHPA while simultaneously increas-
ing the pooling attention on a non overlapping subset of D/h
ing the channel dimension, in MLP layers, (at the end of a stage).
channels of the D dimensional input tensor X. Each stage consists of N∗ transformer blocks, denoted in [brackets].
Computational Analysis. Since attention computation
scales quadratically w.r.t. the sequence length, pooling the dimension D to encode the positional information and break
key, query and value tensors has dramatic benefits on the permutation invariance. A learnable class embedding is
fundamental compute and memory requirements of the Mul- appended to the projected image patches.
tiscale Transformer model. Denoting the sequence length The resulting sequence of length of L + 1 is then pro-
reduction factors by fQ , fK and fV we have, cessed sequentially by a stack of N transformer blocks, each
one performing attention (MHA [104]), multi-layer percep-
fj = sjT · sjH · sjW , ∀ j ∈ {Q, K, V }. tron (MLP) and layer normalization (LN [3]) operations.
Considering X to be the input of the block, the output of a
Considering the input tensor to P(; Θ) to have dimensions single transformer block, Block(X) is computed by
D × T × H × W , the run-time complexity of MHPA is
O(T HW D/h(D + T HW/fQ fK )) per head and the mem- X1 = MHA(LN(X)) + X
ory complexity is O(T HW h(D/h + T HW/fQ fK )). Block(X) = MLP(LN(X1 )) + X1 .
This trade-off between the number of channels D and
sequence length term T HW/fQ fK informs our design The resulting sequence after N consecutive blocks is layer-
choices about architectural parameters such as number of normalized and the class embedding is extracted and passed
heads and width of layers. We refer the reader to §C for a through a linear layer to predict the desired output (e.g. class).
detailed analysis and discussions on the runtime-memory By default, the hidden dimension of the MLP is 4D. We
complexity trade-off. refer the reader to [28, 104] for details.
In context of the present paper, it is noteworthy that ViT
3.2. Multiscale Transformer Networks maintains a constant channel capacity and spatial resolution
Building upon Multi Head Pooling Attention (Sec. 3.1), throughout all the blocks (see Table 1).
we describe the Multiscale Transformer model for visual Multiscale Vision Transformers (MViT). Our key con-
representation learning using exclusively MHPA and MLP cept is to progressively grow the channel resolution (i.e. di-
layers. First, we present a brief review of the Vision Trans- mension), while simultaneously reducing the spatiotemporal
former Model that informs our design. resolution (i.e. sequence length) throughout the network. By
Preliminaries: Vision Transformer (ViT). The Vision design, our MViT architecture has fine spacetime (and coarse
Transformer (ViT) architecture [28] starts by dicing the input channel) resolution in early layers that is up-/downsampled
video of resolution T ×H×W , where T is the number of to a coarse spacetime (and fine channel) resolution in late
frames H the height and W the width, into non-overlapping layers. MViT is shown in Table 2.
patches of size 1×16×16 each, followed by point-wise ap- Scale stages. A scale stage is defined as a set of N trans-
plication of linear layer on the flattened image patches to to former blocks that operate on the same scale with identi-
project them into the latent dimension, D, of the transformer. cal resolution across channels and space-time dimensions
This is equivalent to a convolution with equal kernel size D×T ×H×W . At the input (cube1 in Table 2), we project
and stride of 1×16×16 and is shown as patch1 stage in the the patches (or cubes if they have a temporal extent) to a
model definition in Table 1. smaller channel dimension (e.g. 8× smaller than a typical
Next, a positional embedding E ∈ RL×D is added to ViT model), but long sequence (e.g. 4×4 = 16× denser than
each element of the projected sequence of length L with a typical ViT model; cf. Table 1).

6827
 stage operators output sizes stage operators output sizes stage operators output sizes
data stride 8×1×1 8×224×224 data stride 4×1×1 16×224×224 data stride 4×1×1 16×224×224
1×16×16, 768 3×7×7, 96 3×8×8, 128
patch1 768×8×14×14 cube1 96×8×56×56 cube1 128×8×28×28
stride 1×16×16 stride 2×4×4 stride 2×8×8
     
MHA(768) MHPA(96) MHPA(128)
scale2 ×12 768×8×14×14 scale2 ×1 96×8×56×56 scale2 ×3 128×8×28×28
MLP(3072) MLP(384) MLP(512)
   
MHPA(192) MHPA(256)
scale3 ×2 192×8×28×28 scale3 ×7 256×8×14×14
MLP(768) MLP(1024)
   
MHPA(384) MHPA(512)
scale4 ×11 384×8×14×14 scale4 ×6 512×8×7×7
MLP(1536) MLP(2048)
 
MHPA(768)
scale5 ×2 768×8×7×7
MLP(3072)
(a) ViT-B with 179.6G FLOPs, 87.2M param, (b) MViT-B with 70.5G FLOPs, 36.5M param, (c) MViT-S with 32.9G FLOPs, 26.1M param,
16.8G memory, and 68.5% top-1 accuracy. 6.8G memory, and 77.2% top-1 accuracy. 4.3G memory, and 74.3% top-1 accuracy.
Table 3. Comparing ViT-B to two instantiations of MViT with varying complexity, MViT-S in (c) and MViT-B in (b). MViT-S operates at
a lower spatial resolution and lacks a first high-resolution stage. The top-1 accuracy corresponds to 5-Center view testing on K400. FLOPs
correspond to a single inference clip, and memory is for a training batch of 4 clips. See Table 2 for the general MViT-B structure.
At a stage transition (e.g. scale1 to scale2 to in Table 2), Skip connections. Since the channel dimension and se-
the channel dimension of the processed sequence is up- quence length change inside a residual block, we pool the
sampled while the length of the sequence is down-sampled. skip connection to adapt to the dimension mismatch between
This effectively reduces the spatiotemporal resolution of the its two ends. MHPA handles this mismatch by adding the
underlying visual data while allowing the network to assimi- query pooling operator P(·; ΘQ ) to the residual path. As
late the processed information in more complex features. shown in Fig. 3, instead of directly adding the input X of
MHPA to the output, we add the pooled input X to the
Channel expansion. When transitioning from one stage
output, thereby matching the resolution to attended query Q.
to the next, we expand the channel dimension by increas-
ing the output of the final MLP layer in the previous stage For handling the channel dimension mismatch between
by a factor that is relative to the resolution change intro- stage changes, we employ an extra linear layer that operates
duced at the stage. Concretely, if we down-sample the on the layer-normalized output of our MHPA operation. Note
space-time resolution by 4×, we increase the channel di- that this differs from the other (resolution-preserving) skip-
mension by 2×. For example, scale3 to scale4 changes reso- connections that operate on the un-normalized signal.
lution from 2D× sTT × H T T H T
8 × 8 to 4D× sT × 16 × 16 in Table 2. 3.3. Network instantiation details
This roughly preserves the computational complexity across
stages, and is similar to ConvNet design principles [93, 50]. Table 3 shows concrete instantiations of the base mod-
els for Vision Transformers [28] and our Multiscale Vision
Query pooling. The pooling attention operation affords
Transformers. ViT-Base [28] (Table 3b) initially projects
flexibility not only in the length of key and value vectors
the input to patches of shape 1×16×16 with dimension
but also in the length of the query, and thereby output, se-
D = 768, followed by stacking N = 12 transformer
quence. Pooling the query vector P(Q; k; p; s) with a kernel
blocks. With an 8×224×224 input the resolution is fixed to
s ≡ (sQ Q Q
T , sH , sW ) leads to sequence reduction by a factor of 768×8×14×14 throughout all layers. The sequence length
sT · sH · sQ
Q Q
W . Since, our intention is to decrease resolution (spacetime resolution + class token) is 8 · 14 · 14 + 1 = 1569.
at the beginning of a stage and then preserve this resolution Our MViT-Base (Table 3b) is comprised of 4 scale stages,
throughout the stage, only the first pooling attention operator each having several transformer blocks of consistent channel
of each stage operates at non-degenerate query stride sQ > 1, dimension. MViT-B initially projects the input to a channel
with all other operators constrained to sQ ≡ (1, 1, 1). dimension of D = 96 with overlapping space-time cubes of
Key-Value pooling. Unlike Query pooling, changing the se- shape 3×7×7. The resulting sequence of length 8 ∗ 56 ∗ 56 +
quence length of key K and value V tensors, does not change 1 = 25089 is reduced by a factor of 4 for each additional
the output sequence length and, hence, the space-time resolu- stage, to a final sequence length of 8 ∗ 7 ∗ 7 + 1 = 393 at
tion. However, they play a key role in overall computational scale4 . In tandem, the channel dimension is up-sampled by
requirements of the pooling attention operator. a factor of 2 at each stage, increasing to 768 at scale4 . Note
We decouple the usage of K, V and Q pooling, with that all pooling operations, and hence the resolution down-
Q pooling being used in the first layer of each stage and sampling, is performed only on the data sequence without
K, V pooling being employed in all other layers. Since the involving the processed class token embedding.
sequence length of key and value tensors need to be identical We set the number of MHPA heads to h = 1 in the scale1
to allow attention weight calculation, the pooling stride used stage and increase the number of heads with the channel
on K and value V tensors needs to be identical. In our dimension (channels per-head D/h remain consistent at 96).
default setting, we constrain all pooling parameters (k; p; s) At each stage transition, the previous stage output MLP
to be identical i.e. ΘK ≡ ΘV within a stage, but vary s dimension is increased by 2× and MHPA pools on Q tensors
adaptively w.r.t. to the scale across stages. with sQ = (1, 2, 2) at the input of the next stage.

6828
 model pre-train top-1 top-5 FLOPs×views Param model pretrain top-1 top-5 GFLOPs×views Param
Two-Stream I3D [14] - 71.6 90.0 216 × NA 25.0 SlowFast 16×8 +NL [34] - 81.8 95.1 234×3×10 59.9
ip-CSN-152 [102] - 77.8 92.8 109×3×10 32.8 X3D-M - 78.8 94.5 6.2×3×10 3.8
SlowFast 8×8 +NL [34] - 78.7 93.5 116×3×10 59.9 X3D-XL - 81.9 95.5 48.4×3×10 11.0
SlowFast 16×8 +NL [34] - 79.8 93.9 234×3×10 59.9 ViT-B-TimeSformer [8] IN-21K 82.4 96.0 1703×3×1 121.4
X3D-M [33] - 76.0 92.3 6.2×3×10 3.8 ViT-L-ViViT [1] IN-21K 83.0 95.7 3992×3×4 310.8
X3D-XL [33] - 79.1 93.9 48.4×3×10 11.0 MViT-B, 16×4 - 82.1 95.7 70.5×1×5 36.8
ViT-B-VTN [84] ImageNet-1K 75.6 92.4 4218×1×1 114.0 MViT-B, 32×3 - 83.4 96.3 170×1×5 36.8
ViT-B-VTN [84] ImageNet-21K 78.6 93.7 4218×1×1 114.0 MViT-B-24, 32×3 - 84.1 96.5 236×1×5 52.9
ViT-B-TimeSformer [8] ImageNet-21K 80.7 94.7 2380×3×1 121.4 Table 5. Comparison with previous work on Kinetics-600.
ViT-L-ViViT [1] ImageNet-21K 81.3 94.7 3992×3×4 310.8
ViT-B (our baseline) ImageNet-21K 79.3 93.9 180×1×5 87.2
The first Table 4 section shows prior art using ConvNets.
ViT-B (our baseline) - 68.5 86.9 180×1×5 87.2 The second section shows concurrent work using Vision
MViT-S - 76.0 92.1 32.9×1×5 26.1 Transformers [28] for video classification [84, 8]. Both ap-
MViT-B, 16×4 - 78.4 93.5 70.5×1×5 36.6 proaches rely on ImageNet pre-trained base models. ViT-B-
MViT-B, 32×3 - 80.2 94.4 170×1×5 36.6 VTN [84] achieves 75.6% top-1 accuracy, which is boosted
MViT-B, 64×3 - 81.2 95.1 455×3×3 36.6 by 3% to 78.6% by merely changing the pre-training from
Table 4. Comparison with previous work on Kinetics-400. We
ImageNet-1K to ImageNet-21K. ViT-B-TimeSformer [8]
report the inference cost with a single “view" (temporal clip with
spatial crop) × the number of views (FLOPs×viewspace ×viewtime ). shows another 2.1% gain on top of VTN, at higher cost of
Magnitudes are Giga (109 ) for FLOPs and Mega (106 ) for Param. 7140G FLOPs and 121.4M parameters. ViViT improves
Accuracy of models trained with external data is de-emphasized. accuracy further with an even larger ViT-L model.
The third section in Table 4 shows our ViT baselines. We
We employ K, V pooling in all MHPA blocks, with first list our ViT-B, also pre-trained on the ImageNet-21K,
ΘK ≡ ΘV and sK,V = (1, 8, 8) in scale1 and adaptively which achieves 79.3%, thereby being 1.4% lower than ViT-B-
decay this stride w.r.t. to the scale across stages such that the TimeSformer, but is with 4.4× fewer FLOPs and 1.4× fewer
K, V tensors have consistent scale across all blocks. parameters. This result shows that simply fine-tuning an
off-the-shelf ViT-B model from ImageNet-21K [28] provides
4. Experiments: Video Recognition a strong baseline on Kinetics. However, training this model
from-scratch with the same fine-tuning recipe will result
Datasets. We use Kinetics-400 [64] (K400) (∼240k train-
in 34.3%. Using our “training-from-scratch” recipe will
ing videos in 400 classes) and Kinetics-600 [12]. We fur-
produce 68.5% for this ViT-B model, using the same 1×5,
ther assess transfer learning performance for on Something-
spatial × temporal, views for video-level inference.
Something-v2 [43], Charades [92], and AVA [44].
The final section of Table 4 lists our MViT results. All our
We report top-1 and top-5 classification accuracy (%) on
models are trained-from-scratch using this recipe, without
the validation set, computational cost (in FLOPs) of a single,
any external pre-training. Our small model, MViT-S pro-
spatially center-cropped clip and the number of clips used.
duces 76.0% while being relatively lightweight with 26.1M
Training. By default, all models are trained from random param and 32.9×5=164.5G FLOPs, outperforming ViT-B
initialization (“from scratch”) on Kinetics, without using by +7.5% at 5.5× less compute in identical train/val setting.
ImageNet [25] or other pre-training. Our training recipe and Our base model, MViT-B provides 78.4%, a +9.9% accu-
augmentations follow [34, 101]. For Kinetics, we train for racy boost over ViT-B under identical settings, while having
200 epochs with 2 repeated augmentation [55] repetitions. 2.6×/2.4×fewer FLOPs/parameters. When changing the
We report ViT baselines that are fine-tuned from Ima- frame sampling from 16×4 to 32×3 performance increases
geNet, using a 30-epoch version of the training recipe in [34]. to 80.2%. Finally, we take this model and fine-tune it for 5
For the temporal domain, we sample a clip from the full- epochs with longer 64 frame input, after interpolating the
length video, and the input to the network are T frames with temporal positional embedding, to reach 81.2% top-1 using
a temporal stride of τ ; denoted as T × τ [34]. 3 spatial and 3 temporal views for inference (it is sufficient
Inference. We apply two testing strategies following [34, test with fewer temporal views if a clip has more frames).
33]: (i) Temporally, uniformly samples K clips (e.g. K=5) Further quantitative and qualitative results are in §A.
from a video, scales the shorter spatial side to 256 pixels and Kinetics-600 [12] is a larger version of Kinetics. Results
takes a 224×224 center crop, and (ii), the same as (i) tempo- are in Table 5. We train MViT from-scratch, without any
rally, but take 3 crops of 224×224 to cover the longer spatial pre-training. MViT-B, 16×4 achieves 82.1% top-1 accu-
axis. We average the scores for all individual predictions. racy. We further train a deeper 24-layer model with longer
All implementation specifics are in §D. sampling, MViT-B-24, 32×3, to investigate model scale on
this larger dataset. MViT achieves state-of-the-art of 83.4%
4.1. Main Results
with 5-clip center crop testing while having 56.0× fewer
Kinetics-400. Table 4 compares to prior work. From top- FLOPs and 8.4× fewer parameters than ViT-L-ViViT [1]
to-bottom, it has four sections and we discuss them in turn. which relies on large-scale ImageNet-21K pre-training.

6829
 model pretrain top-1 top-5 FLOPs×views Param model pretrain val mAP FLOPs Param
TSM-RGB [76] IN-1K+K400 63.3 88.2 62.4×3×2 42.9 SlowFast, 4×16, R50 [34] 21.9 52.6 33.7
MSNet [68] IN-1K 64.7 89.4 67×1×1 24.6 SlowFast, 8×8, R101 [34] 23.8 138 53.0
TEA [73] IN-1K 65.1 89.9 70×3×10 - MViT-B, 16×4 K400 24.5 70.5 36.4
ViT-B-TimeSformer [8] IN-21K 62.5 - 1703×3×1 121.4 MViT-B, 32×3 26.8 170 36.4
ViT-B (our baseline) IN-21K 63.5 88.3 180×3×1 87.2 MViT-B, 64×3 27.3 455 36.4
SlowFast R50, 8×8 [34] 61.9 87.0 65.7×3×1 34.1
SlowFast, 8×8 R101+NL [34] 27.1 147 59.2
SlowFast R101, 8×8 [34] 63.1 87.6 106×3×1 53.3
SlowFast, 16×8 R101+NL [34] 27.5 296 59.2
MViT-B, 16×4 K400 64.7 89.2 70.5×3×1 36.6
X3D-XL [33] 27.4 48.4 11.0
MViT-B, 32×3 67.1 90.8 170×3×1 36.6 K600
MViT-B, 64×3 67.7 90.9 455×3×1 36.6
MViT-B, 16×4 26.1 70.5 36.3
MViT-B, 16×4 66.2 90.2 70.5×3×1 36.6 MViT-B, 32×3 27.5 170 36.4
MViT-B, 32×3 K600 67.8 91.3 170×3×1 36.6 MViT-B-24, 32×3 28.7 236 52.9
MViT-B-24, 32×3 68.7 91.5 236×3×1 53.2 Table 8. Comparison with previvous work on AVA v2.2. All
Table 6. Comparison with previous work on SSv2. methods use single center crop inference following [33].

Something-Something-v2 (SSv2) [43] is a dataset with

videos containing object interactions, which is known as 4.2. Ablations on Kinetics
a ‘temporal modeling‘ task. Table 6 compares our method We carry out ablations on Kinetics-400 (K400) using 5-
with the state-of-the-art. We first report a simple ViT-B clip center 224×224 crop testing. We show top-1 accuracy
(our baseline) that uses ImageNet-21K pre-training. Our (Acc), as well as computational complexity measured in
MViT-B with 16 frames has 64.7% top-1 accuracy, which is GFLOPs for a single clip input of spatial size 2242 . Infer-
better than the SlowFast R101 [34] which shares the same ence computational cost is proportional as a fixed number
setting (K400 pre-training and 3×1 view testing). With more of 5 clips is used (to roughly cover the inferred videos with
input frames, our MViT-B achieves 67.7% and the deeper T ×τ =16×4 sampling.) We also report Parameters in M(106 )
MViT-B-24 achieves 68.7% using our K600 pre-trained and training GPU memory in G(109 ) for a batch size of 4. By
model of above. In general, Table 6 verifies the capability of default all MViT ablations are with MViT-B, T ×τ =16×4
temporal modeling for MViT. and max-pooling in MHSA.
model pretrain mAP FLOPs×views Param model shuffling FLOPs (G) Param (M) Acc
Nonlocal [107] 37.5 544×3×10 54.3 MViT-B 77.2
IN-1K+K400
STRG +NL [108] 39.7 630×3×10 58.3 70.5 36.5
MViT-B X 70.1 (−7.1)
Timeception [61] 41.1 N/A×N/A N/A
ViT-B 68.5
LFB +NL [111] 42.5 529×3×10 122 179.6 87.2
ViT-B X 68.4 (−0.1)
SlowFast 50, 8×8 [34] 38.0 65.7×3×10 34.0
SlowFast 101+NL, 16×8 [34] 42.5 234×3×10 59.9
Table 9. Shuffling frames in inference. MViT-B severely drops
K400 (−7.1%) for shuffled temporal input, but ViT-B models appear to
X3D-XL [33] 43.4 48.4×3×10 11.0
MViT-B, 16×4 40.0 70.5×3×10 36.4 ignore temporal information as accuracy remains similar (−0.1%).
MViT-B, 32×3 44.3 170×3×10 36.4
MViT-B, 64×3 46.3 455×3×10 36.4 Frame shuffling. Table 9 shows results for randomly shuf-
SlowFast R101+NL, 16×8 [34] 45.2 234×3×10 59.9 fling the input frames in time during testing. All models are
X3D-XL [33] 47.1 48.4×3×10 11.0 trained without any shuffling and have temporal embeddings.
MViT-B, 16×4 K600 43.9 70.5×3×10 36.4 We notice that our MViT-B architecture suffers a significant
MViT-B, 32×3 47.1 170×3×10 36.4 accuracy drop of -7.1% (77.2 → 70.1) for shuffling infer-
MViT-B-24, 32×3 47.7 236×3×10 53.0
Table 7. Comparison with previous work on Charades.
ence frames. By contrast, ViT-B is surprisingly robust for
shuffling the temporal order of the input.
Charades [92] is a dataset with longer range activities. We
validate our model in Table 7. With similar FLOPs and kernel k pooling func Param Acc
s max 36.5 76.1
parameters, our MViT-B 16×4 achieves better results (+2.0
2s + 1 max 36.5 75.5
mAP) than SlowFast R50 [34]. As shown in the Table, the s+1 max 36.5 77.2
performance of MViT-B is further improved by increasing s+1 average 36.5 75.4
the number of input frames and MViT-B layers and using s+1 conv 36.6 78.3
K600 pre-trained models. 3×3×3 conv 36.6 78.4
Table 10. Pooling function: Varying the kernel k as a function of
AVA [44] is a dataset with for spatiotemporal-localization
stride s. Functions are average or max pooling and conv which is a
of human actions. We validate our MViT on this detection learnable, channel-wise convolution.
task. Details about the detection architecture of MViT can
be found in §D.2. Table 8 shows the results of our MViT Pooling function. The ablation in Table 10 looks at the
models compared with SlowFast [34] and X3D [33]. We kernel size k w.r.t. the stride s, and the pooling function
observe that MViT-B can be competitive to SlowFast and (max/average/conv). First, we see that having equivalent
X3D using the same pre-training and testing strategy. kernel and stride k=s provides 76.1%, increasing the kernel

6830
size to k=2s+1 decays to 75.5%, but using a kernel k=s+1 model Acc FLOPs (G) Param (M)
gives a clear benefit of 77.2%. This indicates that overlap- RegNetZ-4GF [27] 83.1 4.0 28.1
RegNetZ-16GF [27] 84.1 15.9 95.3
ping pooling is effective, but a too large overlap (2s+1) hurts. EfficientNet-B7 [99] 84.3 37.0 66.0
Second, we investigate average instead of max-pooling and DeiT-S [101] 79.8 4.6 22.1
observe that accuracy decays by from 77.2% to 75.4%. DeiT-B [101] 81.8 17.6 86.6
Third, we use conv-pooling by a learnable, channelwise DeiT-B ↑ 3842 [101] 83.1 55.5 87.0
convolution followed by LN. This variant has +1.2% over Swin-B (concurrent) [79] 83.3 15.4 88.0
Swin-B ↑ 3842 (concurrent) [79] 84.2 47.0 88.0
max pooling and is used for all experiments in §4.1 and §5.
MViT-B-16, max-pool 82.5 7.8 37.0
MViT-B-16 83.0 7.8 37.0
model clips/sec Acc FLOPs×views Param
MViT-B-24 84.0 14.7 72.7
X3D-M [33] 7.9 74.1 4.7×1×5 3.8 MViT-B-24-3202 84.8 32.7 72.9
SlowFast R50 [34] 5.2 75.7 65.7×1×5 34.6
Table 12. Comparison to prior work on ImageNet. RegNet
SlowFast R101 [34] 3.2 77.6 125.9×1×5 62.8
ViT-B [28] 3.6 68.5 179.6×1×5 87.2
and EfficientNet are ConvNet examples that use different training
MViT-S, max-pool 12.3 74.3 32.9×1×5 26.1
recipes. DeiT/MViT are ViT-based and use identical recipes [101].
MViT-B, max-pool 6.3 77.2 70.5×1×5 36.5
MViT-S, conv-pool 9.4 76.0 32.9×1×5 26.1
The bottom section in Table 12 shows results for our
MViT-B, conv-pool 4.8 78.4 70.5×1×5 36.6
Table 11. Speed-Accuracy tradeoff on Kinetics-400. Training Multiscale Vision Transformer (MViT) models.
throughput is measured in clips/s. MViT is fast and accurate. We show models of different depth, MViT-B-Depth, (16
and 24 layers), where MViT-B-16 is our base model and the
Speed-Accuracy tradeoff. In Table 11, we analyze the deeper variant is simply created by repeating the number of
speed/accuracy trade-off of our MViT models, along with blocks N∗ in each scale stage (cf. Table 3b) and using a larger
their counterparts vision transformer (ViT [28]) and Con- channel dimension of D = 112. All our models are trained
vNets (SlowFast 8×8 R50, SlowFast 8×8 R101 [34], & using the identical 300-epoch recipe as DeiT [101], except
X3D-L [33]). We measure training throughput as the num- repeated augmentation which we found not beneficial.
ber of video clips per second on a single M40 GPU.
We observe that both MViT-S and MViT-B models are We make the following observations:
not only significantly more accurate but also much faster (i) Our lightweight MViT-B-16 achieves 82.5% top-1
than both the ViT-B baseline and convolutional models. Con- accuracy, with only 7.8 GFLOPs, which outperforms the
cretely, MViT-S has 3.4× higher throughput speed (clips/s), DeiT-B counterpart by +0.7% with lower computation cost
is +5.8% more accurate (Acc), and has 3.3× fewer param- (2.3×fewer FLOPs and Parameters). If we use conv instead
eters (Param) than ViT-B. Using a conv instead of max- of max-pooling, this number is increased by +0.5% to 83.0%.
pooling in MHSA, we observe a training speed reduction of (ii) Our deeper model MViT-B-24, provides a gain of
∼20% for convolution and additional parameter updates. +1.0% accuracy at slight increase in computation.
(iii) A larger model, MViT-B-24-3202 with input resolu-
5. Experiments: Image Recognition tion 3202 reaches 84.8%, corresponding to a +1.7% gain, at
We apply our video models on static image recognition by 1.7×fewer FLOPs, over DeiT-B↑3842 . These results suggest
using them with single frame, T = 1, on ImageNet-1K [25]. that Multiscale Vision Transformers have an architectural
advantage over Vision Transformers.
Training. Our recipe is identical to DeiT [101] and summa-
rized in §D.5. Training is for 300 epochs and results improve Finally, compared to the best model of concurrent
for training longer [101]. Swin [79] (which was designed for image recognition),
MViT has +0.6% better accuracy at 1.4×less computation.
5.1. Main Results
For this experiment, we take our models which were de- 6. Conclusion
signed by ablation studies for video classification on Kinetics
and simply remove the temporal dimension. Then we train We have presented Multiscale Vision Transformers that
and validate them (“from scratch”) on ImageNet. aim to connect the fundamental concept of multiscale feature
Table 12 shows the comparison with previous work. hierarchies with the transformer model. MViT hierarchically
From top to bottom, the table contains RegNet [87] and expands the feature complexity while reducing visual reso-
EfficientNet [99] as ConvNet examples, and DeiT [101], lution. In empirical evaluation, MViT shows a fundamental
with DeiT-B being identical to ViT-B [28] but trained with advantage over single-scale vision transformers for video
the improved recipe in [101]. Therefore, this is the vision and image recognition. We hope that our approach will foster
transformer counterpart we are interested in comparing to. further research in visual recognition.

6831
References [17] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Hee-
woo Jun, David Luan, and Ilya Sutskever. Generative pre-
[1] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen training from pixels. In Proc. ICML, pages 1691–1703.
Sun, Mario Lučić, and Cordelia Schmid. Vivit: A video PMLR, 2020. 2
vision transformer. arXiv preprint arXiv:2103.15691, 2021.
[18] Yunpeng Chen, Haoqi Fang, Bing Xu, Zhicheng Yan, Yannis
2, 3, 6, 14
Kalantidis, Marcus Rohrbach, Shuicheng Yan, and Jiashi
[2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hin- Feng. Drop an octave: Reducing spatial redundancy in con-
ton. Layer normalization. arXiv preprint arXiv:1607.06450, volutional neural networks with octave convolution. arXiv
2016. 1, 17 preprint arXiv:1904.05049, 2019. 2
[3] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. [19] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever.
Layer normalization. arXiv:1607.06450, 2016. 4 Generating long sequences with sparse transformers. arXiv
[4] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. preprint arXiv:1904.10509, 2019. 3
Neural machine translation by jointly learning to align and [20] Krzysztof Choromanski, Valerii Likhosherstov, David Do-
translate. arXiv preprint arXiv:1409.0473, 2014. 1 han, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter
[5] Josh Beal, Eric Kim, Eric Tzeng, Dong Huk Park, Andrew Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser,
Zhai, and Dmitry Kislyuk. Toward transformer-based object et al. Rethinking attention with performers. arXiv preprint
detection. arXiv preprint arXiv:2012.09958, 2020. 2 arXiv:2009.14794, 2020. 3
[6] Irwan Bello. Lambdanetworks: Modeling long-range in- [21] Xiangxiang Chu, Bo Zhang, Zhi Tian, Xiaolin Wei, and
teractions without attention. International Conference on Huaxia Xia. Do we really need explicit position encodings
Learning Representations, 2021. 3 for vision transformers? arXiv preprint arXiv:2102.10882,
[7] Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, 2021. 2
and Quoc V Le. Attention augmented convolutional net- [22] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V
works. International Conference on Computer Vision, 2019. Le. Randaugment: Practical automated data augmentation
2 with a reduced search space. In Proc. CVPR, 2020. 17, 19
[8] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is [23] Zihang Dai, Guokun Lai, Yiming Yang, and Quoc V
space-time attention all you need for video understanding? Le. Funnel-transformer: Filtering out sequential redun-
arXiv preprint arXiv:2102.05095, 2021. 2, 3, 6, 7, 14 dancy for efficient language processing. arXiv preprint
[9] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Sub- arXiv:2006.03236, 2020. 3
biah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakan- [24] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li,
tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. and Li Fei-Fei. Imagenet: A large-scale hierarchical image
Language models are few-shot learners. arXiv preprint database. In Proc. CVPR, pages 248–255. Ieee, 2009. 2, 3
arXiv:2005.14165, 2020. 1 [25] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.
[10] Peter J Burt and Edward H Adelson. The laplacian pyramid ImageNet: A large-scale hierarchical image database. In
as a compact image code. In Readings in computer vision, Proc. CVPR, 2009. 6, 8, 18
pages 671–679. Elsevier, 1987. 1 [26] Karan Desai and Justin Johnson. Virtex: Learning visual
[11] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas representations from textual annotations. arXiv preprint
Usunier, Alexander Kirillov, and Sergey Zagoruyko. End- arXiv:2006.06666, 2020. 2
to-end object detection with transformers. In Proc. ECCV, [27] Piotr Dollár, Mannat Singh, and Ross Girshick. Fast and
pages 213–229. Springer, 2020. 2 accurate model scaling. In Proc. CVPR, 2021. 8
[12] Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe [28] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov,
Hillier, and Andrew Zisserman. A short note about Kinetics- Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner,
600. arXiv:1808.01340, 2018. 2, 6 Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl-
[13] João Carreira, Eric Noland, Chloe Hillier, and Andrew Zis- vain Gelly, et al. An image is worth 16x16 words: Trans-
serman. A short note on the kinetics-700 human action formers for image recognition at scale. arXiv preprint
dataset. arXiv preprint arXiv:1907.06987, 2019. 13 arXiv:2010.11929, 2020. 1, 2, 3, 4, 5, 6, 8, 14, 17, 18
[14] Joao Carreira and Andrew Zisserman. Quo vadis, action [29] Sergey Edunov, Myle Ott, Michael Auli, and David Grang-
recognition? a new model and the kinetics dataset. In Proc. ier. Understanding back-translation at scale. arXiv preprint
CVPR, 2017. 2, 6 arXiv:1808.09381, 2018. 1
[15] Chun-Fu Chen, Quanfu Fan, Neil Mallinar, Tom Sercu, and [30] Alaaeldin El-Nouby, Natalia Neverova, Ivan Laptev, and
Rogerio Feris. Big-little net: An efficient multi-scale feature Hervé Jégou. Training vision transformers for image re-
representation for visual and speech recognition. arXiv trieval. arXiv preprint arXiv:2102.05644, 2021. 2
preprint arXiv:1807.03848, 2018. 2 [31] Haoqi Fan, Yanghao Li, Bo Xiong, Wan-Yen Lo, and
[16] Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Christoph Feichtenhofer. PySlowFast. https://github.
Adeli, Yan Wang, Le Lu, Alan L Yuille, and Yuyin Zhou. com/facebookresearch/slowfast, 2020. 2, 17
Transunet: Transformers make strong encoders for medi- [32] Haoqi Fan, Tullie Murrell, Heng Wang, Kalyan Vasudev Al-
cal image segmentation. arXiv preprint arXiv:2102.04306, wala, Yanghao Li, Yilei Li, Bo Xiong, Nikhila Ravi, Meng
2021. 2 Li, Haichuan Yang, Jitendra Malik, Ross Girshick, Matt

6832
 Feiszli, Aaron Adcock, Wan-Yen Lo, and Christoph Fe- [47] Boris Hanin and David Rolnick. How to start training:
ichtenhofer. PyTorchVideo: A deep learning library for The effect of initialization and architecture. arXiv preprint
video understanding. In Proceedings of the 29th ACM arXiv:1803.01719, 2018. 17, 18
International Conference on Multimedia, 2021. https: [48] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Gir-
//pytorchvideo.org/. 2 shick. Mask R-CNN. In Proc. ICCV, 2017. 18
[33] Christoph Feichtenhofer. X3d: Expanding architectures for [49] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
efficient video recognition. In Proc. CVPR, pages 203–213, Delving deep into rectifiers: Surpassing human-level per-
2020. 2, 6, 7, 8, 14 formance on imagenet classification. In Proc. CVPR, 2015.
[34] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and 1
Kaiming He. SlowFast networks for video recognition. In [50] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Proc. ICCV, 2019. 2, 6, 7, 8, 14, 15, 17, 18 Deep residual learning for image recognition. In Proc.
[35] Christoph Feichtenhofer, Haoqi Fan, Jitendra Ma- CVPR, 2016. 5, 17
lik, and Kaiming He. SlowFast networks for [51] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
video recognition in ActivityNet challenge 2019. Identity mappings in deep residual networks. In Proc. ECCV,
http://static.googleusercontent.com/ 2016. 2
media/research.google.com/en//ava/2019/ [52] Shuting He, Hao Luo, Pichao Wang, Fan Wang, Hao Li,
fair_slowfast.pdf, 2019. 13 and Wei Jiang. Transreid: Transformer-based object re-
[36] Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman. identification. arXiv preprint arXiv:2102.04378, 2021. 2
Convolutional two-stream network fusion for video action [53] Dan Hendrycks and Kevin Gimpel. Gaussian error linear
recognition. In Proc. CVPR, 2016. 2 units (GELUs). arXiv preprint arXiv:1606.08415, 2016. 17
[37] Kunihiko Fukushima and Sei Miyake. Neocognitron: A [54] Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya
self-organizing neural network model for a mechanism of Sutskever, and Ruslan R Salakhutdinov. Improving neural
visual pattern recognition. In Competition and cooperation networks by preventing co-adaptation of feature detectors.
in neural nets, pages 267–285. Springer, 1982. 1 arXiv:1207.0580, 2012. 17
[38] Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia [55] Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten
Schmid. Multi-modal transformer for video retrieval. In Hoefler, and Daniel Soudry. Augment your batch: Improving
Proc. ECCV, volume 5. Springer, 2020. 2 generalization through instance repetition. In Proc. CVPR,
[39] Shanghua Gao, Ming-Ming Cheng, Kai Zhao, Xin-Yu pages 8129–8138, 2020. 6, 17, 18
Zhang, Ming-Hsuan Yang, and Philip HS Torr. Res2net: [56] Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, and Yichen
A new multi-scale backbone architecture. IEEE PAMI, 2019. Wei. Relation networks for object detection. In Proc. CVPR,
2 pages 3588–3597, 2018. 2
[40] Rohit Girdhar, Joao Carreira, Carl Doersch, and Andrew [57] Han Hu, Zheng Zhang, Zhenda Xie, and Stephen Lin. Local
Zisserman. Video action transformer network. In Proc. relation networks for image recognition. In Proc. ICCV,
CVPR, 2019. 2 pages 3464–3473, 2019. 2
[41] Ross Girshick. Fast R-CNN. In Proc. ICCV, 2015. 18 [58] Ronghang Hu and Amanpreet Singh. Transformer is all
[42] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noord- you need: Multimodal multitask learning with a unified
huis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, transformer. arXiv preprint arXiv:2102.10772, 2021. 2
Yangqing Jia, and Kaiming He. Accurate, large minibatch [59] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q
SGD: training ImageNet in 1 hour. arXiv:1706.02677, 2017. Weinberger. Deep networks with stochastic depth. In Proc.
17, 18 ECCV, 2016. 17, 18
[43] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michal- [60] DH Hubel and TN Wiesel. Receptive fields of optic nerve
ski, Joanna Materzynska, Susanne Westphal, Heuna Kim, fibres in the spider monkey. The Journal of physiology,
Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz 154(3):572–580, 1960. 1
Mueller-Freitag, et al. The “Something Something" video [61] Noureldien Hussein, Efstratios Gavves, and Arnold WM
database for learning and evaluating visual common sense. Smeulders. Timeception for complex action recognition. In
In ICCV, 2017. 2, 6, 7, 18 Proc. CVPR, 2019. 7
[44] Chunhui Gu, Chen Sun, David A. Ross, Carl Von- [62] Boyuan Jiang, MengMeng Wang, Weihao Gan, Wei Wu, and
drick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijaya- Junjie Yan. Stm: Spatiotemporal and motion encoding for
narasimhan, George Toderici, Susanna Ricco, Rahul Suk- action recognition. In Proc. CVPR, pages 2000–2009, 2019.
thankar, Cordelia Schmid, and Jitendra Malik. AVA: A video 2
dataset of spatio-temporally localized atomic visual actions. [63] Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsuan
In Proc. CVPR, 2018. 2, 6, 7, 18 Yang, Erik Learned-Miller, and Jan Kautz. Super slomo:
[45] Meng-Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang High quality estimation of multiple intermediate frames for
Mu, Ralph R Martin, and Shi-Min Hu. Pct: Point cloud video interpolation. In Proc. CVPR, 2018. 18
transformer. arXiv preprint arXiv:2012.09688, 2020. 2 [64] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang,
[46] Zhang Hang, Chongruo Wu, Zhongyue Zhang, Yi Zhu, Zhi Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola,
Zhang, Haibin Lin, and Yue Sun. Resnest: Split-attention Tim Green, Trevor Back, Paul Natsev, et al. The kinetics
networks. 2020. 2 human action video dataset. arXiv:1705.06950, 2017. 2, 6

6833
[65] Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. [82] Ilya Loshchilov and Frank Hutter. Fixing weight decay
Reformer: The efficient transformer. arXiv preprint regularization in adam. 2018. 17, 18
arXiv:2001.04451, 2020. 3 [83] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert:
[66] Jan J Koenderink. The structure of images. Biological Pretraining task-agnostic visiolinguistic representations for
cybernetics, 50(5):363–370, 1984. 1 vision-and-language tasks. arXiv preprint arXiv:1908.02265,
[67] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2019. 2
ImageNet classification with deep convolutional neural net- [84] Daniel Neimark, Omri Bar, Maya Zohar, and Dotan As-
works. In NIPS, 2012. 2 selmann. Video transformer network. arXiv preprint
[68] Heeseung Kwon, Manjin Kim, Suha Kwak, and Minsu Cho. arXiv:2102.00719, 2021. 2, 3, 6, 14
Motionsqueeze: Neural motion feature learning for video [85] Zhaofan Qiu, Ting Yao, and Tao Mei. Learning spatio-
understanding. In Proc. ECCV, pages 345–362. Springer, temporal representation with pseudo-3d residual networks.
2020. 7 In Proc. ICCV, 2017. 2
[69] Hung Le, Doyen Sahoo, Nancy F Chen, and Steven CH [86] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya
Hoi. Multimodal transformer networks for end-to- Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry,
end video-grounded dialogue systems. arXiv preprint Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning
arXiv:1907.01166, 2019. 2 transferable visual models from natural language supervi-
[70] Yann LeCun, Bernhard Boser, John S Denker, Donnie sion. arXiv preprint arXiv:2103.00020, 2021. 2
Henderson, Richard E Howard, Wayne Hubbard, and [87] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick,
Lawrence D Jackel. Backpropagation applied to handwritten Kaiming He, and Piotr Dollár. Designing network design
zip code recognition. Neural computation, 1(4):541–551, spaces. In Proc. CVPR, June 2020. 2, 8
1989. 1, 2 [88] Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan
[71] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, Bello, Anselm Levskaya, and Jonathon Shlens. Stand-
and Kai-Wei Chang. Visualbert: A simple and perfor- alone self-attention in vision models. arXiv preprint
mant baseline for vision and language. arXiv preprint arXiv:1906.05909, 2019. 2
arXiv:1908.03557, 2019. 2
[89] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott
[72] Rui Li, Chenxi Duan, and Shunyi Zheng. Linear attention
Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya
mechanism: An efficient attention for semantic segmenta-
Sutskever. Zero-shot text-to-image generation. arXiv
tion. arXiv preprint arXiv:2007.14902, 2020. 3
preprint arXiv:2102.12092, 2021. 2
[73] Yan Li, Bin Ji, Xintian Shi, Jianguo Zhang, Bin Kang, and
[90] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.
Limin Wang. Tea: Temporal excitation and aggregation for
Faster R-CNN: Towards real-time object detection with re-
action recognition. In Proc. CVPR, pages 909–918, 2020. 7
gion proposal networks. In NIPS, 2015. 18
[74] Zhenyang Li, Kirill Gavrilyuk, Efstratios Gavves, Mihir Jain,
[91] Azriel Rosenfeld and Mark Thurston. Edge and curve de-
and Cees GM Snoek. VideoLSTM convolves, attends and
tection for visual scene analysis. IEEE Transactions on
flows for action recognition. Computer Vision and Image
computers, 100(5):562–569, 1971. 1
Understanding, 166:41–50, 2018. 2
[75] Ji Lin, Chuang Gan, and Song Han. Temporal shift module [92] Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali
for efficient video understanding. In Proc. ICCV, 2019. 18 Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in
[76] Ji Lin, Chuang Gan, and Song Han. Tsm: Temporal shift homes: Crowdsourcing data collection for activity under-
module for efficient video understanding. In Proc. CVPR, standing. In ECCV, 2016. 2, 6, 7, 18
pages 7083–7093, 2019. 7 [93] Karen Simonyan and Andrew Zisserman. Two-stream con-
[77] Kevin Lin, Lijuan Wang, and Zicheng Liu. End-to-end hu- volutional networks for action recognition in videos. In
man pose and mesh reconstruction with transformers. arXiv NIPS, 2014. 2, 5
preprint arXiv:2012.09760, 2020. 2 [94] Karen Simonyan and Andrew Zisserman. Very deep convo-
[78] Wei Liu, Sihan Chen, Longteng Guo, Xinxin Zhu, and Jing lutional networks for large-scale image recognition. In Proc.
Liu. Cptr: Full transformer network for image captioning. ICLR, 2015. 2
arXiv preprint arXiv:2101.10804, 2021. 2 [95] Chen Sun, Abhinav Shrivastava, Carl Vondrick, Kevin Mur-
[79] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, phy, Rahul Sukthankar, and Cordelia Schmid. Actor-centric
Zheng Zhang, Stephen Lin, and Baining Guo. Swin trans- relation network. In ECCV, 2018. 18
former: Hierarchical vision transformer using shifted win- [96] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet,
dows. arXiv preprint arXiv:2103.14030, 2021. 8 Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent
[80] Francesco Locatello, Dirk Weissenborn, Thomas Un- Vanhoucke, and Andrew Rabinovich. Going deeper with
terthiner, Aravindh Mahendran, Georg Heigold, Jakob convolutions. In Proc. CVPR, 2015. 2
Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object- [97] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet,
centric learning with slot attention. arXiv preprint Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent
arXiv:2006.15055, 2020. 2 Vanhoucke, and Andrew Rabinovich. Going deeper with
[81] Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gra- convolutions. In Proc. CVPR, 2015. 17
dient descent with warm restarts. arXiv:1608.03983, 2016. [98] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe,
17, 18 Jonathon Shlens, and Zbigniew Wojna. Rethinking the in-

6834
 ception architecture for computer vision. arXiv:1512.00567, captioning. IEEE transactions on circuits and systems for
2015. 17, 18 video technology, 30(12):4467–4480, 2019. 2
[99] Mingxing Tan and Quoc V Le. Efficientnet: Rethinking [115] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi,
model scaling for convolutional neural networks. arXiv Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-
preprint arXiv:1905.11946, 2019. 2, 8 to-token vit: Training vision transformers from scratch on
[100] Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da- imagenet. arXiv preprint arXiv:2101.11986, 2021. 2
Cheng Juan. Sparse sinkhorn attention. In Proc. ICML, [116] Zhenxun Yuan, Xiao Song, Lei Bai, Wengang Zhou, Zhe
pages 9438–9447. PMLR, 2020. 3 Wang, and Wanli Ouyang. Temporal-channel transformer for
[101] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco 3d lidar-based video object detection in autonomous driving.
Massa, Alexandre Sablayrolles, and Hervé Jégou. Training arXiv preprint arXiv:2011.13628, 2020. 2
data-efficient image transformers & distillation through at- [117] Boxiang Yun, Yan Wang, Jieneng Chen, Huiyu Wang, Wei
tention. arXiv preprint arXiv:2012.12877, 2020. 1, 2, 6, 8, Shen, and Qingli Li. Spectr: Spectral transformer for hy-
17, 18 perspectral pathology image segmentation. arXiv preprint
[102] Du Tran, Heng Wang, Lorenzo Torresani, and Matt Feiszli. arXiv:2103.03604, 2021. 2
Video classification with channel-separated convolutional [118] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk
networks. In Proc. ICCV, 2019. 2, 6 Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regu-
[103] Jeya Maria Jose Valanarasu, Poojan Oza, Ilker Hacihaliloglu, larization strategy to train strong classifiers with localizable
and Vishal M Patel. Medical transformer: Gated axial- features. In Proc. ICCV, 2019. 17, 18
attention for medical image segmentation. arXiv preprint [119] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and
arXiv:2102.10662, 2021. 2 David Lopez-Paz. Mixup: Beyond empirical risk minimiza-
[104] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- tion. In Proc. ICLR, 2018. 17, 18
reit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Il- [120] Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun. Exploring
lia Polosukhin. Attention is all you need. arXiv preprint self-attention for image recognition. In Proc. CVPR, pages
arXiv:1706.03762, 2017. 1, 2, 3, 4 10076–10085, 2020. 2
[105] Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, [121] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip Torr, and
and Liang-Chieh Chen. Max-deeplab: End-to-end panop- Vladlen Koltun. Point transformer. arXiv preprint
tic segmentation with mask transformers. arXiv preprint arXiv:2012.09164, 2020. 2
arXiv:2012.00759, 2020. 2 [122] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and
[106] Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Yi Yang. Random erasing data augmentation. In Proceedings
Hao Ma. Linformer: Self-attention with linear complexity. of the AAAI Conference on Artificial Intelligence, volume 34,
arXiv preprint arXiv:2006.04768, 2020. 3 pages 13001–13008, 2020. 17, 19
[107] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaim- [123] Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Tor-
ing He. Non-local neural networks. In Proc. CVPR, 2018. ralba. Temporal relational reasoning in videos. In ECCV,
2, 7 2018. 2
[108] Xiaolong Wang and Abhinav Gupta. Videos as space-time
region graphs. In Proc. ECCV, 2018. 7
[109] Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua
Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-
to-end video instance segmentation with transformers. arXiv
preprint arXiv:2011.14503, 2020. 2
[110] Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan,
Peizhao Zhang, Masayoshi Tomizuka, Kurt Keutzer, and
Peter Vajda. Visual transformers: Token-based image repre-
sentation and processing for computer vision. arXiv preprint
arXiv:2006.03677, 2020. 2
[111] Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaim-
ing He, Philipp Krähenbühl, and Ross Girshick. Long-term
feature banks for detailed video understanding. In Proc.
CVPR, 2019. 2, 7
[112] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and
Kevin Murphy. Rethinking spatiotemporal feature learning
for video understanding. arXiv:1712.04851, 2017. 2
[113] Sen Yang, Zhibin Quan, Mu Nie, and Wankou Yang. Trans-
pose: Towards explainable human pose estimation by trans-
former. arXiv preprint arXiv:2012.14214, 2020. 2
[114] Jun Yu, Jing Li, Zhou Yu, and Qingming Huang. Multimodal
transformer with multi-view visual representation for image

6835