Masked Autoencoders Are Scalable Vision Learners掩码自编码器是可扩展的视觉学习器Abstract摘要This paper shows that masked autoencoders (MAE) are scalable self-supervised learners for computer vision. Our MAE approach is simple: we mask random patches of the input image and reconstruct the missing pixels. It is based on two core designs. First, we develop an asymmetric encoder-decoder architecture, with an encoder that operates only on the visible subset of patches (without mask tokens), along with a lightweight decoder that reconstructs the original image from the latent representation and mask tokens. Second, we find that masking a high proportion of the input image, e.g., 75%, yields a nontrivial and meaningful selfsupervisory task. Coupling these two designs enables us to train large models efficiently and effectively: we accelerate training (by 3× or more) and improve accuracy. Our scalable approach allows for learning highcapacity models that generalize well: e.g., a vanilla ViT-Huge model achieves the best accuracy (87.8%) among methods that use only ImageNet-1K data. Transfer performance in downstream tasks outperforms supervised pretraining and shows promising scaling behavior.本文表明，掩码自编码器（Masked Autoencoders，MAE）是一种具有良好可扩展性（Scalability）的计算机视觉自监督学习（Self-Supervised Learning）方法。我们的 MAE 方法非常简单：随机遮蔽输入图像中的部分图像块（Patches），然后重建缺失的像素。该方法基于两个核心设计。首先，我们设计了一种非对称编码器-解码器架构（Asymmetric Encoder-Decoder Architecture）：其中，编码器（Encoder）仅处理可见的图像块子集，而不使用掩码标记（Mask Tokens）；与此同时，一个轻量级的解码器（Decoder）根据潜在表征（Latent Representation）和掩码标记重建原始图像。其次，我们发现，对输入图像中的较高比例区域进行遮蔽，例如遮蔽 75%，能够构成一个具有一定难度且有意义的自监督任务（Self-Supervisory Task）。将这两种设计结合起来，使我们能够高效且有效地训练大规模模型：训练速度提升了 3 倍甚至更多，同时模型准确率也得到提高。我们的可扩展方法能够学习具有较大模型容量（Model Capacity）且具有良好泛化能力（Generalization Ability）的模型。例如，一个标准的ViT-Huge模型在仅使用 ImageNet-1K 数据的方法中取得了最高的准确率（87.8%）。在下游任务（Downstream Tasks）上的迁移性能（Transfer Performance）超越了督式预训练（Supervised Pretraining），并展现出令人期待的规模扩展特性（Scaling Behavior）。Introduction1. 引言Deep learning has witnessed an explosion of architectures of continuously growing capability and capacity [33, 25, 57]. Aided by the rapid gains in hardware, models today can easily overfit one million images [13] and begin to demand hundreds of millions of-often publicly inaccessible-labeled images [16].深度学习已经见证了各种架构的迅速涌现，其能力和容量也在持续增长 [33, 25, 57]。在硬件性能快速提升的推动下，如今的模型可以轻易地在一百万张图像上产生过拟合（Overfitting）[13]，并开始需要数以亿计的、且往往无法公开获取的标注图像（Labeled Images）[16]。This appetite for data has been successfully addressed in natural language processing (NLP) by self-supervised pretraining. The solutions, based on autoregressive language modeling in GPT [47, 48, 4] and masked autoencoding in BERT [14], are conceptually simple: they remove a portion of the data and learn to predict the removed content. These methods now enable training of generalizable NLP models containing over one hundred billion parameters [4].对于这种巨大的数据需求，自然语言处理（Natural Language Processing，NLP）领域已经通过自监督预训练（Self-Supervised Pretraining）成功加以解决。相关方法在概念上十分简单，例如GPT [47, 48, 4] 中基于自回归语言建模（Autoregressive Language Modeling）的方法，以及 BERT [14] 中基于掩码自编码（Masked Autoencoding）的方法：它们都会移除一部分数据，并学习预测被移除的内容。如今，这些方法已经能够训练包含超过一千亿个参数、并具有良好泛化能力（Generalizability）的 NLP 模型 [4]。Figure 1: Figure 1. Our MAE architecture. During pre-training, a large random subset of image patches (e.g., 75%) is masked out. The encoder is applied to the small subset of visible patches. Mask tokens are introduced after the encoder, and the full set of encoded patches and mask tokens is processed by a small decoder that reconstructs the original image in pixels. After pre-training, the decoder is discarded and the encoder is applied to uncorrupted images (full sets of patches) for recognition tasks.图 1：我们的 MAE 架构。在预训练期间，大部分图像块的随机子集（例如 75%）会被遮蔽。编码器应用于少量可见图像块子集。掩码 Tokens 在编码器之后引入，完整的已编码图像块集合和掩码 Tokens 由一个小型解码器处理，以像素形式重建原始图像。预训练后，丢弃解码器，并将编码器应用于未损坏的图像（完整的图像块集合）以执行识别任务。The idea of masked autoencoders, a form of more general denoising autoencoders [58], is natural and applicable in computer vision as well. Indeed, closely related research in vision [59, 46] preceded BERT. However, despite significant interest in this idea following the success of BERT, progress of autoencoding methods in vision lags behind NLP. We ask: what makes masked autoencoding different between vision and language? We attempt to answer this question from the following perspectives:(i) Until recently, architectures were different. In vision, convolutional networks [34] were dominant over the last decade [33]. Convolutions typically operate on regular grids and it is not straightforward to integrate ‘indicators’ such as mask tokens [14] or positional embeddings [57] into convolutional networks. This architectural gap, however, has been addressed with the introduction of Vision Transformers (ViT) [16] and should no longer present an obstacle.(ii) Information density is different between language and vision. Languages are human-generated signals that are highly semantic and information-dense. When training a model to predict only a few missing words per sentence, this task appears to induce sophisticated language understanding. Images, on the contrary, are natural signals with heavy spatial redundancy-e.g., a missing patch can be recovered from neighboring patches with little high-level understanding of parts, objects, and scenes. To overcome this difference and encourage learning useful features, we show that a simple strategy works well in computer vision: masking a very high portion of random patches. This strategy largely reduces redundancy and creates a challenging selfsupervisory task that requires holistic understanding beyond low-level image statistics. To get a qualitative sense of our reconstruction task, see Figures 2-4.(iii) The autoencoder’s decoder, which maps the latent representation back to the input, plays a different role between reconstructing text and images. In vision, the decoder reconstructs pixels, hence its output is of a lower semantic level than common recognition tasks. This is in contrast to language, where the decoder predicts missing words that contain rich semantic information. While in BERT the decoder can be trivial (an MLP) [14], we found that for images, the decoder design plays a key role in determining the semantic level of the learned latent representations.掩码自编码器（Masked Autoencoders）的思想本质上属于更广义的去噪自编码器（Denoising Autoencoders）[58]，这一思想同样自然地适用于计算机视觉。事实上，在视觉领域，与之密切相关的研究 [59, 46] 甚至早于 BERT 出现。然而，尽管 BERT 成功之后这一思路受到了广泛关注，视觉领域中的自编码方法，其发展进度仍然落后于 NLP。于是，我们提出一个问题：为什么掩码自编码（Masked Autoencoding）在视觉和语言领域中的表现会有所不同？我们尝试从以下几个方面回答这一问题。(i) 直到不久之前，两者所使用的架构仍然不同。在视觉领域，过去十年中占据主导地位的是卷积神经网络（Convolutional Networks）[34, 33]。卷积通常作用于规则网格，因此，要将掩码标记（Mask Tokens）[14]、位置嵌入（Positional Embeddings）[57] 等“指示信息”集成到卷积神经网络中，并不是一件直接的事情。不过，随着视觉 Transformer（Vision Transformers，ViT）[16] 的提出，这种架构上的差异已经得到解决，因此不应再构成障碍。(ii) 语言和视觉之间的信息密度（Information Density）不同。语言是由人类生成的信号，具有高度的语义性和很高的信息密度。当训练模型仅预测一个句子中少量缺失的词语时，这项任务似乎就能够促使模型形成较为复杂的语言理解能力（Language Understanding）。相比之下，图像是一种自然信号，具有很强的空间冗余（Spatial Redundancy）。例如，一个缺失的图像块往往可以仅根据其相邻图像块进行恢复，而无需对图像中的部件、物体和场景形成高层次的理解。为了克服这种差异，并促进模型学习有用的特征，我们表明，一种简单的策略在计算机视觉中非常有效：随机遮蔽非常高比例的图像块（Random Patches）。这种策略能够大幅减少冗余，并构造出一个更具挑战性的自监督任务（Self-Supervisory Task），迫使模型形成超越低层图像统计信息的整体性理解（Holistic Understanding）。若想直观了解我们的重建任务，可以参见图 2 至图 4。(iii)自编码器（Autoencoder）中的解码器（Decoder）负责将潜在表征（Latent Representation）映射回输入空间，而它在文本重建和图像重建中所扮演的角色并不相同。 在视觉任务中，解码器需要重建像素，因此其输出所对应的语义层次低于常见的 识别任务（Recognition Tasks）。这与语言任务形成鲜明对比：在语言任务中，解码器预测的是缺失的词语，而这些词语本身包含丰富的语义信息。在 BERT 中，解码器可以非常简单，例如仅使用一个多层感知机（Multilayer Perceptron，MLP）[14]。但我们发现，对于图像而言，解码器的设计在决定所学习到的潜在表征（Latent Representations）的语义层次方面起着关键作用。图 2：ImageNet 验证图像上的示例结果。对于每组三联图，我们展示了掩码图像（左）、我们的 MAE 重建结果（中）以及真实图像（右）。掩码比例为 80%，196 个图块中仅保留 39 个。更多示例见附录。 由于可见图块上不计算损失，模型在可见图块上的输出在视觉质量上较差。只需将输出与可见图块叠加即可提升视觉质量。我们有意不这样做，以便能够更全面地展示该方法的行为。图 3：图 3。COCO 验证图像上的示例结果，使用在 ImageNet 上训练的 MAE（与图 2 中相同的 Model Weights）。请注意最右侧两个示例的重建结果，尽管它们与真实图像不同，但在语义上是合理的。Driven by this analysis, we present a simple, effective, and scalable form of a masked autoencoder (MAE) for visua representation learning. Our MAE masks random patches from the input image and reconstructs the missing patches in the pixel space. It has an asymmetric encoderdecoder design. Our encoder operates only on the visible subset of patches (without mask tokens), and our decoder is lightweight and reconstructs the input from the latent representation along with mask tokens (Figure 1). Shifting the mask tokens to the small decoder in our asymmetric encoder-decoder results in a large reduction in computation. Under this design, a very high masking ratio (e.g., 75%) can achieve a win-win scenario: it optimizes accuracy while allowing the encoder to process only a small portion (e.g., ) of patches. This can reduce overall pre-training time by 3× or more and likewise reduce memory consumption, enabling us to easily scale our MAE to large models.受上述分析的推动，我们提出了一种用于视觉表征学习（Visual Representation Learning）的简单、有效且可扩展的掩码自编码器（Masked Autoencoder，MAE）。我们的 MAE 会对输入图像中的随机图像块（Patches）进行遮蔽，并在像素空间（Pixel Space）中重建缺失的图像块。它采用了一种非对称编码器-解码器设计（Asymmetric Encoder-Decoder Design）。我们的编码器（Encoder）仅处理可见的那部分图像块子集（不包含掩码标记（Mask Tokens）），而解码器（Decoder）则是轻量级的，它结合潜在表征（Latent Representation）和掩码标记来重建输入（见图 1）。在这种非对称的编码器-解码器设计中，将掩码标记转移到较小的解码器中，可以大幅减少计算量。在这一设计下，非常高的掩码比例（Masking Ratio）（例如 75%）能够实现一种双赢局面：一方面提升准确率，另一方面使编码器只需处理很小一部分图像块（例如 25%）。这可以将整体预训练时间（Pre-training Time）缩短 3 倍或更多，同时相应减少内存消耗（Memory Consumption），从而使我们能够轻松地将 MAE 扩展到大规模模型。Our MAE learns very high-capacity models that generalize well. With MAE pre-training, we can train datahungry models like ViT-Large/-Huge [16] on ImageNet-1K with improved generalization performance. With a vanilla ViT-Huge model, we achieve 87.8% accuracy when finetuned on ImageNet-1K. This outperforms all previous results that use only ImageNet-1K data. We also evaluate transfer learning on object detection, instance segmentation, and semantic segmentation. In these tasks, our pre-training achieves better results than its supervised pre-training counterparts, and more importantly, we observe significant gains by scaling up models. These observations are aligned with those witnessed in self-supervised pretraining in NLP [14, 47, 48, 4] and we hope that they will enable our field to explore a similar trajectory.我们的 MAE 能够学习到容量非常高且具有良好泛化能力（Generalization Ability）的模型。借助 MAE预训练（Pre-training），我们可以在 ImageNet-1K 上训练像 ViT-Large / ViT-Huge [16] 这样对数据需求很大的模型，并获得更好的泛化表现。对于一个标准的ViT-Huge模型，我们在 ImageNet-1K 上进行微调（Fine-tuning）后，达到了 87.8% 的准确率。这一结果超过了此前所有仅使用 ImageNet-1K 数据的方法。我们还在目标检测（Object Detection）、实例分割（Instance Segmentation）语义分割（Semantic Segmentation）任务上评估了迁移学习（Transfer Learning）表现。在这些任务中，我们的预训练方法优于对应的 监督式预训练（Supervised Pre-training） 方法。更重要的是，我们观察到，随着模型规模增大，性能提升十分显著。这些观察结果与 NLP 中自监督预训练（Self-Supervised Pre-training）的现象一致 [14, 47, 48, 4]。我们希望，这将推动本领域沿着类似的发展轨迹继续探索。Figure 4: Figure 4. Reconstructions of ImageNet validation images using an MAE pre-trained with a masking ratio of 75% but applied on inputs with higher masking ratios. The predictions differ plausibly from the original images, showing that the method can generalize.图 4：图 4。使用以 75% 掩码比例预训练的 MAE 对 ImageNet 验证图像进行重建，但将其应用于具有更高掩码比例的输入。预测结果与原始图像存在合理的差异，表明该方法能够泛化。Related Work2. 相关工作Masked language modeling and its autoregressive counterparts, e.g., BERT [14] and GPT [47, 48, 4], are highly successful methods for pre-training in NLP. These methods hold out a portion of the input sequence and train models to predict the missing content. These methods have been shown to scale excellently [4] and a large abundance of evidence indicates that these pre-trained representations generalize well to various downstream tasks.掩码语言建模（Masked Language Modeling）及其对应的自回归方法（Autoregressive Counterparts），例如 BERT [14] 和 GPT [47, 48, 4]，都是自然语言处理（NLP）中非常成功的预训练（Pre-training）方法。这些方法会从输入序列中留出一部分内容，并训练模型预测缺失的部分。已有研究表明，这些方法具有出色的规模扩展能力（Scaling Ability）[4]，而且大量证据显示，通过这些方法得到的预训练表征（Pre-trained Representations）能够很好地泛化到各种下游任务（Downstream Tasks）。Autoencoding is a classical method for learning representations. It has an encoder that maps an input to a latent representation and a decoder that reconstructs the input. For example, PCA and k-means are autoencoders [29]. Denoising autoencoders (DAE) [58] are a class of autoencoders that corrupt an input signal and learn to reconstruct the original, uncorrupted signal. A series of methods can be thought of as a generalized DAE under different corruptions, e.g., masking pixels [59, 46, 6] or removing color channels [70]. Our MAE is a form of denoising autoencoding, but different from the自编码（Autoencoding）是一种经典的表征学习（Representation Learning）方法。它包含一个编码器（Encoder），负责将输入映射为潜在表征（Latent Representation），一个解码器（Decoder），负责重建输入。例如，主成分分析（Principal Component Analysis，PCA）和k 均值聚类（k-means）都可以看作自编码器 [29]。去噪自编码器（Denoising Autoencoders，DAE）[58] 是一类对输入信号进行破坏，并学习重建原始、未受破坏信号的自编码器。一系列方法都可以视为在不同破坏方式下的广义 DAE，例如对像素进行遮蔽 [59, 46, 6]，或者移除颜色通道 [70]。我们的掩码自编码器（Masked Autoencoder，MAE）属于去噪自编码器（Denoising Autoencoders，DAE）的一种形式，但它在许多方面不同于经典的 DAE。Masked image encoding methods learn representations from images corrupted by masking. The pioneering work of [59] presents masking as a noise type in DAE. Context Encoder [46] inpaints large missing regions using convolutional networks. Motivated by the success in NLP, related recent methods [6, 16, 2] are based on Transformers [57]. iGPT [6] operates on sequences of pixels and predicts unknown pixels. The ViT paper [16] studies masked patch prediction for selfsupervised learning. Most recently, BEiT [2] proposes to predict discrete tokens [44, 50].掩码图像编码（Masked Image Encoding）方法通过对被掩码破坏后的图像进行学习来获得表征。开创性的工作 [59] 将掩码作为 DAE 中的一种噪声类型。上下文编码器（Context Encoder）[46] 使用卷积神经网络对大面积缺失区域进行图像修复（Inpainting）。受到 NLP 成功经验的推动，近期相关方法 [6, 16, 2] 开始基于Transformer[57]。iGPT[6] 直接处理像素序列，并预测未知像素。ViT 论文 [16] 研究了用于自监督学习（Self-Supervised Learning）的掩码图像块预测（Masked Patch Prediction）。最近，BEiT[2] 提出了预测离散标记（Discrete Tokens）[44, 50] 的方法。Self-supervised learning approaches have seen significant interest in computer vision, often focusing on different pretext tasks for pre-training [15, 61, 42, 70, 45, 17]. Recently, contrastive learning [3, 22] has been popular, e.g., [62, 43, 23, 7], which models image similarity and dissimilarity (or only similarity [21, 8]) between two or more views. Contrastive and related methods strongly depend on data augmentation [7, 21, 8]. Autoencoding pursues a conceptually different direction, and it exhibits different behaviors as we will present.自监督学习（Self-Supervised Learning）方法在计算机视觉领域受到了广泛关注，这类方法通常聚焦于设计不同的代理任务（Pretext Tasks）来进行预训练 [15, 61, 42, 70, 45, 17]。近年来，对比学习（Contrastive Learning）/