七年前，一篇题为《Attention is all you need》的论文在深度学习领域引发了一场革命，提出了Transformer架构，它颠覆了传统的神经网络结构。如今，几乎所有的先进模型都是基于Transformer架构构建的。尽管如此，Transformer内部的工作原理仍然是一个谜。
去年，Transformer论文的作者之一Llion Jones创立了Sakana AI，并发表了一篇论文，将Transformer层比作画家作画流水线。论文通过实验探究了预训练Transformer中的信息流，并对仅包含解码器或仅包含编码器的Transformer模型进行了测试。
研究人员发现，Transformer模型中的中间层似乎共享一个共同的表征空间，这表明除了最外层和最内层之外，中间层的功能是相似的。实验表明，可以跳过一些中间层而不导致模型性能出现灾难性下降，这意味着不是所有的层都是必须的。
此外，研究人员测试了替换中间层的权重是否会影响模型性能。结果表明，即使替换了中间层的权重，模型在基准测试上的表现也会迅速下降，这表明每个层都有其独特的作用，不能简单地被其他层所取代。
这项研究不仅加深了对Transformer内部工作机制的理解，也为优化模型结构和提高模型性能提供了新的视角。随着人工智能技术的不断发展，这些发现有望为未来的模型设计和应用提供重要的指导。
英语如下：
Title: “Unveiling the Transformer: A New Perspective on Deep Learning Inspired by an Artist’s Painting Process”
Keywords: Transformer, Architecture, Understanding
News Content: Seven years ago, a paper titled “Attention is all you need” sparked a revolution in the field of deep learning, introducing the Transformer architecture which revolutionized traditional neural network structures. Today, almost all advanced models are built upon the Transformer architecture. Despite this, the inner workings of the Transformer remain a mystery.
Last year, one of the authors of the Transformer paper, Llion Jones, founded Sakana AI and published a paper that likened the Transformer layer to an artist’s painting pipeline. The paper explored the information flow within pre-trained Transformers through experiments and tested Transformer models that only contained decoders or only contained encoders.
Researchers discovered that the intermediate layers of the Transformer model seem to share a common representation space, indicating that the functions of the middle layers are similar, aside from the outermost and innermost layers. Experiments showed that skipping some intermediate layers did not lead to a catastrophic drop in model performance, suggesting that not all layers are necessary.
Furthermore, the researchers tested whether replacing the weights of the intermediate layers would affect model performance. The results showed that even with the weights of the intermediate layers replaced, the model’s performance on benchmark tests would quickly decline, indicating that each layer plays a unique role and cannot be simply replaced by others.
This research not only deepens our understanding of the Transformer’s inner workings but also offers a new perspective on optimizing model structures and enhancing model performance. As artificial intelligence technology continues to develop, these findings are expected to provide important guidance for future model design and applications.
【来源】https://www.jiqizhixin.com/articles/2024-08-07-3
Views: 1