OpenNLPLab 团队近日发布了一款名为 Lightning Attention-2 的新一代线性注意力机制，旨在一劳永逸地解决大语言模型长序列问题。该机制让长序列的训练和推理成本与 1K 序列长度的一致，使得在遇到显存瓶颈之前，无限增大序列长度不会对模型训练速度产生负面影响。这一突破使得无限长度预训练成为可能，并将极大地降低大语言模型的推理成本。
Lightning Attention-2 机制的提出，将改变现有大语言模型在处理长序列时的性能瓶颈，为人工智能领域带来更大规模、更高效的语言模型。这一技术突破在大数据时代背景下具有重要意义，将为自然语言处理、机器翻译、文本生成等应用场景提供更强大的支持。
英文翻译：
News Title: Breakthrough in Next-Generation Attention Mechanism Solves Long Sequence Problem in Large Language Models
Keywords: Lightning Attention-2, Linear Attention Mechanism, Large Language Models
News Content:
The OpenNLPLab team has recently released a new generation of linear attention mechanism called Lightning Attention-2, aiming to solve the long sequence problem in large language models once and for all. This mechanism enables the training and inference cost of long sequences to be consistent with that of 1K sequence lengths, allowing infinite sequence length pre-training to become possible before hitting memory bottlenecks. This breakthrough will significantly reduce the inference cost of current large language models.
The introduction of Lightning Attention-2 is set to overcome the performance bottleneck of existing large language models in handling long sequences, enabling the creation of larger and more efficient language models in the era of big data. This technological breakthrough holds great significance in fields such as natural language processing, machine translation, and text generation, where it will provide stronger support.
【来源】https://www.jiqizhixin.com/articles/2024-01-18-5
Views: 5