近日，上海交通大学发布了一款名为PowerInfer-2的手机推理框架，旨在优化内存有限的智能手机上的推理运算能力。据了解，该框架能够流畅运行高达470亿参数的模型，使得Mixtral 47B模型在手机上的运行速度达到惊人的每秒处理11个令牌（tokens/s）。
上海交大IPADS实验室推出的这一大模型推理引擎——PowerInfer-2.0，让学界和工业界为之一振。与当前热门的开源推理框架llama.cpp相比，PowerInfer-2.0的推理加速比平均达到了惊人的25倍，最高甚至可以达到29倍。这一突破性的技术将极大地推动人工智能在手机等移动设备上的普及和应用。
专家表示，PowerInfer-2.0的推出将极大地促进人工智能技术在移动设备上的发展。这一创新性的技术将使得更多的大型模型能够在手机上流畅运行，从而推动人工智能技术在移动场景下的广泛应用。
此次上海交大发布的PowerInfer-2.0框架无疑为移动人工智能的发展开辟了新的道路，未来有望引领人工智能技术在手机上的新一轮发展热潮。
英语如下：
News Title: Shanghai Jiao Tong University Launches New Mobile Inference Framework PowerInfer-2: Speed Exceeds Popular Open Source Inference Frameworks!
Keywords: Shanghai Jiao Tong University, Mobile Inference, PowerInfer-2 Accelerated Inference Framework
News Content:
Shanghai Jiao Tong University has released an efficient mobile inference framework named PowerInfer-2.0, which achieves accelerated running of the Mixtral 47B model on mobile phones.
Recently, Shanghai Jiao Tong University has released a mobile inference framework called PowerInfer-2 aimed at optimizing inference computations on memory-limited smartphones. It is understood that this framework can smoothly run models with up to 47 billion parameters, achieving an impressive processing speed of 11 tokens per second on the Mixtral 47B model.
The big model inference engine – PowerInfer-2.0, launched by the IPADS Lab at Shanghai Jiao Tong University, has caused a stir in both academic and industrial circles. Compared with the current popular open-source inference framework llama.cpp, PowerInfer-2.0’s inference acceleration ratio averages an astonishing 25 times, with a maximum of 29 times. This breakthrough technology will greatly promote the popularization and application of artificial intelligence on mobile devices.
Experts indicate that the launch of PowerInfer-2.0 will greatly promote the development of artificial intelligence technology on mobile devices. This innovative technology will enable more large models to run smoothly on mobile phones, thereby promoting the widespread use of artificial intelligence in mobile scenarios.
The PowerInfer-2.0 framework released by Shanghai Jiao Tong University undoubtedly opens up a new path for the development of mobile artificial intelligence, and is expected to lead a new wave of development in artificial intelligence on mobile phones in the future.
【来源】https://www.qbitai.com/2024/06/153436.html
Views: 3