【南加大研究揭示ChatGPT参数规模疑云：或仅为70亿】
据量子位报道，近日，南加州大学的一支研究团队在探索OpenAI的ChatGPT模型时，取得了一项重要发现。该团队的三位研究人员成功破解了未公开的gpt-3.5-turbo模型的嵌入向量维度，其数值为4096或4608。这一发现引发了对ChatGPT参数规模的重新评估。
通常情况下，开源大模型如Llama和Mistral，在嵌入向量维度为4096时，其参数规模大约为70亿。这一比例被认为是保持模型性能的理想平衡，过宽或过窄的网络结构都会对模型的效能产生负面影响。基于此，南加大团队推测，除非gpt-3.5-turbo采用了特殊的MoE（Mixture of Experts）架构，否则其参数规模很可能也接近70亿。
MoE架构是一种先进的深度学习技术，允许模型在不同任务中动态分配计算资源，从而在保持性能的同时，可能实现更高效的参数利用。然而，OpenAI尚未公开确认ChatGPT是否采用了这种架构。
这一研究结果对于理解ChatGPT的性能和效率具有重要意义，也为未来大模型的设计提供了参考。尽管如此，确切的参数规模仍需等待OpenAI的官方确认。南加大团队的研究无疑为公众揭开了ChatGPT神秘面纱的一角，引发了业界对模型优化和参数效率的深入讨论。
英语如下：
**News Title:** “USC Study Casts Doubt on ChatGPT’s Parameter Size: Could It Be Just 7 Billion, Challenging Standard Proportions for Large Models?”
**Keywords:** USC study, ChatGPT parameters, model performance
**News Content:**
**”USC Study Reveals Mystery Surrounding ChatGPT’s Parameter Scale: Potentially Only 7 Billion”**
According to Quantum Bit, a research team from the University of Southern California (USC) recently made a significant discovery while exploring OpenAI’s ChatGPT model. The trio of researchers successfully decrypted the embedding vector dimension of the unreleased gpt-3.5-turbo model, which stands at either 4096 or 4608. This finding has sparked a reevaluation of ChatGPT’s parameter size.
In general, open-source large models like Llama and Mistral, with an embedding vector dimension of 4096, are estimated to have around 7 billion parameters. This ratio is considered an optimal balance for maintaining model performance, as overly wide or narrow network structures can negatively impact efficiency. Based on this, the USC team speculates that unless gpt-3.5-turbo employs a specialized MoE (Mixture of Experts) architecture, its parameter count is likely to be in the neighborhood of 7 billion.
MoE architecture is an advanced deep learning technique that enables models to dynamically allocate computational resources for different tasks, potentially allowing for higher parameter efficiency while maintaining performance. However, OpenAI has not officially confirmed whether ChatGPT utilizes this architecture.
This research outcome holds significant implications for understanding ChatGPT’s performance and efficiency and provides a reference for future large model design. Nonetheless, the exact parameter size awaits official confirmation from OpenAI. The USC team’s investigation has undeniably shed light on one aspect of ChatGPT’s enigma, fostering a deeper industry discourse on model optimization and parameter efficiency.
【来源】https://mp.weixin.qq.com/s/y0RQ0aOrHGLzLJKxbyGxMw
Views: 1