正文：
随着人工智能技术的不断进步，智能体（Agent）成为了大型语言模型社区中的研究热点。用户只需提出需求，智能体框架就能调度多个大型语言模型（LLMs），支持多智能体协作或竞争，完成复杂的任务。近日，CAMEL AI社区主导开发的CRAB（Cross-environment Agent Benchmark）框架问世，为多模态语言模型智能体提供了跨环境性能评估的基准。
CRAB框架采用基于图的细粒度评估方法，提供了高效的任务和评估器构建工具。它不仅支持单一环境任务，更能够评估智能体在跨平台环境下的表现，如同时操作电脑和手机。CRAB框架还推出了首个跨平台测试数据集CRAB Benchmark-v0，包含100个任务，涵盖了从单一设备到跨平台任务的各种场景。
该框架的推出，旨在解决现有智能体性能评估基准的局限性，如任务构建和测试环境的复杂性，以及评价指标的单一性等问题。通过CRAB，研究人员和开发者可以更全面地测试智能体的性能，推动多模态语言模型智能体的发展。
CRAB框架的实验结果显示，使用GPT-4作为推理引擎的单智能体结构在测试点完成率上表现最佳，达到35.26%。这一成就为智能体在真实世界中同时操作多个设备提供了可能，有望提高工作效率，简化复杂软件操作。
未来，随着CRAB框架的不断完善和普及，多模态语言模型智能体将在跨平台环境中发挥更大作用，为用户提供更加智能和便捷的服务。
英语如下：
News Title: “The Revolution of Intelligent Agents: New Breakthrough in Multi-task Cross-system Control”
Keywords: Intelligent Agent, Multimodal, Collaboration-Competition
News Content:
Title: CRAB, a Cross-platform Intelligent Agent Benchmark, Launches to Advance Multimodal Language Model Development
In the ongoing evolution of artificial intelligence technology, intelligent agents (Agents) have emerged as a focal point of research in the large language model community. With the agent framework, users can simply articulate their needs, and the system can orchestrate multiple large language models (LLMs) to support collaborative or competitive interactions among multiple agents, tackling complex tasks. Recently, the CRAB (Cross-environment Agent Benchmark) framework, developed by the CAMEL AI community, has been introduced as a benchmark for evaluating the performance of multimodal language model agents across environments.
CRAB employs a fine-grained, graph-based evaluation method and provides efficient tools for constructing tasks and evaluators. It not only supports single-environment tasks but also evaluates agents’ performance in cross-platform settings, such as operating both computers and smartphones simultaneously. CRAB has also released the first cross-platform test dataset, CRAB Benchmark-v0, which includes 100 tasks covering a range of scenarios from single-device tasks to cross-platform operations.
The launch of CRAB aims to address the limitations of existing intelligent agent performance evaluation benchmarks, such as the complexity of task construction and testing environments, and the single-mindedness of evaluation metrics. Through CRAB, researchers and developers can comprehensively test the performance of agents, thus driving the development of multimodal language model agents.
Experimental results with CRAB show that a single agent structure using GPT-4 as the inference engine achieved the best completion rate at test points, reaching 35.26%. This achievement opens up the possibility of agents operating multiple devices in the real world, promising to enhance productivity and simplify complex software operations.
As CRAB continues to evolve and gain wider adoption, multimodal language model agents will play a more significant role in cross-platform environments, offering users smarter and more convenient services.
【来源】https://www.jiqizhixin.com/articles/2024-08-14-4
Views: 2