Empirical Testing of PlanBench Planning Capabilities in OpenAI o1 AI Model

A research team from Arizona State University recently utilized the cutting-edge benchmarking platform, PlanBench, to conduct a comprehensive evaluation of OpenAI's newly released o1 model's performance in the planning domain. This study not only highlights the significant advancements achieved by the o1 model in specific tasks but also reveals various limitations and challenges it faces in practical applications.

PlanBench Benchmark: A Litmus Test for Planning Capabilities

Developed meticulously in 2022, PlanBench is an evaluation framework specifically designed to measure the effectiveness of artificial intelligence systems in planning tasks. Its core consists of 600 challenging tasks derived from the Blocksworld domain, requiring intelligent agents to precisely stack blocks in a predetermined sequence, thereby testing their logical and strategic planning abilities.

OpenAI o1 Model: Concerns Beneath High Scores

Under the stringent evaluation of PlanBench, OpenAI's o1 model stood out with a 97.8% accuracy rate, significantly outperforming the previous leader—the LLaMA 3.1 405B model, which achieved only a 62.6% accuracy in standard Blocksworld tasks. However, when faced with the more intricate and encrypted version, "Mystery Blocksworld," the o1 model maintained a 52.8% accuracy rate, indicating underlying challenges that cannot be overlooked.

Randomized Variant Testing: Notable Performance Discrepancies

To further verify whether the o1 model's performance is influenced by training data, the research team introduced a novel randomized variant test. The results showed a sharp decline in the o1 model's accuracy to 37.3% under these conditions. Nevertheless, it still significantly outperformed competitors, which scored nearly zero, highlighting the o1 model's degree of generalization capability.

Complexity Challenges: Warning Signs of Performance Decline

As task complexity increases, the o1 model's performance weaknesses become more apparent. Specifically, in advanced tasks requiring 20 to 40 planning steps, its accuracy plummeted from a peak of 97.8% to 23.63%, indicating significant bottlenecks in handling highly complex problems. Additionally, the o1 model's accuracy in identifying unsolvable tasks was only 27%, with nearly half of the generated complete plans being theoretically non-executable, further exposing limitations in its decision-making mechanisms.

"Quantum Improvements" and Cost Considerations

Although the o1 model achieved so-called "quantum improvements" in benchmark testing, the correctness of its solutions is not fundamentally guaranteed. In contrast, traditional planning algorithms like the Fast Downward algorithm can achieve perfect accuracy in a shorter time, highlighting a dual advantage in efficiency and precision. Furthermore, the study points out that the high cost of running the o1 model—nearly $1900—stands in stark contrast to the almost zero operational costs of classical algorithms, prompting a profound reevaluation of the cost-effectiveness of AI systems.