Artificial intelligence has captivated the world, with large language models (LLMs) like ChatGPT demonstrating remarkable capabilities in understanding and generating human language. However, a recent study by MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) reveals that these models may not be as adept at reasoning as previously thought, especially when faced with unfamiliar tasks.
Researchers at CSAIL scrutinized the performance of LLMs across a variety of tasks, uncovering intriguing insights into the interplay between memorization and reasoning skills. They found that the reasoning abilities of these models are often overestimated. The study compared “default tasks,” which are the standard tasks on which models are trained and evaluated, with “counterfactual scenarios,” hypothetical situations that deviate from normal conditions. Models like GPT-4 and Claude, which are generally expected to handle these variations, struggled significantly.
Instead of creating entirely new tasks, the researchers modified existing ones to push the models beyond their comfort zones. They used various datasets and benchmarks designed to test different capabilities, such as arithmetic, chess, code evaluation, and logic questions. Typically, when users perform arithmetic with language models, it’s in base-10, a familiar territory for the models. However, excelling in base-10 might misleadingly suggest strong addition skills. True proficiency would entail consistent high performance across all numerical bases, akin to calculators. The study revealed that these models are not as capable as presumed.
Their stellar performance on familiar tasks plummets dramatically in counterfactual scenarios, highlighting a lack of generalizable arithmetic ability. This trend extended to other tasks like musical chord fingering, spatial reasoning, and chess, where altered starting positions of pieces presented a challenge. While human players can adapt given time, the models fared no better than random guesses, indicating poor generalization to unfamiliar situations. Much of their success on standard tasks appears rooted in overfitting to or memorizing training data rather than demonstrating true task proficiency.
“We’ve uncovered a fascinating aspect of large language models: they excel in familiar scenarios, almost like a well-worn path, but struggle when the terrain gets unfamiliar,” remarked Zhaofeng Wu, an MIT PhD student in electrical engineering and computer science, CSAIL affiliate, and the lead author on the new paper. “As AI is becoming increasingly ubiquitous in our society, it must reliably handle diverse scenarios, whether familiar or not. We hope these insights will one day inform the design of future LLMs with improved robustness.”
Despite the valuable insights, the researchers acknowledge limitations in their study. The focus on specific tasks and conditions doesn’t encompass the wide array of challenges LLMs might face in real-world applications, underscoring the need for more diverse testing scenarios. Future work could involve expanding the range of tasks and counterfactual conditions to uncover more potential weaknesses. This could mean looking at more complex and less common scenarios. The team also aims to improve interpretability by creating methods to better comprehend the rationale behind the models’ decision-making processes.
According to Hao Peng, an assistant professor from the University of Illinois at Urbana-Champaign, the more a language model scales up, the more difficult it is to understand how the model is trained. The community still is in the dark about whether language models truly generalize to unseen tasks or superficially succeed by memorizing the training dataset. The paper makes major strides toward answering this question and gives new insights into the capabilities of state-of-the-art LLMs.
The research, supported by the MIT–IBM Watson AI Lab, the MIT Quest for Intelligence, and the National Science Foundation, was presented last month at the North American Chapter of the Association for Computational Linguistics. Results show that some real future research is still needed to see the robustness and adaptability of large language models—desiring reassurances that it can perform across a wide array of situations as reliably as possible in a fast-changing technological environment.
More for you:
- Reasoning skills of large language models are often overestimated
- Large language models like ChatGPT struggle as soon as they encounter unfamiliar problems, MIT study finds!