Investigating the Inverse Task of Redefinition in Multi-Agent LLM Systems

Abstract

The capabilities of Large Language Models (LLMs) in natural language and reasoning tasks are constantly evolving. However, the question remains whether these capabilities reflect genuine logical thinking. Previous experiments demonstrate the inability of LLMs to adapt to changes that conflict with their training data. The Redefinition task, in which the model is asked to replace the value of a scientific constant with a new, different value, is a characteristic example of this weakness. At the same time, to address inherent limitations of LLMs, multi-agent architectures have been proposed, in which LLM-based agents interact to achieve a common goal, just as human teams are formed to tackle a problem more effectively. This thesis studies the Redefinition problem for mathematical and physical constants and units of measurement in five different multi-agent settings. Its aim is to investigate whether a model's tendency to adhere to the values it learned during training (anchoring) persists, is mitigated, or is amplified when multiple instances of the same model are called upon to work together to answer the questions. The questions require the use of the new, redefined value and are of escalating difficulty, while the experiments are carried out in different response formats (Free-Form and Multiple-Choice), using various prompting techniques. Our results show that interaction between agents alone is not enough to limit the models' tendency to anchor to established values, ignoring the new instructions. Regarding constants, only one architecture managed to mitigate the anchoring phenomenon, under both Free-Form and Multiple-Choice questions. The same architecture is also the only one that reduces anchoring in units of measurement redefinitions for Free-Form questions. Under Multiple-Choice, however, most architectures were beneficial for units of measurement.