Clueing up LLMs with Tool-Augmented Deductive Reasoning

Problem

The paper addresses a gap in the capability of large language models (LLMs) to perform logically consistent deductive reasoning over extended interactions. This limitation is particularly evident in complex scenarios requiring multi-turn reasoning, which is critical for applications in interactive environments. The work is presented as a preprint and has not undergone peer review.

Method

The authors propose a tool-augmented approach implemented in a text-based, multi-agent version of the board game Clue. The system consists of six LLM-based agents, specifically utilizing GPT-4o-mini and Gemini-2.5-Flash, with three agents from each model family participating in turn-based gameplay. A key innovation is the introduction of a structured possibility matrix that transforms the implicit game state into an explicit representation. This matrix serves two primary functions: it encodes extended-turn memory, allowing agents to retain information across multiple turns, and it imposes deductive constraints that guide the reasoning process of the agents.

Results

The available text does not report quantitative results. The authors establish baseline performance through repeated games with the LLM agents, but no specific metrics or comparisons against established benchmarks are provided. The evaluation of the tool-augmented approach is discussed in terms of reasoning quality and task success, yet again, no numerical results or detailed comparisons to other methods are disclosed.

Limitations

The authors do not report any limitations in their work. However, the absence of quantitative results and comparisons to existing methods may hinder the assessment of the proposed approach's effectiveness relative to other state-of-the-art techniques in deductive reasoning.

Why it matters

This research has significant implications for the development of LLMs capable of more sophisticated reasoning in interactive settings. By enhancing deductive reasoning through structured representations, the findings could inform future work on LLM applications in complex decision-making tasks, potentially improving their utility in domains such as game AI, automated reasoning systems, and interactive storytelling.