Retrieval Stability and Deferred Commitment in Multi-Turn Retrieval-Augmented Generation Systems

Abstract

Multi-turn conversational Retrieval-Augmented Generation (RAG) remains substantially harder than its single-turn counterpart: retrieval degrades across turns because of non-standalone queries, coreference and implicit topic carryover, while generation must stay faithful to the evidence and abstain when a request cannot be answered. This diploma thesis addresses these challenges through SemEval-2026 Task 8 (MTRAGEval), covering passage retrieval (Task A), reference-grounded generation (Task B) and end-to-end RAG (Task C) on the MTRAG benchmark. For retrieval, we replace heterogeneous retriever ensembles with query diversity: five complementary LLM-based reformulations are issued to a single corpus-aligned sparse retriever (ELSER v1) and fused through a hierarchical Nested Reciprocal Rank Fusion scheme. For generation, we propose a five-stage agentic pipeline (answerability triage, verbatim evidence span extraction, dual-candidate drafting, judge-based selection and micro-adjustment), guided by a 4-gram extractiveness shaping function and extended for end-to-end RAG with a multi-judge answerability gate. Development-set ablations indicate that query diversity over a single aligned retriever outperforms heterogeneous ensembling under the official relevance judgments, and that evidence extraction before generation improves faithfulness. On the test set, answerability calibration becomes the decisive end-to-end factor as the share of unanswerable requests nearly triples, while top-rank retrieval errors remain a substantial second source of failure. Our system ranked 1st of 38 systems in Task A (nDCG@5: 0.5776, +20.5% over the strongest baseline), 2nd of 26 in Task B (Harmonic Mean: 0.7698) and 11th of 29 in Task C (Harmonic Mean: 0.5409).