From Prediction to Understanding: Retrieval-Augmented Generation for Explaining Synthetic Clinical Trajectory Forecasts

Abstract

Clinical trajectory models forecast future diagnoses as sequences of ICD-10 (International Classification of Diseases, 10th Revision) codes, whose clinical implications can be difficult to interpret. This thesis designs and evaluates a retrieval-augmented generation (RAG) system that converts synthetic forecasts into personalised explanations linked to clinical evidence. The trajectory module uses a Delphi-style demonstration checkpoint trained on synthetic UK Biobank-format sequences, not the published population model. A fixed two-phase language model pipeline first generates an explanation without retrieved documents, then passes it to a second call that adds structured ICD knowledge, semantic retrieval, cross-references, and PubMed evidence. The evaluation comprises ten main benchmark runs over sixteen synthetic patient profiles, with 38–47 explained events per condition, and one three-profile pilot run executed mid-campaign. It compares single-pass and two-phase generation, full retrieval and ablations, and medium-capability and weaker generators. Three language model judges from different providers score the anonymised explanations across three rounds with randomised presentation order. Each system explanation is judged side by side with a medium-model single-pass baseline. The mean score difference Δ (with bootstrap confidence intervals) is therefore a pairwise preference against a comparator-dependent baseline score, not an absolute quality gain. On the normal-selected event pool, mean score differences are +0.009 for single-pass full RAG and +1.222 for two-phase full RAG. Score differences concentrate in Medical Depth and Personalization, while temporal accuracy and hallucination-check differences are near zero. The judged two-phase text is largely written in the first phase (almost entirely so without retrieval), and its different prompt is the likeliest explanation, though not isolated. Single-pass generation, which shares the baseline’s prompt, is not distinguishable from it. Within two-phase runs, retrieval scores lower than no retrieval on normal-selected events but higher on rare-selected events. The interaction contrast is +0.59 (95% bootstrap confidence interval [+0.25, +0.93]). Its cross-pool interpretation remains exploratory because baseline-freezing rules differ and independent cell resampling omits matched-event and shared-profile dependencies. On the nine events identical in both pools, which cannot differ in rarity, the same contrast is +0.99, so run-level differences alone can produce an interaction of this size and the pattern does not isolate rarity. For the medium generator, an exploratory check on events with the same draft in both runs (14/38 rare, 25/47 normal with FAISS-only retrieval) detected no clear change from adding evidence. With the weaker generator, whose drafts were identical in both arms, retrieval raised the explanation’s own score (+0.19) and narrowed the deficit relative to the baseline, although both weaker-model configurations remain below it. The contribution is a documented multi-source explanation pipeline and comparative evidence on architecture and retrieval within this benchmark. Limitations include synthetic profiles and a demonstration forecaster, reliance on language model judges with possible format and verbosity biases, no length-controlled ablation or clinician validation, and incomplete verification of citation faithfulness. The findings motivate further research on how the first write is prompted and on retrieval measured against a fixed draft. Adaptive gating was not implemented or validated. They do not establish clinical safety, and the system does not constitute medical advice.