Picture a hospital ward at 3 a.m. A monitoring agent notices a patient’s vitals drifting in a pattern that looks unremarkable on its own — but matches an early-sepsis signature buried in thousands of past cases. It flags the trend, pulls the relevant protocol, and hands its assessment to a second agent that drafts an alert for the on-call physician, all before a human has opened the chart. This is no longer a thought experiment: it is the direction medical AI is moving in 2026, as systems shift from single-purpose classifiers to increasingly autonomous *agents* — software that retrieves evidence, reasons across a patient record, and hands off conclusions to other agents before a clinician ever sees a final output. This shift exposes a gap that earlier explainability debates largely avoided: the difference between an *interpretable* system, whose internal logic can be inspected, and an *explainable* one, which merely produces a plausible-sounding account of its reasoning. For agentic systems built on large language models (LLMs), this distinction is no longer academic — it is becoming the central technical and regulatory bottleneck.
The Core Problem: Unfaithful Reasoning
Most LLM-based medical agents generate a chain-of-thought (CoT) — a step-by-step natural-language justification — alongside their output. Clinicians and regulators alike tend to treat this narrative as a window into the model’s actual decision process. Research increasingly shows this assumption is unsafe.
Lanham et al. (2023) demonstrated that CoT explanations can be systematically unfaithful: truncating or perturbing the stated reasoning often leaves the final answer unchanged, indicating the explanation was not load-bearing [1]. In medical domains specifically, a 2025 analysis of clinical LLM reasoning traces identified a recurring failure mode termed “CoT mismatch,” in which a model’s stated rationale supports one diagnosis while the model selects another [2]. A 2026 review of reasoning-driven LLMs in clinical medicine confirms the pattern: models can produce reasoning that is fluent, terminologically correct, and structurally familiar to clinicians, while drawing on erroneous causal links between symptoms and conditions [3]. The explanation is not a window into the computation — it is a second, separately generated output that resembles one.
Compounding Effects in Multi-Agent Systems
This problem scales badly in agentic architectures. A 2025 review of LLM-based biomedical agents notes that systems increasingly chain specialised agents — for triage, evidence retrieval, documentation, and coding — with each agent’s output feeding the next [4]. If an intermediate agent’s stated reasoning is unfaithful, downstream agents (and the clinician reviewing the final output) inherit an error that is invisible at every individual step. The attribution problem this creates — which agent’s “judgment” actually drove an adverse outcome — currently has no standard methodology for resolution.
Current Technical Approaches
Three approaches dominate current practice, with markedly different maturity levels.
Evidence-grounded reasoning
Frameworks combining CoT with retrieval-augmented generation (RAG) ground each reasoning step in retrievable clinical evidence — guidelines, prior notes, lab values — and some convert the resulting trace into a structured, clinician-reviewable “evidence chain” rather than free text [5]. This improves traceability to sources but does not by itself guarantee that the trace reflects the model’s actual computation.
Process supervision
Training approaches that reward correct intermediate reasoning steps (rather than only correct final answers) are intended to make verbalised reasoning more verifiable and reduce the gap between stated and actual reasoning [6]. This remains largely a training-time intervention rather than something verifiable at inference time in deployed systems.
Mechanistic interpretability
The most direct response to the faithfulness problem is to examine a model’s internal computations rather than its self-report. Recent interpretability work has shown it is possible, in constrained settings, to identify internal features corresponding to genuine intermediate computation and distinguish these from cases where a model’s stated reasoning does not match its internal activity [7]. This is the only approach that addresses faithfulness directly, but it remains research-stage and has not yet been integrated into clinical product documentation at scale.
For imaging-based (non-LLM) models, feature-attribution methods such as saliency maps remain the default and are still routinely cited in regulatory submissions [8] — but they answer “where did the model look,” not “what was its reasoning,” and do not transfer to agentic, language-based systems.
Regulatory Implications
Current transparency frameworks largely presuppose that a documented explanation is a meaningful proxy for the model’s logic. The EU AI Act classifies most diagnostic, triage and clinical decision-support systems as high-risk, requiring providers to supply instructions enabling users to interpret outputs (Article 13), with core obligations applying from August 2026 and a broader individual right to explanation (Article 86) deferred to 2027 [9]. In the US, FDA guidance on transparency and on Predetermined Change Control Plans for AI-enabled devices similarly expects manufacturers to document “algorithm logic” as a basis for post-market modification pathways [10].
Neither framework currently distinguishes between an explanation that is faithful and one that is merely plausible. As faithfulness research matures, this gap is likely to become a substantive compliance issue: a documented explanation that can be shown, post hoc, to misrepresent the system’s actual basis for a recommendation may not satisfy a “meaningful explanation” standard, regardless of its fluency.
Outlook
Closing the faithfulness gap will likely require three developments operating together: standardised faithfulness-auditing methods that can be applied to deployed models, not just research benchmarks; agent-level audit logs that record which agent acted on what evidence with what confidence, enabling error attribution across multi-agent pipelines; and regulatory criteria that explicitly require evidence of faithfulness, not merely the presence of a generated explanation. Until these converge, the explanations attached to medical AI agents should be treated as hypotheses about the system’s reasoning — useful for clinical review, but not yet reliable evidence of it.
References
- Lanham, , Chen, A., Radhakrishnan, A., et al. (2023). Measuring Faithfulness in Chain-of-Thought Reasoning. Anthropic. arXiv:2307.13702.
- Sim, Z. Y., & Chen, T. (2025). Critique of impure reason: Unveiling the reasoning behaviour of medical large language models. Elife, 14, e106187.
- Wang, , Xiong, Z., Zou, K., Srinivasan, S., Lo, T. W. S., Wu, Y., … & Tham, Y. C. (2026). Reasoning-driven large language models in medicine: opportunities, challenges, and the road ahead. The Lancet Digital Health.
- Xu, , & Sankar, R. (2025). Large language model agents for biomedicine: a comprehensive review of methods, evaluations, challenges, and future directions. Information, 16(10), 894.
5] Du, J., Li, X., Liu, Y., Lv, T., Liu, H., & Yin, H. (2025). Large language models driven reliable clinical decision-making: Framework and application. Informatics and Health.
- Lei, , Xu, J., Liang, C.X., Bi, Z., Li, X., Zhang, D., Song, J., & Yu, Z. (2025). Reasoning in Large Language Models: From Chain-of-Thought to Massively Decomposed Agentic Processes.
Preprints.org. Available at: https://doi.org/10.20944/preprints202512.2242.v1
- (2025). Tracing the Thoughts of a Large Language Model. Anthropic Research.
Available at: https://www.anthropic.com/research/tracing-thoughts-language-model
- (2026). EU MDR & AI Act Compliance for AI Medical Devices. Available at: https://intuitionlabs.ai/articles/ai-medical-device-compliance-eu-mdr-ai-act
- European Union. (2024). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act), Articles 13 and Available at: https://eur-lex.europa.eu/eli/reg/2024/1689/oj
- S. Food and Drug Administration. (2024). Transparency for Machine Learning-Enabled Medical Devices: Guiding Principles. Available at: https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-software-medical-device
- S. Food and Drug Administration. (2025). Marketing Submission Recommendations for a Predetermined Change Control Plan for Artificial Intelligence-Enabled Device Software Functions (Final Guidance). Available at: https://www.fda.gov/regulatory-information/search-fda-guidance-documents/marketing-submission-recommendations-predetermined-change-control-plan-artificial-intelligence
