Evaluating Reasoning Faithfulness
Evaluating Reasoning Faithfulness in Medical Vision-Language Models using Multimodal Perturbations
Evaluating Reasoning Faithfulness in Medical Vision-Language Models using Multimodal Perturbations
"Chain-of-thought (CoT) offers a potential boon for AI safety as it allows monitoring a model’s CoT to try to understand its intentions and reasoning processes. However, the effectiveness of such monitoring hinges on CoTs faithfully representing models’ actual reasoning processes."
(Chen et al., 2025, "Reasoning Models Don’t Always Say What They Think")
Abstract
Vision-language models (VLMs) often produce chain-of-thought (CoT) explanations that sound plausible yet fail to reflect the underlying decision process, undermining trust in high-stakes clinical use. Existing evaluations rarely catch this misalignment, prioritizing answer accuracy or adherence to formats. We present a clinically grounded framework for chest X-ray visual question answering (VQA) that probes CoT faithfulness via controlled text and image modifications across three axes: clinical fidelity, causal attribution, and confidence calibration.
In a reader study (n=4), evaluator-radiologist correlations fall within the observed inter-radiologist range for all axes, with strong alignment for attribution (Kendall's τb=0.670), moderate alignment for fidelity (τb=0.387), and weak alignment for confidence tone (τb=0.091), which we report with caution.
Benchmarking six VLMs shows that answer accuracy and explanation quality are decoupled, acknowledging injected cues does not ensure grounding, and text cues shift explanations more than visual cues. While some open-source models match final answer accuracy, proprietary models score higher on attribution (25.0% vs. 1.4%) and often on fidelity (36.1% vs. 31.7%), highlighting deployment risks and the need to evaluate beyond final answer accuracy.
Key Contributions
🏥 Clinically Grounded Dataset
A chest X-ray VQA dataset derived from expert annotations with clinically realistic, reasoning-oriented questions, reducing leakage risk and weak-label bias.
Click for details ▼🔍 Faithfulness Evaluation Framework
A perturbation-based protocol that assesses reasoning along clinical fidelity, causal attribution, and confidence calibration under both bias-inducing and evidence-manipulating cues.
Click for details ▼👩🏼⚕️ Expert Validation
A reader study with four board-certified radiologists showing our automatic metrics align with expert judgments.
Click for details ▼📊 Benchmarking
We evaluate six VLMs spanning proprietary, open-source, general-domain, reasoning, and medicine-specific models, revealing vulnerabilities of CoT explanations under a range of controlled text and image modifications.
Click for details ▼
Core Findings
- ⛓️💥 Accuracy and Explanation Quality Are Decoupled: High answer accuracy does not guarantee faithful or grounded chain-of-thought explanations.
- 💬 Disclosure ≠ Grounding: Models may acknowledge influence (attribution) without truly integrating evidence; explicit checks are needed.
- 📝 Textual Cues Dominate: Text-based prompts shift explanations more than visual cues, so standardized prompting is crucial for clinical reliability.
BibTeX
@article{evaluating-2025,
title={Evaluating Reasoning Faithfulness in Medical Vision-Language Models using Multimodal Perturbations},
author={Moll, Johannes and Graf, Markus and Lemke, Tristan and Lenhart, Nicolas and Truhn, Daniel and Delbrouck, Jean-Benoit and Pan, Jiazhen and Rueckert, Daniel and Adams, Lisa C. and Bressem, Keno K.},
journal={arXiv preprint arXiv:2510.11196},
url={https://arxiv.org/abs/2510.11196},
year={2025}
}