An explanation can make health AI feel more trustworthy without making its answer more reliable. Research led by MIT collaborators found that diagnostic interfaces affected non-experts and primary-care providers in sharply different ways, especially when a model was wrong.
In a dermatology study published in Nature Medicine, non-experts generally became more accurate with AI assistance, but largely because they deferred to the system. Clinicians performed best when shown a prediction without an explanation. The findings challenge the idea that adding a heat map, similar image or fluent rationale automatically creates responsible AI.
The same explanation changes different users differently
Participants saw skin images alongside an AI prediction. One interface displayed the result and confidence without explanation. Others showed similar cases, highlighted image regions with heat maps or generated plain-language reasoning with a large language model.
Non-experts judged whether moles were cancerous, while clinicians produced differential diagnoses across a broader set of dermatological conditions. Every explanation format improved non-expert accuracy in the tested setting, especially for recognising non-cancerous moles.
Yet improvement came with vulnerability. A fairness-constrained model increased accuracy and reduced disparities across skin tones, but non-experts relied heavily on its recommendations. Incorrect outputs harmed their performance more than correct outputs helped it.
LLM explanations created the strongest deference. Participants trusted them whether the underlying prediction was correct or wrong, and vague or generic explanations could appear especially convincing. Users receiving language-model assistance also reported greater confidence in incorrect answers.
That combination—plausible prose, misplaced confidence and limited medical knowledge—is particularly important for consumer health products. A polished rationale may become the basis of a user’s opinion rather than evidence they can critically assess.
Clinicians benefited from a more restrained interface
Primary-care providers were more resilient when the AI recommendation or explanation was wrong. Their best performance came from seeing the model’s prediction without added reasoning, while LLM-generated explanations delivered the smallest improvement among the tested clinician interfaces.
The study does not prove that explanations are useless in clinical care. It shows that an explanation designed for one audience may interfere with another audience’s decision process. Clinicians often form an initial diagnosis and compare the AI result with existing knowledge; a non-expert may use the same rationale to form the initial belief.
Timing changed behaviour too. Users deferred more when the explanation appeared before they had an opportunity to make their own assessment. An interface could instead ask for an initial hypothesis, then reveal the model’s output and alternatives. That sequence preserves independent judgement before introducing an anchor.
The people who relied most heavily on AI were also the weakest performers without it. They may gain the most from accurate assistance while facing the greatest harm from an error. Interface design must therefore handle benefit and dependence together rather than assuming confidence is evidence of understanding.
Health AI needs role-specific safeguards and failure testing
Product teams should define the user before choosing an explanation method. A clinician may need calibrated predictions, comparable cases and uncertainty that support professional review. A patient may need clear limits, escalation advice and warnings against treating generated reasoning as diagnosis.
Evaluation must deliberately include wrong model outputs. Measuring only how users behave when AI is correct hides automation bias. Useful tests should track whether users detect errors, how explanations change confidence and whether performance varies with expertise, skin tone or atypical symptoms.
The research found that AI performed better when symptoms were subtle, while humans did better when images contained atypical or unrelated features. Interfaces could use that complementarity by encouraging disagreement and showing when a case differs from the model’s familiar patterns.
This was a dermatology study with specific tasks and participant groups; its results should not be generalised mechanically to every medical specialty. Clinical consequences, user training and available evidence vary widely.
The central design lesson is nevertheless concrete: explainability is an interaction, not a property attached to a model. The right question is not whether an AI system provides an explanation, but whether that explanation helps this user make a safer decision when the system is both right and wrong.
