Beyond accuracy: an educational benchmarking study of task fragility and reasoning stability of large language models on dermatology board-style questions
New papers tested clinical LLMs on stability, bias and reproducibility rather than headline accuracy.
TL;DR
- A benchmarking study indexed on PubMed examined task fragility and reasoning stability of large language models on dermatology board-style questions.
- Other new papers tested implicit bias in LLM diagnosis of eating disorders and the accuracy and reproducibility of ChatGPT, DeepSeek and Gemini on patient-centred lipedema questions.
- On arXiv, PCQC proposed privileged counterfactual question credit for multi-turn medical dialogue.
A new study went 'beyond accuracy', benchmarking task fragility and reasoning stability of large language models on dermatology board-style questions. [1]
An experimental vignette study examined implicit bias in large language model diagnosis of eating disorders. [2]
Another paper evaluated the accuracy and reproducibility of ChatGPT, DeepSeek and Gemini in responding to patient-centred lipedema questions. [3]
On the methods side, an arXiv paper introduced PCQC, a privileged counterfactual question credit approach for multi-turn medical dialogue. [4]
Why it matters
Evaluation of clinical LLMs is moving past headline accuracy to stability, bias and reproducibility, the properties regulators and hospital buyers will want evidence on before trusting AI tools in patient-facing roles.
Editor's note
Cited at the metadata level; findings were not summarised from abstracts and were not reproduced. This is not medical advice.