andrewzw20.bsky.social @andrewzw20.bsky.social · 29/06/2026Big thanks to amazing collaborators @moberst.bsky.social and @sethzhangjs.bsky.social on the work! 020 2
andrewzw20.bsky.social @andrewzw20.bsky.social · 29/06/2026For more details, see the full paper at arxiv.org/abs/2509.19671, and at @CHILconference , where I’ll be presenting this work on June 29-30!arxiv.orgRevisiting Performance Claims for Chest X-Ray Models Using Clinical ContextPublic datasets of Chest X-Rays (CXRs) have long been a popular benchmark for developing machine learning (ML) computer vision models in healthcare. However, the reported strong average-case performan... 000 0
andrewzw20.bsky.social @andrewzw20.bsky.social · 29/06/2026These findings argue for the integration of context into future evaluation pipelines. To ensure models add diagnostic value to human decision-makers, their potential use cases in the diagnostic workflow need to be evaluated during development to assess their true contribution. 000 0
andrewzw20.bsky.social @andrewzw20.bsky.social · 29/06/2026We find that models struggle on these cases with performance drops in both setups. This suggests that for patients with similar clinical risk profiles, models struggle to resolve this uncertainty. Additionally, these models may not be using visual cues independent of context. 000 0
andrewzw20.bsky.social @andrewzw20.bsky.social · 29/06/2026We then construct evaluation sets via Inverse Probability Weighting (IPW) to statistically decouple the context from the disease label, and Matched Neighbors to pair positively and negatively labeled images based on similar context risk. 000 0
andrewzw20.bsky.social @andrewzw20.bsky.social · 29/06/2026We first find that model performance drops significantly for images with increasing pre-test probabilities, suggesting that they may be most useful in triaging low-risk cases, as opposed to images whose contextual information is less conclusive. 000 0
andrewzw20.bsky.social @andrewzw20.bsky.social · 29/06/2026We use discharge summaries occurring before each CXR to calculate a “pre-test probability” for various disease labels. This acts as a proxy for the context and risk profiles clinicians already have before the image is even taken. 000 0
andrewzw20.bsky.social @andrewzw20.bsky.social · 29/06/2026We developed a framework to evaluate how well models actually complement prior knowledge available to a clinician from their context, and assess whether diagnostic vision models perform reliably across clinically meaningful subpopulations and scenarios. 700 7
andrewzw20.bsky.social @andrewzw20.bsky.social · 29/06/2026Public datasets of Chest X-Rays (CXRs) are a popular benchmark for ML computer vision models, which often report strong average-case performance. But does this aggregate performance reflect actual utility in the clinic? 233 3
