AI in healthcare ‘double-edged sword’, can fool clinicians with data: Study

AI models in health care are a doubleedged sword with models improving diagnostic decisions for some demographics but worsening decisions for others when the model has absorbed biased medical data suggests a studyGiven the very real life and death risks of clinical decisionmaking researchers and policymakers are taking steps to ensure AI models are safe secure and trustworthy and that their use will lead to improved outcomesHowever the new study published in JAMA finds that even with provided AI explanations clinicians can be fooled by biassed AI modelsldquoThe problem is that the clinician has to understand what the explanation is communicating and the explanation itselfrdquo said Sarah Jabbour doctoral candidate in computer science and engineering at the College of Engineering at the University of MichiganThe team studied AI models and AI explanations in patients with acute respiratory failureldquoDetermining why a patient has respiratory failure can be difficult In our study we found clinicians baseline diagnostic accuracy to be around 73 per centrdquo said Michael Sjoding Associate Professor of internal medicine at the varsityrsquos Medical SchoolldquoDuring the normal diagnostic process we think about a patientrsquos history lab tests and imaging results and try to synthesise this information and come up with a diagnosis It makes sense that a model could help improve accuracyrdquo he addedThe team designed a study to evaluate the diagnostic accuracy of 457 hospitalist physicians nurse practitioners and physician assistants with and without assistance from an AI modelEach clinician was asked to make treatment recommendations based on their diagnoses Half were randomised to receive an AI explanation with the AI model decision while the other half received only the AI decision with no explanationClinicians were then given real clinical vignettes of patients with respiratory failure as well as a rating from the AI model on whether the patient had pneumonia heart failure or Chronic obstructive pulmonary disease COPDIn the half of participants who were randomised to see explanations the clinician was provided a heatmap or visual representation of where the AI model was looking in the chest radiograph which served as the basis for the diagnosisThe team found that clinicians who were presented with an AI model trained to make reasonably accurate predictions but without explanations had their own accuracy increase by 29 percentage points When provided an explanation their accuracy increased by 44 percentage pointsHowever to test whether an explanation could enable clinicians to recognise when an AI model is clearly biased or incorrect the team also presented clinicians with models intentionally trained to be biased for example a model predicting a high likelihood of pneumonia if the patient was 80 years old or olderWhen clinicians were shown the biased AI model however it decreased their accuracy by 113 percentage points and explanations which explicitly highlighted that the AI was looking at nonrelevant information such as low bone density in patients over 80 years did not help them recover from this serious decline in performanceThis story has been sourced from a third party syndicated feed agencies Midday accepts no responsibility or liability for its dependability trustworthiness reliability and data of the text Midday managementmiddaycom reserves the sole right to alter delete or remove without notice the content in its absolute discretion for any reason whatsoever