Some artificial intelligence-driven apps are now about as good as a specialist dermatologist at diagnosing skin lesions and significantly better than a trainee, at least in an experimental setting.
Nevertheless, it appears human clinicians still have the last word, particularly when it comes to treatment decisions, where AI can struggle, Sydney University researchers say.
The verdict is based on a study led by dermatologist Professor Scott Menzies, which put two smartphone-based AI apps up against human doctors to test the accuracy of their diagnostic and management decisions.
The combined Australian-Austrian study funded by AI developer MetaOptima Technology, included 172 suspicious pigmented lesions (84 malignant) from 124 patients, photographed with a mobile phone using a simple dermoscopic attachment and analysed with the two AI programs.
The diagnosis was based on seven pigmented lesion categories: melanoma, melanocytic naevus, basal cell carcinoma, pigmented actinic keratosis or intraepithelial carcinoma, benign keratotic lesion, benign vascular lesion, and dermatofibroma.
Patients were also physically examined by each clinician (five specialists in skin lesion diagnosis and 18 dermatology trainees) separately, with no verbal communication or history-taking allowed prior to excision and testing.
Comparison with histopathology showed strong potential for the apps, with one – described as a ‘new 7-class AI algorithm’ developed by MetaOptima correctly diagnosing 74% of the lesions.
By contrast, the specialist dermatologists accurately diagnosed 73% of lesions and the trainees just 52%.
“This trial is the first prospective study to support the potential of AI-based skin cancer diagnosis of dermoscopy images compared with clinicians in a clinical setting for all clinically relevant classes of pigmented lesions,” they wrote in The Lancet Digital Health (link here).
“The importance of the study is highlighted because the results were obtained on the basis of a simple mobile phone technology without expensive hardware, in contrast to previous, more expensive, stand-alone devices undergoing large pivotal studies.”
Nevertheless, the other app, using the International Skin Imaging Collaboration (ISIC) AI algorithm, achieved an accuracy rate of only 61%, the researchers reported.
This difference in accuracy was likely a reflection of the fact the ISIC AI had been trained on a smaller database of images with less diverse sources, although it had performed better than humans in previous studies of different design, they said.
“However, a major difference between our trial and the ISIC reader study is that, in the ISIC reader study, humans were compared with the AI by diagnosing computer images from multiple sources that differed in polarisation, colour, and magnification, and without the aid of palpation,” the authors noted.
“By contrast, in our trial, clinicians applied their routine methods to diagnose, using familiar dermoscopy devices, without any time constraint, and face-to-face with the patient.”
And neither AI algorithm did was all as a human specialist when it came to management decisions, which were tested by giving the program (or human doctor) the choice of either dismissal, monitoring or biopsy of each lesion presented.
But considering the diagnosis algorithm was slightly superior to specialists in the diagnostic trial, a more intelligent conversion from the 7-class diagnosis AI to the management decision than the current method might be achieved, the researchers said.
Summing up, they said the mixed results showed the potential of AI technology on consumer devices like smart phones, but underlined that caution was still needed.
“Mobile phone-powered AI has the clear potential to be a simple, inexpensive, and accurate intervention for the diagnosis of pigmented skin cancer,” they concluded.
“In the regulatory policy setting, caution extrapolating experimental to real clinical settings is advised. Further research on the development of management AI is required.”