Latest general-purpose artificial intelligence (AI) models recorded lower accuracy in a skin lesion diagnosis experiment than Google’s image recognition model released in 2016. AI performance is advancing rapidly, but analysis suggests that in medical fields requiring expertise such as skin cancer, specialised training methods may matter more than a model’s size or recency.

On Oct. 26, the online media outlet Gigazine reported that Magnus Lynch (마그너스 린치), principal investigator at the Lynch Institute at King's College London, compared GPT-6 Astra, Claude Fable 5.1 and Google's Inception v3 using 200 skin lesion images. Inception v3 is an image recognition model released in 2016 and underwent 20 minutes of additional training using 68,011 skin images.

The images used in the experiment consisted of 100 lesions requiring examination or treatment and 100 lesions that could be sent home without further action. Each model selected the most likely option among 6 diagnoses.

The results favoured the older model. Inception v3 correctly identified 125 of 200 cases, an accuracy rate of 62.5 percent. GPT-6 Astra identified 84 cases, or 42 percent, and Claude Fable 5.1 identified 85 cases, or 42.5 percent. Inception v3 led the two general-purpose AI models by about 20 percentage points.

A gap also appeared in the ability to identify melanoma, a deadly form of skin cancer. Inception v3 correctly diagnosed 13 of 30 melanoma cases. GPT-6 Astra identified 8 cases and Claude Fable 5.1 identified 11.

The difference was also pronounced in distinguishing benign lesions. Inception v3 correctly classified 37 of 50 nevi cases and 26 of 50 seborrheic keratosis cases. GPT-6 Astra correctly identified 16 and 13 cases, respectively, while Claude Fable 5.1 identified 33 and 6.

Individual cases also showed the risk of dangerous misdiagnosis. On an image of invasive melanoma smaller than 0.8 mm with almost no pigmentation, GPT-6 Astra produced an answer of basal cell carcinoma with 95 percent confidence. Claude Fable 5.1 and Inception v3 judged the lesion as melanoma. In a case of seborrheic keratosis on a leg, all three models produced incorrect answers.

When allowing up to the second choice even if the correct answer was not included as the top choice, Inception v3’s cases including the correct answer rose to 173. GPT-6 Astra had 134 cases and Claude Fable 5.1 had 138.

Lynch assessed that an older, smaller specialised neural network showed far better performance in skin lesion recognition than two cutting-edge general-purpose models. He also stressed that the training conditions for the three models were not the same. Inception v3 underwent additional training on skin images, but no separate additional training was applied to the two general-purpose models.

He explained that state-of-the-art AI for skin cancer diagnosis uses image transformation algorithms trained on vast data, and that in specialised medical tasks, the right data and training methods are important.

It is difficult to interpret the results as meaning the latest general-purpose AI lags behind across the medical field. That is because the comparison targets and training conditions were limited and the evaluation included only 200 cases. The test still shows that even if general-purpose AI performance improves, task-specific training can determine competitiveness in areas such as medical imaging that require specialised knowledge and precise discrimination. Larger-scale verification and safety assessments are needed for use in real clinical settings.