LLM Skin Cancer Benchmark: Why Diagnosis Accuracy and Triage Tell Different Stories

Dermatologist Magnus Lynch published a skin-lesion comparison on October 25 in which a fine-tuned Inception v3 model classified more cases correctly than GPT-6 Astra or Claude Fable 5.1. The useful finding goes beyond an older architecture beating newer models: naming a condition and deciding what happens next are different evaluation tasks.

This is a small, author-published experiment, not evidence that any of the systems is ready for clinical use. Its results are worth examining because a headline accuracy score can hide a very different pattern of errors in the decision that follows.

What the skin cancer benchmark measures

The released cases and model outputs contain 200 lesions across six diagnostic categories. The sample deliberately balances 100 cases labelled safe to discharge against 100 labelled unsafe. It is a constructed evaluation set, rather than an estimate of the conditions a typical clinic or consumer app would encounter.

Both LLMs received a clinical photograph and a dermoscopic image. The clinician-style prompt requested a diagnosis, confidence, second choice and a separate discharge decision. The accompanying record specifies one run per case, default settings, no system prompt and no reply-length cap. GPT-6 Astra used OpenAI's API; Claude Fable 5.1 used OpenRouter.

That setup measures the response to a particular instruction and image pair. Changing the task description, available information or decision rule could change the result. A comparison needs to preserve those choices alongside the model names.

The accuracy result and the decision result

The published comparison reports the following counts. The LLM columns can also be checked directly against the released structured replies; the CNN figures remain the author's reported results.

Measure	GPT-6 Astra	Claude Fable 5.1	Inception v3
Correct first diagnosis, out of 200	84	85	125
Unsafe cases retained, out of 100	97	96	77 at preset threshold
Melanomas discharged, out of 30	3	4	18 at preset threshold

The CNN leads on classification while its preset discharge threshold produces the weakest result on the last two measures. A model can name more conditions correctly overall and still make worse decisions about which cases require further attention.

Lynch also reports that changing the CNN threshold to retain 98 or 99 unsafe cases reduces discharged melanomas to two or one. That is a different operating point, not a correction to the original classification score. It illustrates why threshold selection belongs in the comparison rather than in a footnote.

Why the dataset changes the interpretation

The underlying MILK10k dataset, published in 2025, contains 10,480 paired images representing 5,240 lesions. Its documentation says 95.7% were biopsied or excised and that most patients had middle-range skin tones. The archive includes broader diagnostic categories and metadata than this six-category test uses.

Those details constrain what a result can mean. A dataset rich in biopsied cases does not reproduce the mix of ordinary photographs uploaded to an app. A balanced test also cannot tell a service how many alerts it will generate when the proportion of concerning cases changes.

For a developer, this creates two separate validation questions: whether the model separates the relevant categories, and whether the chosen operating point remains useful in the intended population. Improving one score does not answer both.

An old architecture is not an untouched old model

Inception's original architecture paper was submitted in December 2015. It explored efficient convolutional-network design for image classification, including ways to use computation more effectively. Its age describes the architecture's origin, not the age of the domain-specific training used here.

Lynch says his comparison fine-tuned the network on 68,011 dermatology images, excluding the test patients and lesions. This is therefore a specialized classifier compared with general-purpose multimodal models under one prompting setup. It does not establish a universal ranking between convolutional networks and language models.

The practical implication is narrower and more useful: a strong general model should still be compared with a task-specific baseline. The baseline helps establish whether the added flexibility of conversational output improves the actual job being evaluated.

What can be inspected, and what remains open

The downloadable record includes case labels, exact prompts, LLM replies and scoring scripts. It does not include CNN prediction files or its training pipeline, so it supports checking the LLM counts without independently reproducing the complete comparison. No fresh model inference was performed for this article. Lynch discloses that Claude Code assisted with his analysis and initial drafts, a useful detail when reading an evaluation that includes an Anthropic model.

One response per case leaves run-to-run variation unresolved. More importantly, a small retrospective test cannot establish clinical safety or replace validation in the intended workflow. Readers should not use these results or an online model to decide whether a skin concern needs medical assessment.

The next useful evidence would connect reproducible predictions, a declared decision threshold and a separate validation population. For AI builders, that is the lesson worth carrying forward: choose the metric around the decision the product will make, then show the tradeoff rather than promoting whichever score looks strongest.

I'm Paras. I work across AI, engineering and distribution.

I help technical teams explain research, evaluation results and product decisions clearly, with their evidence and limitations intact.

Book a 30-minute call.