How Do You Know an AI Learned the Right Thing?

The horse that could count

In the early 1900s, a horse in Berlin named Hans drew crowds by doing arithmetic. Ask him what four plus three was, and he tapped his hoof seven times. Experts examined him and found no trick.

A careful experimenter noticed that Hans was right only when the person asking already knew the answer. The horse was not counting. He read tiny, unconscious changes in his questioner’s posture as the number of taps approached the right answer.

Machine learning researchers still use his name, Clever Hans, for a model that gets the right answer for the wrong reason.¹ Hans was often right. However, he was right for a reason that would fail the moment his questioner did not know the answer.

AI health tools face the same test. The first article in this series, AI Can’t Rescue a Bad Measurement, asked what a model is fed. This one asks the next question: when a model gets the answer right, did it get it right for the right reason?

This is where measurement turns into understanding. A score tells you how often a model was right. It does not tell you what the model learned to be right.

A model learns whatever works

During training, a model is rewarded for one thing: matching the answers in its examples. Nothing in that process tells which pattern is the right one. If two patterns separate the answers equally well, one rooted in the disease and one in an accident of how the data were collected, the model has no reason to prefer the first. It tends to take whichever is easiest to find. Researchers call this shortcut learning.¹

Chemists know a version of this problem. Suppose you calibrate a detector on standards that all happen to share the same trace contaminant. The instrument can end up responding to the contaminant rather than the compound you care about. Everything looks fine, run after run, until you measure a sample that does not contain it.

Analogy has one important limit. A chemist can usually name the contaminant and test it. A model rarely tells you what it latched onto, because its pattern is spread across an enormous number of internal settings. The shortcut must be found from the outside, by experiment.

The purple ink test

In 2018, dermatologists at the University of Heidelberg reported that deep learning convolutional neural networks (CNN) trained on dermoscopic images (i.e., magnified photographs of skin lesions) outperformed most of 58 dermatologists at telling melanoma from harmless moles in standardized photo-only evaluations.² The network scored 0.86 on the area under the curve (AUC), against the dermatologists’ average 0.79.² The AUC measures how well a model ranks a sick case above a healthy one: 0.5 is a coin toss, 1.0 is perfect.

A year later, the same research group tested a different CNN, one approved as a medical device in Europe.³ Often, before a suspicious mole is removed, it is first circled with a purple surgical skin marker. To investigate the association between the purple marker and the diagnostic performance by the CNN, the team photographed the same 130 skin lesions (107 harmless moles and 23 melanomas) three ways: unmarked, marked with purple ink, and marked but cropped so the ink fell out of frame.³

Without ink, the CNN flagged about 17 of the 107 harmless moles as possible melanoma, a specificity of 84.1% (the share of harmless moles it correctly left alone).³ With ink on the very same moles, it flagged about 58 of them, and specificity fell to 45.8%.³ It still caught the melanomas: 22 of 23 unmarked and all 23 marked.³ The ink made it see melanoma where there was none. Cropping the ink out brought specificity up to 97.2%, or about 3 false alarms.³

Nothing about the moles had changed. Only the ink had. Heat maps, which highlight the parts of an image that drive the CNN’s answer, pointed at the purple markings as the source of the false alarms.³ The authors explain that marked lesions may have been more common among the melanomas in the CNN’s training images, so it came to treat purple ink as a sign of cancer.³

The CNN had found a real pattern. It was simply a pattern about what doctors do to moles they already suspect, not about moles themselves.

Three ways to learn the wrong thing

The purple ink is a shortcut: an easy cue that happens to travel with the answer in the training data. It is the most vivid way a model can learn the wrong thing, but not the only one.

The second is leakage, when the answers slip into the exam. A model is supposed to be tested on cases it never saw while it was being built. If the same patient appears in both the training and the test data, or information from the test set quietly shapes how the model is designed, the test stops being a test. It is like grading students on an exam they saw the answers the night before. A survey of research that uses machine learning found leakage in 17 fields, affecting 294 papers and in some cases producing wildly overoptimistic conclusions.⁴ When the errors were corrected in one field the authors re-examined, complex models did no better than decades-old statistical ones.⁴

Biological data makes leakage especially easy to miss. Samples are often related in ways invisible in a spreadsheet (e.g., repeated tissue or imaging samples from the same person), and those hidden dependencies let information cross the training-testing boundary unnoticed.⁴^,⁵ The result is a performance estimate that looks far better than anything the model will achieve in real world use.⁵

The third is overfitting: memorizing rather than learning. A model with many adjustable settings and few examples can fit the quirks of its training data so closely that it describes those patients perfectly and new patients poorly. Think of a student who memorized last year’s answer key: perfect on that test, lost on a new one. The COVID-19 pandemic ran an unplanned test of this. In an early update of a living systematic review in the BMJ, 232 prediction models for COVID-19 were evaluated, and every single one was rated at high or unclear risk of bias, a formal rating of how much a study’s design could skew its results, with overfitting among the most common flaws; subsequent updates appraised over 700 models with similarly discouraging results.⁶ A separate review of 62 models that read chest X-rays and CT scans found none of potential clinical use.⁷

Right prediction, wrong lesson

Even a model that learns a genuine pattern, from honest data, can teach the wrong lesson. In a classic case, researchers trained models on records from about 14,000 pneumonia patients to predict who would die.⁸ One model learned a rule: patients with asthma had a lower risk of dying from pneumonia.⁸

The data were right. Asthmatic patients with pneumonia were usually admitted straight to intensive care, and that aggressive care lowered their risk.⁸ If the rule had been used to decide who could safely be treated at home, the patients it labeled low risk would have lost the very care that made them low risk.

Statisticians draw this line sharply. A model built to predict is not a model built to explain, and doing one well does not mean doing the other.⁹ Prediction answers “what is likely to happen?” It does not answer “why?” or “what happens if we act?”

Language models, the systems behind chatbots, including health chatbots, show a related gap. They learn the patterns of text, and their fluency is real. So is a particular failure. When researchers gave twelve models medical questions where none of the offered answers was correct, the models still answered confidently rather than admitting the gap.¹⁰

Researchers call these arbitrary, plausible-sounding errors confabulations. Some of them can be caught by asking a model the same question several times, and if its answers disagree in meaning, the answer is less likely to be trustworthy.¹¹

Passing the exam, failing the conversation

Health chatbots make a useful test case, because on paper their medical knowledge is impressive. Researchers wrote ten everyday medical scenarios, the kind where someone at home has to work out what might be wrong and whether to rest, see a doctor, or call an ambulance. When the full scenario was typed straight into the language models, they named a relevant condition in about 95 of every 100 cases.¹²

Then the researchers gave the same models to real people. In a randomized trial, 1,298 adults in the UK worked through the scenarios. Some used a chatbot; the rest used whatever they would normally turn to at home, usually a search engine or a health website. The people with a chatbot named a relevant condition in fewer than 35 of every 100 cases, which was worse than the comparison group. They were no better at deciding what to do next.¹²

The knowledge was there, but it got lost in the conversation. In about half of the conversations the researchers examined closely, people’s first message left out details that mattered. The chatbots, in turn, usually offered two or three possibilities, and only about one in three suggestions was correct. People then had to choose among them, and often chose wrong.¹²

Think of a driving test. Acing the written exam shows you know the rules; it doesn’t show you can merge onto a busy highway. The chatbots had, in effect, passed the written exam, and the standard medical knowledge tests used to evaluate them did not predict this gap.¹² The analogy has a limit: on the road the driver acts alone, while here the failure was shared between the person and the machine.

This is the distinction this series keeps returning to: performance as measured versus usefulness as claimed. A model can learn the patterns of medical text without learning how to help a worried person who doesn’t know which details matter.

What a high score can and cannot tell you

None of this means the skin-cancer CNN was useless. On unmarked images, its accuracy was real, and so was the earlier CNN’s performance against dermatologists.²˒³ The lesson is narrower and more useful:

Looking inside a model rarely settles that question. What settles it is the oldest tool in the laboratory, the controlled experiment. Change one thing, the ink, hold everything else fixed, and watch what happens. The Heidelberg team did exactly that, and the shortcut revealed itself in a single comparison.³

A few checks travel well beyond dermatology. Was the model tested in cases it never saw while it was being built, ideally from somewhere else? Does its accuracy look too good to be true? Reported scores for COVID-19 diagnostic models reached above 0.99, the kind of number that should prompt a second look rather than a celebration.⁶ Was it tested on how people use it, or only on an exam? ¹²

Open questions remain. How do you find a shortcut you have not thought of looking for? Who runs the purple-ink test before a tool reaches the clinic, and who pays for it? These are the questions laboratories ask of any new method before trusting its results, and AI deserves the same scrutiny.

So, when a model’s score impresses you, ask one more question.

What did it actually learn?

This article is for informational and educational purposes only and does not constitute medical advice. Scientific claims are grounded in the peer-reviewed literature cited; individual findings should be interpreted within the scope and limitations of those sources.

José C Bozelli Jr., PhD, is a scientist working at the intersection of biomarkers, omics, data science, and AI. He writes about health intelligence: how biological measurements become evidence, insight, and better decisions.

bozelli.ca

References

1. Geirhos R, et al. Shortcut learning in deep neural networks. Nat Mach Intell. 2020;2(11):665–73. DOI: 10.1038/s42256-020-00257-z

2. Haenssle HA, et al. Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists. Ann Oncol. 2018;29(8):1836–42. DOI: 10.1093/annonc/mdy166

3. Winkler JK, et al. Association between surgical skin markings in dermoscopic images and diagnostic performance of a deep learning convolutional neural network for melanoma recognition. JAMA Dermatol. 2019;155(10):1135–41. DOI: 10.1001/jamadermatol.2019.1735

4. Kapoor S, Narayanan A. Leakage and the reproducibility crisis in machine-learning-based science. Patterns. 2023;4(9):100804. DOI: 10.1016/j.patter.2023.100804

5. Bernett J, et al. Guiding questions to avoid data leakage in biological machine learning applications. Nat Methods. 2024;21(8):1444–53. DOI: 10.1038/s41592-024-02362-y

6. Wynants L, et al. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. BMJ. 2020;369:m1328. DOI: 10.1136/bmj.m1328

7. Roberts M, et al. Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans. Nat Mach Intell. 2021;3(3):199–217. DOI: 10.1038/s42256-021-00307-0

8. Caruana R, et al. Intelligible models for healthcare: predicting pneumonia risk and hospital 30-day readmission. In: Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; 2015:1721–30. DOI: 10.1145/2783258.2788613

9. Shmueli G. To explain or to predict? Stat Sci. 2010;25(3):289–310. DOI: 10.1214/10-STS330

10. Griot M, et al. Large language models lack essential metacognition for reliable medical reasoning. Nat Commun. 2025;16(1):642. DOI: 10.1038/s41467-024-55628-6

11. Farquhar S, et al. Detecting hallucinations in large language models using semantic entropy. Nature. 2024;630(8017):625–30. DOI: 10.1038/s41586-024-07421-0

12. Bean AM, et al. Reliability of LLMs as medical assistants for the general public: a randomized preregistered study. Nat Med. 2026;32(2):609–15. DOI: 10.1038/s41591-025-04074-y

Author & Editorial Note: This article was conceived and editorially directed by the author, who defined its argument, audience, voice, narrative structure, and scientific framing. AI tools, including large language model assistants, supported literature discovery and synthesis, citation checking, and generated draft prose within that direction. The author reviewed the evidence, edited the manuscript, and retains responsibility for the interpretation and final published text.