Bias in Medical AI: Where It Comes From and Why It Matters

Bias in medical AI is a concrete problem: it means algorithms that return systematically wrong answers for some groups of patients more often than others. Understanding where it comes from means taking a step back first, to see exactly what bias is and how it differs from two concepts it often gets confused with: error and noise.

Error, Noise and Bias: What's the Difference

Three concepts get confused often when people talk about AI going wrong: error, noise and bias. Picture a round of darts thrown at a target, where the goal is to hit the bullseye. Each dart stands for a decision, a medical diagnosis, say, or a patient's risk assessment.

Error is simply missing the bullseye. If the darts land far from it, the decision doesn't match the correct one. On its own, error says nothing about its cause. It only tells us the result is wrong.

Noise in Medicine

Noise, a concept developed by Kahneman, Sibony and Sunstein (2021), describes the scatter of decisions rather than their accuracy. Picture the darts landing at random around the target: some to the right, some to the left, some high, some low, with no consistent pattern. That randomness means the same situation produces a different decision every time.

In medicine, noise shows up when two doctors reach different diagnoses looking at the same clinical case, or when the same doctor reaches different conclusions on different occasions. The problem here isn't a systematic direction of error. It's the inconsistency of the decision-making process itself.

How Bias Produces Inequality

Bias is a distinct kind of error. Now picture the darts landing close together, but as a group shifted away from the bullseye. The errors aren't random anymore: something about the way the aim is set pushes every throw in the same direction. The error becomes predictable and structural, a systematic deviation in the decision-making process that produces persistent inequality rather than a string of independent mistakes.

Bias and noise can coexist. A decision-making system can carry a lot of noise and little bias, a lot of bias and little noise, or both at once: the darts can be scattered and, as a group, shifted off-centre. This distinction explains why bias is particularly serious in AI systems. It doesn't just produce errors. It produces systematic errors that hit the same groups of people again and again.

Bias and AI: Four Stages, Four Different Problems

There's no such thing as a single, generic “AI bias” in medicine, a simplification that's easy to reach for but misses the point. Distortions can emerge at different stages in an AI system's life cycle: in the data used to train the model, in the labels that teach it which answer is correct, in the design choices developers make, or in the way clinicians actually use the system.

Data Bias: When Data Doesn't Represent Every Patient

The first type is data bias. Machine learning systems learn by recognising patterns in data. When some patient groups are underrepresented, the model learns the traits of larger groups with more precision and those of underrepresented groups with less.

Cardiology offers a real example. In 2024, Straw and colleagues analysed a large set of algorithms built to predict cardiovascular disease. Women, they found, were systematically underrepresented in the training datasets, and many studies didn't even report performance broken down by sex.

When the algorithms were tested separately on men and women, the result was troubling: in 13 of the 16 models analysed, the rate of false negatives was significantly higher for women. In practice, the algorithms were more likely to miss heart disease in female patients, mirroring an imbalance already present in the training data (Straw et al., 2024). It's the same problem that shaped the S-RACE platform at UniSR, designed to catch clinical data quality issues before a model is ever trained.

Label Bias: The Hidden Error in Diagnostic Labels

A second type of distortion is label bias. In supervised models, the AI doesn't learn the presence of a disease directly. It learns the labels humans provide: a diagnosis, a test result, a treatment decision. If those labels already carry bias, the model inherits it.

Chang and colleagues described a phenomenon they call disparate censorship. Many algorithms are trained using diagnostic test results as their reference point, but not every patient gets tested with the same frequency. When a group is tested less often, cases of disease go unconfirmed and end up recorded as negative. The algorithm doesn't learn the real distribution of the disease; it learns the distribution of doctors' decisions about who gets tested, carrying forward whatever disparities already exist in clinical practice (Chang et al., 2022).

Algorithmic Bias: Distortion Built Into Design Choices

A third category covers algorithmic bias. Here the data can be accurate and the labels reliable, yet the distortion still emerges, this time from choices made during model design. The best-known case is the one described by Obermeyer and colleagues in 2019. The algorithm was used by numerous US hospitals to identify patients for intensive care management programmes.

The goal was to find patients with the greatest healthcare needs, but developers had chosen future healthcare costs as the variable to predict, assuming that higher spending tracked with worse clinical condition. That assumption turned out to be wrong.

Because of unequal access to care, Black patients generated lower average healthcare costs than white patients with the same level of illness. The algorithm concluded that Black patients had lower care needs when, in fact, they were on average sicker. Simply replacing the target variable with a more direct measure of health status sharply reduced the bias (Obermeyer et al., 2019).

Implementation Bias: The Error Born in Clinical Use

Implementation bias, finally, covers the distortions that surface once an algorithm reaches clinical practice. Even an accurate system can produce unwanted effects if clinicians change their behaviour in response to its recommendations. Automation bias is one example: the tendency to place excessive trust in an AI system's suggestions.

Gaube and colleagues studied this by asking general practitioners to evaluate a series of clinical cases with AI support. When the system proposed an incorrect diagnosis, many doctors abandoned a diagnosis they had originally gotten right, deferring to the algorithm instead.

The AI wasn't producing the error directly; it was leading the clinician into making it. The study shows that the way a system gets built into the decision-making process can itself become a source of bias (Gaube et al., 2021). It's also why, in UniSR's own predictive AI projects, including the model that estimates heart attack risk from cardiac imaging, responsibility for the clinical decision always stays with the physician.

Four Types of Bias, Four Different Fixes

These examples show that a generic label like “AI bias in medicine” hides more than it reveals. Each type of bias emerges at a different stage of a system's life cycle and calls for a different fix: more representative data, more reliable labels, better-designed models, or ways of using AI that preserve the clinician's critical judgement.

Bibliography

Chang, T., Sjoding, M. W., & Wiens, J. (2022, December). Disparate censorship & undertesting: A source of label bias in clinical machine learning. In Machine Learning for Healthcare Conference (pp. 343-390). PMLR.

Gaube, S., Suresh, H., Raue, M., Merritt, A., Berkowitz, S. J., Lermer, E., ... & Ghassemi, M. (2021). Do as AI say: susceptibility in deployment of clinical decision-aids. NPJ digital medicine, 4(1), 31.

Kahneman, D., Sibony, O., & Sunstein, C. R. (2021). Noise: A flaw in human judgment. Hachette UK.

Motterlini, M., & Crupi, V. (2026). Decisioni mediche: Un punto di vista cognitivo (Nuova ed.). Milano: Raffaello Cortina Editore.

Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447-453.

Straw, I., Rees, G., & Nachev, P. (2024). Sex-based performance disparities in machine learning algorithms for cardiac disease prediction: exploratory study. Journal of Medical Internet Research, 26, e46936.