Bias and Fairness
AI is a mirror of the data it learned from, and if the mirror is warped, so is the reflection.
Lesson 2 said AI learns from examples, and "garbage in, garbage out." This lesson follows that idea to one of its most important consequences: when the examples reflect unfair patterns from the past, AI can repeat those patterns, quietly and at enormous scale, while looking objective.
This isn't a story about evil programmers. Most AI bias is accidental. That's exactly what makes it hard to see, and why it matters that ordinary people understand it. AI now helps decide who gets job interviews, loans, medical attention, and bail.
You'll see real cases, some where harm was done, some where it was caught and fixed, and one where the attempted fix went wrong. You'll also see that "fair" itself turns out to have more than one meaning, and that reasonable people disagree about which to use.
How to use this page
- There are five sections. Each takes about 9 to 12 minutes.
- Start anywhere. If you only read one, read Section 2 (real-world examples). The cases make everything else concrete.
- Press Focus at the top to show just one section at a time. Your checkmarks are saved on this device.
- Every section follows the same pattern: one-sentence summary → an analogy → the details → what to watch out for.
Focus mode is on: one section at a time. Press Focus again to see everything.
Bias can slip into AI at every stage (the history the data records, who's included in the data, how it's labeled, what the system is told to aim for, and how it's used), and usually no one intends it.
Analogy: the funhouse mirror
A funhouse mirror doesn't invent you; it reflects you. But its curves stretch some parts and shrink others. Stand in front of it long enough and you might start to believe the stretched reflection is how you really look.
AI reflects the world through the data it was given. If that data is curved (by history, by gaps, by the way it was collected), the reflection is curved too. And because it comes from a computer, people may trust the distorted picture more than they should.
What "bias" means here
In everyday speech, "bias" often means personal prejudice. In AI, it means something broader: a system that produces systematically worse or unfair results for some groups of people than others. The system doesn't need to "dislike" anyone. It only needs to have learned a pattern that works worse for some people.
Six ways bias gets in
| Source | What happens | Everyday example |
|---|---|---|
| 1. Historical bias | The data accurately records an unfair past, and AI learns to repeat it. | A hiring tool trained on a company's past hires learns to favor whoever it hired before. |
| 2. Representation bias | Some groups are missing or scarce in the data, so the AI works worse for them. | A skin-cancer tool trained mostly on light skin misses cancers on dark skin. |
| 3. Measurement bias | The thing measured is a poor stand-in for what we actually care about. | Using healthcare spending to estimate healthcare need (see Section 2). |
| 4. Labeling bias | The people who label training examples bring their own assumptions. | Labelers rating certain dialects or accents as "less professional." |
| 5. Design choices | What the builders tell the system to optimize can create unfair side effects. | An ad system told to minimize cost shows high-paying job ads to fewer women because reaching them costs more. |
| 6. Deployment bias | A tool built for one setting is used in another where it doesn't fit. | A medical model trained at a big-city hospital used at a rural clinic with different patients. |
The sneaky one: proxies
You might think the fix is simple: just don't tell the AI anyone's race, sex, or age. But AI is very good at finding proxies, other information that tracks closely with those traits:
- Zip code can stand in for race, because of the long history of housing segregation.
- Graduation year can stand in for age.
- Hobbies, first names, or college names can stand in for sex or ethnicity.
- Gaps in employment can stand in for caregiving, disability, or illness.
Removing a sensitive category doesn't remove its shadow. Sometimes it just makes the bias harder to detect.
Bias in chatbots and image generators
Generative AI learned from the internet, which reflects society's stereotypes. Studies have found image generators asked for "a CEO" or "a doctor" producing mostly white men, and "a nurse" or "a housekeeper" producing mostly women and people of color, often exaggerating the real-world imbalance. Chatbots can make similar assumptions in stories, examples, and advice. Companies work to correct this, and as Section 5 shows, the corrections can create problems of their own.
Watch out for
- "The data is just the data." Every dataset reflects choices: what to collect, from whom, and how to label it. Data is a record of the past, not a neutral truth.
- "We removed race, so it can't be biased." Proxies can carry the same pattern. Only testing results by group can show whether bias remains.
Documented cases show AI bias affecting who gets hired, who gets credit, who gets medical attention, and who is judged risky by the courts, and they also show that bias can be detected, argued over, and in some cases fixed.
Analogy: the well-worn path
In a field, people walk where others have walked before, and the path gets deeper and more obvious. Soon everyone uses it, not because it's the best route, but because it's the worn one.
AI trained on past decisions follows the worn path, and by making thousands of decisions a day, wears it deeper still.
Hiring
Amazon's recruiting experiment (reported 2018). Amazon built an AI tool to rate job applicants, trained on ten years of résumés sent to the company, mostly from men, reflecting the tech industry. The tool learned to downgrade résumés containing the word "women's" (as in "women's chess club captain") and graduates of two all-women's colleges. Engineers tried to fix it, couldn't be confident they had, and scrapped the project. It's a clear example of historical bias, and of a company catching it before wide use.
Mobley v. Workday (ongoing). A job seeker over 40 sued Workday, a company whose software screens job applications for many employers, alleging its AI tools unfairly rejected older applicants. In 2025 a federal judge allowed the case to proceed as a nationwide collective action for applicants 40 and over, and in 2026 rejected a key part of Workday's defense. The case could affect how the law treats companies that build hiring AI. Workday denies the claims; the outcome is not yet decided.
Lending
Mortgage approvals. A 2021 investigation by the news organization The Markup, analyzing federal mortgage data, found lenders were significantly more likely to deny loans to Black, Latino, Asian, and Native American applicants than to white applicants with similar financial profiles; Black applicants were about 80% more likely to be denied. The lending industry disputed the findings, noting that the public data lacked credit scores. The debate itself shows how hard it is to measure fairness, and why access to data matters.
The Apple Card (2019). After some couples reported that husbands received much higher credit limits than wives with similar or better finances, New York regulators investigated. In 2021 they found no unlawful discrimination, but criticized the lack of transparency: customers couldn't get a clear explanation of how their limits were set. The lesson: even when a system isn't found to be illegal, being unexplainable erodes trust.
Healthcare
The healthcare cost algorithm (2019). Researchers publishing in the journal Science examined an algorithm used by U.S. hospitals and insurers to decide which patients got extra care-management help, affecting many millions of people. It predicted future health costs as a stand-in for health needs. But because less money had historically been spent on Black patients with the same conditions (due to unequal access to care), the algorithm judged Black patients as healthier than equally sick white patients.
The researchers estimated that correcting the measure would raise the share of Black patients flagged for extra help from about 18% to 47%. They worked with the algorithm's maker to reduce the bias, a hopeful example of research leading to a fix.
A warning from outside AI: pulse oximeters. During the COVID-19 pandemic, a 2020 study found that these fingertip oxygen sensors missed dangerously low oxygen levels in Black patients about three times as often as in white patients, because the devices had been tested mostly on lighter skin. It isn't AI, but it's the same lesson: a tool is only as fair as the people it was tested on.
Criminal justice
Risk scores in court (2016). Some U.S. courts use software to estimate a defendant's risk of committing another crime, informing decisions about bail and sentencing. A 2016 ProPublica investigation of one widely used tool, COMPAS, found that among defendants who did not go on to reoffend, Black defendants were nearly twice as likely as white defendants to have been labeled higher risk (about 45% versus 23%).
The company responded that its scores were equally accurate for both groups: a given score meant about the same chance of reoffending regardless of race. Here's the remarkable part: both claims were true. Mathematicians then proved that when two groups have different underlying rates in the data, a tool usually can't satisfy both kinds of fairness at once. (Section 4 explains why this matters.)
Facial recognition arrests. As Lessons 4 and 7 described, several Americans have been wrongly arrested after facial recognition matches, including Robert Williams in Detroit (2020).
Government decisions
- The Netherlands (2013–2019): tax authorities used a risk-scoring system that, among other things, treated having dual nationality as a risk factor for childcare benefits fraud. Tens of thousands of families were wrongly accused and ordered to repay large sums, pushing many into financial ruin. The scandal led the entire Dutch government to resign in January 2021.
- United Kingdom (2020): when COVID-19 cancelled exams, an algorithm estimated students' grades, partly based on their schools' past results. Nearly 40% of teacher-predicted grades were lowered, disproportionately hurting high-achieving students at historically lower-performing schools. After public protests, the government scrapped the algorithm's results within days.
Watch out for
- Assuming these are all ancient history. Several of these cases led to reforms, but similar tools are in wider use than ever.
- Assuming every accusation of bias is proven. Some cases, like the mortgage study and the Workday lawsuit, are genuinely disputed. Look at the evidence on both sides.
AI works best for the people most represented in its data (typically younger, English-speaking, urban, wealthier, and online) and worse for everyone underrepresented, including older adults, rural communities, speakers of other languages and dialects, people with disabilities, and people with little digital footprint.
Analogy: the one-size-fits-all glove
A glove factory measures a thousand hands, but they happen to be mostly from young men. The "one size fits all" gloves fit them perfectly. Everyone else gets fingers that are too long, a thumb in the wrong place, a grip that slips.
Nobody at the factory meant to exclude anyone. They just measured the hands in front of them. Who you measure decides who it fits.
Faces: the Gender Shades study
In 2018, researchers Joy Buolamwini and Timnit Gebru tested commercial face-analysis systems from major tech companies on how well they identified gender. The results by group were striking:
Error rates for the worst-performing system tested. Source: Buolamwini & Gebru, "Gender Shades," 2018.
The datasets used to build these systems had been dominated by lighter-skinned faces. After the study was published, the companies updated their systems and error gaps shrank considerably, a strong example of how measuring a problem publicly can drive change.
Voices
A 2020 study of speech recognition systems from five major tech companies found they made roughly twice as many errors transcribing Black speakers as white speakers. Voice systems can also struggle with strong regional accents, non-native speakers, older voices, and people with speech differences from conditions like stroke or Parkinson's disease.
Groups often underrepresented
| Group | Why they're left out | What can go wrong |
|---|---|---|
| Older adults | Less represented in app usage data, online text, and many medical studies | Voice tools misunderstand; health tools calibrated for younger bodies; age-based hiring filters |
| Rural communities | Fewer sensors, less broadband, less data collected | Maps, health, and farm tools that fit cities better |
| Non-English speakers and dialect speakers | The internet is heavily English; many languages have little digital text | Weaker, costlier AI (Lesson 3, Section 2); dialects flagged as "errors" |
| People with disabilities | Rarely included when collecting training data | Tools that misread unusual movement, speech, or typing; hiring AI that penalizes gaps or nonstandard responses |
| People with thin credit or digital records | Millions of Americans have little or no credit history | Treated as risky simply for being unknown |
| Women (in some fields) | Historical research and testing often centered on men | Medical tools less accurate for women's symptoms and bodies |
A long-standing pattern, not just an AI problem
For decades, car crash-test dummies were modeled on an average man, and studies later found women faced higher injury risks in comparable crashes. Many early medical studies included mostly men. Author Caroline Criado Perez documented dozens of such gaps in her book Invisible Women. AI inherits these old gaps and can spread them faster.
The digital divide makes it worse
People who use technology less generate less data, so AI learns less about them, so it serves them worse, so they may use it even less. It's another feedback loop. That's one reason AI literacy efforts that reach older adults, rural communities, and people new to technology matter so much: their participation and feedback help shape better tools.
Ask this
When an organization tells you an AI system is "accurate," ask: "Accurate for whom? Was it tested on people like me?" Good developers can answer with results broken down by age, sex, race, language, and disability. If they can't, that's worth knowing.
Watch out for
- Average accuracy hiding group gaps. A system that's "95% accurate" overall can still fail badly for a small group, because that group barely affects the average.
- Blaming yourself. If a voice assistant never understands you, or a face scanner never recognizes you, the problem may be the tool's training, not you. Say so; feedback helps.
Every AI decision rests on human choices (what data to use, what goal to set, which kind of fairness to prioritize, where to draw the line), so "the computer decided" really means "people decided, and then automated it."
Analogy: the referee and the rulebook
A referee can call a game perfectly by the rules. But if the rulebook itself favors one team (say, the basket is lower on one side), perfect refereeing just applies the unfairness perfectly.
A computer can follow its "rules" with flawless consistency. Consistency isn't the same as fairness. The question is always: who wrote the rulebook, and what did they put in it?
The human choices behind every "automated" decision
- What problem to solve. Deciding to use AI to flag welfare fraud, rather than to find people eligible for benefits they're missing, is itself a choice about priorities.
- What data to use, and whose data counts.
- What to predict. Health costs or health needs? Arrests or actual crimes? These stand-ins matter enormously (Section 2).
- Where to set the cutoff. A risk score of 6 or 7 might be the line for denial. Moving it changes who is affected.
- Which errors to accept. Is it worse to wrongly deny a loan to someone who'd repay, or wrongly approve someone who won't? Every system trades one kind of error for another.
- How much human review there is, and whether humans can actually overrule the system.
Fairness has more than one definition
Remember the COMPAS case, where both sides were right? That's because they were using different definitions of fairness:
"Equal accuracy" fairness
A given score should mean the same thing for everyone. If a score of 7 means a 60% chance of reoffending, that should be true for every group.
This is what the company emphasized.
"Equal error" fairness
People who won't reoffend should be equally likely to be wrongly flagged, whatever group they belong to.
This is what ProPublica emphasized.
When groups differ in the underlying data, it's usually mathematically impossible to satisfy both at once. So someone must choose which kind of fairness matters more in that situation. That choice is a question of values, not of math, and it belongs in public debate, not only in engineering meetings.
The feedback loop
Automation bias: trusting the machine too much
People tend to defer to computer recommendations, even when their own judgment says otherwise. This is called automation bias. A loan officer, doctor, or judge who sees "high risk" on a screen may accept it without question, especially when they're busy. "A human reviews every decision" means little if the human rubber-stamps the machine.
Scale changes the stakes
One biased hiring manager affects dozens of applicants. One biased hiring algorithm, used by hundreds of companies, can affect millions, often invisibly, since rejected applicants rarely learn why. And because the decisions look scientific, they can be harder to challenge.
But humans are biased too
This is an important and fair point. Research has documented bias in human hiring, lending, medicine, and sentencing for decades. Some argue that well-designed, carefully tested AI could be fairer than people: more consistent, easier to audit, and possible to correct in ways that human hunches aren't. Both things can be true: AI can reduce some human biases while introducing or amplifying others. The goal isn't "AI versus humans," but systems of both, with real checks.
Ask this: when an automated system decides about you
- "Was an automated system used in this decision?"
- "What were the main factors?"
- "Can a person review it?"
- "How do I appeal or correct wrong information?"
Under U.S. law, if you're denied credit, lenders must tell you the main reasons, even when an algorithm was involved. Several states and the EU give additional rights to explanation or human review for important automated decisions.
Watch out for
- "It's just math." The math carries out choices made by people. Ask who made them.
- "There's a human in the loop." Ask whether that human has the time, information, and authority to disagree with the machine.
Bias can be reduced (through better data, testing results by group, independent audits, transparency, meaningful human oversight, and laws requiring accountability), but it can't be eliminated once and forgotten, and fixes themselves involve judgment calls that can go wrong.
Analogy: tuning a piano
A piano doesn't stay in tune forever. Temperature, humidity, and use slowly pull the strings off. A good piano tuner doesn't tune once and walk away; they come back regularly, listen carefully to every key, and adjust.
Keeping AI fair works the same way: test, adjust, and keep testing, because the world, the data, and the people using the system keep changing.
Technical approaches
- Better, broader data: deliberately collecting examples from underrepresented groups, as face-recognition companies did after the Gender Shades study.
- Testing by group: breaking results down by age, sex, race, language, and disability, not just reporting one overall accuracy number.
- Choosing better targets: predicting what actually matters (health need, not spending), as in the healthcare algorithm fix.
- Hunting for proxies: checking whether harmless-looking inputs are standing in for protected traits.
- Fairness adjustments: techniques that rebalance training data or adjust outputs to reduce unfair gaps.
- Red-teaming: deliberately trying to provoke biased or stereotyped outputs so they can be fixed before release.
Process and transparency
- Independent audits by outside experts, not just the company that built the tool.
- Documentation: "model cards" and "datasheets" that describe what a system was trained on, what it's for, how it was tested, and where it performs poorly.
- Diverse teams who are more likely to notice blind spots.
- Involving affected communities in design and testing, not just in complaints afterward.
- Meaningful human oversight: reviewers with the time, training, and authority to overrule the system.
- Appeals: clear ways for people to challenge decisions and correct errors.
When the fix goes wrong: a cautionary tale
In early 2024, Google's Gemini image generator, adjusted to show more diversity, produced historically inaccurate images, such as racially diverse depictions of 1940s German soldiers and America's Founding Fathers. After widespread criticism, Google paused its generation of images of people and acknowledged the tool had "missed the mark." The episode showed that crude corrections can create new distortions, and that "fair" doesn't mean the same thing in every context. Accuracy about history and fairness in hiring are different goals.
Laws and rules
- Existing anti-discrimination laws apply. U.S. federal agencies have stated that laws against discrimination in employment, lending, and housing apply whether a decision is made by a person or by an algorithm.
- New York City (since 2023) requires employers using automated hiring tools to have them audited for bias each year and to notify candidates.
- Colorado passed one of the first broad state AI anti-discrimination laws in 2024. After delays, legal challenges, and debate, it was replaced in 2026 with a narrower law focused on transparency: telling people when automated systems make important decisions about them, explaining bad outcomes, and offering human review. It's scheduled to take effect in 2027.
- The EU AI Act classifies AI used in hiring, credit, education, and law enforcement as "high-risk," requiring testing, documentation, and human oversight.
The ongoing debate
How to reduce bias, and even what counts as bias, is contested:
Some argue for more active correction
Past discrimination is baked into data, so simply "following the data" repeats injustice.
Strong rules and required audits are needed, because companies won't prioritize fairness on their own.
Others urge caution about correction
Adjusting AI outputs can introduce the builders' own values and slants, and fairness goals can conflict with accuracy.
Heavy regulation may slow useful tools, and rules differ state by state. In the U.S., the federal government has pushed back on some state AI laws and on what it calls ideological adjustments in AI.
Most people who study this agree on a core: test systems honestly, show the results, and give people a way to challenge decisions. The disagreements are mostly about how far to go beyond that, and who decides.
Do this: what you can do
- Ask questions when an automated system affects you (Section 4).
- Report problems. Use feedback buttons when AI gets something wrong about you or produces stereotypes. Companies do track these.
- File complaints about possible discrimination: employment with the EEOC, lending with the Consumer Financial Protection Bureau, housing with HUD, and your state attorney general.
- Speak up locally when schools, city governments, or employers consider AI tools. Ask how they were tested, and on whom.
Watch out for
- "Our AI is bias-free." No honest developer claims that. Ask for testing results by group instead.
- Fairness as a one-time checkbox. A tool audited once, years ago, may not be fair today.
Putting it together
Five ideas to take with you:
- Bias enters at every stage: history, gaps, labels, goals, and use. It's usually accidental, and removing a sensitive category doesn't remove its proxies.
- Real cases span hiring, lending, healthcare, justice, and government, and some were caught and fixed, which shows testing and scrutiny work.
- Who you measure decides who it fits. Older adults, rural communities, other languages, people with disabilities, and people with thin records are often left out.
- "The computer decided" means people decided, then automated it. Fairness has several definitions that can conflict, and choosing among them is a question of values.
- Bias can be reduced but never finished: better data, testing by group, audits, transparency, real human oversight, appeals, and laws, applied again and again.
Check yourself
Try to answer each question in your head, then tap to see a sample answer.
1. What does "bias" mean in AI?
A system that produces systematically worse or unfair results for some groups of people than others, whether or not anyone intended it.
2. What is a proxy, and why does it matter?
Information that stands in for a protected trait, like zip code for race or graduation year for age. Removing the trait itself doesn't stop the AI from using its proxies.
3. Why did Amazon's recruiting tool penalize women?
It learned from ten years of mostly male résumés, so it treated signs of being a woman, like "women's chess club," as negatives.
4. What went wrong with the healthcare cost algorithm?
It used past spending as a stand-in for health need. Less had historically been spent on Black patients, so they were judged healthier than equally sick white patients.
5. In the COMPAS debate, how could both sides be right?
They used different definitions of fairness (equal accuracy of scores vs. equal rates of wrongly flagging people). When groups differ in the data, both usually can't be satisfied at once.
6. What is automation bias?
The human tendency to defer to computer recommendations, even over one's own judgment.
7. What did the Gemini image episode show?
That crude attempts to correct bias can create new distortions, and that fairness means different things in different contexts.
8. Name three ways to reduce AI bias.
Any three of: broader data, testing results by group, better prediction targets, checking for proxies, independent audits, documentation, diverse teams, community involvement, real human oversight, appeals, and laws.
Glossary
- Algorithmic bias
- Systematically unfair outcomes for some groups produced by an automated system.
- Audit (AI)
- An examination of an AI system's performance and fairness, ideally by an independent party.
- Automation bias
- The tendency to trust a computer's recommendation over one's own judgment.
- Datasheet / model card
- Documentation describing a dataset or AI model: its sources, purpose, testing, and limits.
- Feedback loop
- When an AI's decisions create the data it later learns from, reinforcing its patterns.
- Fairness metric
- A mathematical definition of fairness used to test a system. Different metrics can conflict.
- Historical bias
- Unfairness recorded in accurate data about an unfair past.
- Measurement bias
- Bias from measuring a poor stand-in for what actually matters.
- Protected characteristic
- A trait, such as race, sex, age (40+ in U.S. employment law), religion, or disability, that anti-discrimination law protects.
- Proxy
- Data that closely tracks a protected characteristic.
- Red-teaming
- Deliberately trying to make a system fail or misbehave in order to fix it.
- Representation bias
- Bias from some groups being missing or scarce in the data.