AI copies its examples
A model learns from data that people produced, and people's past decisions are not always fair. If the examples carry unfairness, the model learns it and repeats it, quickly and at scale. This is bias.
Real kinds of harm
- Hiring: a model trained on a company's past hires, who were mostly men, may score women's CVs lower.
- Faces: face recognition trained mostly on lighter-skinned faces has been shown to make more mistakes with darker-skinned faces.
- Loans: a model may learn that people from certain areas were refused in the past, and keep refusing them, whatever their real ability to pay.
- Language: a model trained mostly on English may serve speakers of Swahili, Yoruba or Tagalog worse.
Why it is easy to miss
Nobody typed "be unfair". The bias hides inside millions of examples. And because a computer gave the answer, people tend to trust it more than they should.
Reducing bias
- Better data: include people of different genders, ages, regions, skin tones and languages.
- Test by group: check accuracy separately for each group, not only overall. A model that is 95% right overall may be 99% right for one group and 80% for another.
- A person decides: for jobs, loans, school places and health, AI can help, but a person should make and explain the final decision.
- A way to appeal: people affected should be able to ask for a human review.
Guided practice
A face-unlock feature is right 98% of the time for group A and 85% for group B. The company reports "92% accurate". What's the problem?
The single overall number hides that group B is failed much more often. The model should be tested and improved for each group, with better data for group B.