Joy Buolamwini: The Researcher Who Exposed Bias in Face AI

Joy Buolamwini is a computer scientist who co-authored the 2018 Gender Shades study, which found that commercial gender-classification systems made their largest errors on darker-skinned women. She also founded the Algorithmic Justice League (AJL), a group whose stated goal is a world with more equitable and accountable technology. Her site credits her research with the 2020 retreats by IBM and Amazon on face recognition. Al Jazeera's coverage places Amazon's move alongside the 2020 protests, so the causal link is disputed and no source read settles it.

What Joy Buolamwini Found, and What She Built

Joy Buolamwini is the computer scientist whose Gender Shades study showed that all three commercial systems it tested made the most errors on darker-skinned women. She founded the Algorithmic Justice League, which works toward a world with more equitable and accountable technology. The paper tested Microsoft's Cognitive Services Face API, IBM's Watson Visual Recognition API and Face++.

For darker-skinned women, error rates ran from 20.8% for Microsoft to 34.7% for IBM. For lighter-skinned men, the error rate was 0.0% for Microsoft and 0.3% for IBM. The largest gap between the best and worst groups was 34.4%.

The authors argue that overall averages hide these gaps. Overall gender accuracy on their benchmark ranged from 87.9% to 93.7%. The paper says a company "might justify the market readiness of a classifier by presenting" figures like these.

Her own site lists a Bachelor of Science in computer science from the Georgia Institute of Technology, a master's in education from the University of Oxford, and a master's and doctoral degree in Media Arts and Sciences from the Massachusetts Institute of Technology (MIT). It also lists her as a Rhodes Scholar, a Fulbright Fellow and a Stamps President's Scholar. The Algorithmic Justice League's own about page lists her as Founder and President.

Her site says she serves on the Global Tech Panel, which advises world leaders and executives on reducing AI harms.

How the Gender Shades Study Worked, and Its Limits

The study ran in two parts. First, the authors audited two existing facial-analysis benchmarks, named IJB-A and Adience. They found both were overwhelmingly lighter-skinned, at 79.6% for IJB-A and 86.2% for Adience.

Second, they built the Pilot Parliaments Benchmark (PPB), with 1,270 unique individuals. It uses official profile photos of parliamentarians, balanced by gender and skin type. Lighter-skinned subjects make up 53.6% of PPB and darker-skinned subjects make up 46.4%.

Darker-skinned female subjects are 21.3% of PPB, compared with 4.4% of Adience. The paper names Iceland, Finland and Sweden as selected countries for lighter skin, and Senegal and South Africa for darker skin. The groups come from the Fitzpatrick Skin Type classification system, which the paper calls dermatologist approved.

The authors collapsed it into lighter and darker, then crossed it with gender. That produced four groups: darker female, darker male, lighter female and lighter male. The groups are defined by skin type, not race.

The paper's key figures, as printed, are these.

The authors name their own limits. They say PPB "is highly constrained" because it uses official profile photos of parliamentarians. Those photos come from cooperative subjects, with fixed pose, constant illumination and neutral or smiling expressions.

The authors write that "error rates would be higher on more challenging unconstrained datasets." Two more concessions appear in the paper. All the evaluated companies used "the binary sex labels of female and male." The authors say this "reductionist view of gender does not adequately capture the complexities of gender or address transgender identities." They also call the Fitzpatrick categories "fairly coarse" for the world's population, while defending them as "a scientifically based starting point." The study measured gender classification, not face recognition.

The paper says future work should explore "intersectional error analysis of facial detection, identification and verification." Using the 34.7% figure as a face recognition error rate would misstate the study.

Coded Bias, the Book, and Her Testimony

The publisher's page says Buolamwini coined the term "the coded gaze" for the bias she documented in commercial AI services. Her spoken-word visual audit, "AI, Ain't I A Woman?", shows AI failures on the faces of Serena Williams, Oprah Winfrey and Michelle Obama, according to her site. That work has been exhibited from Ars Electronica to the Barbican Centre in the United Kingdom.

The documentary Coded Bias, about her and the Algorithmic Justice League's work, premiered at the Sundance Film Festival in 2020 and went to Netflix. Her site calls it "critically-acclaimed" and says it "examines AI threats to civil rights and democracy." Her TED Featured Talk on algorithmic bias has more than 1.4 million views, according to the same site. Her book Unmasking AI: My Mission to Protect What Is Human in a World of Machines came out as an ebook and audiobook on October 31, 2023.

The paperback followed on November 19, 2024, at 336 pages, with International Standard Book Number (ISBN) 9780593241844. The publisher calls it a national bestseller and a Los Angeles Times Best Book of the Year, and says it was shortlisted for the Inc. Non-Obvious Book Award. On May 22, 2019, she testified before the U.S. House Committee on Oversight and Reform at the hearing "Facial Recognition Technology: Part I, Its Impact on Our Civil Rights and Liberties." She submitted her written statement as Founder of the Algorithmic Justice League.

One secondary source also names the House Science, Space, and Technology Committee in 2019, but the sources this page draws on don't confirm that appearance. In a National Public Radio (NPR) interview, she said there were second and third hearings on the topic.

What Changed Afterwards: IBM and Amazon in 2020

Several events came before the 2020 decisions. In 2019, a follow-up study by Inioluwa Deborah Raji and Buolamwini found that the August 2018 version of Amazon's Rekognition had a 31% error rate classifying the gender of darker-skinned women. For lighter-skinned men, the error rate was 0%.

In December 2019, the National Institute of Standards and Technology (NIST) published NISTIR 8280, "Face Recognition Vendor Test (FRVT) Part 3: Demographic Effects." It evaluated 189 algorithms from 99 developers on 18.27 million images of 8.49 million people. The images came from State Department, Department of Homeland Security (DHS) and Federal Bureau of Investigation (FBI) operational databases. NIST found that in one-to-one matching, Asian and African American faces produced false positives at "a factor of 10 to 100 times" the rate for Caucasian faces, depending on the algorithm.

In one-to-many matching, African American women had higher false positive rates, tested on an FBI database of 1.6 million domestic mugshots. Those findings point in the same direction as her work. On June 9, 2020, IBM said it would exit the facial recognition business, citing concerns about mass surveillance and racial profiling.

On June 10, Amazon announced a "one-year moratorium" on police use of its facial recognition software. Amazon said: "We hope this one-year moratorium might give Congress enough time to implement appropriate rules." The pause covered only police use and lasted one year. Amazon didn't explain why it acted at that moment.

Microsoft had not made that commitment. As of June 11, 2020, it had called for regulation to prevent human rights abuses, but it had not said it would stop selling to law enforcement. It did not respond to a request for comment.

Buolamwini said at the time: "Microsoft also needs to take a stand." Buolamwini called Amazon's announcement a "welcomed though unexpected announcement" and urged lawmakers to act. The first reading says her research drove the change. Her site says her MIT thesis methodology uncovered large racial and gender bias in AI services from Microsoft, IBM and Amazon.

It says that "in 2020 these companies stepped back from selling facial recognition to law enforcement." Wikipedia links the 2020 pauses to her organization's advocacy. Al Jazeera's contemporaneous reporting supports a narrower version, saying her findings "spurred Microsoft and IBM to improve their systems." The second reading says the protests drove the timing. Al Jazeera places Amazon's announcement among activists' concerns about unjust arrests during the protests over police brutality and the death of George Floyd.

The article says it "does not say Amazon cited the protests as a reason." It does not connect her research to Amazon's decision. No source the page draws on establishes causation either way. Whether her research, the protests, or both drove the pauses is unsettled.

A separate outcome came in February 2022. The Internal Revenue Service (IRS) agreed to halt its ID.me facial recognition program after a petition by the Algorithmic Justice League, filed with Fight for the Future and the Electronic Privacy Information Center (EPIC). The sources this page draws on don't establish whether Microsoft, IBM or Face++ closed the gaps the paper found.

Amazon claimed improvements in 2019, but no independent verification was found. The 2018 figures describe those products as they were tested in 2018.

The Methodological Criticism, Including Amazon's Response

Pushback came from Amazon. Matt Wood, then general manager of AI at Amazon Web Services (AWS), made three arguments. The table sets out each one with Buolamwini's reply, where the sources give one.

She used a Washington County Sheriff's Office admission that it does not follow Amazon's confidence-threshold recommendation to make the point that threshold advice doesn't describe real deployments. She also argued that a high confidence score can be "confidence in a falsehood." A March 26, 2019 response titled "On recent research auditing commercial facial analysis technology" was published on the MIT Media Lab site and credited to "Concerned Researchers." It said Amazon officials Matthew Wood and Michael Punke wrote "a series of blog posts attempting to refute the results of the study." The response said those posts misrepresented the study's technical details and the state of the art. Amazon had publicly attacked her methods the year before the 2020 pause.

It also disputed a past study showing Rekognition struggled to identify the gender of darker-skinned people. Her co-author, Timnit Gebru, was employed at Microsoft Research when the paper was written. Microsoft was one of the three companies audited.

The paper prints that affiliation on its first page. Her earnings aren't published in the sources this page draws on. Those sources don't show a salary, speaking fee or book advance, so this page doesn't estimate any of them.

The Algorithmic Justice League's about page doesn't name any corporate AI funders, and it routes donations through ActBlue. A critique also comes from outside the companies. Os Keyes's paper "The Misgendering Machines" argues that automated gender recognition operationalizes gender in a trans-exclusive way, whatever its accuracy.

On that argument, closing an accuracy gap leaves the underlying harm in place. Keyes has since argued for banning the technology rather than improving it. The text of the paper wasn't retrieved for this page, so the argument is summarized, not quoted.

The reform-versus-abolition split is a real disagreement, and this page doesn't settle it. For the ethics side, see facial recognition ethics, and for the wider debate, see whether AI is racist. Read the facial recognition laws guide next to see how the policy side has developed.

Sources

• Gender Shades paper, Proceedings of Machine Learning Research, 2018
• Joy Buolamwini's official site, About page
• Algorithmic Justice League, About
• Penguin Random House, Unmasking AI
• U.S. House Committee on Oversight and Reform, written testimony, May 22, 2019
• MIT Media Lab, "On recent research auditing commercial facial analysis technology"
• Joy Buolamwini's response to Amazon Rekognition, Medium
• Biometric Update, Buolamwini responds to Amazon
• NIST, NISTIR 8280 news release, December 2019
• Al Jazeera, Amazon pauses police use of facial recognition software, June 2020
• Wikipedia, Algorithmic Justice League
• Os Keyes, "The Misgendering Machines"