AI Bias: Consequences and Mitigation

What has become known as the ‘wisdom of the crowd’ phenomenon suggests that the combined judgments of a diverse group, drawing from varying levels of knowledge and experience, could outperform even the most knowledgeable individual experts. Over the century since, extensive research has validated this principle across many fields, from business to science.

Today, as artificial intelligence becomes increasingly influential in our lives, Galton's insights take on new significance. The capabilities of advanced AI systems depend heavily on the data used to train them. Just as the fair's crowd harnessed collective human knowledge, training AI requires drawing from a wide range of human perspectives and experiences.

The path to unbiased, equitable AI lies in embracing the wisdom of the crowd, recognizing that diverse input is essential to unlocking the full potential of these transformative technologies.

The Consequences of AI Bias

Much like the human mind, AI systems are influenced by the information they consume and as their use in informing decision making processes becomes increasingly routine, it is crucial that any harbored biases are identified and addressed. If left unaddressed, the content, predictions, recommendations, and classifications generated by biased models can have serious consequences.

In Healthcare

As of March of this year, the FDA has approved 1,016 medical devices with AI integration. However, recent studies exemplify how skewed training data results in suboptimal care for certain medical patient populations.

In a 2019 study, researchers discovered that Optum, an algorithm used to predict which patients would benefit from additional medical care, exhibited racial bias. Only 18% of patients flagged to receive extra care were black, compared to about 82% of white patients. This discrepancy failed to account for an additional 48,772 chronic disease cases suffered by the black community. The bias has been attributed to the use of financial data to determine the level of care, a flawed approach as minority groups within the U.S. have historically had less access to healthcare and experience higher rates of mortality.

In Employment

More than 32% of consumers believe AI bias has negatively impacted employment opportunities and their opinion is not unwarranted.

In 2014, Amazon tested a recruitment algorithm that was biased based on applicant gender, according to a Reuters report. The AI system, trained on 10 years of submitted resumes, mostly from male applicants, developed a preference for male candidates. It even penalized resumes containing the word "women." Although 500 models were used to recognize roughly 50,000 key terms within past candidate resumes, its emphasis on gender caused Amazon to abandon the hiring tool completely.

In Prison Sentencing

Law enforcement agencies across the country are utilizing AI to strategically dispatch officers in hopes of decreasing response times to criminal activity. However, research organizations argue that these practices target low-income and minority communities disproportionately.

Facial recognition software, used to identify criminals, has been shown to be unreliable on numerous occasions, particularly when used against women of color with an error rate of more than 34%. A 2020 study conducted by the ACLU states that this inaccuracy compounds upon itself as black people are more likely to be arrested for minor crimes, creating a larger data set that can mismatch identities.

Additionally, risk assessment algorithms that generate a recidivism score, have been criticized for finding correlations rather than causations in the historical crime data they are trained on. A ProRepublica article details their comparison of the recidivism score calculated by Northpointe's COMPAS tool to the actual recidivism rate of over 7,000 criminal defendants over a two-year period. The analysis found that the algorithm only had a 20% success rate in predicting those who would commit violent crimes within the time period. Furthermore, black defendants were nearly twice as likely to be flagged as future criminals while white defendants were more often rated as low risk overall. Even after isolating race as a factor, black defendants were still 77% more likely to be rated at a higher score for committing a future violent crime and 45% more likely to commit a crime of any nature.

In International Affairs

In the current divisive political climate, both parties had similar views over AI's influence on the 2024 U.S. election, with 56% of Republicans and 58% of Democrats expressing high concern.

A recent report published by the American Sunlight Project unveils the vast extent of Russian efforts to invoke civil unrest worldwide. A large-scale investigation across 182 domains and subdomains has classified them as the nodes of the "Pravda" network, which is duplicating content at an alarming rate. It is believed that the network is being used to poison LLMs with pro-Russian content and execute a mass disinformation campaign, generating an average of 3.6 million articles per year.

This finding coincides with an audit of the ten most popular chatbots, performed by NewsGuard, which found that the LLMs collectively repeated Russian disinformation narratives 31.75% of the time.

Although Pravda's initial target was West and West-aligned countries, researchers have monitored expansions into other areas of the world, especially those that underwent geopolitical shifts in order to control the narrative of events.

Mitigating AI Bias

Mitigating AI bias requires a multifaceted approach that addresses both technical and societal challenges. The risk of creating negative feedback loops, where AI systems amplify existing biases, necessitates a proactive approach to ensure fairness and equity. This includes:

• Diverse and Representative Data: AI systems must be trained on diverse datasets that reflect the full spectrum of human experiences and perspectives. This includes actively seeking out and incorporating underrepresented viewpoints to counterbalance historical biases.

- Transparency and Accountability: The development and deployment of AI systems must be transparent and accountable. This includes clear documentation of data sources, algorithmic processes, and potential biases, as well as mechanisms for external audits and public scrutiny.

- Collaboration and Ethical Guidelines: Addressing AI bias requires collaboration among stakeholders, including policymakers, researchers, industry leaders, and civil society. This includes developing and enforcing ethical guidelines and standards that prioritize fairness, accountability, and human rights in AI development and deployment.

- Ongoing Monitoring and Evaluation: AI systems must be continuously monitored and evaluated for bias and discriminatory outcomes to stop its progression in its tracks. This includes implementing feedback mechanisms and corrective measures to address identified biases and ensure ongoing fairness and equity.

By embracing these principles and actively working to mitigate AI bias, we can harness the transformative potential of AI while upholding democratic values and social justice. The path forward requires a collective commitment to responsible AI development and deployment that prioritizes human well-being and societal equity.

How HackerOne Combats AI Bias

The very nature of HackerOne's crowdsourced security model embodies the 'wisdom of the crowd' due to our diverse community of ethical hackers.

In 2021, as part of DEF CON 29's AI Village, HackerOne partnered with X (formerly known as Twitter) to launch the first algorithmic bias bounty challenge within the platform's image-cropping algorithm. Through this collaborative effort, numerous biases were brought to the attention of X:

• The winning submission demonstrated that the model showed a bias towards subjects that were slimmer, younger, feminine, and lighter in skin tone.
• The second place submission discovered that the algorithm was also biased towards disabled individuals, those that had white hair, and possessed traits of albinism.
• The third place submission found that the model exhibited bias in overlaid text, preferring English to Arabic.

One submission even revealed that the cropping algorithm preferred lighter skinned emojis.

Since then, HackerOne has continued to define the standard for quality AI security and safety evaluations, developing the industry’s leading AI Red Teaming methodology. Grounded in the principles of the OWASP Top 10 for LLM Applications Project, MITRE ATLAS Framework, and the NIST AI Risk Management Framework, our AI services take a proactive approach to ensure flaws in AI systems are identified and addressed early.

Additionally, our testing processes adhere to the EU AI Act, ISED Voluntary Code of Conduct on the Responsible Development and Management of Advanced Generative AI Systems, US state laws, and will continuously adapt to future regulatory requirements in this new environment. With the leading AI security specialists at our disposal, HackerOne is in a unique position to act as the bridge between ethical hackers, developers, and policy makers alike.

Recently, HackerOne supported Snap Inc. in their effort to stress-test their Lens and My AI Text2Image generative AI products. Snap selected 21 researchers from around the globe to participate in their red team exercise, which provided valuable insights into their models' behavior in regard to harmful imagery generation.

The Path Forward

Mitigating bias in AI systems will require consistent maintenance and a strategic approach. Not only is it a newfound technical issue, but it is also an issue of human nature. One that we have yet to solve ourselves.

We are at risk of creating negative feedback loops in which human originating biases are introduced to AI, which then further solidifies them in the public realm of thought. Although AI has far-reaching potential to better the lives of everyone, they can also amplify prejudices or societal inequalities if we are not careful.

The historical data that AI absorbs in order to learn can be corrupted with outdated ideas and beliefs. Because of this, a commitment to providing a diverse pool of data is required, a collection that reflects both commonly held and alternative viewpoints.

In the modern gold rush that is the AI landscape, it is also important that development is transparent.  A permanent solution may never exist and will require everyone across the world to keep the problem at bay. Meaning collaboration between stakeholders, policy makers, researchers, industry leaders, as well as average end users is vital to establish ethical guidelines and standards to ensure AI systems do not spiral out of control.