Two horror cases highlight the dangers of blind faith in what AI generates

In a previous RealKM Magazine article series, Dr Rachad Najjar identified 35 knowledge management (KM) processes that can be augmented by generative artificial intelligence (AI). Coinciding with a new article¹ in The Conversation reporting that generative AI has moved far enough along the hype cycle to where it can become genuinely useful, notable examples of these applications are starting to emerge. They include the skills development example from the recent KM4Dev Knowledge Café 37, Vanessa Liu’s expert knowledge example to be presented on 28 August, Stuart French’s personal assistant example to be presented on 4 September, and the expert identification example explored in recent research².

However, in a recent RealKM Magazine article, Adi Gaskell reports on research³ which supports the need for the KM community to proceed with caution in regard to generative AI. The research looks at risks associated with using generative AI chatbots for content generation. The authors delve into what they refer to as “botshit,” which they define as inaccurate or fabricated content produced by chatbots. They state that:

Our paper explains that when this jumble of truth and falsehood is used for work tasks, it can become botshit. For chatbots to be used reliably, it is important to recognize that their responses can best be thought of as provisional knowledge

The two cases below highlight the horrific outcomes that can result when organizations and leaders have such a level of blind faith in AI and technology that botshit-style outputs become entrenched as key inputs into what then becomes exceptionally bad decision-making.

Forthcoming RealKM Magazine articles will put forward approaches that can help the KM community to avoid such dreadful mistakes while still maximizing the potential for generative AI to support KM.

Case #1: Australian Robodebt AI scandal

The former Liberal-led Australian Government’s heinous ‘Robodebt’ scheme⁴ is a frightening example of what happens when AI outputs are treated as fact, rather than “provisional knowledge” as the research referenced above advises.

As Noel Cressie writes⁵ in The Conversation, the Robodebt AI system was designed to catch people exploiting welfare. It compared welfare recipients’ reported fortnightly income with their tax office-reported yearly income, the latter of which was averaged to provide fortnightly figures. If the difference showed an overpayment, a red flag was raised. The AI system then issued a debt notice and put the onus on the recipient to prove they weren’t exploiting the welfare system.

However, the AI system got many of the assessments seriously wrong to such an extent that it triggered numerous suicides⁶. But the flaws in the scheme were ignored, even when concerns were raised multiple times⁷. As Meredith Lewis reports in a 2019 RealKM Magazine series, a knowledge-based community protest campaign eventually saw what was deemed an illegal scheme shut down.

In 2022, the new incoming Labor-led Australian Government then launched a Royal Commission into the scheme. The Royal Commission findings have slammed the scheme and those responsible for it⁸. The Australian Government subsequently announced that it would implement all of the Commissioner’s recommendations⁹, including commencing investigations into 16 public servants, and putting in place clear review processes where decisions have been automated and an oversight body that can audit automated decisions.

Case #2: UK Post Office Horizon scandal

As summarized¹⁰ in BBC News and chronicled in Wikipedia¹¹, more than 900 sub-postmasters in the United Kingdom (UK) were prosecuted for stealing between 1999 and 2015. Many of the sub-postmasters went to prison for false accounting and theft, and many were financially ruined. The government-owned Post Office itself took many cases to court, prosecuting 700 people. Another 283 cases were brought by other bodies, including the Crown Prosecution Service (CPS).

However, many were wrongly prosecuted because of incorrect information from the Post Office Horizon computer system. Horizon was produced by Japanese information technology (IT) company Fujitsu in the 1990s. Fujitsu was aware that Horizon contained software bugs as early as 1999, but the Post Office insisted that Horizon was robust and failed to disclose knowledge of the faults in the system during criminal and civil cases. While it was an IT system rather than AI, the same blindness to problems or ‘incomplete curiosity’¹² could occur with AI.

When news of the faults finally broke publicly, it triggered legal action by sub-postmasters. The judge’s rulings led to sub-postmasters challenging their convictions in the courts and the government setting up an independent inquiry in 2020. The Post Office Horizon IT Inquiry was converted into a statutory public inquiry the following year. The public inquiry is ongoing and the Metropolitan Police are investigating personnel from the Post Office and Fujitsu.

Courts began to quash the sub-postmasters’ convictions in December 2020. By February 2024, 100 of the convictions had been overturned. Those wrongfully convicted became eligible for compensation, as did more than 2,750 sub-postmasters who had been affected by the scandal but not convicted. The final cost of compensation is expected to exceed £1 billion. In May 2024, the UK Parliament passed a law overturning the convictions of sub-postmasters in England, Wales and Northern Ireland. Scotland passed a similar law the same month.

As a result, the Post Office Horizon scandal has been called the UK’s most widespread miscarriage of justice.

Header image source: Created by Bruce Boyes with Perchance AI Photo Generator.

References: