Are you really doing enough to detect “botshit” in your AI-generated content?

This article is part of an ongoing series looking at artificial intelligence (AI) in relation to knowledge management (KM), and KM in relation to AI.

Research and practice in regard to the use of artificial intelligence (AI) in knowledge management (KM), and conversely also the use of KM in AI, continues to accelerate at a rapid rate. This can can be seen in the numerous articles in the long-running RealKM Magazine artificial intelligence (AI) series.

However, a 2024 article by Adi Gaskell reported on research¹ supporting the need for the KM community to proceed with caution in regard to AI. The research looks at risks associated with using generative AI chatbots for content generation. The authors delve into what they refer to as “botshit,’ which they define as inaccurate or fabricated content produced by chatbots. They state that:

Our paper explains that when this jumble of truth and falsehood is used for work tasks, it can become botshit. For chatbots to be used reliably, it is important to recognize that their responses can best be thought of as provisional knowledge

To highlight the dangers of using botshit in decision-making, I then wrote the article “Two horror cases highlight the dangers of blind faith in what AI generates,” reporting on the Australian Robodebt AI and UK Post Office Horizon scandals which have damaged or destroyed the lives of many people.

Reinforcing the findings of the botshit paper authors, notable KM pioneer Tom Davenport advises² that critical thinking to be able to decide if outputs are correct is a key capability needed in using these new technologies. However, as I have documented in a further troubling case, at least some in the KM community currently lack the key capability of critical thinking, not only in regard to AI outputs, but knowledge generally.

The botshit paper authors alert to the need for checking, thinking about, and questioning AI outputs if horrors such as those in the case studies linked above are to be avoided. Are you doing this for your AI outputs? Or are you just using AI without any consideration for its accuracy?

This article puts forward a further case study showing just how much checking, thinking about, and questioning is needed to detect botshit in AI outputs.

Checking, thinking about, and questioning

As I reported in a March 2023 RealKM Magazine “In the know” article, I had trialled the use of AI content summarizer WordTune Read in writing several articles in the “Open access to scholarly knowledge in the digital era” series and one article in the “Introduction to knowledge graphs” series. At the end of each article, I acknowledged the use of WordTune Read in preparing the article, consistent with the WAME Recommendations³ and similar guidance in other academic journals. Such acknowledgements are now mandatory for all RealKM Magazine articles where AI has been used in their preparation, as discussed in our information for authors.

As I noted in the March 2023 “In the know”, there had been significant mistakes in each of the summaries initially drafted by WordTune Read, caused by condensing, merging, or truncating sentences in a way that completely changed their meaning. So I had still needed to check and edit the summaries before publication.

An example is shown in Figure 1. At first glance, the Wordtune Read summary excerpt may appear to be accurate, and I expect that many users of Wordtune Read or other similar AI would just assume this accuracy and immediately proceed to use the AI-generated summary.

But closer checking, thinking about, and questioning reveals that the AI has made significant errors. Firstly, health sciences, human sciences, and applied social sciences DO NOT comprise 67 percent of the articles in SciELO Brazil journals. Rather, they comprise 67 percent of the articles in all of SciELO, with the Brazil journals aspect coming later in the source chapter but then inappropriately added to the earlier sentence. Secondly, it IS NOT agricultural topics that are prominent in Costa Rica and South Africa, but rather biological sciences.

These errors are not trivial, because in a complex and uncertain world, any such knowledge distortion can lead to very serious consequences. The corrected summary as it appears in the published article is shown in Figure 2.

A further example is shown in Figure 3. In this summary excerpt, the national systems of repositories developed by nine Latin American agencies ARE NOT known as La Referencia. Rather, as indicated by the adverb “also” in the source chapter, La Referencia is a distinct entity also started by the same agencies who had agreed to national systems of repositories, and is not the name for those systems of repositories.

An article⁴ republished from The Conversation advises that:

Learning from text data means systems such as ChatGPT are language models, not knowledge models. While it is truly amazing how much knowledge gets encoded via the mathematical training process, these models are not always reliable when asked knowledge questions.

Their real strength is working with language.

However, neglecting to consider the “also” in the example in Figure 3 means that at least some AI aren’t even good language models. The corrected summary as it appears in the published article is shown in Figure 4.

The effort that is really required, and cost-benefit considerations

As can be seen from the two examples above, the checking, thinking about, and questioning needed to detect botshit requires considerable effort, and if this effort is not applied, then false and potentially very destructive knowledge will be circulated.

So, as the botshit paper authors advise, ALL AI-generated content needs to be thought of as provisional knowledge until it has been rigorously checked and any errors corrected.

In the March 2023 “In the know” article, I had advised that, even with needing to do this checking and editing, WordTune Read was saving me time, and that this was very significant for a small non-profit with limited resources such as RealKM Cooperative. I also wrote that I intended to keep using WordTune Read going forward, and to investigate the use of other potentially suitable AI options.

However, the initial summaries from which I had drawn that conclusion proved to not be representative of the summaries that would come later, with a number of the remaining source chapters having a relatively complicated writing style meaning that Wordtune Read’s errors were more numerous and required a greater effort to detect. At this point, the time cost of using Wordtune Read outweighed the benefits, so I abandoned it and returned to doing manual summaries myself.

But this does not mean that I have abandoned AI completely. I have for some time been using AI to generate the header images for many RealKM Magazine articles, including this one, because this often involves a considerable time saving over searching for suitable images in copyright-free repositories. Into the future, I also still plan to further investigate other potentially suitable AI options for RealKM Magazine, but will only implement them if the benefits exceed the cost involved in ensuring their accuracy.

Header image source: Created by Bruce Boyes with Microsoft Designer Image Creator.

References: