Disgraceful AI acts are growing. Will the knowledge management community be next?

This article is part of an ongoing series looking at AI in KM, and KM in AI.

As Dr Arthur Shelley wrote¹ in an inspiring RealKM Magazine article last week, artificial intelligence (AI) can make highly beneficial knowledge initiatives possible, and there are further examples in our long-running ongoing series on AI in KM, and KM in AI.

However, this potential is being seriously hampered by a continuing saga of disgraceful AI acts. I’ve reported previously² on such acts, yet, as I document below, not only do they keep happening, but they are arguably getting worse! This is despite the development of not only abundant guidance in regard to ethical and responsible AI, but also media reporting of more and more cases of disgraceful AI acts. As a comment in response to one of these new acts very appropriately laments³, “How much of this kind of thing does one need to see? Honestly, people need to start paying attention!”

Apparently, people need to see a lot more of it, including in the knowledge management (KM) community, where in the past few days yet more content with unchecked hallucinations has been circulated.

Case 1. So-called “definitive proof” retracted

Shortly after the release of ChatGPT in 2022, I found myself under attack from an international education sector leader. He had made the accusation on LinkedIn that education authorities in an Australian state were banning the use of AI in schools because they wanted to protect human jobs, and not because of their stated concerns in regard to potential risks. The ban was intended to be temporary, to allow time for a thorough evaluation of both the possible applications and potential risks of AI. This caution seemed a wise approach to me, but when I said that in a comment on the LinkedIn post, I was made out to be the worst in the world.

This would be my very first, but far from the very last, experience of AI overhype and the loud moral certainty⁴ that is accompanying this overhype. Anyone who dares to question the assumptions underpinning AI research or deployment is very quickly branded a heretic who is trying to prevent progress.

However, a notable example of disgraceful AI research exposed last month goes to the heart of this AI in education issue. On 22 April, the academic journal Humanities and Social Sciences Communications, part of the prestigious Springer Nature group, issued a Retraction Note⁵ for the paper “The effect of ChatGPT on students’ learning performance, learning perception, and higher-order thinking: insights from a meta-analysis” which had been written by Jin Wang and Wenxiang Fan from Hangzhou Normal University. The Retraction Note states that:

The Editor has decided to retract this paper owing to concerns regarding discrepancies in the meta-analysis. These issues ultimately undermine the confidence the Editor can place in the validity of the analysis and resulting conclusions. The authors have not responded to correspondence regarding this retraction.

As the following article⁶ reports, Wang & Fan’s paper has been seen and promoted as definitive proof of the value of ChatGPT in education, being cited by other researchers more than 500 times and widely circulated on social media.

Those more than 500 citations include at least 20 papers discussing knowledge management (KM) or aspects of KM such as knowledge acquisition, knowledge application, knowledge silos, and knowledge integration. These papers include:

• “Harnessing generative AI for next-gen education: a cognitive load and knowledge-based view approach to fuel innovation,” published⁷ in the Journal of Knowledge Management, which references Wang & Fan’s paper three times.
• “System 0: Transforming Artificial Intelligence into a Cognitive Extension,” published⁸ in Cyberpsychology, Behavior, and Social Networking, which references Wang & Fan’s paper 10 times.
• “Students’ perspectives on the educational benefits of ChatGPT: a quantitative exploration,” published⁹ in Innoeduca. International Journal of Technology and Educational Innovation, which references Wang & Fan’s paper once.

These papers should all now be corrected by their authors, and the journals that have published them should require the corrections to be made. In the case of the paper which references Wang & Fan’s paper 10 times, a retraction may be more appropriate. Corrections may also be need to be made to any other papers that have in turn cited these papers. KM and organizational learning practitioners also need to make sure that in their work they don’t use or reference Wang & Fan’s paper or any papers citing it. Or, if they have done this already, corrections need to be made to reflect the retraction.

Thankfully though, not all of the 500+ citations have blindly used Wang & Fan’s work. In an example of the critical analysis that has contributed to the retraction, Ilkka Tuomi alerts in the paper “What counts as evidence in AI & ED: Towards Science-for-Policy 3.0” published¹⁰ in the European Journal of Education Policy and Practice that:

[The] study by Wang and Fan … uses the same methodology as the Deng et al. study, to the extent that it copies their search pattern with the original spelling mistakes. Already a quick review of the journals where the original studies have been published, show that low-quality and potentially predatory journals are included.

Case 2. National AI policy cites fake research created by AI

Wang & Fan overhyped ChatGPT paper fails to meet established standards in regard to research quality and information integrity. This second case is also an example of such failures.

As Dr Nomalanga Mashinini writes¹¹ in the following article from The Conversation, South Africa’s Department of Communications and Digital Technologies published the Draft South Africa National Artificial Intelligence Policy for public comment. However, checks by the media found hallucinated references – both academic journals that do not exist, and real journals in which the referenced research articles were never published.

Mashinini reports that much of the public commentary has treated this as an embarrassment: the policy meant to govern AI was itself undermined by AI.  As a senior lecturer in cyber law, including the regulation of AI, she argues that framing this episode as an embarrassment obscures what needs to be examined. It misses the main point of what is at stake. It is also why I have chosen to use the word “disgraceful” in this article rather than “embarrassing.” Mashinini advises that:

Responsible AI needs accountability, transparency, and explainability. These are non-negotiable conditions, echoed by the Organisation for Economic Co-operation and Development principles and the Smart Africa AI Blueprint that the policy draws on. These governance principles are not just for AI system designers. They bind any institution that uses AI, including use in the production of public documents.

Is the KM community a coming case study of disgraceful AI?

In the past six months, I’ve experienced three cases of content with unchecked hallucinations being circulated in the KM community, most recently in the past few days. If this trend continues, it’s only a matter of time before such a case attracts wider attention, carrying the risk of serious long-term reputational damage to KM and the KM community.

Amplifying the risk, the KM community currently lacks key capabilities needed for safe AI use, as I alert in an exposé¹² of how a prominent KM organization has become the victim of organized crime because it failed to detect and then published readily identifiable scam content.

So, which of the following two scenarios will become the reality for the KM community? Will it continue along its current path towards AI disgrace? Or will it become an AI leader by actively taking up Dr Nomalanga Mashinini’s advice and embracing and promoting ethical and responsible AI in KM?

Acknowledgement: Thank you to Professor Eric Tsui for alerting me to the retraction of Wang & Fan’s paper.

Header image source: Nicola Barts on Pexels.

References: