Nature has retracted a paper that claimed AI had a positive impact on student learning.\u00a0The original paper, titled \u201cThe effect of ChatGPT on students\u2019 learning performance, learning perception, and higher-order thinking: insights from a meta-analysis,\u201d was originally published in May of last year by Jin Wang and Wenxiang Fan of the Hangzhou Normal University in China. It is a meta-analysis, meaning it combines data from 51 research studies published between November 2022 and February 2025 on the effectiveness of ChatGPT in education. The paper claimed it found that ChatGPT had a large or moderately positive impact on \u201cstudents\u2019 learning performance, learning perception, and higher-order thinking.\u201d\u201cThe Editor has decided to retract this paper owing to concerns regarding discrepancies in the meta-analysis,\u201d Nature said in its retraction note. \u201cThese issues ultimately undermine the confidence the Editor can place in the validity of the analysis and resulting conclusions. The authors have not responded to correspondence regarding this retraction.\u201d\u00a0The researchers did not immediately respond to a request for comment.\u00a0\u201cI first noticed the paper published just a day or two after it came out on 6 May 2025,\u201d Ben Williamson, a senior lecturer in digital education at the University of Edinburgh, told me in an email. \u201cIt rapidly picked up a lot of attention on social media, especially on LinkedIn, as it appeared to offer some of the first hard evidence that ChatGPT improves what the authors called \u2018learning performance.\u2019 Within a month it had been accessed online almost 400,000 times and had an Altmetric score of 365 after being shared hundreds of times on X and Bluesky. It was very much helped by some very influential individuals sharing it on social media as good evidence to support promoting AI in education.\u201dThe retraction note did not provide more details on Nature’s decision, but a 2025 study published in European Journal of Education Policy and Practice shows that the method Wang and Fan used is often flawed, and highlighted the issues in their paper before it was retracted.\u00a0\u201cExisting empirical evidence on AIED [AI in educations] suggests some positive effects, but a closer look reveals methodological and conceptual problems and leads to the conclusion that existing evidence should not be used to guide policy or practice,\u201d the paper, written Ilkka Tuomi and titled \u201cWhat counts as evidence in AI & ED: Towards Science-for-Policy 3.0,\u201d said.\u00a0One problem according to Tuomi is that these meta-analysis studies use any paper that was peer-reviewed, but that a closer look at each individual paper reveals that they vary in quality or that the data doesn\u2019t show AI improves learning outcomes.\u00a0\u201cDespite its apparent methodological quality and apparent rigour, the heterogeneity of the analysed studies makes the quantitative results of the Deng et al. meta-analysis meaningless,\u201d Tuomi said, referring to another study about ChatGPT enhancing student learning. \u201cVery similar problems underpin another viral article that has been interpreted to provide final proof that ChatGPT has positive impacts on learning. This study, by Wang and Fan (2025), uses the same methodology as the Deng et al. study, to the extent that it copies their search pattern with the original spelling mistakes. Already a quick review of the journals where the original studies have been published, show that low-quality and potentially predatory journals are included.\u201d\u00a0\u201cThis meta analysis on ChatGPT effects on learning appeared only two and a half years after ChatGPT was launched,\u201d Williamson said. \u201cSo what we are supposed to believe is that in the intervening period, dozens of high quality studies of the effect of ChatGPT on learning performance took place, were written up, submitted for peer review, and published, which the meta analysis authors then painstakingly synthesized using robust methods. What appeared actually to be the case is that the meta analysis aggregated a whole bunch of very low quality research published in disreputable journals. Ultimately, the meta analysis recycled junk science into headline-grabbing claims about the benefits of ChatGPT for learners. And those claims were simply unfounded due to methodological problems with the conduct of the study, as the retraction now appears to indicate.\u00a0\u00a0“The retraction of this study should serve as a crucial reminder to the education community,\u201d Jake Baskin, executive director of the Computer Science Teachers Association, told me in an email. \u201cWe need to teach students how this technology actually works, not just how to use it, and rigorously evaluate if and how generative AI genuinely improves teaching and learning.”\u00a0Our reporting has repeatedly shown that large language models are prone to errors that can make education frustrating to both students and teachers. Multiple teachers have told us that ChatGPT has completely upended their ability to educate students and grade their work which is increasingly AI-generated. My investigation into Alpha School, the leading \u201cAI-powered\u201d school, used AI generated lesson plans that included errors and flawed questions.\u00a0Despite these problems, AI companies and lawmakers continue to push AI products into schools.\u201cChatGPT and other generative AI applications have been incredibly disruptive in education for several years,\u201d Williamson said. \u201cWhat educators, parents and policy officials really needed was high quality data and evidence to help guide them. What they have had to deal with instead is some substandard research.\u201d<\/p>\n","protected":false},"excerpt":{"rendered":"