TEQnation 2024: Using AI to save time and sanity – Rens van de Schoot

In today’s academic world, the number of scientific papers on any topic is growing. While this wealth of textual data is a treasure trove for data scientists, it simultaneously presents significant challenges for anyone who wants to screen this literature systematically. Whether for medical guidelines, review papers, or for evidence-based policy recommendations, one needs to systematically sift through thousands of studies to identify the rare relevant ones. This tedious, time-consuming, and error-prone process (e.g., due to screening fatigue) usually demands months of work. Despite this effort, systematic overviews are indispensable for scholars and clinicians, policymakers, journalists, and the public at large. The rapidly evolving field of AI offers promising solutions to the literature screening challenge using machine learning models and, very recently, large language models (LLMs). However, many of these AI-driven solutions emerge from tech companies that publish new and, hopefully, better models at an unprecedented rate. The rapidity of advancements in the field of AI outpaces meticulous scientific evaluations, leaving many methods unrefined and unproven. Moreover, we know that the broader scientific community (and beyond) has concerns about using LLMs for research: Should we replace human screeners with an AI (as predicted by OpenAI), or should we keep a human in the loop? And, if we want to use an AI, what model is best for maximizing performance and under what conditions? How do humans interact with machines, and what types of bias are introduced by these new workflows? At The AI-aided knowledge discovery lab at Utrecht University, we facilitate research to answer such questions and develop fully open and transparent infrastructure to test and apply new models to real-world applications, including the open-source software ASReview (https://github.com/asreview). During the presentation, Rens will present several use cases where comprehensive literature overviews are essential (think of medical guidelines, side-effects of drugs, and dosage recommendations for children). He will show how machine learning models can accelerate the screening task, and then we will discuss whether AI can fully replace humans for this task… p.s., Become a stargazer of the ASReview project and help us to increase visibility!