Reviewing scientific literature is an important part of advancing a research field. A comprehensive analysis of existing research provides the current state of integration and identifies gaps in knowledge where future research can focus. However, writing a well-written review article is a great thing.
Researchers often comb through large volumes of academic work. They should select studies that are not old, but avoid recency bias. Then comes the intensive work of assessing the quality of the research, extracting relevant data from the work that makes the cut, analyzing the data to gain insights, and writing a compelling narrative that summarizes the past while looking to the future. Research synthesis is a field of study in itself, and even great scientists may not be able to write good literature reviews.
Enter artificial intelligence. As in many industries, a number of startups have emerged that are leveraging AI to accelerate, simplify, and revolutionize the scientific literature review process. Many of these startups are positioning themselves as academic research-focused AI search engines, each with differentiated product features and target audiences.
Elicit invites searchers to “analyze research papers at superhuman speed” and highlights the use of Elicit by professional researchers at institutions such as Google, NASA, and the World Bank. Scite says it has built the largest citation database by continuously monitoring 200 million academic sources and provides ‘smart citations’ that categorize evidence into supporting or contrasting implications. Consensus offers a homepage demo that appears to be intended to help the public gain a clearer understanding of a given question, describes the product as “Google Scholar meets ChatGPT” and provides a consensus meter that summarizes key takeaways. These are just a few of many.
But can AI replace high-quality systematic scientific literature reviews?
Research synthesis experts tend to agree that these AI models are currently very good at performing qualitative analysis, that is, creating narrative summaries of scientific literature. The not so great parts are the more complex quantitative layers that make the review truly organized. These quantitative syntheses typically involve statistical methods, such as meta-analysis, which analyze numerical data from multiple studies to draw stronger conclusions.
“AI models are almost as good as humans at summarizing key points and crafting fluid arguments,” said Joshua Polanin, co-founder of the Methods of Synesis and Integration Center (MOSAIC) at the American Institutes for Research. “It can be 100% excellent.” “But we are not even 20 percent of the way to quantitative synthesis,” he says. “Real meta-analyses follow a rigorous process in how they search for studies and quantify the results. These numbers are the basis for evidence-based conclusions. AI is not even close to being able to do that.”
A question of quantification
The quantification process can be difficult even for experienced experts, explains Polanin. Both humans and AI can generally read research and summarize its content. Study A found an effect, Study B did not find an effect. The tricky part is placing a numeric value for the degree of effect. Moreover, because there are often many different ways to measure effectiveness, researchers must identify research and measurement designs that are consistent with the premises of the research question.
Polanin says models must first identify and extract relevant data and then make nuanced decisions about how to compare and analyze it. “As human experts, we try to make decisions in advance, but we may end up having to change our minds on the fly,” he says. “That’s not something a computer can do well.”
Given the arrogance found within AI and startup culture, one might expect companies building these AI models to protest Polanin’s assessment. But you won’t hear the following argument from Consensus co-founder Eric Olson: “I honestly couldn’t agree more,” he says.
In Polanin’s view, Consensus intentionally “goes beyond other tools to give people the foundational knowledge for rapid insights,” Olson adds. He sees the typical user as a graduate student, someone with an intermediate knowledge base who is working toward becoming an expert. Consensus can be one of many tools for true subject matter experts, and can help non-scientists stay informed, such as consensus users in Europe who keep their research up to date on rare genetic disorders in their children. “He spent hundreds of hours on Google Scholar as a non-researcher. He said he had dreamed of something like this for 10 years and that it had changed his life. Now he uses it every day,” Olson says.
At Elicit, the team targets different types of ideal customers. “Someone working in industry, perhaps in a biomedical company, in an R&D environment trying to decide whether to move forward with the development of a new medical intervention,” says James Brady. , Head of Engineering.
Elicit keeps high-risk users in mind and clearly shows users’ causal claims and the evidence that supports them. This tool breaks down the complex task of literature review into manageable parts that humans can understand and provides greater transparency than a typical chatbot. Researchers can see how the AI model arrived at its answers and check them against the source.
The future of scientific review tools
Brady agrees that current AI models do not provide a full Cochrane-style systematic review, but says this is not a fundamental technical limitation. Rather, it is a question of future advancements in AI and better, faster engineering. “I don’t think, in principle, that our brains can do anything that computers can’t do,” says Brady. “And this also applies to the systematic review process.”
Roman Lukyanenko, a University of Virginia professor who specializes in research methods, agrees that developing methods to support the initial prompting process to get better answers should be a major focus in the future. He also points out that while current models tend to prioritize freely accessible journal articles, there is plenty of high-quality research behind paywalls. Still, he is optimistic about the future.
“I believe that AI is very revolutionary in this field, on a massive level,” says Lukyanenko, who co-authored a pre-ChatGPT 2022 study on AI and literature review with Gerit Wagner and Guy Paré. “We are bombarded with information, but human biology limits what we can do with that information. These tools represent great potential.”
He says scientific advances often come from interdisciplinary approaches, and this is where AI’s potential may be greatest. “We have the term ‘Renaissance Man’. I like to think of it as ‘Renaissance AI’ – something that has access to a significant part of our knowledge and can make connections,” says Lukyanenko. . “We must work hard to make serendipitous, unexpected, and remote discoveries across disciplines.”
From your site article
Related articles on the web
