4 questions with Joanna Ng

What has inspired this research?

My poster presentation evaluates the Rayyan artificial intelligence (AI) tool for systematic literature review screening.

AI is a growing topic of interest within healthcare and various areas of healthcare are seeking to integrate it at different ways, from patient care to data aggregation.

At Cencora, we frequently conduct systematic literature reviews, a process that demands significant time and effort. Typically, sorting through the vast amount of literature in the title/abstract screening phase requires many hours. We aimed to explore ways to streamline the title/abstract screening phase with the assistance of the Rayyan AI tool.

Was there a hypothesis that was confirmed through the research?

Our objective was to evaluate whether the AI tool could effectively identify relevant literature in the title/abstract screening phase. Researchers face the challenge of reading and assessing each piece of literature within a heavy volume of search results for relevance to their topic; therefore, we sought to evaluate the performance and efficiencies of the Rayyan AI tool in the systematic literature review title/abstract screening phase. Across various topics, including clinical, humanistic, and economic fields, the tool performed exceptionally well in selecting the relevant articles of interest. This consistency reassured us of its reliability.

What is the key takeaway from your research?

We trained the Rayyan AI tool on 20% of the references for two systematic literature reviews that were previously conducted by human reviewers. The tool then predicted the relevance of the remaining references on a scale of 1-5, ranging from “most likely to exclude” to “most likely to include.” We compared its ratings with those of human reviewers and found that the tool’s sensitivity ranged from 79% to 100% across all outcomes. Specificity ranged from 8% to 62% across all outcomes. The research confirmed that the Rayyan AI tool has a high sensitivity, effectively identifying relevant articles. However, it has low to moderate specificity, meaning it struggles to exclude irrelevant articles.

There is potential for time savings with using Rayyan AI tool. Time savings ranged from 6%-47% for a single reviewer using the Rayyan AI tool. However, time savings for dual screening which is the typical standard for SLRs, with 1 human reviewer and Rayyan AI tool, were modest (3%-23%).

What’s next for this research?

While Rayyan AI tool provides moderate time savings and high sensitivity, we are exploring ways to increase time savings and specificity. Additionally, we are considering how to further integrate AI tools into the broader systematic literature review process. My evaluation focused on abstract and title screening, but there may be other stages of the systematic literature review process where AI tools could be beneficial.

To learn more about Joanna’s research, view her poster here.

Citations relevant to the content described herein are provided in the article linked here. Readers should review all available information related to the topics mentioned herein and rely on their own experience and expertise in making decisions related thereto.