A Comparative Analysis of AI Tools for Research Support: ChatGPT, Google Gemini, and Microsoft Copilot

Abstract

This presentation is based on a study that compared public ChatGPT 3.5, commercial ChatGPT 4, Google Gemini, and Microsoft Copilot in answering reference questions from the Rider University Library and Learning Commons in 2024. A total of 28 user-initiated questions were extracted from the chat reference transaction log during the 2024 academic year and input into the four AI applications as prompts. The responses from each AI tool were rated on a scale of 1 to 10 based on relevance, accuracy, friendliness, and the quality of information literacy instruction. The purpose of the study was to determine which AI application best supports research and information literacy. ANOVA (Analysis of Variance), a statistical test used to analyze differences between the means of more than two groups, was employed in the analysis. The results indicate that Google Gemini provides stronger support for research and information literacy education. In contrast, Microsoft Copilot tends to deliver brief and cursory responses. All of the AI tools attempt to manage hallucinations by avoiding citations. However, both ChatGPT and Google Gemini outperformed librarians in educating users about Boolean searches and recommending resources. To sum up, AI has made great progress in responses if we compare this study to the findings from those in 2023.

(presented on 14 August 2025 at "AI in the Academic Field: Supporting Research and Learning" session)