GLASGOW, Scotland ― ChatGPT failed to pass the UK's National Primary Care examinations in a new study, highlighting how artificial intelligence (AI) does not necessarily match human perceptions of medical complexity.
ChatGPT also provided novel explanations ― it frequently "hallucinates" ― by describing inaccurate information as if they were facts, according to Shathar Mahmood, BA, a fifth-year medical student at the University of Cambridge School of Clinical Medicine, Cambridge, UK, who presented the findings at the Royal College of General Practitioners (RCGP) Annual Conference 2023. The study was published in JMIR Medical Education earlier this year.
"Artificial intelligence has generated impressive results across medicine, and with the release of ChatGPT there is now discussion about these large language models taking over clinicians' jobs," Arun James Thirunavukarasu, MB BChir, of the University of Oxford and Oxford University Hospitals NHS Foundation Trust, who is the lead author of the study, told Medscape Medical News.
Performance of AI on medical school examinations has prompted much of this discussion, often because performance does not reflect real-world clinical practice, he said. "We used the Applied Knowledge Test instead, and this allowed us to explore the potential and pitfalls of deploying large language models in primary care and to explore what further development of medical large language model applications is required."
