Study highlights the exceptional clinical reasoning of ChatGPT-4 in specific situations

In a study published in JAMA Network Open, researchers from Beth Israel Deaconess Medical Center in Boston found that in specific restricted scenarios, ChatGPT-4 exhibited exceptional clinical reasoning and outperformed human clinicians. The researchers conducted a comparison between ChatGPT-4 and clinicians using a set of five medical cases. These cases were previously given to clinicians in a survey on probabilistic reasoning. ChatGPT-4 was given the same prompt 100 times and asked to provide the likelihood of a specific diagnosis based on the patient’s presentation. The chatbot was also asked to update its estimates in response to certain test results, such as mammography for breast cancer. The team then compared the chatbot’s probabilistic estimates with responses from over 550 human practitioners in the survey. The study found that in all five cases, ChatGPT-4 showed superior accuracy compared to human clinicians when assessing pretest and post-test probability following a negative test result. However, the performance of the large language model was not as strong when dealing with positive test results.