To evaluate ChatGPT's effectiveness in medical imaging interpretation within urology, addressing the critical need for safe AI application in healthcare by identifying its strengths and limitations as a diagnostic and educational resource.

Using publicly available cases from Radiopaedia.com, we entered 1-3 CT or MRI images into ChatGPT. A standard prompt instructed the model to provide a differential diagnosis ranked by probability. This task was repeated a second time with organ guidance (OG), which provided the organ of diagnostic interest to the model (eg, kidney). Primary outcomes included whether the model’s top or differential diagnosis correctly identified the underlying pathology.

ChatGPT correctly identified the pathologic condition as its top diagnosis in 14% of CT (7/50) and 28% (14/50) of MRI cases (P = .08). OG increased the model’s ability to recognize the top diagnosis by 18% (P = .03) when interpreting CT images, a benefit not shared when interpreting MRI images (P = .4). At baseline the differential diagnosis contained the final diagnosis for 30% and 56% of CT and MRI cases (P = .03). With the inclusion of OG, the model’s differential diagnosis was able to correctly identify the underlying condition in 62% of both CT and MRI cases (CT: P = .001, MRI: P = .31).

ChatGPT's effectiveness in medical imaging diagnostics is initially limited, yet it substantially benefits from the addition of user guidance. The study underscores AI's current shortcomings but also its considerable capacity to improve clinical operations when enriched with more data and expert direction.