Mental health disorders have a high Disability-Adjusted Life Years in the Middle East and North Africa. This rise has led to a surge in related publications, prompting researchers to use AI tools like ChatGPT to reduce time spent on routine tasks. Our study aimed to validate an AI-assisted Critical Appraisal (CA) tool by comparing it with human raters.

We developed customized GPT models using ChatGPT-4. These models were tailored to evaluate studies using the Newcastle-Ottawa Scale (NOS) or the Jadad Scale in one model, while another model evaluated STROBE or CONSORT guidelines.

Our results showed a moderate to good agreement between human CA and our GPTs for the NOS for cohort, case control and cross-sectional studies and for the Jadad scale, with an ICC of 0.68 [95%CI: 0.24 - 0.82], 0.69 [95%CI: 0.31 - 0.88], 0.76 [95%CI: 0.47 - 0.90] and 0.84 [95%CI: 0.57 - 0.94] respectively. There was also a moderate to substantial agreement between the two methods for STROBE in cross sectional, cohort, case control studies, and for CONSORT in trial design, with a K of 0.63 [95%CI: 0.56 -,0.70], 0.57 [95%CI: 0.47 - 0.66], 0.48 [95%CI: 0.38 - 0.50] and 0.70 [95%CI: 0.63 - 0.77] respectively. Our custom GPT models produced hallucinations in 6.5% and 4.9% of cases, respectively. Human raters took an average of 19.6 ± 4.3 minutes per article, whereas our customized GPTs took only 1.4.

ChatGPT could be a useful tool for handling repetitive tasks yet its effective application relies on the critical expertise of researchers.