Individual Gestalt Is Unreliable for the Evaluation of Quality in Medical Education Blogs: A METRIQ Study - 29/08/17
, Stefanie S. Sebok-Syer, PhD c, Keeth Krishnan, BHSc d, Marshall Siemens a, N. Seth Trueger, MD, MPH e, Isabelle Colmers-Gray, MD, MSc f, Rob Woods, MD, MMEd a, Emil Petrusa, PhD b, Teresa Chan, MD, MHPE gthe
METRIQ Study Collaborators†
Abstract |
Study objective |
Open educational resources such as blogs are increasingly used for medical education. Gestalt is generally the evaluation method used for these resources; however, little information has been published on it. We aim to evaluate the reliability of gestalt in the assessment of emergency medicine blogs.
Methods |
We identified 60 English-language emergency medicine Web sites that posted clinically oriented blogs between January 1, 2016, and February 24, 2016. Ten Web sites were selected with a random-number generator. Medical students, emergency medicine residents, and emergency medicine attending physicians evaluated the 2 most recent clinical blog posts from each site for quality, using a 7-point Likert scale. The mean gestalt scores of each blog post were compared between groups with Pearson’s correlations. Single and average measure intraclass correlation coefficients were calculated within groups. A generalizability study evaluated variance within gestalt and a decision study calculated the number of raters required to reliably (>0.8) estimate quality.
Results |
One hundred twenty-one medical students, 88 residents, and 100 attending physicians (93.6% of enrolled participants) evaluated all 20 blog posts. Single-measure intraclass correlation coefficients within groups were fair to poor (0.36 to 0.40). Average-measure intraclass correlation coefficients were more reliable (0.811 to 0.840). Mean gestalt ratings by attending physicians correlated strongly with those by medical students (r=0.92) and residents (r=0.99). The generalizability coefficient was 0.91 for the complete data set. The decision study found that 42 gestalt ratings were required to reliably evaluate quality (>0.8).
Conclusion |
The mean gestalt quality ratings of blog posts between medical students, residents, and attending physicians correlate strongly, but individual ratings are unreliable. With sufficient raters, mean gestalt ratings provide a community standard for assessment.
Le texte complet de cet article est disponible en PDF.Plan
| Please see page 395 for the Editor’s Capsule Summary of this article. |
|
| Supervising editor: Peter C. Wyer, MD |
|
| Author contributions: BT, KK, and TC conceived the study. BT, SSS-S, RW, EP, and TC designed the trial. BT, RW, and TC obtained research funding. BT and MS supervised the conduct of the trial and data collection and managed the data, including quality control. BT, KK, MS, NST, IC-G, RW, and TC undertook recruitment of participants. SSS-S, EP, and TC provided statistical advice on study design and analyzed the data. BT drafted the article and all authors contributed substantially to its revision. BT takes responsibility for the paper as a whole. |
|
| All authors attest to meeting the four ICMJE.org authorship criteria: (1) Substantial contributions to the conception or design of the work; or the acquisition, analysis, or interpretation of data for the work; AND (2) Drafting the work or revising it critically for important intellectual content; AND (3) Final approval of the version to be published; AND (4) Agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. |
|
| Funding and support: By Annals policy, all authors are required to disclose any and all commercial, financial, and other relationships in any way related to the subject of this article as per ICMJE conflict of interest guidelines (see www.icmje.org/). Funding for this research was provided by the Canadian Association of Emergency Physicians (Junior Investigator Grant) and the Royal College of Physicians and Surgeons of Canada (Robert Maudsley Fellowship for Studies in Medical Education). Dr. Thoma (ALiEM.com, Debrief2Learn.org, and CanadiEM.org), Dr. Trueger (MDaware.org), and Dr. Chan (ALiEM.com, CanadiEM.org, FeminEM.org, and ICENet.com) edit or operate medical education blogs. Dr. Trueger reports receiving stipends as social media editor for Annals of Emergency Medicine and Emergency Physicians Monthly. |
|
| Readers: click on the link to go directly to a survey in which you can provide QT3RT9P to Annals on this particular article. |
Vol 70 - N° 3
P. 394-401 - septembre 2017 Retour au numéro
Article précédent
- Paramedic Intubation: Does Practice Make Perfect?
- Jestin N. Carlson, Henry E. Wang
- Practical Application of Educational Theory for Learning Technical Skills in Emergency Medicine
- Andrew J. Davis, Lizveth Fierro, Mindi Guptill, Michael Kiemeney, Lance Brown, Dustin D. Smith, Timothy P. Young
Bienvenue sur EM-consulte, la référence des professionnels de santé.
L’accès au texte intégral de cet article nécessite un abonnement.
Déjà abonné à cette revue ?