Machine learning algorithms are essential for predicting severe outcomes during public health crises like COVID-19. However, the dynamic nature of diseases requires continual evaluation and updating of these algorithms. This study aims to compare three update strategies for predicting severe COVID-19 outcomes postdiagnosis: “naive” (a single initial model), “frequent” (periodic retraining), and “context-driven” (retraining informed by clinical insights). The goal is to determine the most effective timing and approach for adapting algorithms to evolving disease dynamics and emerging data.

A dataset of 1.11 million COVID-19 patients from diverse U.S. regions was used to develop and validate an XGBoost algorithm for predicting severe outcomes upon diagnosis. Data included patient demographics, vital signs, comorbidities, and immunity-related factors (prior infection and vaccination status) from January 2007 to November 2021. The study analyzed the performance of the three update strategies from March 2020 to November 2021.

Predictive features changed over the pandemic, with comorbidities and vitals being significant initially, and geography, demographics, and immunity-related variables gaining importance later. The “naive” strategy had an average area under the curve (AUC) of 0.77, the “frequent” strategy maintained stability with an average AUC of 0.81, and the “context-driven” strategy averaged an AUC of 0.80, outperforming the “naive” strategy and aligning closely with the “frequent” strategy.

A context-driven approach, guided by clinical insights, can enhance predictive performance and offer cost-effective solutions for dynamic public health challenges. These findings have significant implications for efficiently managing healthcare resources during evolving disease outbreaks.