To use artificial intelligence (AI) to predict billing code levels for emergency department (ED) encounters.

We accessed ED encounters from our health system from January to September 2023. We developed an ensemble model using natural language processing and machine learning techniques to predict billing codes from clinical notes combined with clinical characteristics and orders. Explainable AI techniques were used to help determine the important model features. The main endpoint was to predict evaluation and management professional billing codes (levels 2 to 5 [Current Procedural Terminology codes 99282 to 99285] and critical care). Secondary endpoints included predicting professional billing codes at different decision boundary thresholds and generalizability of the model at other EDs.

There were 321,893 adult ED encounters coded at levels 2 (<1%), 3 (5%), 4 (38%), 5 (51%), and critical care (5%). Model performance for professional billing code levels of 4 and 5 yielded area under the receiver operating characteristic curve values of 0.94 and 0.95, accuracy values of 0.80 and 0.92, and F1-scores of 0.79 and 0.91, respectively. At a 95% decision boundary threshold, level 5 predicted charts had a precision/positive predictive value of 0.99 and recall/sensitivity of 0.57. The most important features using Shapley Additive Explanations values were critical care note, number of orders, discharge disposition, cardiology, and psychiatry.

Currently available AI models accurately predict billing code levels for ED encounters based on clinical notes, clinical characteristics, and orders. This has the potential to automate coding of ED encounters and save administrative costs and time.