External evaluation of the Dynamic Criticality Index: A machine learning model to predict future need for ICU care in hospitalized pediatric patients

Authors

Anita K. Patel, Department of Pediatrics, Division of Critical Care Medicine, Children's National Health System, Washington, DC, United States of America.
Eduardo Trujillo-Rivera, Department of Pediatrics, Division of Critical Care Medicine, Children's National Health System, Washington, DC, United States of America.
James M. Chamberlain, George Washington University School of Medicine and Health Sciences, Washington, DC, United States of America.
Hiroki Morizono, Children's National Research Institute, Children's National Hospital, Washington, DC, United States of America.
Murray M. Pollack, Department of Pediatrics, Division of Critical Care Medicine, Children's National Health System, Washington, DC, United States of America.

Document Type

Journal Article

Publication Date

1-1-2024

Journal

PloS one

Volume

19

Issue

1

DOI

10.1371/journal.pone.0288233

Abstract

OBJECTIVE: To assess the single site performance of the Dynamic Criticality Index (CI-D) models developed from a multi-institutional database to predict future care. Secondarily, to assess future care-location predictions in a single institution when CI-D models are re-developed using single-site data with identical variables and modeling methods. Four CI-D models were assessed for predicting care locations >6-12 hours, >12-18 hours, >18-24 hours, and >24-30 hours in the future. DESIGN: Prognostic study comparing multi-institutional CI-D models' performance in a single-site electronic health record dataset to an institution-specific CI-D model developed using identical variables and modelling methods. The institution did not participate in the multi-institutional dataset. PARTICIPANTS: All pediatric inpatients admitted from January 1st 2018 -February 29th 2020 through the emergency department. MAIN OUTCOME(S) AND MEASURE(S): The main outcome was inpatient care in routine or ICU care locations. RESULTS: A total of 29,037 pediatric hospital admissions were included, with 5,563 (19.2%) admitted directly to the ICU, 869 (3.0%) transferred from routine to ICU care, and 5,023 (17.3%) transferred from ICU to routine care. Patients had a median [IQR] age 68 months (15-157), 47.5% were female and 43.4% were black. The area under the receiver operating characteristic curve (AUROC) for the multi-institutional CI-D models applied to a single-site test dataset was 0.493-0.545 and area under the precision-recall curve (AUPRC) was 0.262-0.299. The single-site CI-D models applied to an independent single-site test dataset had an AUROC 0.906-0.944 and AUPRC range from 0.754-0.824. Accuracy at 0.95 sensitivity for those transferred from routine to ICU care was 72.6%-81.0%. Accuracy at 0.95 specificity was 58.2%-76.4% for patients who transferred from ICU to routine care. CONCLUSION AND RELEVANCE: Models developed from multi-institutional datasets and intended for application to individual institutions should be assessed locally and may benefit from re-development with site-specific data prior to deployment.

APA Citation

Patel, Anita K.; Trujillo-Rivera, Eduardo; Chamberlain, James M.; Morizono, Hiroki; and Pollack, Murray M., "External evaluation of the Dynamic Criticality Index: A machine learning model to predict future need for ICU care in hospitalized pediatric patients" (2024). GW Authored Works. Paper 4112.
https://hsrc.himmelfarb.gwu.edu/gwhpubs/4112

Department

Pediatrics