Browse the corpus

Clinically applicable approach for predicting mechanical ventilation in patients with COVID-19. BACKGROUND: Patients with coronavirus disease 2019 (COVID-19) requiring mechanical ventilation have high mortality and resource utilisation. The ability to predict which patients may require mechanical ventilation allows increased acuity of care and targeted interventions to potentially mitigate deterioration. METHODS: We included hospitalised patients with COVID-19 in this single-centre retrospective observational study. Our primary outcome was mechanical ventilation or death within 24 h. As clinical decompensation is more recognisable, but less modifiable, as the prediction window shrinks, we also assessed 4, 8, and 48 h prediction windows. Model features included demographic information, laboratory results, comorbidities, medication administration, and vital signs. We created a Random Forest model, and assessed performance using 10-fold cross-validation. The model was compared with models derived from generalised estimating equations using discrimination. RESULTS: Ninety-three (23%) of 398 patients required mechanical ventilation or died within 14 days of admission. The Random Forest model predicted pending mechanical ventilation with good discrimination (C-statistic=0.858; 95% confidence interval, 0.841-0.874), which is comparable with the discrimination of the generalised estimating equation regression. Vitals sign data including SpO2/FiO2 ratio (Random Forest Feature Importance Z-score=8.56), ventilatory frequency (5.97), and heart rate (5.87) had the highest predictive utility. In our highest-risk cohort, the number of patients needed to identify a single new case was 3.2, and for our second quintile it was 5.0. CONCLUSION: Machine learning techniques can be leveraged to improve the ability to predict which patients with COVID-19 are likely to require mechanical ventilation, identifying unrecognised bellwethers and providing insight into the constellation of accompanying signs of respiratory failure in COVID-19.

For this retrospective observational study performed at our academic quaternary care centre, we obtained Institutional Review Board approval (University of Michigan, Ann Arbor, MI, USA; HUM00052066). As no patient care interventions were made through conducting the study, patient consent was waived. This manuscript follows multidisciplinary guidelines for reporting machine learning predictive models in biomedical research.14 Study outcomes, data collection, and statistical analyses were established a priori and presented at a multidisciplinary peer-review forum on May 20, 2020 before data access.15

ent was waived. This manuscript follows multidisciplinary guidelines for reporting machine learning predictive models in biomedical research.14 Study outcomes, data collection, and statistical analyses were established a priori and presented at a multidisciplinary peer-review forum on May 20, 2020 before data access.15 For all patients with COVID-19 admitted to the hospital, the electronic health record (Epic Systems, Verona, WI, USA) was queried for patient characteristics, baseline comorbidities, vital signs, laboratory values, medication administration record, and processes of care. The full list of features included in our model can be found in Supplementary Table S1. Medical comorbidities were categorised according to International Classification of Diseases-9/10 diagnoses present upon admission according to a previously described and validated classification system.16,17 Patients were excluded if they were receiving mechanical ventilation on arrival (via hospital transfer) or were intubated within 4 h of hospital admission. Data were grouped into 4 h windows and extended to the next window, if no new data were recorded. If supplementary O2 was expressed in L min−1, instead of FiO2, then L min−1 flow was converted to FiO2 by adding 0.038 for every L min−1 of supplemental oxygen.18 Hi-Flow nasal cannula and Venturi masks are recorded in the medical record as FiO2. Non-rebreather masks were considered to supply FiO2=0.70. The actual FiO2 for face masks and nasal cannula will vary from person to person depending on factors such as tidal volume and ventilatory frequency18; we used these conversion factors to be consistent across all patients. Data at a given time window, data from the immediately preceding time window, and the change between them (delta) were incorporated into our model. If preceding data were not available, data were imputed to population mean and the delta value was set to zero. Data for all patients were censored at 14 days after hospital admission.

ven time window, data from the immediately preceding time window, and the change between them (delta) were incorporated into our model. If preceding data were not available, data were imputed to population mean and the delta value was set to zero. Data for all patients were censored at 14 days after hospital admission. Our target output (primary outcome) was mechanical ventilation or death within 24 h. As the clinical decompensation is likely more recognisable and less modifiable as the time window decreases, we also assessed and characterised the predictive utility of our model to predict mechanical ventilation or death within 4 and 8 h, and, for more notice, 48 h as secondary outcomes. Each outcome extended from whenever the prediction was being made to the end of the designated prediction window. Predictions were made every 4 h through the first 14 days of a patient's hospitalisation (or until the outcome was reached). For example at the 8 h prediction point, the primary outcome was intubation before the 32 h mark and 12, 16, and 56 h for the secondary outcomes. At the 24 h prediction point, the primary outcome was intubation before the 48 h mark, and the secondary outcomes 28, 32, and 72 h. The decision to intubate was left to the discretion of the clinical care team (typically fellowship-trained intensivists). There were no institutional criteria for intubation. Bi-level positive airway pressure was used as an escalation of respiratory management, but was not included as a primary outcome (i.e. invasive mechanical ventilation). The initial prediction window (i.e. 0 h) began with the first documented vital signs (which may have occurred upon presentation to the emergency department, before hospital admission).

was used as an escalation of respiratory management, but was not included as a primary outcome (i.e. invasive mechanical ventilation). The initial prediction window (i.e. 0 h) began with the first documented vital signs (which may have occurred upon presentation to the emergency department, before hospital admission). Clinical data were summarised using means and standard deviations (sd) for normally distributed continuous covariates, medians and inter-quartile range for non-normally distributed continuous variables, and counts and percentages for categorical covariates. Statistical analysis was performed in SAS for Windows 9.4 (SAS Institute Inc., Cary, NC, USA).

ing means and standard deviations (sd) for normally distributed continuous covariates, medians and inter-quartile range for non-normally distributed continuous variables, and counts and percentages for categorical covariates. Statistical analysis was performed in SAS for Windows 9.4 (SAS Institute Inc., Cary, NC, USA). A Random Forest is a classification algorithm characterised by a set of many decision ‘trees’ uncorrelated to each other.19 A Random Forest was trained to predict when a patient would require mechanical ventilation (using randomForest V4.6-14 in R version 3.5.1; R Foundation for Statistical Computing, Vienna, Austria) using 500 trees and default parameters.20 For classifier training, 398 patients were monitored across 4-h time intervals resulting in 27 282 observations. The Random Forest used 73 predictive features grouped into demographic features, comorbidities, laboratory values, vital signs, and medications (Supplementary Table S1). The Comorbidities included in our static variables were derived from International Classification of Diseases (ICD)-9/10 diagnostic codes present upon admission (from previous hospitalisations, rather than the patient's current hospitalisation), the goal being to only include data that would be available to the clinical provider in real time at the point when the prediction is being made. Groupings for each class (such as renal failure and cardiac arrhythmias) are composite variables based upon these ICD-9/10 codes using previously validated Elixhauser Comorbidity Index.16 Missing laboratory values and vital signs were giving the average across all non-missing features. Missing medication values were given a value of zero. Delta values based on missing values were imputed to zero. The classifier was assessed for sensitivity, specificity, and balanced accuracy using 10-fold cross-validation. To ensure that performance was not overestimated, all time points from the same patient were restricted to the same fold.

were given a value of zero. Delta values based on missing values were imputed to zero. The classifier was assessed for sensitivity, specificity, and balanced accuracy using 10-fold cross-validation. To ensure that performance was not overestimated, all time points from the same patient were restricted to the same fold. The Random Forest Feature Importance Z-score19 was used to rank all candidate features. As data from the immediately preceding time window, and the change between them (delta) were also included, this is larger than the feature list presented in Supplementary Table S2. Briefly, the Random Forest Feature Importance Z-score calculates the number of correct votes on the out-of-bag cases for a particular model feature compared with a randomly permuted set of values from that same feature. [In Breiman's original implementation of the random forest algorithm, each tree is trained on about two-thirds of the total training data.19 As the forest is built, each tree can thus be tested (similar to leave one out cross-validation) on the samples not used in building that tree. This is the out of bag error estimate – an internal error estimate of a random forest as it is being constructed.]

is trained on about two-thirds of the total training data.19 As the forest is built, each tree can thus be tested (similar to leave one out cross-validation) on the samples not used in building that tree. This is the out of bag error estimate – an internal error estimate of a random forest as it is being constructed.] During the initial development, we considered several machine learning approaches but ultimately selected a Random Forest. Although a deep neural network would in theory provide the highest performance for real-time classification, fewer than 400 patients would not be a sufficient number of training examples to properly train the model. In addition, Random Forests are more capable of handling categorical features compared with support vector machines (SVMs). Random Forests are more interpretable and transparent than deep learning or SVMs. To facilitate interpretability of our model, predictive features were ranked according to Z-score. In addition, the highest predictive score for each patient was graphed with visualisation of the primary outcome after the time that score occurred.

rests are more interpretable and transparent than deep learning or SVMs. To facilitate interpretability of our model, predictive features were ranked according to Z-score. In addition, the highest predictive score for each patient was graphed with visualisation of the primary outcome after the time that score occurred. The Random Forest model was then compared with generalised estimating equations (GEE) models at each of the four prediction windows. GEE was selected to account for the longitudinal structure of the data. To create this model, we first used least absolute shrinkage and selection operator (LASSO) using the proc hpgenselect procedure in SAS to select variables for inclusion at each prediction window as previously described.17 LASSO regression also provided the reported c-statistics, as GEE does not provide these. In brief, this method estimates the parameters of a generalised linear regression model by using maximum likelihood techniques with exchangeable correlation structure and logit link. The hpgenselect procedure is a high-performance procedure that provides model fitting and model building for generalised linear models. It fits models for standard distributions in the exponential family, such as the binomial distributions.

For all patients with COVID-19 admitted to the hospital, the electronic health record (Epic Systems, Verona, WI, USA) was queried for patient characteristics, baseline comorbidities, vital signs, laboratory values, medication administration record, and processes of care. The full list of features included in our model can be found in Supplementary Table S1. Medical comorbidities were categorised according to International Classification of Diseases-9/10 diagnoses present upon admission according to a previously described and validated classification system.16,17 Patients were excluded if they were receiving mechanical ventilation on arrival (via hospital transfer) or were intubated within 4 h of hospital admission. Data were grouped into 4 h windows and extended to the next window, if no new data were recorded. If supplementary O2 was expressed in L min−1, instead of FiO2, then L min−1 flow was converted to FiO2 by adding 0.038 for every L min−1 of supplemental oxygen.18 Hi-Flow nasal cannula and Venturi masks are recorded in the medical record as FiO2. Non-rebreather masks were considered to supply FiO2=0.70. The actual FiO2 for face masks and nasal cannula will vary from person to person depending on factors such as tidal volume and ventilatory frequency18; we used these conversion factors to be consistent across all patients. Data at a given time window, data from the immediately preceding time window, and the change between them (delta) were incorporated into our model. If preceding data were not available, data were imputed to population mean and the delta value was set to zero. Data for all patients were censored at 14 days after hospital admission.

Our target output (primary outcome) was mechanical ventilation or death within 24 h. As the clinical decompensation is likely more recognisable and less modifiable as the time window decreases, we also assessed and characterised the predictive utility of our model to predict mechanical ventilation or death within 4 and 8 h, and, for more notice, 48 h as secondary outcomes. Each outcome extended from whenever the prediction was being made to the end of the designated prediction window. Predictions were made every 4 h through the first 14 days of a patient's hospitalisation (or until the outcome was reached). For example at the 8 h prediction point, the primary outcome was intubation before the 32 h mark and 12, 16, and 56 h for the secondary outcomes. At the 24 h prediction point, the primary outcome was intubation before the 48 h mark, and the secondary outcomes 28, 32, and 72 h. The decision to intubate was left to the discretion of the clinical care team (typically fellowship-trained intensivists). There were no institutional criteria for intubation. Bi-level positive airway pressure was used as an escalation of respiratory management, but was not included as a primary outcome (i.e. invasive mechanical ventilation). The initial prediction window (i.e. 0 h) began with the first documented vital signs (which may have occurred upon presentation to the emergency department, before hospital admission).

Clinical data were summarised using means and standard deviations (sd) for normally distributed continuous covariates, medians and inter-quartile range for non-normally distributed continuous variables, and counts and percentages for categorical covariates. Statistical analysis was performed in SAS for Windows 9.4 (SAS Institute Inc., Cary, NC, USA).

A Random Forest is a classification algorithm characterised by a set of many decision ‘trees’ uncorrelated to each other.19 A Random Forest was trained to predict when a patient would require mechanical ventilation (using randomForest V4.6-14 in R version 3.5.1; R Foundation for Statistical Computing, Vienna, Austria) using 500 trees and default parameters.20 For classifier training, 398 patients were monitored across 4-h time intervals resulting in 27 282 observations. The Random Forest used 73 predictive features grouped into demographic features, comorbidities, laboratory values, vital signs, and medications (Supplementary Table S1). The Comorbidities included in our static variables were derived from International Classification of Diseases (ICD)-9/10 diagnostic codes present upon admission (from previous hospitalisations, rather than the patient's current hospitalisation), the goal being to only include data that would be available to the clinical provider in real time at the point when the prediction is being made. Groupings for each class (such as renal failure and cardiac arrhythmias) are composite variables based upon these ICD-9/10 codes using previously validated Elixhauser Comorbidity Index.16 Missing laboratory values and vital signs were giving the average across all non-missing features. Missing medication values were given a value of zero. Delta values based on missing values were imputed to zero. The classifier was assessed for sensitivity, specificity, and balanced accuracy using 10-fold cross-validation. To ensure that performance was not overestimated, all time points from the same patient were restricted to the same fold.

The Random Forest model was then compared with generalised estimating equations (GEE) models at each of the four prediction windows. GEE was selected to account for the longitudinal structure of the data. To create this model, we first used least absolute shrinkage and selection operator (LASSO) using the proc hpgenselect procedure in SAS to select variables for inclusion at each prediction window as previously described.17 LASSO regression also provided the reported c-statistics, as GEE does not provide these. In brief, this method estimates the parameters of a generalised linear regression model by using maximum likelihood techniques with exchangeable correlation structure and logit link. The hpgenselect procedure is a high-performance procedure that provides model fitting and model building for generalised linear models. It fits models for standard distributions in the exponential family, such as the binomial distributions.

A total of 398 patients met our inclusion criteria, with 90 patients requiring mechanical ventilation (23%) and three patients dying without mechanical ventilation (0.8%). The dataset included patients admitted from March 1, 2020 to May 5, 2020. After compiling dynamic model features into 4 h increments, we assessed our primary outcome at 27 282 observations, with 431 positive observations. For our secondary outcomes, we had 93, 171, and 715 positive observations at 4, 8, and 48 h, respectively. Patients meeting our composite outcome tended to be older (mean [sd]: 65 [14] vs 59 [17], P=0.001), male (70% vs 48%, P<0.001), and had higher incidence of: renal failure (58% vs 28%, P<0.001), diabetes (58% vs 37%, P<0.001), and cardiac arrhythmias (69% vs 47%, P<0.001). Furthermore, patients meeting the composite outcome had higher serum creatinine (mean, 2.2 vs 1.4; P=0.019) and ventilatory frequency (23 [6] vs 20 [4], P<0.001) and lower SpO2 (94% [3%] vs 96% [3%], P<0.001) and SpO2/FiO2 ratio (271 [114] vs 367 [107], P<0.001) upon presentation than those not meeting the outcome. Patients requiring subsequent ventilation were administered tocilizumab (15% vs 7%, P=0.021) and norepinephrine (10% vs 2%, P=0.002) more frequently than those not progressing to ventilation or death. Additional details on our patient population can be found in Table 1.Table 1Characteristics of patients requiring intubation or dying within 24 h. Laboratory studies and vital signs are presenting or initial values. Note that not all patients have full laboratory results or vital signs within the first 4 h of admission. The medications counts/percentages listed are based upon administration at any point from admission until data collection was censored at either primary outcome or 14 days after admission.

enting or initial values. Note that not all patients have full laboratory results or vital signs within the first 4 h of admission. The medications counts/percentages listed are based upon administration at any point from admission until data collection was censored at either primary outcome or 14 days after admission. COPD, chronic obstructive pulmonary disease; FiO2, fraction of inspired oxygen; sd, standard deviation; SpO2, blood oxygen saturation level .Table 1VariableLevelAll data (n=398)Control group (n=305)Ventilation or death (n=93)P-valueN%MeansdN%MeansdN%Meansdχ2t-testAge (yr)398100.06017305100.0591793100.0065140.001BMI (kg m−2)398100.031.58.530510031.28.69310032.68.30.171Height (cm)398100.0170.011.4305100.0169.511.593100.0171.710.90.105Weight (kg)398100.091.126.5305100.089.727.193100.095.624.30.063SexFemale18747.015952.12830.1<0.001Male21153.014647.96569.9RaceAfrican American13934.99932.54043.00.433American Indian10.310.300.0Asian133.3113.622.2Caucasian20852.316654.44245.2Other164.0134.333.2Unknown215.3154.966.5Elixhauser comorbiditiesAlcohol abuse215.3175.644.30.631Blood loss anaemia5213.13611.81617.20.176Cardiac arrhythmias20752.014346.96468.8<0.001COPD14035.210434.13638.70.415Coagulopathy9924.97424.32526.90.609Congestive heart failure9724.46722.03032.30.043Anaemia (iron deficiency)7719.36019.71718.30.766Depression13233.210333.82931.20.643Complicated diabetes mellitus9423.66220.33234.40.005Uncomplicated diabetes mellitus16641.711236.75458.1<0.001Drug abuse287.0258.233.20.101Fluid and electrolyte disorders22456.315149.57378.5<0.001Complicated hypertension12130.48026.24144.10.001Uncomplicated hypertension26666.819162.67580.60.001Hypothyroidism6716.84916.11819.40.458Liver disease6616.64815.71819.40.412Metastatic cancer6616.65016.41617.20.854Obesity15839.711437.44447.30.086Neurological disorders10325.97424.32931.20.182Peripheral vascular disorders7819.66220.31617.20.507Pulmonary/circulation disorder8020.15417.72628.00.031Renal failure13934.98527.95458.1<0.001Solid tumour without metastasis7418.66120.01314.00.191Valvular diseases of the heart4611.63712.199.70.517Weight loss9724.47323.92425.80.713Laboratory studiesAlanine transaminase (ALT)34987.760.0181.426867.351.4136.98187.188.4282.10.258(Initial/Presenting)Aspartate transaminase (AST)34987.767.8131.726867.357.794.98187.1101.2209.60.073Brain natriuretic peptide12731.9300.7808.89323.4296.1843.43436.6313.2717.20.916Serum creatinine (Cr)37895.01.61.929273.41.41.48692.52.22.90.019C-reactive protein26466.311.89.519448.711

26867.351.4136.98187.188.4282.10.258(Initial/Presenting)Aspartate transaminase (AST)34987.767.8131.726867.357.794.98187.1101.2209.60.073Brain natriuretic peptide12731.9300.7808.89323.4296.1843.43436.6313.2717.20.916Serum creatinine (Cr)37895.01.61.929273.41.41.48692.52.22.90.019C-reactive protein26466.311.89.519448.711 .49.27075.313.210.30.174D-dimer24260.83.67.217644.24.07.86671.02.65.10.123Glucose37694.5143.576.828872.4140.075.68894.6154.879.70.115High-sensitivity troponin22556.562.6205.517042.757.5221.05559.178.4148.20.514Total bilirubin34185.70.71.126165.60.71.28086.00.70.50.940White blood cell37494.08.64.829072.98.64.58490.38.75.60.865Procalcitonin25664.32.210.318947.52.511.86772.01.43.90.472Vital signsVentilatory frequency (bpm)36792.221528471.42048389.2236<0.001(Initial/Presenting)Systolic blood pressure (mm Hg)39699.51342230376.11352393100.0131210.194Diastolic blood pressure (mm Hg)39699.5731230376.1741293100.072110.129Heart rate (beats min−1)37093.0871728772.187178389.288180.452Temperature (°C)35589.237.10.628070.437.10.67580.637.20.60.021SpO2 (%)36692.096328371.19638389.2943<0.001SpO2/FiO236692.034511628371.13671078389.2271114<0.001MedicationsHydrocortisone92.382.611.10.379Heparin (s.c.)8721.95819.02931.20.013Heparin (i.v.)5313.34113.41212.90.893Enoxaparin164.0113.655.40.447Tocilizumab369.0227.21415.10.021Remdesivir225.5196.233.20.267Norepinephrine164.072.399.70.002Hydroxychloroquine9223.16822.32425.80.482

92.034511628371.13671078389.2271114<0.001MedicationsHydrocortisone92.382.611.10.379Heparin (s.c.)8721.95819.02931.20.013Heparin (i.v.)5313.34113.41212.90.893Enoxaparin164.0113.655.40.447Tocilizumab369.0227.21415.10.021Remdesivir225.5196.233.20.267Norepinephrine164.072.399.70.002Hydroxychloroquine9223.16822.32425.80.482 Characteristics of patients requiring intubation or dying within 24 h. Laboratory studies and vital signs are presenting or initial values. Note that not all patients have full laboratory results or vital signs within the first 4 h of admission. The medications counts/percentages listed are based upon administration at any point from admission until data collection was censored at either primary outcome or 14 days after admission. COPD, chronic obstructive pulmonary disease; FiO2, fraction of inspired oxygen; sd, standard deviation; SpO2, blood oxygen saturation level .

tions counts/percentages listed are based upon administration at any point from admission until data collection was censored at either primary outcome or 14 days after admission. COPD, chronic obstructive pulmonary disease; FiO2, fraction of inspired oxygen; sd, standard deviation; SpO2, blood oxygen saturation level . The Random Forest algorithm found several variables associated with receipt of mechanical ventilation or death. The variables with the best predictive ability were: (1) current SpO2/FiO2 (Z-score=8.55), (2) previous SpO2/FiO2 (Z=6.25), (3) current ventilatory frequency (Z=5.97), (4) current heart rate (Z=5.87), (5) previous heart rate (Z=5.83), (6) current diastolic blood pressure (Z=5.76), and (7) current blood glucose (Z=5.76) (Supplementary Table S2). Our algorithm is able to predict subsequent ventilation or death with very good discrimination (c-statistic for the 4 h time window=0.885, 95% confidence interval [CI], 0.858–0.924; 8 h window=0.881, 95% CI 0.856–0.906; 24 h window=0.858, 95% CI 0.841–0.874; and 48 h window=0.839, 95% CI 0.825–0.854). The areas under the precision recall curve were 0.038, 0.060, 0.106, and 0.147 at 4, 8, 24, and 48 h prediction windows, respectively. Receiver operator characteristic curves and precision–recall curves for each of our prediction windows are shown in Figure 1. Notably at Youden's point, the sensitivity for the 24 h prediction window was 0.77 and the specificity was 0.80 (compared with sensitivity of 0.84 and specificity of 0.80 for the 4 h prediction window).Fig 1Receiver operator characteristic curves and precision recall curves for each prediction window. (a) Four hour prediction window. (b) Eight hour prediction window. (c) Twenty-four hour prediction window. (d) Forty-eight hour prediction window.Fig 1

vity of 0.84 and specificity of 0.80 for the 4 h prediction window).Fig 1Receiver operator characteristic curves and precision recall curves for each prediction window. (a) Four hour prediction window. (b) Eight hour prediction window. (c) Twenty-four hour prediction window. (d) Forty-eight hour prediction window.Fig 1 Receiver operator characteristic curves and precision recall curves for each prediction window. (a) Four hour prediction window. (b) Eight hour prediction window. (c) Twenty-four hour prediction window. (d) Forty-eight hour prediction window. Next we graphed the maximum predicted score for each patient, along with their receipt of mechanical ventilation or death (Fig 2). By quintiles of machine learning scores, 8.5%, 6.5%, 8.8%, 20.0%, and 31.6% of patients (Fig 2) required mechanical ventilation or died within the subsequent 24 h.Fig 2Maximum predicted score for each patient, along with ventilation requirement or death. Quintile 1, 8.5% died or required mechanical ventilation within 24 h. Quintile 2, 6.5% died or required mechanical ventilation within 24 h. Quintile 3, 8.8% died or required mechanical ventilation within 24 h. Quintile 4, 20.0% died or required mechanical ventilation within 24 h. Quintile 5, 31.6% died or required mechanical ventilation within 24 h.Fig 2

chanical ventilation within 24 h. Quintile 2, 6.5% died or required mechanical ventilation within 24 h. Quintile 3, 8.8% died or required mechanical ventilation within 24 h. Quintile 4, 20.0% died or required mechanical ventilation within 24 h. Quintile 5, 31.6% died or required mechanical ventilation within 24 h.Fig 2 Maximum predicted score for each patient, along with ventilation requirement or death. Quintile 1, 8.5% died or required mechanical ventilation within 24 h. Quintile 2, 6.5% died or required mechanical ventilation within 24 h. Quintile 3, 8.8% died or required mechanical ventilation within 24 h. Quintile 4, 20.0% died or required mechanical ventilation within 24 h. Quintile 5, 31.6% died or required mechanical ventilation within 24 h.

ed mechanical ventilation within 24 h. Quintile 2, 6.5% died or required mechanical ventilation within 24 h. Quintile 3, 8.8% died or required mechanical ventilation within 24 h. Quintile 4, 20.0% died or required mechanical ventilation within 24 h. Quintile 5, 31.6% died or required mechanical ventilation within 24 h. Using GEE, nine features were found to be significantly associated with ventilation or death within 24 h (c-statistic=0.866; 95% CI, 0.863–0.869). The demographic features: age (adjusted odds ratio [aOR]=1.025; 95% CI, 1.008–1.043; P=0.005), male sex (aOR=2.817; 95% CI, 1.582–5.025; P<0.001), and BMI (aOR=1.035; 95% CI, 1.004–1.067; P=0.026) were all associated with mechanical ventilation or death. The laboratory findings of high sensitivity troponin (aOR=1.005; 95% CI, 1.001–1.010; P=0.014) and D-Dimer (aOR=0.983; 95% CI, 0.972–0.994; P=0.002) were also associated with our primary outcome. The vital signs – (1) previous ventilatory frequency (aOR=1.010; 95% CI, 1.003–1.017; P=0.004), (2) current ventilatory frequency (aOR=1.014; 95% CI, 1.007–1.021; P<0.001), (3) previous SpO2/FiO2 (aOR=0.999; 95% CI, 0.998–1.000; P=0.005), and (4) current SpO2/FiO2 (aOR=0.998; 95% CI, 0.998–0.999; P<0.001) – were also associated with our primary outcome. As the prediction window increased from 4 to 48 h, the discrimination remained similar (c-statistic: 4 h time window=0.865, 95% CI 0.862–0.868; 8 h window=0.854, 95% CI 0.850–0.856; 24 h window=0.866, 95% CI 0.863–0.869; 48 h window=0.840, 95% CI 0.837–0.843); and an increasing number of variables were selected (4 h: four significant variables, 8 h: five variables, 24 h: nine variables, 48 h: 11 variables). Sex, high-sensitivity troponin, previous ventilatory frequency, current ventilatory frequency, and previous SpO2/FiO2 and SpO2/FiO2 occurred consistently across multiple prediction windows. The full results of the GEE for each of the prediction windows can be seen in Supplementary Table S3.

The Random Forest algorithm found several variables associated with receipt of mechanical ventilation or death. The variables with the best predictive ability were: (1) current SpO2/FiO2 (Z-score=8.55), (2) previous SpO2/FiO2 (Z=6.25), (3) current ventilatory frequency (Z=5.97), (4) current heart rate (Z=5.87), (5) previous heart rate (Z=5.83), (6) current diastolic blood pressure (Z=5.76), and (7) current blood glucose (Z=5.76) (Supplementary Table S2). Our algorithm is able to predict subsequent ventilation or death with very good discrimination (c-statistic for the 4 h time window=0.885, 95% confidence interval [CI], 0.858–0.924; 8 h window=0.881, 95% CI 0.856–0.906; 24 h window=0.858, 95% CI 0.841–0.874; and 48 h window=0.839, 95% CI 0.825–0.854). The areas under the precision recall curve were 0.038, 0.060, 0.106, and 0.147 at 4, 8, 24, and 48 h prediction windows, respectively. Receiver operator characteristic curves and precision–recall curves for each of our prediction windows are shown in Figure 1. Notably at Youden's point, the sensitivity for the 24 h prediction window was 0.77 and the specificity was 0.80 (compared with sensitivity of 0.84 and specificity of 0.80 for the 4 h prediction window).Fig 1Receiver operator characteristic curves and precision recall curves for each prediction window. (a) Four hour prediction window. (b) Eight hour prediction window. (c) Twenty-four hour prediction window. (d) Forty-eight hour prediction window.Fig 1

Using GEE, nine features were found to be significantly associated with ventilation or death within 24 h (c-statistic=0.866; 95% CI, 0.863–0.869). The demographic features: age (adjusted odds ratio [aOR]=1.025; 95% CI, 1.008–1.043; P=0.005), male sex (aOR=2.817; 95% CI, 1.582–5.025; P<0.001), and BMI (aOR=1.035; 95% CI, 1.004–1.067; P=0.026) were all associated with mechanical ventilation or death. The laboratory findings of high sensitivity troponin (aOR=1.005; 95% CI, 1.001–1.010; P=0.014) and D-Dimer (aOR=0.983; 95% CI, 0.972–0.994; P=0.002) were also associated with our primary outcome. The vital signs – (1) previous ventilatory frequency (aOR=1.010; 95% CI, 1.003–1.017; P=0.004), (2) current ventilatory frequency (aOR=1.014; 95% CI, 1.007–1.021; P<0.001), (3) previous SpO2/FiO2 (aOR=0.999; 95% CI, 0.998–1.000; P=0.005), and (4) current SpO2/FiO2 (aOR=0.998; 95% CI, 0.998–0.999; P<0.001) – were also associated with our primary outcome. As the prediction window increased from 4 to 48 h, the discrimination remained similar (c-statistic: 4 h time window=0.865, 95% CI 0.862–0.868; 8 h window=0.854, 95% CI 0.850–0.856; 24 h window=0.866, 95% CI 0.863–0.869; 48 h window=0.840, 95% CI 0.837–0.843); and an increasing number of variables were selected (4 h: four significant variables, 8 h: five variables, 24 h: nine variables, 48 h: 11 variables). Sex, high-sensitivity troponin, previous ventilatory frequency, current ventilatory frequency, and previous SpO2/FiO2 and SpO2/FiO2 occurred consistently across multiple prediction windows. The full results of the GEE for each of the prediction windows can be seen in Supplementary Table S3.

In the setting of COVID-19, the Random Forest algorithm is able to predict ventilation or death with high sensitivity (0.77) and specificity (0.80). Furthermore, we have very good discrimination (c-statistic=0.858; 95% CI, 0.841–0.874) for predicting our primary target (24 h prediction window), which improves as our prediction window narrows (4 h window, c-statistic=0.885; 95% CI, 0.858–0.924). [Interpretation of the c-statistic: 0.5–0.6 for a poor model, 0.6–0.7 for a good model, 0.8–0.9 for a very good model, and 0.9–1.0 for an excellent model.] Of the 10 features with the highest predictive value, nine are vital signs. By capturing the clinical trajectory, these dynamic features enable greater predictive utility to detect changes through the course of a hospitalisation. We have selected a list which can be easily and automatically extracted for potential integration into a clinical support system.21 In addition, we demonstrate consistent significance of key features (age, sex, BMI, high sensitivity troponin, blood glucose, SpO2/FiO2, and ventilatory frequency) across two independent modelling methodologies (Random Forest and GEE) and multiple prediction windows (4, 8, 24, and 48 h). This suggests a robust signal that can be leveraged for prediction of mechanical ventilation.

tures (age, sex, BMI, high sensitivity troponin, blood glucose, SpO2/FiO2, and ventilatory frequency) across two independent modelling methodologies (Random Forest and GEE) and multiple prediction windows (4, 8, 24, and 48 h). This suggests a robust signal that can be leveraged for prediction of mechanical ventilation. Our highest utility predictor, SpO2/FiO2, has been used as a proxy for Pao2/FiO2 – which occurs in the diagnosis and grading of acute respiratory distress syndrome.22,23 As SpO2/FiO2 can be easily calculated, without the need for arterial blood draw and can be used to monitor continuously, this may represent a promising metric to assess for respiratory deterioration in general care patients, not just patients with COVID-19. Similar to other studies, we found that older,7 heavier,24 or male25 patients are more likely to require mechanical ventilation. Although other studies have found associations between renal failure, congestive heart failure, hypertension, diabetes, and cardiac arrhythmias critical illness or death,7,10,26 we found these to have only small utility in the machine learning algorithm and not associated with outcome in the GEE. Our lack of finding these previously reported associations may be attributable to different patient populations, different clinical practices, or to our more comprehensive list of potential factors. Both C-reactive protein24 and aspartate aminotransferase (AST)27 – which we have identified in our Random Forest model – have also been included in previous severity models. Tachypnoea is a well characterised clinical sign of respiratory decompensation.7 The discrimination of our ventilation model was also similar that reported in a critical illness model (c-statistic=0.88).10

ST)27 – which we have identified in our Random Forest model – have also been included in previous severity models. Tachypnoea is a well characterised clinical sign of respiratory decompensation.7 The discrimination of our ventilation model was also similar that reported in a critical illness model (c-statistic=0.88).10 Our algorithm can be integrated into a clinical support software with the ultimate goal of identifying patients before clinical decompensation.21 Our primary target (24 h prediction window) was selected to allow appropriate time for interventions, while still providing evidence of deterioration in dynamic features. The advantages of identifying potential respiratory failure 24 or 48 h in advance, include: (1) enrolment in clinical trials, (2) aggressive therapeutic interventions such as prone positioning or noninvasive mechanical ventilation, and (3) planning for appropriate ventilator allocation and utilisation. To identify the prediction window that optimises the trade-off between detection and potential intervention, we also quantified discrimination at 4, 8, and 48 h prediction windows. In our Random Forest model, we have the greatest discrimination to predict within 4 h (c-statistic=0.885; 95% CI, 0.858–0.924) and the lowest, but still very good, discrimination when predicting within 48 h (c-statistic=0.839; 95% CI, 0.825–0.854). This is expected because evidence of the imminent respiratory failure has likely started to manifest, improving the ability to predict, but a 4 h prediction window unfortunately allows the least opportunity for meaningful intervention. We have shown high discrimination (for the Random Forest Model) at 24 h. This can inform when the model is most useful. However, the utility of the model must account for both discrimination of the model and clinical actionability. In addition to the high discrimination, 24 h notice also allows the clinician an opportunity to make modifications in clinical care and preparation in resources for potential decompensation.

l is most useful. However, the utility of the model must account for both discrimination of the model and clinical actionability. In addition to the high discrimination, 24 h notice also allows the clinician an opportunity to make modifications in clinical care and preparation in resources for potential decompensation. The Random Forest model has a sensitivity of 0.77 and a specificity of 0.80. Determination of the optimal identification threshold should weigh the risk of falsely identifying a patient as at risk for mechanical ventilation (increased monitoring and resource utilisation, aggressive intervention) vs failing to identify a patient who is susceptible to future deterioration (missed opportunity to alter clinical trajectory and a delay in recognising the need for increased acuity of care). The number of patients needed to identify (NNI) is 3.2 for the highest quintile and 5.0 for the second highest quintile, which are reasonable numbers that limit false positives while identifying patients in need of life-saving, but invasive, therapy.

ry and a delay in recognising the need for increased acuity of care). The number of patients needed to identify (NNI) is 3.2 for the highest quintile and 5.0 for the second highest quintile, which are reasonable numbers that limit false positives while identifying patients in need of life-saving, but invasive, therapy. For additional insight into patient characteristics our algorithm is likely to misclassify, we reviewed the patients with the lowest predictive score, who ultimately required mechanical ventilation within 24 h (‘false negatives’), and patients with high predictive scores who never required ventilation (‘false positives’). Patients the algorithm failed to identify were disproportionately missing data for highly predictive features, such as Pao2/FiO2, ventilatory frequency, heart rate, and SpO2. Specifically, seven of the 10 patients with the lowest predicted scores, who received mechanical ventilation within 24 h (i.e. the false negatives), were found to be missing data for key features. Our algorithm was programmed to overcome this pitfall, by propagating values from the previous time window, when no new values are recorded. Therefore, these false negative cases skew early in their hospital course, where no prior values are recorded and missing values are imputed to population mean. As with any predictive metric, our algorithm is inherently limited by the quality of data recorded. Furthermore, the absence of regularly recorded vital signs may be associated with unrecognised decompensation, because of lower prioritisation of medical documentation in an emergency situation or as a reflection of the medical care team's attentiveness. Because of inherent limitations secondary to incomplete data, we have characterised missing data in Supplementary Table S4. Static variables (e.g. age, height, weight, and comorbidities such as chronic pulmonary disease) have no missing values across our dataset. This contrasts with dynamic variables which have some missing values. For laboratory values and vital signs, this likely reflects how often they were clinically indicated. For example, SpO2, which is missing in 47% of our 4 h prediction windows, may be typically checked less frequently than every 4 h in a stable, general care patient; however, we do not have the reasons why SpO2 was not recorded. Future studies may benefit from including absence or presence of a value as part of the algorithm.

ample, SpO2, which is missing in 47% of our 4 h prediction windows, may be typically checked less frequently than every 4 h in a stable, general care patient; however, we do not have the reasons why SpO2 was not recorded. Future studies may benefit from including absence or presence of a value as part of the algorithm. We also reviewed the patients with the highest predictive scores who did not require ventilation within 24 h (‘false positive’). Five of the 10 patients with the highest predictive scores ultimately required mechanical ventilation during their hospital course, suggesting our algorithm was successful in detecting future respiratory decline, but not within the pre-specified prediction window. To assess the utility of our predictions on a patient level, we quantified the percentage of patients in each risk quintile requiring ventilation or dying within 24 h of their maximum risk score (Fig 2). Patients in risk quintiles 1, 2, and 3 had an 8.5%, 6.5%, and 8.8% risk, respectively. This compares with 20.0% risk in the fourth quintile and 31.6% risk for a patient in the fifth quintile. Even though a patient in the highest risk quintile still has less than a 1 in 3 chance of requiring mechanical ventilation within the next 24 h, the clinical provider may decide that because of the high mortality in patients requiring mechanical ventilation, the increased patient risk (31.6% compared with <10% in the three lowest risk quintiles or 15.1% in our overall cohort) merits closer attention or more aggressive care.

cal ventilation within the next 24 h, the clinical provider may decide that because of the high mortality in patients requiring mechanical ventilation, the increased patient risk (31.6% compared with <10% in the three lowest risk quintiles or 15.1% in our overall cohort) merits closer attention or more aggressive care. In our highest risk cohort, the NNI a single new case of mechanical ventilation was 3.2, and for our second risk quintile the NNI was 5.0. This means that for every three patients our algorithm identifies as being in the highest risk group (or five in the second quintile), we will correctly detect one new case requiring mechanical ventilation in the next 24 h. Given the high mortality associated with mechanical ventilation,7, 8, 9 an NNI <11 may be considered reasonable, particularly if the intervention is low-risk or low-cost. The intervention may be as low-risk and low-cost as using continuous monitoring with SpO2 rather than intermittent monitoring, thus detecting a decrease in the SpO2/FiO2 ratio, our strongest indicator of risk for mechanical ventilation or death. If desired, the desired threshold can be adjusted up or down based on type of intervention and availability of resources.

t as using continuous monitoring with SpO2 rather than intermittent monitoring, thus detecting a decrease in the SpO2/FiO2 ratio, our strongest indicator of risk for mechanical ventilation or death. If desired, the desired threshold can be adjusted up or down based on type of intervention and availability of resources. The Random Forest identified initiation of intravenous heparin (Z-score=1.60) and hydroxychloroquine (1.37) in the algorithm. Other pharmacologic agents, such as tocilizumab (0.85), remdesivir (0.15), and hydrocortisone (0), had very low association. No pharmacologic agents were selected in the GEE models. Potential reasons include inadequate statistical power, differences in patient population, or a reflection of pharmacologic utility. High-sensitivity troponin was included in the Random Forest Model (4.59) and was selected in multiple GEE models. Although the mechanism of respiratory deterioration remains unresolved, the association between myocardial injury, myocarditis, myocardial infarction, and thromboembolic events has been previously described and merits further study and incorporation into predictive models.28

59) and was selected in multiple GEE models. Although the mechanism of respiratory deterioration remains unresolved, the association between myocardial injury, myocarditis, myocardial infarction, and thromboembolic events has been previously described and merits further study and incorporation into predictive models.28 Our study used two very dissimilar techniques (Random Forest and GEE) for analysing the data and found similar discrimination and similar factors being associated with mechanical ventilation and death. Our study possessed several limitations. First, we were unable to account for all predictive features that may contribute to pending respiratory failure. In our study, we included some features, such as SpO2/FiO2, which had not been previously characterised in the progression of COVID-19, and included basic relationships between features (change in values); however, other features and more complex relationships were potentially missed by our methodology. The lack of institutional criteria for intubation also introduces heterogeneity in our primary outcome, although the variability in provider practice likely also increases the generalisability of our model.

ge in values); however, other features and more complex relationships were potentially missed by our methodology. The lack of institutional criteria for intubation also introduces heterogeneity in our primary outcome, although the variability in provider practice likely also increases the generalisability of our model. Additional limitations to our study include those inherent to our single-centre, observational study design: our conclusions require prospective multicentre validation. We also failed to explore the causal relationship between our predictive features and the outcome. In addition, the model's positive predictive value is a function of outcome incidence. As the pandemic has progressed, the fraction of infected individuals who require mechanical ventilation or die has decreased.29 This means the positive predictive value will be lower and the NNI will be higher if the model were applied to the current, less critically ill patient population as compared with the patients in our dataset. Overfitting was another potential concern. This was addressed through our selection of generalised linear modelling, which adjusts standard error estimates by an estimated overfitting parameter. To mitigate this potential issue within our Random Forest model, cross-validation was independent, with all time points corresponding to a single patient restricted to the same fold.

ssed through our selection of generalised linear modelling, which adjusts standard error estimates by an estimated overfitting parameter. To mitigate this potential issue within our Random Forest model, cross-validation was independent, with all time points corresponding to a single patient restricted to the same fold. Although we demonstrated that tachypnoea, hypotension, and hypoxia are associated with impending respiratory decline, we do not address whether addressing these homeostatic imbalances through vasopressors or supplemental oxygen mitigate progression of respiratory decline. A final limitation is lack of external validation of our models. To mitigate this intrinsic issue, independent cross-validation was performed. Randomly dividing all the time points to different folds would result in time points from the same patient in many different folds. We would like to estimate how well the model generalises to completely independent samples. To ensure a conservative estimate of how well the model generalises during cross-validation, we have ensured that all time points from the same patient are restricted to the same fold.

m the same patient in many different folds. We would like to estimate how well the model generalises to completely independent samples. To ensure a conservative estimate of how well the model generalises during cross-validation, we have ensured that all time points from the same patient are restricted to the same fold. Another limitation of this study is that the rapidly evolving understanding of COVID-19 and advances in clinical management, necessitate re-calibration of the machine learning model at regular time intervals. This is an important consideration when applying this model to new data and an additional limitation of this study. For example, even though hydroxychloroquine was associated with the outcome in the Random Forest Model, that association probably does not hold today because of evolving practice patterns.30

Our highest utility predictor, SpO2/FiO2, has been used as a proxy for Pao2/FiO2 – which occurs in the diagnosis and grading of acute respiratory distress syndrome.22,23 As SpO2/FiO2 can be easily calculated, without the need for arterial blood draw and can be used to monitor continuously, this may represent a promising metric to assess for respiratory deterioration in general care patients, not just patients with COVID-19. Similar to other studies, we found that older,7 heavier,24 or male25 patients are more likely to require mechanical ventilation. Although other studies have found associations between renal failure, congestive heart failure, hypertension, diabetes, and cardiac arrhythmias critical illness or death,7,10,26 we found these to have only small utility in the machine learning algorithm and not associated with outcome in the GEE. Our lack of finding these previously reported associations may be attributable to different patient populations, different clinical practices, or to our more comprehensive list of potential factors. Both C-reactive protein24 and aspartate aminotransferase (AST)27 – which we have identified in our Random Forest model – have also been included in previous severity models. Tachypnoea is a well characterised clinical sign of respiratory decompensation.7 The discrimination of our ventilation model was also similar that reported in a critical illness model (c-statistic=0.88).10

Our algorithm can be integrated into a clinical support software with the ultimate goal of identifying patients before clinical decompensation.21 Our primary target (24 h prediction window) was selected to allow appropriate time for interventions, while still providing evidence of deterioration in dynamic features. The advantages of identifying potential respiratory failure 24 or 48 h in advance, include: (1) enrolment in clinical trials, (2) aggressive therapeutic interventions such as prone positioning or noninvasive mechanical ventilation, and (3) planning for appropriate ventilator allocation and utilisation. To identify the prediction window that optimises the trade-off between detection and potential intervention, we also quantified discrimination at 4, 8, and 48 h prediction windows. In our Random Forest model, we have the greatest discrimination to predict within 4 h (c-statistic=0.885; 95% CI, 0.858–0.924) and the lowest, but still very good, discrimination when predicting within 48 h (c-statistic=0.839; 95% CI, 0.825–0.854). This is expected because evidence of the imminent respiratory failure has likely started to manifest, improving the ability to predict, but a 4 h prediction window unfortunately allows the least opportunity for meaningful intervention. We have shown high discrimination (for the Random Forest Model) at 24 h. This can inform when the model is most useful. However, the utility of the model must account for both discrimination of the model and clinical actionability. In addition to the high discrimination, 24 h notice also allows the clinician an opportunity to make modifications in clinical care and preparation in resources for potential decompensation.

For additional insight into patient characteristics our algorithm is likely to misclassify, we reviewed the patients with the lowest predictive score, who ultimately required mechanical ventilation within 24 h (‘false negatives’), and patients with high predictive scores who never required ventilation (‘false positives’). Patients the algorithm failed to identify were disproportionately missing data for highly predictive features, such as Pao2/FiO2, ventilatory frequency, heart rate, and SpO2. Specifically, seven of the 10 patients with the lowest predicted scores, who received mechanical ventilation within 24 h (i.e. the false negatives), were found to be missing data for key features. Our algorithm was programmed to overcome this pitfall, by propagating values from the previous time window, when no new values are recorded. Therefore, these false negative cases skew early in their hospital course, where no prior values are recorded and missing values are imputed to population mean. As with any predictive metric, our algorithm is inherently limited by the quality of data recorded. Furthermore, the absence of regularly recorded vital signs may be associated with unrecognised decompensation, because of lower prioritisation of medical documentation in an emergency situation or as a reflection of the medical care team's attentiveness. Because of inherent limitations secondary to incomplete data, we have characterised missing data in Supplementary Table S4. Static variables (e.g. age, height, weight, and comorbidities such as chronic pulmonary disease) have no missing values across our dataset. This contrasts with dynamic variables which have some missing values. For laboratory values and vital signs, this likely reflects how often they were clinically indicated. For example, SpO2, which is missing in 47% of our 4 h prediction windows, may be typically checked less frequently than every 4 h in a stable, general care patient; however, we do not have the reasons why SpO2 was not recorded. Future studies may benefit from including absence or presence of a value as part of the algorithm.

Our study used two very dissimilar techniques (Random Forest and GEE) for analysing the data and found similar discrimination and similar factors being associated with mechanical ventilation and death. Our study possessed several limitations. First, we were unable to account for all predictive features that may contribute to pending respiratory failure. In our study, we included some features, such as SpO2/FiO2, which had not been previously characterised in the progression of COVID-19, and included basic relationships between features (change in values); however, other features and more complex relationships were potentially missed by our methodology. The lack of institutional criteria for intubation also introduces heterogeneity in our primary outcome, although the variability in provider practice likely also increases the generalisability of our model. Additional limitations to our study include those inherent to our single-centre, observational study design: our conclusions require prospective multicentre validation. We also failed to explore the causal relationship between our predictive features and the outcome. In addition, the model's positive predictive value is a function of outcome incidence. As the pandemic has progressed, the fraction of infected individuals who require mechanical ventilation or die has decreased.29 This means the positive predictive value will be lower and the NNI will be higher if the model were applied to the current, less critically ill patient population as compared with the patients in our dataset.

has progressed, the fraction of infected individuals who require mechanical ventilation or die has decreased.29 This means the positive predictive value will be lower and the NNI will be higher if the model were applied to the current, less critically ill patient population as compared with the patients in our dataset. Overfitting was another potential concern. This was addressed through our selection of generalised linear modelling, which adjusts standard error estimates by an estimated overfitting parameter. To mitigate this potential issue within our Random Forest model, cross-validation was independent, with all time points corresponding to a single patient restricted to the same fold. Although we demonstrated that tachypnoea, hypotension, and hypoxia are associated with impending respiratory decline, we do not address whether addressing these homeostatic imbalances through vasopressors or supplemental oxygen mitigate progression of respiratory decline. A final limitation is lack of external validation of our models. To mitigate this intrinsic issue, independent cross-validation was performed. Randomly dividing all the time points to different folds would result in time points from the same patient in many different folds. We would like to estimate how well the model generalises to completely independent samples. To ensure a conservative estimate of how well the model generalises during cross-validation, we have ensured that all time points from the same patient are restricted to the same fold.

A Random Forest Machine learning approach and a GEE approach, using demographic data, vital signs, medication records, laboratory studies, and medical comorbidities can be leveraged to predict which patients with COVID-19 are likely to require mechanical ventilation. Of the 10 features with highest predictive value, nine are vital signs. SpO2/FiO2 can be easily estimated and monitored continuously, providing a promising metric to assess for respiratory collapse in patients with COVID-19. Future studies will (1) validate the algorithm on a larger number of patients across additional healthcare systems, (2) integrate the complexity of the model within clinician workflow, and (3) assess if clinical features identified by the algorithm may provide targets for medical intervention to alter the clinical course.

Study conception: NJD, CBD, MCE Study design: NJD, CBD, MRM, CP, KKT, MCE Data interpretation: NJD, CBD, GM, MRM, CP, KKT, MCE Data analysis (Random Forest Model): CBD Data analysis (logistic regression and GEE models): GM Developing the initial and final drafts of the manuscript: NJD, MCE Assimilation of intellectual content from all co-authors: NJD, MCE Critical revision of the work for important intellectual content: CBD, GM, MRM, CP, KKT