Development and validation of a clinical-friendly model for predicting 180-day mortality in the older with community-acquired pneumonia

Abstract

Objective

Currently, there is no effective way to identify older patients with community-acquired pneumonia (CAP) at high risk of long-term death. We aimed to develop and validate a pneumonia scoring system to predict 180-day mortality, and compare its performance with the commonly used CURB-65 score.

Methods

The prospective cohort study enrolled patients aged 65 years and older with CAP from 10 medical centers in China between April 2021 and December 2023. The primary outcome was 180-day mortality. A Cox proportional hazards model was used to develop a new pneumonia scoring system, and the area under the time-dependent curve (AUC) was used to assess its discriminatory power. Internal validation was performed using both bootstrap resampling and 10-fold cross-validation. The model was visualized by a nomogram and a questionnaire. The optimal cutoff value of the nomogram was determined based on the maximum Youden index for the 180-day mortality prediction, dividing patients into high- and low-risk groups. The performance of model in predicting both short- and long-term mortality was compared with CURB-65 using AUC, sensitivity, specificity, negative predictive value and positive predictive value.

Results

A total of 619 patients, with a median age of 78 years (IQR: 70.5–85.0), were included in the analysis. The 180-day mortality was 6.9%. The model was developed using six variables, including age, the ratio of pulse oximetry saturation (SpO₂) to the fraction of inspired oxygen (FiO₂), loneliness, Barthel index, Clinical Frailty Scale and malnutrition. The AUC of the model for predicting 180-day, 90-day and 30-day mortality were 0.829, 0.832 and 0.904, respectively. The cut-off value for the model was 142, while it was 2 for CURB-65. Using the cut-off values, the AUC of the model for predicting 180-day mortality was 0.768 (95% confidence interval [CI]: 0.695–0.842), significantly higher than that of CURB-65(AUC: 0.573, 95%CI: 0.488–0.659). A similar trend was observed for predictions of 90-day, 30-day and in-hospital mortality.

Conclusions

This study developed and validated a prediction model for long-term mortality in older patients with CAP, showing better discriminatory power and accuracy over CURB-65.

Introduction

Community-acquired pneumonia (CAP) remains a major cause of morbidity and mortality worldwide [1, 2], with a high incidence and mortality rate in the older. Currently, there is a growing recognition that CAP is not only an acute illness but also has an impact on long-term outcomes [1]. Particularly with the global epidemic of Covid-19 in recent years, the long-term effects of acute infection have received more attention and research, and have been termed post-acute infection syndromes (PAIS) [2]. In addition, CAP is associated with long-term death, with death at 1 year occurring in approximately 30% of all hospitalized patients [1]. It is important to identify early at high risk of long-term death, which is the first step to reduce the long-term mortality rate.

Several pneumonia severity scoring systems have been developed, such as CURB-65 [3], CRB-65, Pneumonia Severity Index (PSI) [4], A-DROP [5], and SMART-COP [6], but they have been developed to predict adverse outcomes in the acute phase, including mortality, the need for intensive care unit or mechanical ventilation. The performance of current systems in predicting long-term mortality was unclear. CRB-65 and CURB-65 are easy to calculate and widely used, but it has been reported that these two systems may underestimate the risk of mortality in the older. Other systems, such as the PSI, A-DROP and SMART-COP scores, require invasive tests and imaging, making them less practical for routine clinical use. These limitations have highlighted the need for a simpler and more effective tool to predict the long-term mortality in the older with CAP.

To address this research gap, the aim of this study was to develop and validate a clinically applicable tool for predicting 180-day mortality in the older patients with CAP. In addition, the performance of the new tool was compared with the CURB-65 risk score.

Methods

Study design and participants

The study was reported according to the Transparent Reporting of multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) guidelines [7]. The checklist for this article is provided in Supplementary file 1.

Participants

The study was a multi-center prospective study which was conducted at 10 hospitals in China between April 2021 and December 2023 (Supplementary file 2 Table S1). Patients aged 65 years and older with CAP were consecutively enrolled in the study. The diagnosis of CAP was established in accordance with the guidelines of the American Thoracic Society and the Infectious Diseases Society of America (ATS–IDSA) [8]. Patients were excluded if they had a totally dependent functional status, deafness, severe cognitive impairment, advanced malignancy or an expected survival of less than 1 year. Written informed consent was obtained from all participants or their family members before the survey.

Data collection

At the time of admission, data on demographic characteristics, including age, sex, history of alcohol consumption, and cigarette use, as well as vital signs, such as blood pressure, heart rate, respiratory rate, peripheral oxygen saturation (SpO₂), and fraction of inspired oxygen (FiO₂) and laboratory tests were collected. Additionally, geriatric assessments were performed.

Geriatric assessments

Before patients were enrolled in the study, two qualified physicians determined the eligibility according to the inclusion and exclusion criteria at the time of admission. A trained nurse then started the geriatric assessment.

Comorbidity was assessed by self-reported and physician-recorded conditions. The Charlson comorbidity Index (CCI) was used to quantify the comorbidity burden [9].

Physical function was reflected by the basic activities of daily living (BADL), which was assessed using the Barthel Index (BI) [10]. The BI measured several activities of daily living, including feeding, bathing, grooming, dressing, bowel and bladder control, toileting, transferring from bed to chair and vice versa, walking on level surfaces, and climbing stairs. Higher scores indicated better functional status.

Loneliness was defined as self-reported feelings of loneliness.

Frailty was assessed using the validated Clinical Frailty Scale (CFS), a 9-point item tool that evaluates specific domains including comorbidity, functional status, and cognition, resulting in a frailty score ranging from 1 (very fit) to 9 (terminally ill), which has been well validated in previous studies. A score of ≥ 5 is considered indicative of frailty [11, 12].

Malnutrition was defined as a body mass index (BMI) < 18.5 kg/m² or a weight loss of 5 kg in the previous three months according to the recommendations of the European Society of Clinical Nutrition and Metabolism (ESPEN) [13].

Follow up

Events during hospitalization were extracted from the electronic medical records. The medical staff in our study contacted the patients or their family members by phone at 30, 90, and 180 days after discharge to collect their survival status. Patients were considered lost to follow-up if they had no in-hospital assessments, no medical records and no response after three consecutive phone call attempts during the follow-up period.

Outcomes

The primary outcome was the mortality rate within 180 days following disease onset (180-day mortality). Secondary outcomes included mortality rates at 90 days, 30 days and during hospitalization.

Sample size calculation

In our study, the 180-day mortality rate of CAP was 0.069. Assuming 10 candidate predictor parameters and using a conservative estimate of 15% of the maximum Cox-Snell R², we determined that the minimum sample size required to fit the regression models was 549, with 38 events, resulting in approximately 3.79 events per predictor parameter.

Statistical analysis

Patients lost to follow-up (11/630, 1.7%) were excluded from the analysis. Given the low proportion of loss to follow-up, their exclusion is unlikely to significantly affect the study results and complete-case analysis was recommended [14].

Continuous variables were presented as median (interquartile range), whereas categorical variables were expressed as absolute values (percentages), respectively. For comparisons between survivors and non-survivors, the Mann-Whitney U test was used for continuous variables, and the χ2 test or Fisher’s exact test was used for categorical variables, as appropriate.

Through literature review, clinical input and application for clinical implementation, potential predictors were shortlisted in univariable Cox regression excluding laboratory test results [12, 15,16,17]. For clinical applicability and interpretability, the optimal SpO2/FiO2 ratio cut-off was based on the maximum Youden index in predicting 180-day mortality prediction, which was stratified into two categories: <450 and ≥ 450 (Supplementary file 2, Figure S2). Variables with a P value < 0.05 in the univariate analysis were included in the full model and subsequently selected using a bidirectional stepwise approach to derive the model with the lowest Akaike information criterion. Finally, we assessed the performance of the model by examining its discrimination and calibration. Discrimination was assessed using ROC curves and quantified by the area under the time-dependent ROC curve (AUC). An AUC value greater than 0.7 indicates a reasonable level of predictive accuracy. Internal validation was conducted using bootstrap resampling with 500 repetitions and ten-fold cross-validation. Calibration was evaluated using calibration curves. Survival curves were generated using the Kaplan-Meier method and compared with the log-rank test. The performance of the nomogram and CURB-65 was compared in terms of AUC, sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV). All statistical analyses were conducted using R software (version 4.2.2), with a two-tailed P value < 0.05 considered statistically significant.

Result

Characteristics of study population

After excluding 11 patients who were lost to follow-up, a total of 619 patients were included in the final analysis (Supplementary file 2, Figure S1), and their baseline demographic and clinical characteristics are shown in Table 1. The median age was 78.0 years (IQR: 70.5–85.0) and 65.9% (408/619) were male. The mortality rates at 30 days, 90 days, and 180 days were 0.6%, 4.0% and 6.9%, respectively (Fig. 1D). Compared to survivors, non-survivors at 180 days had a higher median age (86.0 vs. 77.0 years, p < 0.001), a higher proportion with SpO₂/FiO₂ ratio below 450 (69.8% vs. 42.9%, p < 0.001), and a significantly higher prevalence of frailty (69.8% vs. 34.4%, p < 0.001), self-reported loneliness (26.2% vs. 11.8%, p = 0.007), malnutrition (32.6% vs. 12.7%, p < 0.001) and higher CCI scores. Non-survivors also had lower BI scores compared to survivors. Additionally, survivors had a higher median albumin level on admission than non-survivors (34.0 g/L vs. 37.0 g/L, p = 0.002). There was no difference in the distribution of CURB-65 scores between the two groups.

Fig. 1

Visualization and validation of the model. (A) The nomogram of the mortality prediction model; (B) The ROC curve of the model; (C) The calibration plot of the model; (D) The Kaplan‒Meier curve for 180-day mortality demonstrating significant differences between the low-risk and high-risk groups

Table 1 Baseline characteristics of the older with CAP

Development, validation and visualization of the model

Six predictors were identified through stepwise multivariate Cox regression analysis, including age, self-reported loneliness, BI score, SpO2/FiO2 ratio, CFS and malnutrition. Among these variables, age and BI score were analyzed as continuous variables. The hazard ratios (HRs) for the predictors are presented in Table 2.

Table 2 Cox regression analysis for 180-day mortality. Values are hazard ratios and 95% confidence intervals

The nomogram for predicting mortality with the older patients with CAP was constructed based on the multivariate Cox regression analysis (Fig. 1A). Each variable in the nomogram is assigned a score based on its hazard ratio, reflecting its contribution to mortality prediction. The scores for the six variables are summed to calculate a total score, which is then converted into an estimated mortality probability based on the patient’s clinical profile. To improve clinical utility, the model was also translated into a questionnaire format following the same principles of the nomogram (Supplementary file 2, Table S3). Patients were stratified into high-risk and low-risk groups using an optimal cut-off value of 142, determined by the maximum Youden index. Kaplan-Meier survival analysis demonstrated a significantly poorer survival probability in the high-risk group compared to the low-risk group (Fig. 1D).

The model demonstrated well predictive performance, with AUC values of 0.829 (95% CI: 0.776–0.882) for 180-day mortality, 0.832 (95%CI: 0.765–0.899) for 90-day mortality and 0.904 (95%CI: 0.799–1.008) for 30-day mortality (Fig. 1B). For internal validation, the model was assessed using tenfold cross-validation and bootstrapping methods. The tenfold cross-validation yielded an average AUC of 0.830 in the training set and 0.818 in the test set for 180-day mortality prediction. Bootstrap resampling analysis produced an optimism-corrected AUC of 0.807. These results across both validation methods confirmed the robust internal validity and reliable predictive performance of the model. Furthermore, calibration plots demonstrated that the predicted probabilities aligned with the actual probabilities (Fig. 1C). However, based on the characteristics of the data set, the model is applicable to older patients with CAP aged 65 to 105 years.

The performance comparison between nomogram and CURB-65

The performance comparison between the nomogram and CURB-65 in predicting mortality is presented in Table 3. Based on the Youden index, the optimal cut-off value of CURB-65 for predicting 180-day mortality was 2. The AUC of the nomogram for predicting 180-day mortality was 0.768(95%CI: 0.695–0.842), higher than that of CURB-65(AUC: 0.573, 95% CI: 0.488–0.659). The sensitivity of the nomogram for 180-day mortality was also higher than that of CURB-65, while both tools showed good performance in NPV, with scores above 0.9. The trend was also in 90-day mortality, 30-day mortality and in-hospital mortality.

Table 3 Performance comparison between nomogram and CURB-65

In summary, the nomogram demonstrates high sensitivity, making it particularly effective for early screening of high-risk patients. Additionally, the model’s high NPV makes it useful for excluding low-risk patients. Compared to CURB-65, it has lower specificity, which may lead to a higher likelihood of misdiagnosis. Combining the nomogram with CURB-65 may improve diagnostic accuracy and optimize clinical decision-making.

Discussion

The study developed a non-invasive model to identify the older patients with CAP at high risk of long-term death. The final predictive model incorporated six non-invasive variables, making it practical for clinical use. Additionally, the model demonstrated strong discriminatory power, with AUCs for 180-day, 90-day, and 30-day mortality all exceeding 0.8. Compared to CURB-65, the model not only showed higher discriminatory power but also improved sensitivity, NPV and PPV.

The model includes specific predictors of geriatric conditions, such as frailty, loneliness and nutritional status, alongside common mortality risk factors from previous pneumonia severity scoring systems, such as age and oxygenation assessment. Among these, frailty has gained increasing attention in various geriatric diseases. Previous studies have identified frailty as a risk factor for hospitalization, rapid disease progression and poor outcomes in the older patients with respiratory infections [18, 19]. These findings suggested that frailty assessment should be a routine part of the management of older adults with CAP, although it has not yet received sufficient attention in clinical practice in the area of infection. In addition, the prevalence of loneliness in our study was significantly higher among non-survivors, highlighting the importance of psychosocial factors in the prognosis of the older patients with CAP. Although research on the impact of loneliness on pneumonia outcomes in the older is lacking, existing evidence consistently links loneliness with various adverse outcomes [20].A meta-analysis found that both actual and perceived social isolation (self-reported loneliness) were associated with an increased risk of early mortality, with loneliness being linked to a 26% higher risk [15]. Another study found that people with the highest levels of loneliness were 30% more likely to be diagnosed with cardiovascular diseases(CVD) and 48% more likely to be hospitalized for CVD than those with the lowest levels of loneliness [21]. These findings have emphasized the importance of assessing loneliness in older patients, a factor that has often been overlooked in the context of infections.

The prediction model demonstrated good discriminatory ability and calibration for both short- and long-term mortality. Compared to CURB-65, the nomogram achieved higher AUC and sensitivity, as well as a relatively high NPV, making it more accurate in ruling out mortality in low-risk patients. With both higher PPV and NPV, the nomogram consistently provided more reliable mortality predictions than CURB-65. However, in terms of calibration performance, it slightly underestimated the risk of 30-day mortality. This underestimation is probably due to the fact that the model was primarily designed to predict the risk of long-term mortality risk, coupled with the fact that there were fewer cases of short-term mortality in the data. However, although outcome definitions vary, the model still exhibited useful discrimination and potential clinical utility, especially in predicting the long-term prognosis.

To ensure the simplicity of clinical assessment, we have excluded indicators that required invasive testing, allowing for timely risk stratification even before laboratory results are available. However, even when albumin was included in the model, the AUC was 0.828(95%CI: 0.774–0.882) (Supplementary file 2, Figure S3), indicating no significant improvement in the discriminative ability of the model. Therefore, this model may also aid in optimizing resource allocation by identifying which patients may require urgent laboratory test and imaging studies.

Despite the promising results, our study has several limitations. First, the lack of external data prevented us from performing external validation of the established model. However, we plan to collect sufficient data for external validation in future research. In addition, the follow-up period of this study was 180 days andfuture studies could extend the follow-up duration to assess longer-term outcomes and apply the prognostic model more broadly. Finally, although we demonstrated the model’s superiority over CURB-65, it remains unclear whether it outperforms other common pneumonia severity scores, such as PSI, SMART-COP and A-DROP etc. In addition, in our study, the construction of a random forest model was hindered by class imbalance and limited sample size, leading to significant overfitting. Specifically, while the model achieved an AUC close to 1 on the training set, its performance on the test set was markedly lower (Supplementary file 3, Figure S1), even after hyperparameter optimization (Supplementary file 3, Figure S2). Despite efforts to mitigate class imbalance and fine-tune hyperparameters (Supplementary file 3, Figure S3), the learning curve revealed fluctuations in test set performance as the sample size increased, with occasional performance degradation. This instability may stem from distributional discrepancies between the training and test sets, where larger sample sizes introduced additional noise or inconsistencies. Consequently, the model became overly reliant on training set-specific features, compromising its generalizability and resulting in poor test set performance. Although the random forest model demonstrated a higher AUC for predicting 180-day mortality compared to the Cox regression model, its propensity for overfitting prompted its exclusion from the main analysis. Nevertheless, machine learning remains a promising approach, and future studies with larger sample sizes may allow for the development of a more robust and generalizable model.

Conclusions

This study developed and internally validated a prediction model for long-term mortality in older adults with CAP, while also demonstrating significant discriminatory power for short-term mortality. This model offers a valuable tool for early risk assessment, especially in ruling out mortality in low-risk patients and surpasses CURB-65 in both discriminatory ability and accuracy.