Risk prediction models for falls in hospitalized older patients: a systematic review and meta-analysis

Abstract

Background

Existing fall risk assessment tools in clinical settings often lack accuracy. Although an increasing number of fall risk prediction models have been developed for hospitalized older patients in recent years, it remains unclear how useful these models are for clinical practice and future research.

Objectives

To systematically review published studies of fall risk prediction models for hospitalized older adults.

Methods

A search was performed of the Web of Science, PubMed, Cochrane Library, CINAHL, MEDLINE, and Embase databases: to retrieve studies of predictive models related to falls in hospitalized older adults from their inception until January 11, 2024. Extraction of data from included studies, including study design, data sources, sample size, predictors, model development and performance, etc. Risk of bias and applicability were assessed using the Prediction Model Risk of Bias Assessment Tool (PROBAST) checklist.

Results

A total of 8086 studies were retrieved, and after screening, 13 prediction models from 13 studies were included. Four models were externally validated. Eight models reported discrimination metrics and two models reported calibration metrics. The most common predictors of falls were mobility, fall history, medications, and psychiatric disorders. All studies indicated a high risk of bias, primarily due to inadequate study design and methodological flaws. The AUC values of 8 models ranged from 0.630 to 0.851.

Conclusions

In the present study, all included studies had a high risk of bias, primarily due to the lack of prospective study design, inappropriate data analysis, and the absence of robust external validation. Future studies should prioritize the use of rigorous methodologies for the external validation of fall risk prediction models in hospitalized older adults.

Trial registration

The study was registered in the International Database of Prospectively Registered Systematic Reviews (PROSPERO) CRD42024503718.

Background

Falls are a significant global concern, resulting in 684,000 deaths annually, according to the World Health Organization [1]. Falls represent a leading cause of disability among older adults, posing a significant problem even for those in good health. The growing older adults and increasing life expectancy make fall prediction increasingly important. Hospital-acquired falls (HAFs) are a particular concern for healthcare systems [2], with roughly 28% of hospitalized patients reporting a fall within the past year and 15% experiencing one during their stay [3]. It is understood that approximately 1–3% of hospitalized patients who experience falls may suffer from fractures as a result [4]. In addition, falls may also lead to subdural hematomas and hemorrhages, which not only have a significant impact on the health and quality of life of older adults but also place a heavy burden on families and the healthcare system.

Despite a focus on fall reduction in many studies, current fall risk assessment tools and evidence-based practices have limitations in effectiveness [5, 6]. This includes the potential for a time-consuming assessment process and the influence of subjective judgments by healthcare professionals. Moreover, these assessment tools typically rely on static risk factors and fail to account for the dynamic changes in patients’ conditions during their hospital stay. Therefore, a highly accurate and easy-to-use tool is crucial for identifying fall risks in hospitalized older adults. Additionally, translating research findings into clinical practice is essential to enhance safety for hospitalized older adult patients [7].

In recent years, artificial intelligence (AI) has been playing an increasingly important role in medical diagnosis by analyzing medical records, exams, and test results to identify disease patterns and improve diagnostic accuracy [8]. Prediction models are a significant branch of artificial intelligence and serve as a vital quantitative tool for assessing clinical risks and benefits. However, despite the increasing number of prediction models for fall risk in hospitalized older adults, they commonly face several key challenges, including insufficient data quantity, limitations in clinical validation, and a lack of adaptability to different patient populations. These issues restrict the widespread application of these models in clinical practice. Our study aimed to conduct a systematic assessment of these models, integrate the evidence pertaining to risk factors for falls among hospitalized older adults, and provide valuable references for future research and clinical practice.

Methods

Design

Following the established guidelines for evaluating predictive models [9] and the CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS) [10], we conducted a systematic review. The protocol for this review was prospectively registered on the PROSPERO International Prospective Register of Systematic Reviews website (CRD42024503718).

Search strategy

We conducted a comprehensive search of multiple databases and search platforms, including Web of Science, PubMed, Cochrane Library, CINAHL, MEDLINE, and Embase, from their inception until January 11, 2024, that investigated fall risk prediction models in hospitalized older adults aged 65 and older. We also conducted a manual review of the references from the retrieved studies. Our search utilized a combination of medical subject headings (MeSH) and text words, incorporating the following four concepts: (1) inpatients, inpatient, hospital*; (2) aged, elderly, senium, older adults, senior citizen; (3) accidental falls, fall, falling; (4) prediction model, risk score, risk assessment, risk prediction. A complete list of search terms is available in Appendix A. A detailed description of the population, interventions, comparisons, outcomes, timing, and settings (PICOTS) for this systematic review is provided below:

P (Population): ≥65 years old hospitalized older patients.

I (Intervention): Risk prediction models for falls.

C (Comparator): Not applicable.

O (Outcome): Presence of fall.

T (Timing): During the hospitalization.

S (Setting): Hospitalized patients only.

Inclusion and exclusion criteria

To be included in this review, studies had to meet the following criteria: (1) participants were hospitalized patients aged 65 years or older, (2) the study design was observational, (3) the study developed and/or validated a multivariable predictive model with at least two predictors of falls, and (4) the primary outcome of interest was falls during hospitalization. Studies were excluded if they did not meet any of the following criteria: (1) Falls were assessed using an assessment scale, (2) they used a cross-sectional survey design, (3) the outcome measure focused on adverse events due to falls rather than falls themselves, (4) the language of the study was not English, or (5) the full text of the article was not available.

Study selection

Duplicate records were removed using Zotero software. Two independent reviewer pairs (MAL and RMZ) screened titles and abstracts against the inclusion/exclusion criteria for fall prediction model studies. Disagreements were resolved through discussion, with a third reviewer (CSY) consulted when needed. After reaching a consensus, two reviewers (MAL and SJ) independently screened full texts. Additionally, reference lists of included studies were reviewed for potentially relevant articles.

Data extraction

Two reviewers(MAL and SJ) independently extracted data based on the critical appraisal and data extraction for systematic reviews of prediction modelling studies (CHARMS) [10]. Extracted information included basic details like authors, publication year, study design, participants, data sources, and sample size. Specific to predictive modeling, we extracted details on variable selection methods, model development techniques, validation types, performance measures, handling of missing data and continuous variables, predictors used in the final model, and the model presentation format. For studies with multiple models, we focused on the one with the best predictive performance. Any disagreements in data extraction were resolved through discussion (MAL, SJ, and ZHF).

Quality assessment

To assess the risk of bias (ROB) and applicability of prediction models in the included studies, we utilized the Prediction Model Risk of Bias Assessment Tool (PROBAST) [11]. This tool features 20 key questions across four domains: study population, predictors, outcomes, and statistical analysis. The first three domains assess applicability, similar to the Risk of Bias tool but excluding specific risk of bias questions. Each question has answer options like “yes”, “probably yes”, “no”, “probably no”, or “no information”. A domain is considered high risk if it has at least one “no” or “probably no” answer. If one or more domains are unclear and the others are low risk, the overall bias is unclear. Overall low risk of bias requires all domains to be judged low risk. Two authors (MAL and SJ) independently assessed quality using PROBAST. In case of disagreements regarding quality assessment, a discussion involving three authors (MAL, SJ, and ZHF) was held to reach a consensus.

Data analysis

When more than two studies reported the same outcome measure, a meta-analysis was performed. We used the ‘metamisc’ package in R software (version 4.2.3) to estimate unreported AUC confidence intervals and calculate predictive intervals. The random-effects model with the Hartung-Knapp-Sidik-Jonkman (HKSJ) method was used to calculate the 95% confidence interval for the average performance (The HKSJ method can provide more accurate type I error rates and confidence intervals when heterogeneity exists) [12, 13]. Heterogeneity was estimated using the predictive intervals calculated by the HKSJ method, with a wider predictive interval compared to the confidence interval indicating the presence of heterogeneity among the original studies [14]. To investigate the sources of variation, subgroupings of different modelling approaches were performed. Sensitivity analyses were performed to further explore potential sources of this heterogeneity. If heterogeneity cannot be resolved, narrative synthesis will be used in this study to analyze, summarize and compare the included studies. Funnel plots and Egger’s test [15] were employed to assess publication bias. Symmetrical distribution of data points in the funnel plot and a p-value greater than 0.05 from Egger’s test suggest no significant publication bias. In the event of evident publication bias, the trim-and-fill method will be employed to further assess the impact of publication bias on the results of the meta-analysis.

Results

Selection process

Figure 1 presents the PRISMA 2020 flowchart for the literature search and selection process. Our initial search retrieved a total of 8086 records from various databases (Web of Science, PubMed, MEDLINE, EMBASE, CINAHL, and the Cochrane Library) and manual searches (n = 5). After removing duplicates (n = 1094), 6992 records underwent title and abstract screening. Ultimately, 13 studies meeting the inclusion criteria were included in this review, encompassing a total of 13 prediction models. A table summarizing the number of retrieved records from each database is presented in Fig. 1.

Fig. 1

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flowchart of literature search and selection

Study characteristics

Thirteen studies were included in this review. Six employed retrospective cohort designs, five used prospective cohorts, and two were case-control studies (Table 1). Most studies (n = 11) utilized data from rehabilitation organizations, primarily hospitals. Public databases provided data for two studies. One study specifically focused on older adults with dementia (Table 1). The size of the study populations used to build the models ranged from 30 to 72,314 individuals (Table 1).

Table 2 summarizes the characteristics of the models used in the included studies. The studies employed various modeling techniques, including traditional logistic regression (n = 4), machine learning (n = 4), or a combination of both (n = 5). Only four studies incorporated external validation methods, while the remaining eight relied on internal validation (Table 2). Eight studies reported the model’s discrimination performance, with AUC values ranging from 0.630 to 0.851 (Table 2). Two studies used calibration curves to assess calibration, while others reported metrics like sensitivity, specificity, positive predictive rate, and negative predictive rate derived from the confusion matrix (Table 2). The final model predictors fell into four main categories: general demographics, physical and cognitive function, medications, and biochemical markers. The most frequently reported predictors (used in at least two studies each) were activity capacity (n = 7), history of falls (n = 4), medication (n = 4), mental cognition (n = 4), gender (n = 2), disease (n = 2), and vital signs (n = 2).

Table 1 Overview of basic data of the included studies

Table 2 Overview of the information of the included prediction models

Risk of bias and applicability assessment

We used the PROBAST tool to evaluate the risk of bias and applicability of all 13 included models (shown in Fig. 2). A detailed quality assessment is provided in Appendix B.

Our analysis revealed a high risk of bias across all models. Eight studies had a high ROB due to unsuitable data sources (e.g., retrospective design). Similarly, eight studies were rated high ROB in the predictor domain due to the retrospective design lacking blinding, potentially influencing predictor assessment by outcome information. In the outcome domain, nine studies were judged high ROB given that they did not exclude outcome-related factors from the predictor definition, and one study was unclear due to missing information on the time interval between predictor assessment and outcome determination. Finally, all studies except Dormosh et al. [16] had high ROB in the analysis domain. Here, two studies fell short of the recommended sample size (EPV > 20), three studies involved the transformation of continuous data, and three studies excluded a portion of the data from the final analysis. Regarding data samples, two studies lacked data preprocessing (e.g., interpolation), and three used univariate analysis for predictor selection. Evaluation of model performance revealed that five studies omitted discrimination metrics, eleven omitted calibration metrics, and five neglected model fit assessment. Nine out of thirteen studies were classified as low risk for applicability, while four were considered high risk. All high-risk classifications stemmed from the participant domain. One study focused solely on older adults with dementia, a subgroup of the broader target population in this review. The remaining three high-risk studies did not define the age criteria for their older adult participants.

Fig. 2

Risk of bias and applicability assessment of included studies using the Prediction model Risk of Bias Assessment Tool (PROBAST)

Meta-analysis of validation models included in the review

Due to the under-reporting of model assessment metrics, only eight studies were included in the meta-analysis for AUC. Notably, the prediction interval was significantly wider than the confidence interval, indicating substantial heterogeneity among the studies (shown in Fig. 3). Results of the sensitivity analysis (Appendix C) showed that after excluding individual studies in turn, the overall prediction interval was still significantly wider than the confidence interval, implying that there was still large heterogeneity. Subgroup analysis (Appendix D) revealed no significant difference in model performance between traditional logistic regression and machine learning algorithms. However, the within-group prediction interval was still significantly wider than the confidence interval, suggesting significant heterogeneity. Finally, Egger’s test yielded a p-value of 0.102 indicating no significant publication bias.

Fig. 3

Forest plots of fall risk prediction models for hospitalized older adults

Discussion

Hospital-acquired falls are serious adverse events, especially for older patients, leading to injuries, prolonged stays, and increased healthcare costs. Fall prevention is a crucial safety priority for healthcare providers, requiring individual fall risk assessments for each patient. This systematic review identified and assessed the quality of 13 studies on predictive models for falls in hospitalized older adults. The models exhibited significant performance variation in internal/external validation (AUROC: 0.630–0.851). However, the high risk of bias in all studies limits the real-world applicability of these findings.

This systematic review identified several critical methodological issues. Eight studies did not report how they handled missing data, while one study simply excluded it. This can introduce bias in effect size estimates and reduce the models’ discriminative power. Multiple imputation [29] is the preferred approach for handling missing values in both model development and validation due to its accuracy and reduced bias. However, researchers should be mindful of “data leakage” [30]when using this method. Furthermore, four studies converted continuous variables into categorical ones. This can lead to information loss and reduced analytical power, ultimately resulting in lower model performance as documented in the literature [31].

Three of the included models used logistic regression, while the remaining five employed various machine learning algorithms. Machine learning is often viewed as superior to logistic regression for real-world data [32], which can be nonlinear and have complex relationships between features. This allows machine learning to handle large, high-dimensional datasets effectively. However, it should be borne in mind that machine learning models are not always superior [33]. In some cases, logistic regression models can be simpler and more effective. First, its simple form makes it easy to understand and interpret. Second, it can efficiently converge and provide stable results even with smaller datasets. The resulting regression coefficients indicate how strongly each variable influences the outcome. This interpretability is crucial for clinicians, as it allows them to identify key factors in disease development and progression, informing preventive measures or treatment plans. Machine learning models are generally more complex than logistic regression, making them less interpretable, hence the “black box” label [34]. However, advancements are being made to enhance interpretability in these complex algorithms. SHapley Additive exPlanations (SHAP) is a popular example [35]. This game theory-based approach unveils the average contribution of each feature, enabling both global and local interpretability. Local interpretability allows clinicians to tailor rehabilitation programs to individual patients. Therefore, researchers must make trade-offs based on specific data characteristics when selecting modeling methods. To maximize the predictive performance and generalizability of the model, we recommend that researchers consider multiple modeling methods when constructing a prediction model.

Differing from static data, the construction of fall prediction models based on dynamically collected real-time or recent data holds broad prospects for development. In this study, the two studies that employed dynamic data to construct models both demonstrated favorable prediction accuracy (0.731–0.740). By segmenting the data or conducting time-series analysis to capture individual dynamic changes, it is possible to predict fall risk in real-time, which is crucial for the realization of early warnings. However, the data collection process may be plagued by issues of equipment stability and noise interference. The heterogeneity of the data further complicates data processing and increases the difficulty of model training. Consequently, it is imperative for the future to surmount the knowledge barriers between different fields through technological innovation and interdisciplinary collaboration.

Validation studies, both internal and external, can only assess a prediction model’s performance in specific contexts, highlighting the need to confirm model robustness before clinical use [36]. In addition to conducting multicenter studies, researchers can utilize publicly available databases to enhance cost-effectiveness and generalizability by leveraging comprehensive data and larger datasets. However, it is crucial to attend to the temporal sequence between the extracted predictors and the occurrence of outcomes, neglecting this aspect could undermine the stability of the model and elevate the likelihood of missteps in clinical decision processes. Accurate reporting of model results is crucial for informed decision-making, transparency, and continuous model improvement. The PROBAST assessment tool emphasizes reporting on model discrimination (AUC ranges from 0.5 for random chance to 1 for perfect accuracy [37]) and calibration metrics. Additionally, clinical applicability metrics like positive and negative predictive values can provide a more comprehensive assessment. In our study, although two studies reported calibration metrics, the provision of an Observed-to-Expected (O/E) ratio can offer more informative insights into the assessment of model calibration. For imbalanced datasets, the F1-Score and Matthews Correlation Coefficient (MCC) can be employed to comprehensively gauge model performance. Evaluating from multiple perspectives will provide a more holistic reflection of the predictive capabilities of the model, thereby ensuring the effectiveness of the chosen model in real-world applications. To improve reporting quality, researchers should strictly follow the Transparent Reporting of a Multivariate Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) statement [38].

Falls in older adults are a complex issue with both intrinsic (individual characteristics) and extrinsic (environmental) risk factors. Most falls involve a combination of advanced age, health conditions, and interactions with the environment [39]. Due to this complexity, predicting fall risk is challenging. This study summarized the most commonly reported influences on falls based on the final models’ results. The top four factors identified were: mobility limitations, history of falls, medications, and mental cognition. While mobility testing is crucial for fall risk assessment, relying solely on a single test (e.g., Single Leg Stance Test, Timed Up and Go) is insufficient [40, 41]. Combining these tests with other factors improved the accuracy. Indeed, although numerous fall risk assessment tools exist, achieving both high sensitivity and specificity remains difficult [42]. Therefore, a more precise prediction model for hospitalized older adults is urgently needed for clinical application. A history of falls is a strong predictor of future falls and a major focus in clinical assessments [43]. This is likely due to both the physical consequences (functional decline) and psychological impact (fear of falling (FOF)) of falls. Notably, FOF is prevalent, affecting 40–73% of older adults with a history of falls, and even half of those without [44]. Polypharmacy and specific medications like cardiovascular and psychotropic drugs significantly increase fall risk in hospitalized older adults with high comorbidity [45]. Certain medications, including antiepileptic drugs, opioids, and those used in high quantities (polypharmacy), have been associated with an increased risk of falls in older adults. These factors should be considered during fall risk assessment. Cognitive impairment in older adults can impair their ability to cope with their environment, which can be detrimental to balance and gait [46]. However, more research is warranted to determine if there’s a link between cognitive impairment and falling [47, 48]. In addition, falls can be influenced by various characteristics, including gender, medical conditions, and vital signs. Due to the complexity of falls, accurate assessment necessitates considering multiple factors. Future studies should prioritize incorporating well-established risk factors like those discussed above (mobility limitations, fall history, medications, and cognition) into fall risk models. Expanding the model’s predictor base can address misclassification arising from variations in patient characteristics. However, it is important to avoid overfitting the model by introducing excessive complexity.

Strengths and limitations

Our study systematically reviewed multiple databases to evaluate research on fall prediction models for hospitalized older adults and conducted a critical assessment of the retrieved studies, providing comprehensive and objective evidence to support subsequent research. However, this study has several limitations that should be acknowledged. First, by only including English literature, we may have limited the diversity and generalizability of our findings. Additionally, although statistical tests indicated no significant publication bias, funnel plot and the exclusion of relevant studies from the grey literature databases may still lead to potential bias. Second, some studies lacked comprehensive reporting of results, hindering a meta-analysis on the calibration of the predictive models. Finally, the meta-analysis revealed a high degree of heterogeneity, which could be attributed to variations in study design, participant populations, and baseline fall risks. Although the current limitations preclude us from endorsing the clinical application of any specific model, our study can still provide valuable reference points for designing future high-quality studies with transparent reporting practices.

Implications

Our study aggregates and interprets the critical evidence related to fall risk factors in older adults admitted to hospitals, thereby serving as a cornerstone for the future development of precise and clinically actionable fall prediction models. Nevertheless, owing to the limitations in study design quality and the absence of robust model validation, the applied significance of the fall prediction models for hospitalized older adults as included in this research is not yet fully elucidated. The direction of future endeavors should be aimed at meticulous study design and the augmentation of external validation for established prediction models, with the objective of enhancing the broader applicability and generalizability of the research conclusions.

Conclusion

This study identified 13 studies with a total of 13 prediction models for fall risk in hospitalized older adults. The AUC values (0.630–0.851) indicate some discriminative ability. However, all studies exhibited significant methodological shortcomings including a lack of rigorous experimental design or valid external validation. Consequently, we cannot recommend any model for clinical use at this stage. Future research should prioritize rigorous model validation adhering to the PROBAST standards for quality control. Additionally, leveraging big data for external validation can enhance model applicability and generalizability. Continuous optimization is crucial to maximize the model’s practical value.