KES
Articles
- Page Path
- HOME > Endocrinol Metab > Volume 31(1); 2016 > Article
-
Review ArticleHow to Establish Clinical Prediction Models
-
Yong-ho Lee1
, Heejung Bang2, Dae Jung Kim3 -
Endocrinology and Metabolism 2016;31(1):38-44.
DOI: https://doi.org/10.3803/EnM.2016.31.1.38
Published online: March 16, 2016
1Department of Internal Medicine, Yonsei University College of Medicine, Seoul, Korea.
2Division of Biostatistics, Department of Public Health Sciences, University of California Davis School of Medicine, Davis, CA, USA.
3Department of Endocrinology and Metabolism, Ajou University School of Medicine, Suwon, Korea.
- Corresponding author: Dae Jung Kim. Department of Endocrinology and Metabolism, Ajou University School of Medicine, 164 World cup-ro, Yeongtong-gu, Suwon 16499, Korea. Tel: +82-31-219-5128, Fax: +82-31-219-4497, djkim@ajou.ac.kr
- Corresponding author: Yong-ho Lee. Department of Internal Medicine, Yonsei University College of Medicine, 50-1 Yonsei-ro, Seodaemun-gu, Seoul 03722, Korea. Tel: +82-2-2228-1943, Fax: +82-2-393-6884, yholee@yuhs.ac
• Received: January 9, 2016 • Revised: January 14, 2016 • Accepted: January 27, 2016
Copyright © 2016 Korean Endocrine Society
This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.
ABSTRACT
- A clinical prediction model can be applied to several challenging clinical scenarios: screening high-risk individuals for asymptomatic disease, predicting future events such as disease or death, and assisting medical decision-making and health education. Despite the impact of clinical prediction models on practice, prediction modeling is a complex process requiring careful statistical analyses and sound clinical judgement. Although there is no definite consensus on the best methodology for model development and validation, a few recommendations and checklists have been proposed. In this review, we summarize five steps for developing and validating a clinical prediction model: preparation for establishing clinical prediction models; dataset selection; handling variables; model generation; and model evaluation and validation. We also review several studies that detail methods for developing clinical prediction models with comparable examples from real practice. After model development and vigorous validation in relevant settings, possibly with evaluation of utility/usability and fine-tuning, good models can be ready for the use in practice. We anticipate that this framework will revitalize the use of predictive or prognostic research in endocrinology, leading to active applications in real clinical practice.
- Hippocrates emphasized prognosis as a principal component of medicine [1]. Nevertheless, current medical investigation mostly focuses on etiological and therapeutic research, rather than prognostic methods such as the development of clinical prediction models. Numerous studies have investigated whether a single variable (e.g., biomarkers or novel clinicobiochemical parameters) can predict or is associated with certain outcomes, whereas establishing clinical prediction models by incorporating multiple variables is rather complicated, as it requires a multi-step and multivariable/multifactorial approach to design and analysis [1].
- Clinical prediction models can inform patients and their physicians or other healthcare providers of the patient's probability of having or developing a certain disease and help them with associated decision-making (e.g., facilitating patient-doctor communication based on more objective information). Applying a model to a real world problem can help with detection or screening in undiagnosed high-risk subjects, which improves the ability to prevent developing diseases with early interventions. Furthermore, in some instances, certain models can predict the possibility of having future disease or provide a prognosis for disease (e.g., complication or mortality). This review will concisely describe how to establish clinical prediction models, including the principles and processes for conducting multivariable prognostic studies and developing and validating clinical prediction models.
INTRODUCTION
- In the era of personalized medicine, prediction of prevalent or incident diseases (diagnosis) or outcomes for future disease course (prognosis) became more important for patient management by health-care personnel. Clinical prediction models are used to investigate the relationship between future or unknown outcomes (endpoints) and baseline health states (starting point) among people with specific conditions [2]. They generally combine multiple parameters to provide insight into the relative impacts of individual predictors in the model. Evidence-based medicine requires the strongest scientific evidence, including findings from randomized controlled trials, meta-analyses, and systematic reviews [3]. Although clinical prediction models are partly based on evidence-based medicine, the user must also adopt practicality and an artistic approach to establish clinically relevant and meaningful models for targeted users.
- Models should predict specific events accurately and be relatively simple and easy to use. If a prediction model provides inaccurate estimates of future-event occurrences, it will mislead healthcare professionals to provide insufficient management of patients or resources. On the other hand, if a model has high predictability power but is difficult to apply (e.g., with complicated calculation or unfamiliar question/item or unit), time consuming, costly [4] or less relevant (e.g., European model for Koreans, event too far away), it will not be commonly used. For example, a diabetes prediction model developed by Lim et al. [5] has a relatively high area under the receiver operating curve (AUC, 0.77), while blood tests that measure hemoglobin A1c, high density lipoprotein cholesterol, and triglyceride are included in the risk score, which would generally require clinician's involvement so could be a major barrier for use in community settings. When prediction models consist of complicated mathematical equations [67], a web-based application can enhance implementation (e.g., calculating 10-year and lifetime risk for atherosclerotic cardiovascular disease [CVD] is available at http://tools.acc.org/ASCVD-Risk-Estimator/). Therefore, achieving a balance between predictability and simplicity is a key to a good clinical prediction model.
CONCEPT OF CLINICAL PREDICTION MODELS
- There are several reports [18910111213] and a textbook [14] that detail methods to develop clinical prediction models. Although there is currently no consensus on the ideal construction method for prediction models, the Prognosis Research Strategy (PROGRESS) group has proposed a number of methods to improve the quality and impact of model development [215]. Recently, investigators on the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) study have established a checklist of recommendations for reporting on prediction or prognostic models [16]. This review will summarize the analytic process for developing clinical prediction models into five stages.
- Stage 1: preparation for establishing clinical prediction models
- The aim of prediction modeling is to develop an accurate and useful clinical prediction model with multiple variables using comprehensive datasets. First, we have to articulate several important research questions that affect database selection and the approach of model generation. (1) What is the target outcome (event or disease) to predict (e.g., diabetes, CVD, or fracture)? (2) Who is the target patient of the model (e.g., general population, elderly population ≥65 years or patients with type 2 diabetes)? (3) Who is the target user of the prediction model (e.g., layperson, doctor or health-related organization)? Depending on the answers to the above questions, researchers can choose the proper datasets for the model. The category of target users will determine the selection and handling process of multiple variables, which will affect the structure of the clinical prediction model. For example, if researchers want to make a prediction model for laypersons, a simple model with not many user-friendly questions in only a few categories (e.g., yes vs. no) could be ideal.
- Stage 2: dataset selection
- The dataset is one of the most important components of the clinical prediction model—often not under investigators' control—and ultimately determines its quality and credibility; however, there are no general rules for assessing the quality of data [9]. Yet, there is no such thing as perfect data and prefect model. It would be reasonable to search for best-suited dataset. Oftentimes, secondary or administrative data sources must be utilized because a primary dataset with the study endpoint and all of key predictors is not available. Researchers should use different types of datasets, depending on the purpose of the prediction model. For example, a model for screening high-risk individuals with undiagnosed condition/disease can be developed using cross-sectional cohort data. However, such models may have relatively low power for predicting future incidence of disease when different risk factors come into play. Accordingly, longitudinal or prospective cohort datasets should be used for prediction models for future events (Table 1). Models for prevalent events are useful for predicting asymptomatic diseases, such as diabetes or chronic kidney disease, by screening undiagnosed cases, whereas models for incident events are useful for predicting the incidence of relatively severe diseases, such as CVD, stroke, and cancer.
- A universal clinical prediction model for disease does not exist; thus, separate specific models that can individually assess the role of ethnicity, nationality, sex, or age on disease risk are warranted. For example, the Framingham coronary heart disease (CHD) risk score is generated by one of the most commonly used clinical prediction models; however, it tends to overestimate CHD risk by approximately 5-fold in Asian populations [1718]. This indicates that models derived from one ethnicity sample may not be directly applied to populations of other ethnicities. Other specific characteristics of study populations beside ethnicity (e.g., obesity- or culture-related variables) could be important.
- There is no absolute consensus on the minimal requirement for dataset sample size. Generally, large representative, contemporary datasets that closely reflect the characteristics of their target population are ideal for modeling and can enhance the relevance, reproducibility, and generalizability of the model. Moreover, two types of datasets are generally needed: a development dataset and a validation dataset. A clinical prediction model is first derived from analyses of the development dataset and its predictive performance should be assessed in different populations based on the validation dataset. It is highly recommended to use validation datasets from external study populations or cohorts, whenever available [1920]; however, if it is not possible to find appropriate external datasets, an internal validation dataset can be formed by randomly splitting the original cohort into two datasets (if sample size is large) or statistical techniques such as jackknife or bootstrap resampling (if not) [21]. The splitting ratio can vary depending on the researchers' particular goals, but generally, more subjects should be allocated to the development dataset than to the validation dataset.
- Stage 3: handling variables
- Since cohort datasets contain more variables than can reasonably be used in a prediction model, evaluation and selection of the most predictive and sensible predictors should be done. Generally, inclusion of more than 10 variables/questions may decrease the efficiency, feasibility and convenience of prediction models, but expert's judgment that could be somewhat subjective is required to assess the need for each situation. Predictors that were previously found to be significant should normally be considered as candidate variables (e.g., family history of diabetes in diabetes risk score). It should be noted that not all significant predictors need to be included in the final model (e.g., P<0.05); predictor selection must be always guided by clinical relevance/judgement to prevent nonsensical or less relevant or user-unfriendly variables (e.g., socioeconomic status-related) or possible false-positive associations. Additionally, variables which are highly correlated with others may be excluded because they contribute little unique information [22]. On the other hand, variables not statistically significant or with small effect size may still contribute to the model [23]. Depending on researcher discretion, different models that analyze different variables may be developed for targeting distinct users. For example, a simple clinical prediction model that does not require laboratory variables and a comprehensive model that does could both be designed for laypersons and health care providers, respectively [19].
- With regard to variable coding, categorical and continuous variables should be managed differently [8]. For ordered categorical variables, infrequent categories can be merged and similar variables may be combined/grouped. For example, past and current smoker categories can be merged if numbers of subjects who report being a past or current smoker are relatively small and variable unification does not alter the statistical significance of the model materially. Although continuous parameters are usually included in a regression model, assuming linearity, researchers should consider the possibility of non-linear associations such as J- or U-shaped distributions [24]. Furthermore, the relative effect of a continuous variable is determined by the measurement scale used in the model [8]. For example, the impact of fasting glucose levels on the risk of CVD may be interpreted as having a stronger influence when scaled per 10 mg/dL than per 1 mg/dL.
- Researchers often emphasize the importance of not dichotomizing continuous variables in the initial stage of model development because valuable predictive information can be lost during categorization [24]. However, prediction models—is not the same thing as regression models—with continuous parameters may be complex and hard to use or be understood by laypersons, because they have to calculate their risk scores by themselves. A web or computer-based platform is usually required for the implementation of these models. Otherwise, in a later phase, researchers may transform the model into a user-friendly format by categorizing some predictors, if the predictive capacity of the model is retained [81925].
- Finally, missing data is a chronic problem in most data analyses. Missing data can occur various reasons, including uncollected (e.g., by design), not available or not applicable, refusal by respondent, dropout, or "don't know." To handle this issue, researchers may consider imputation technique, dichotomizing the answer into yes versus others, or allow "unknown" as a separate category as in http://www.cancer.gov/bcrisktool/.
- Stage 4: model generation
- Although there are no consensus guidelines for choosing variables and determining structures to develop the final prediction model, various strategies with statistical tools are available [89]. Regression analyses, including linear, logistic, and Cox models are widely used depending on the model and its intended purpose. First, the full model approach is to include all the candidate variables in the model; the benefit of this approach is to avoid overfitting and selection bias [9]. However, it can be impractical to pre-specify all predictors and previously significant predictors may not be in a new population/sample. Second, a backward elimination approach or stepwise selection method can be applied to remove a number of insignificant candidate variables. To check for overfitting of the model, Akaike information criterion (AIC) [26], an index of model fitting that charges a penalty against larger models, may be useful [19]. Lower AIC values indicate a better model fit. Some interpret that AIC addresses explanation and Bayesian information criterion (BIC) addresses prediction, where BIC may be considered a Bayesian counterpart [27].
- If researchers prefer algorithm modeling culture instead of data modeling culture, e.g., formula-based regression [28], a classification and regression tree analysis or recursive partitioning could be considered [282930].
- With regard to determining scores for each predictor in the generation of simplified models, researchers using expert judgment may create a weighted scoring system by converting β coefficients [19] or odds ratios [20] from the final model to integer values, while preserving monotonicity and simplicity. For example, from the logistic regression model built by Lee et al. [19], β coefficients <0.6, 0.7 to 1.3, 1.4 to 2.0, and >2.1 were assigned scores of 1, 2, 3, and 4, respectively.
- Stage 5: model evaluation and validation (internal/external)
- After model generation, researchers should evaluate the predictive power of their proposed model using an independent dataset, where truly external dataset is preferred whenever available. There are several standard performance measures that capture different aspects: two key components are calibration and discrimination [8931]. Calibration can be assessed by plotting the observed proportions of events against the predicted probabilities for groups defined by ranges of individual predicted risk [910]. For example, a common method is to categorize 10 risk groups of equal size (deciles) and then conduct the calibration process [32]. The most ideal calibration plot would show a 45° line, which indicates that the observed proportions of events and predicted probabilities completely overlap over the entire range of probabilities [9]. However, this is not guaranteed when external validation is conducted with a different sample. Discrimination is defined as the ability to distinguish events versus non-events (e.g., dead vs. alive) [8]. The most common discrimination measure is the AUC or, equivalently, concordance (c)-statistic. The AUC is equal to the probability that, given two individuals randomly selected—one who will develop an event and another who will not—the model will assign a higher probability of an event to the former [10]. A c-statistic value of 0.5 indicates a random chance (i.e., flip of a coin). The usual c-statistic range for a prediction model is 0.6 to 0.85; this range can be affected by target-event characteristics (disease) or the study population. A model with a c-statistic ranging from 0.70 to 0.80 has an adequate power of discrimination; a range of 0.80 to 0.90 is considered excellent. Table 2 shows several common statistical measures for model evaluation.
- As usual, selection, application and interpretation of any statistical method and results need great care as virtually all methods entail assumptions and limited capacity. Let us review some here. Predictive values depend on the disease prevalence so direct comparison for different diseases may not be valid. When sample size is very large, P value can be impressively small even for a practically meaningless difference. Net reclassification index and integrated discrimination improvement are known to lead to non-proper scoring and vulnerable to miscalibrated or overfit problems [33]. AUC and R2 are often hard to increase by a new predictor, even with large odds ratio. Despite similar names, AIC and BIC address slightly different issues and information in BIC can be decreased with sample size increases. The Hosmer-Lemeshow test is highly sensitive when sample size is large, which is not an ideal property as a goodness-fit statistic. Calibration plot can easily provide a high correlation coefficient (>0.9), simply because they are computed for predicted versus observed values on grouped data (without random variability). Finally, AUC also needs caution: a high value (e.g., >0.9) may mean excellent discrimination but it can also reflect the situation where prediction is not so relevant: (1) the task is closer to diagnostic or early onset rather than prediction; (2) cases vs. non-cases are fundamentally different with minimal overlap; or (3) predictors and endpoints are virtually the same things (e.g., current blood pressure vs. future blood pressure).
- Despite the long list provided above, we do not think this is a discouraging news to researchers. We may tell us no method is perfect and "one size does not fit all" is also true to statistical methods; thus blinded or automated application can be dangerous.
- It is crucial to separate internal and external validation and to conduct the previously mentioned analyses on both datasets to finalize the research findings (see the following for example reports [192034]). Internal validation can be done using a random subsample or different years from the development dataset or by conducting bootstrap resampling [22]. This approach can particularly assess the stability of selected predictors, as well as prediction quality. Subsequently, external validation should be performed on an independent dataset from that which was previously used to develop the model. For example, datasets can be obtained from populations from other hospitals or centers (see geographic validation [19]) or a more recently collected cohort population (temporal validation [34]). This process is often considered to be a more powerful test for prediction models than internal validation because it evaluates transportability, generalizability and true replication, rather than reproducibility [8]. Poor model performance may occur after use of an external dataset due to differences in healthcare systems, measurement methods/definitions of predictors and/or endpoint, subject characteristics or context (e.g., high vs. low risk).
STEPS TO DEVELOPING CLINICAL PREDICTION MODELS
- For patient-centered perspectives, clinical prediction models are useful for several purposes: to screen high-risk individuals for asymptomatic disease, to predict future events of disease or death, and to assist medical decision-making. Herein, we summarized five steps for developing a clinical prediction model. Prediction models are continuously designed but few have had their predictive performance validated with an external population. Because model development is complex, consultation with statistical experts can improve the validity and quality of rigorous prediction model research. After developing the model, vigorous validation with multiple external datasets and effective dissemination to interested parties should occur before using the model in practice [35]. Web or smartphone-based applications can be good routes for advertisement and delivery of clinical prediction models to the public. For example, Korean risk models for diabetes, fatty liver, CVD, and osteoporosis are readily available at http://cmerc.yuhs.ac/mobileweb/. Simple model may be translated into a one page checklist for patient's self-assessment (e.g., equipped in waiting room in clinic). We anticipate that the framework that we provide/summarize, along with additional assistance from related references or textbooks, will help predictive or prognostic research in endocrinology; this will lead to active application of these practices in real world settings. In light of the personalized- and precision-medicine era, further research is needed to attain individual-level predictions, where genetic or novel biomarkers can play bigger roles, as well as simple generalized predictions which can further help patient-centered care.
CONCLUSIONS
-
Acknowledgements
- This study was supported by a grant from the Korea Healthcare Technology R&D Project, Ministry of Health and Welfare, Republic of Korea (No. HI14C2476). H.B. was partly supported by the National Center for Advancing Translational Sciences, National Institutes of Health, through grant UL1 TR 000002. D.K. was partly supported by a grant of the Korean Health Technology R&D Project, Ministry of Health and Welfare, Republic of Korea (HI13C0715).
ACKNOWLEDGMENTS
-
CONFLICTS OF INTEREST: No potential conflict of interest relevant to this article was reported.
Article information
- 1. Moons KG, Royston P, Vergouwe Y, Grobbee DE, Altman DG. Prognosis and prognostic research: what, why, and how? BMJ 2009;338:b375ArticlePubMed
- 2. Hemingway H, Croft P, Perel P, Hayden JA, Abrams K, Timmis A, et al. Prognosis research strategy (PROGRESS) 1: a framework for researching clinical outcomes. BMJ 2013;346:e5595ArticlePubMedPMC
- 3. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn't. BMJ 1996;312:71–72. ArticlePubMedPMC
- 4. Greenland S. The need for reorientation toward cost-effective prediction: comments on 'Evaluating the added predictive ability of a new marker. From area under the ROC curve to reclassification and beyond' by M. J. Pencina et al., Statistics in Medicine (DOI: 10.1002/sim.2929). Stat Med 2008;27:199–206. ArticlePubMed
- 5. Lim NK, Park SH, Choi SJ, Lee KS, Park HY. A risk score for predicting the incidence of type 2 diabetes in a middle-aged Korean cohort: the Korean genome and epidemiology study. Circ J 2012;76:1904–1910. ArticlePubMed
- 6. Griffin SJ, Little PS, Hales CN, Kinmonth AL, Wareham NJ. Diabetes risk score: towards earlier detection of type 2 diabetes in general practice. Diabetes Metab Res Rev 2000;16:164–171. ArticlePubMed
- 7. Goff DC Jr, Lloyd-Jones DM, Bennett G, Coady S, D'Agostino RB, Gibbons R, et al. 2013 ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. Circulation 2014;129(25 Suppl 2):S49–S73. ArticlePubMed
- 8. Steyerberg EW, Vergouwe Y. Towards better clinical prediction models: seven steps for development and an ABCD for validation. Eur Heart J 2014;35:1925–1931. ArticlePubMedPMCPDF
- 9. Royston P, Moons KG, Altman DG, Vergouwe Y. Prognosis and prognostic research: developing a prognostic model. BMJ 2009;338:b604ArticlePubMed
- 10. Altman DG, Vergouwe Y, Royston P, Moons KG. Prognosis and prognostic research: validating a prognostic model. BMJ 2009;338:b605ArticlePubMed
- 11. Moons KG, Altman DG, Vergouwe Y, Royston P. Prognosis and prognostic research: application and impact of prognostic models in clinical practice. BMJ 2009;338:b606ArticlePubMed
- 12. Laupacis A, Sekar N, Stiell IG. Clinical prediction rules. A review and suggested modifications of methodological standards. JAMA 1997;277:488–494. ArticlePubMed
- 13. Altman DG, Royston P. What do we mean by validating a prognostic model? Stat Med 2000;19:453–473. ArticlePubMed
- 14. Steyerberg EW. Clinical prediction models: a practical approach to development, validation, and updating; New York: Springer; 2009.
- 15. Steyerberg EW, Moons KG, van der Windt DA, Hayden JA, Perel P, Schroter S, et al. Prognosis Research Strategy (PROGRESS) 3: prognostic model research. PLoS Med 2013;10:e1001381ArticlePubMedPMC
- 16. Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement. Ann Intern Med 2015;162:55–63. ArticlePubMed
- 17. Liu J, Hong Y, D'Agostino RB Sr, Wu Z, Wang W, Sun J, et al. Predictive value for the Chinese population of the Framingham CHD risk assessment tool compared with the Chinese Multi-Provincial Cohort Study. JAMA 2004;291:2591–2599. ArticlePubMed
- 18. Jee SH, Jang Y, Oh DJ, Oh BH, Lee SH, Park SW, et al. A coronary heart disease prediction model: the Korean Heart Study. BMJ Open 2014;4:e005025.ArticlePubMedPMC
- 19. Lee YH, Bang H, Park YM, Bae JC, Lee BW, Kang ES, et al. Non-laboratory-based self-assessment screening score for non-alcoholic fatty liver disease: development, validation and comparison with other scores. PLoS One 2014;9:e107584ArticlePubMedPMC
- 20. Bang H, Edwards AM, Bomback AS, Ballantyne CM, Brillon D, Callahan MA, et al. Development and validation of a patient self-assessment score for diabetes risk. Ann Intern Med 2009;151:775–783. ArticlePubMedPMC
- 21. Kotronen A, Peltonen M, Hakkarainen A, Sevastianova K, Bergholm R, Johansson LM, et al. Prediction of non-alcoholic fatty liver disease and liver fat using metabolic and genetic factors. Gastroenterology 2009;137:865–872. ArticlePubMed
- 22. Harrell FE Jr. Regression modeling strategies: with applications to linear models, logistic regression, and survival analysis; New York: Springer; 2001.
- 23. Sun GW, Shook TL, Kay GL. Inappropriate use of bivariable analysis to screen risk factors for use in multivariable analysis. J Clin Epidemiol 1996;49:907–916. ArticlePubMed
- 24. Royston P, Altman DG, Sauerbrei W. Dichotomizing continuous predictors in multiple regression: a bad idea. Stat Med 2006;25:127–141. ArticlePubMed
- 25. Boersma E, Poldermans D, Bax JJ, Steyerberg EW, Thomson IR, Banga JD, et al. Predictors of cardiac events after major vascular surgery: role of clinical characteristics, dobutamine echocardiography, and beta-blocker therapy. JAMA 2001;285:1865–1873. ArticlePubMed
- 26. Sauerbrei W. The use of resampling methods to simplify regression models in medical statistics. J R Stat Soc Ser C Appl Stat 1999;48:313–329.Article
- 27. Shmueli G. To explain or to predict. Stat Sci 2010;289–310.Article
- 28. Heikes KE, Eddy DM, Arondekar B, Schlessinger L. Diabetes risk calculator: a simple tool for detecting undiagnosed diabetes and pre-diabetes. Diabetes Care 2008;31:1040–1045. ArticlePubMed
- 29. Breiman L, Friedman J, Stone CJ, Olshen RA. Classification and regression trees; Belmont: Wadsworth International Group; 1984.
- 30. Breiman L. Statistical modeling: the two cultures (with comments and a rejoinder by the author). Statist Sci 2001;16:199–231.Article
- 31. Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 2010;21:128–138. ArticlePubMedPMC
- 32. Meffert PJ, Baumeister SE, Lerch MM, Mayerle J, Kratzer W, Volzke H. Development, external validation, and comparative assessment of a new diagnostic score for hepatic steatosis. Am J Gastroenterol 2014;109:1404–1414. ArticlePubMedPDF
- 33. Hilden J. Commentary: on NRI, IDI, and "good-looking" statistics with nothing underneath. Epidemiology 2014;25:265–267. ArticlePubMed
- 34. Lee YH, Bang H, Kim HC, Kim HM, Park SW, Kim DJ. A simple screening score for diabetes for the Korean population: development, validation, and comparison with other scores. Diabetes Care 2012;35:1723–1730. ArticlePubMedPMC
- 35. Wyatt JC, Altman DG. Commentary: Prognostic models: clinically useful or quickly forgotten? BMJ 1995;311:1539.ArticlePMC
References
Table 1
![enm-31-38-i001.jpg]()
Characteristics of Different Clinical Prediction Models according to Their Purpose
| Characteristic | Prevalent/concurrent events | Incident/future events |
|---|---|---|
| Data type | Cross-sectional data | Longitudinal/prospective cohort data |
| Application | Useful for asymptomatic diseases for screening undiagnosed cases (e.g., diabetes, CKD) | Useful for predicting the incidence of diseases (e.g., CVD, stroke, cancer) |
| Aim of the model | Detection | Prevention |
| Simplicity in model and use | More important | Less important |
| Example | Korean Diabetes Score [34] | ACC/AHA ASCVD risk equation [7] |
Table 2
![enm-31-38-i002.jpg]()
Statistical Measures for Model Evaluation
Figure & Data
References
Citations
Citations to this article as recorded by 
- Models to predict length of stay in the emergency department: a systematic literature review and appraisal
Raheleh Mahboub Farimani, Hesam Karim, Alireza Atashi, Fariba Tohidinezhad, Kambiz Bahaadini, Ameen Abu-Hanna, Saeid Eslami
BMC Emergency Medicine.2024;[Epub] CrossRef - The MISOPRED score: Development and validation of a clinical scoring system to predict the effectiveness of Misoprostol treatment for early pregnancy loss
Tomer Bar-Noy, Ofer Limonad, Erika Gandelsman, Alon Shrim, Hila Sharabi, Raphy Zarecki, Mordechai Hallak, Ilan Bruchim, Federico Ferrari
PLOS ONE.2024; 19(5): e0303607. CrossRef - The Brilliance, Attitude, Leadership and Materials (BALM) framework of clinical excellence: an adoptable model for sub-Saharan Africa
Taoreed Azeez
Academic Medicine & Surgery.2024;[Epub] CrossRef - Risk prediction models for diabetic nephropathy among type 2 diabetes patients in China: a systematic review and meta-analysis
Wenbin Xu, Yanfei Zhou, Qian Jiang, Yiqian Fang, Qian Yang
Frontiers in Endocrinology.2024;[Epub] CrossRef - Risk Assessment Tool in Predicting the Therapeutic Outcomes of Antiseizure Medication in Adults with Epilepsy
Rose Aniza Rusli, Mohd Makmor Bakry, Noraida Mohamed Shah, Xin Ling Loo, Stefanie Kar Yan Hung
Therapeutics and Clinical Risk Management.2024; Volume 20: 529. CrossRef - Development of an IVF prediction model for donor oocytes: a retrospective analysis of 10 877 embryo transfers
Oisin Fitzgerald, Jade Newman, Luk Rombauts, Alex Polyakov, Georgina M Chambers
Human Reproduction.2024;[Epub] CrossRef - Development and Validation of a Prognostic Model to Predict Hearing Recovery for Patients With Chronic Otitis Media
Fengyang Xie, Xiaoyue Zhen, Haiyuan Zhu, Yan Kou, Changle Li, Ling Guo, Li Shi, Jie Han, Xuanchen Zhou
Ear, Nose & Throat Journal.2023; 102(7): NP327. CrossRef - The reporting of prognostic prediction models for obstetric care was poor: a cross-sectional survey of 10-year publications
Chunrong Liu, Yana Qi, Xinghui Liu, Meng Chen, Yiquan Xiong, Shiyao Huang, Kang Zou, Jing Tan, Xin Sun
BMC Medical Research Methodology.2023;[Epub] CrossRef - Severity of Illness Scores and Biomarkers for Prognosis of Patients with Coronavirus Disease 2019
Rodrigo Cavallazzi, James Bradley, Thomas Chandler, Stephen Furmanek, Julio A. Ramirez
Seminars in Respiratory and Critical Care Medicine.2023; 44(01): 075. CrossRef - Prognostic risk factor of major salivary gland carcinomas and survival prediction model based on random survival forests
Yufan Chen, Guoli Li, Wenmei Jiang, Rong Cheng Nie, Honghao Deng, Yingle Chen, Hao Li, Yanfeng Chen
Cancer Medicine.2023; 12(9): 10899. CrossRef - Semantic Visualization in Functional Recovery Prediction of Intravenous Thrombolysis following Acute Ischemic Stroke in Patients by Using Biostatistics: An Exploratory Study
Chih-Chun Hsiao, Chun-Gu Cheng, Cheng-Chueh Chen, Hung-Wen Chiu, Hui-Chen Lin, Chun-An Cheng
Journal of Personalized Medicine.2023; 13(4): 624. CrossRef - Clinical index to quantify the 1-year risk for common postpartum mental disorders at the time of delivery (PMH CAREPLAN): development and internal validation
Simone N. Vigod, Natalie Urbach, Andrew Calzavara, Cindy-Lee Dennis, Andrea Gruneir, Brett D. Thombs, Mark Walker, Hilary K. Brown
The British Journal of Psychiatry.2023; 223(3): 422. CrossRef - Prediction of Foot Ulcers Using Artificial Intelligence for Diabetic Patients at Cairo University Hospital, Egypt
Khadraa Mohamed Mousa, Farid Ali Mousa, Helalia Shalabi Mohamed, Manal Mohamed Elsawy
SAGE Open Nursing.2023;[Epub] CrossRef - Genetic Studies Investigating Susceptibility to Psoriatic Arthritis: A Narrative Review
Mehreen Soomro, Ryan Hum, Anne Barton, John Bowes
Clinical Therapeutics.2023; 45(9): 810. CrossRef - Scope, design, and reporting of prediction models for antineoplastic drugs‐related adverse drug events: A systematic review of machine learning and traditional modeling
Dan Jiang, Zaiwei Song, Yang Hu, Xinya Li, Rongsheng Zhao
Journal of Evidence-Based Medicine.2023; 16(4): 420. CrossRef - Using the Weibull Accelerated Failure Time Regression Model to Predict Time to Health Events
Enwu Liu, Ryan Yan Liu, Karen Lim
Applied Sciences.2023; 13(24): 13041. CrossRef - Development and validation of a nomogram for predicting low birth weight among pregnant women who had antenatal care visits at Debre Markos Comprehensive and Specialized Hospital, Ethiopia
Bezawit Melak Fente, Getayeneh Antehunegn Tesema, Temesgen Worku Gudayu, Mengstu Melkamu Asaye
Frontiers in Medicine.2023;[Epub] CrossRef - Prediction of psychosis: model development and internal validation of a personalized risk calculator
Tae Young Lee, Wu Jeong Hwang, Nahrie S. Kim, Inkyung Park, Silvia Kyungjin Lho, Sun-Young Moon, Sanghoon Oh, Junhee Lee, Minah Kim, Choong-Wan Woo, Jun Soo Kwon
Psychological Medicine.2022; 52(13): 2632. CrossRef - Designing a Predictive Model for Colorectal Neoplasia Diagnosis Based on Clinical and Laboratory Findings in Colonoscopy Candidate Patients
H. Ghajari, A. Sadeghi, S. Khodakarim, M. Zali, S. S. Hashemi Nazari
Journal of Gastrointestinal Cancer.2022; 53(4): 880. CrossRef - Avoiding immediate whole-body trauma CT: a prospective observational study in stable trauma patients
Elisa Reitano, Stefano Granieri, Fabrizio Sammartano, Stefania Cimbanassi, Miriam Galati, Shailvi Gupta, Angelo Vanzulli, Osvaldo Chiara
Updates in Surgery.2022; 74(1): 343. CrossRef - Survival Estimation, Prognostic Factors Evaluation, and Prognostic Prediction Nomogram Construction of Breast Cancer Patients with Bone Metastasis in the Department of Bone and Soft Tissue Tumor: A Single Center Experience of 8 Years in Tianjin, China
Yao Xu, Haixiao Wu, Guijun Xu, Zhuming Yin, Xin Wang, Vladimir P. Chekhonin, Karl Peltzer, Shu Li, Huiyang Li, Jin Zhang, Wenjuan Ma, Chao Zhang, Sharad Goyal
The Breast Journal.2022; 2022: 1. CrossRef - Machine Learning Model-Based Simple Clinical Information to Predict Decreased Left Atrial Appendage Flow Velocity
Chao Li, Guanhua Dou, Yipu Ding, Ran Xin, Jing Wang, Jun Guo, Yundai Chen, Junjie Yang
Journal of Personalized Medicine.2022; 12(3): 437. CrossRef - Performance of Diabetes and Kidney Disease Screening Scores in Contemporary United States and Korean Populations
Liela Meng, Keun-Sang Kwon, Dae Jung Kim, Yong-ho Lee, Jeehyoung Kim, Abhijit V. Kshirsagar, Heejung Bang
Diabetes & Metabolism Journal.2022; 46(2): 273. CrossRef - The Prediction of Diabetes
Lalit Kumar, Prashant Johri
International Journal of Reliable and Quality E-Healthcare.2022; 11(1): 1. CrossRef - Endoscopic detection of esophageal low‐grade squamous dysplasia: How to predict pathologic upgrades before treatment?
Han Chen, Xiao Ying Zhou, Shuo Li, Liu Qin Jiang, Jie Hua, Xin Min Si, Guo Xin Zhang
Journal of Digestive Diseases.2022; 23(4): 209. CrossRef - Predicting outcomes after traumatic brain injury: A novel hospital prediction model for a patient reported outcome
Rachel S. Morris, Juan F. Figueroa, Courtney J. Pokrzywa, Jason K. Barber, Nancy R. Temkin, Carisa Bergner, Basil S. Karam, Patrick Murphy, Lindsay D. Nelson, Purushottam Laud, Zara Cooper, Marc de Moya, Colleen Trevino, Christopher J. Tignanelli, Terri A
The American Journal of Surgery.2022; 224(4): 1150. CrossRef - Investigating factors affecting musculoskeletal disorders: Predictive models for identifying caregivers at risk
Abdulrahman M. Khamaj, Abdulelah M. Ali, Mohd Mukhtar Alam
Work.2022; 72(4): 1311. CrossRef - A scoping review of complication prediction models in spinal surgery: An analysis of model development, validation and impact
Toros C. Canturk, Daniel Czikk, Eugene K. Wai, Philippe Phan, Alexandra Stratton, Wojtek Michalowski, Stephen Kingwell
North American Spine Society Journal (NASSJ).2022; 11: 100142. CrossRef - Dynamic Predictive Models With Visualized Machine Learning for Assessing Chondrosarcoma Overall Survival
Wenle Li, Gui Wang, Rilige Wu, Shengtao Dong, Haosheng Wang, Chan Xu, Bing Wang, Wanying Li, Zhaohui Hu, Qi Chen, Chengliang Yin
Frontiers in Oncology.2022;[Epub] CrossRef - Prognostic Nomogram of Osteocarcinoma after Surgical Treatment
Qiuli Wu, Canchun Yang, Haolin Yan, Zheyu Wang, Zhilei Zhang, Qiwei Wang, Renyuan Huang, Xumin Hu, Bo Li, Xueliang Wu
Journal of Oncology.2022; 2022: 1. CrossRef - External validation and clinical application of the predictive model for severe hypoglycemia
Jae-Seung Yun, Kyungdo Han, Soo-Yeon Choi, Seon-Ah Cha, Yu-Bae Ahn, Seung-Hyun Ko
Frontiers in Endocrinology.2022;[Epub] CrossRef - Clustering of trauma patients based on longitudinal data and the application of machine learning to predict recovery
Kostas Stoitsas, Saurabh Bahulikar, Leonie de Munter, Mariska A. C. de Jongh, Maria A. C. Jansen, Merel M. Jung, Marijn van Wingerden, Katrijn Van Deun
Scientific Reports.2022;[Epub] CrossRef - Clinical predictors of antipsychotic treatment resistance: Development and internal validation of a prognostic prediction model by the STRATA-G consortium
Sophie E. Smart, Deborah Agbedjro, Antonio F. Pardiñas, Olesya Ajnakina, Luis Alameda, Ole A. Andreassen, Thomas R.E. Barnes, Domenico Berardi, Sara Camporesi, Martine Cleusix, Philippe Conus, Benedicto Crespo-Facorro, Giuseppe D'Andrea, Arsime Demjaha, M
Schizophrenia Research.2022; 250: 1. CrossRef - Factors associated with low-compliance bladder in end-stage renal disease patients and development of a clinical prediction model for urodynamic evaluation: the DUDi score
Teerayut Tangpaitoon, Valeerat Swatesutipun
International Urology and Nephrology.2022; 55(1): 75. CrossRef - Improving Machine Learning Diabetes Prediction Models for the Utmost Clinical Effectiveness
Juyoung Shin, Joonyub Lee, Taehoon Ko, Kanghyuck Lee, Yera Choi, Hun-Sung Kim
Journal of Personalized Medicine.2022; 12(11): 1899. CrossRef - Assessment of Simple Bedside Wound Characteristics for a Prediction Model for Diabetic Foot Ulcer Outcomes
Clara Bender, Simon Lebech Cichosz, Louise Pape-Haugaard, Merete Hartun Jensen, Susan Bermark, Anders Christian Laursen, Ole Hejlesen
Journal of Diabetes Science and Technology.2021; 15(5): 1161. CrossRef - A Risk Score for Predicting the Incidence of Hemorrhage in Critically Ill Neonates: Development and Validation Study
Rozeta Sokou, Daniele Piovani, Aikaterini Konstantinidi, Andreas G. Tsantes, Stavroula Parastatidou, Maria Lampridou, Georgios Ioakeimidis, Antonis Gounaris, Nicoletta Iacovidou, Anastasios G. Kriebardis, Marianna Politou, Petros Kopterides, Stefanos Bono
Thrombosis and Haemostasis.2021; 121(02): 131. CrossRef - Development and validation of a risk assessment nomogram for venous thromboembolism associated with hospitalized postoperative Chinese breast cancer patients
Jing Li, Wan‐Min Qiang, Yan Wang, Xiao‐Yuan Wang
Journal of Advanced Nursing.2021; 77(1): 473. CrossRef - The role of behaviour problems in screening for mental ill-health in adults with intellectual disability
F. Westlake, A. Hassiotis, G. Unwin, V. Totsika
The European Journal of Psychiatry.2021; 35(2): 122. CrossRef - Deep learning model for classifying endometrial lesions
YunZheng Zhang, ZiHao Wang, Jin Zhang, CuiCui Wang, YuShan Wang, Hao Chen, LuHe Shan, JiaNing Huo, JiaHui Gu, Xiaoxin Ma
Journal of Translational Medicine.2021;[Epub] CrossRef - Exploration and Development of a Simpler Respiratory Distress Observation Scale (modRDOS-4) as a Dyspnea Screening Tool: A Prospective Bedside Study
Ru Xin Wong, Ho Shirlynn, Yen Sin Koh, Stella Goh Seow Lin, Daniel Quah, Qingyuan Zhuang
Palliative Medicine Reports.2021; 2(1): 9. CrossRef - Impact of CT convolution kernel on robustness of radiomic features for different lung diseases and tissue types
Sarah Denzler, Diem Vuong, Marta Bogowicz, Matea Pavic, Thomas Frauenfelder, Sandra Thierstein, Eric Innocents Eboulet, Britta Maurer, Janine Schniering, Hubert Szymon Gabryś, Isabelle Schmitt-Opitz, Miklos Pless, Robert Foerster, Matthias Guckenberger, S
The British Journal of Radiology.2021;[Epub] CrossRef - An empirical analysis of dealing with patients who are lost to follow-up when developing prognostic models using a cohort design
Jenna M. Reps, Peter Rijnbeek, Alana Cuthbert, Patrick B. Ryan, Nicole Pratt, Martijn Schuemie
BMC Medical Informatics and Decision Making.2021;[Epub] CrossRef - Predictive Value of Active Sacroiliitis in MRI for Flare Among Chinese Patients with Axial Spondyloarthritis in Remission
Qing Zheng, Wen Liu, Yu Huang, Zhenyu Gao, Yuanhui Wu, Xiaohong Wang, Meimei Cai, Yan He, Shiju Chen, Bin Wang, Lingyu Liu, Shuqiang Chen, Hongjie Huang, Ling Zheng, Rihui Kang, Xiaohong Zeng, Jing Chen, Huaning Chen, Junmin Chen, Zhibin Li, Guixiu Shi
Rheumatology and Therapy.2021; 8(1): 411. CrossRef - Development of a model for predicting the 4-year risk of symptomatic knee osteoarthritis in China: a longitudinal cohort study
Limin Wang, Han Lu, Hongbo Chen, Shida Jin, Mengqi Wang, Shaomei Shang
Arthritis Research & Therapy.2021;[Epub] CrossRef - Digital Communication Biomarkers of Mood and Diagnosis in Borderline Personality Disorder, Bipolar Disorder, and Healthy Control Populations
George Gillett, Niall M. McGowan, Niclas Palmius, Amy C. Bilderbeck, Guy M. Goodwin, Kate E. A. Saunders
Frontiers in Psychiatry.2021;[Epub] CrossRef - Prediction of Multiple Organ Failure Complicated by Moderately Severe or Severe Acute Pancreatitis Based on Machine Learning: A Multicenter Cohort Study
Fumin Xu, Xiao Chen, Chenwenya Li, Jing Liu, Qiu Qiu, Mi He, Jingjing Xiao, Zhihui Liu, Bingjun Ji, Dongfeng Chen, Kaijun Liu, Mirella Giovarelli
Mediators of Inflammation.2021; 2021: 1. CrossRef - Comparison of multiple statistical models for the development of clinical prediction scores to detect advanced colorectal neoplasms in asymptomatic Thai patients
Kamonwan Soonklang, Boonying Siribumrungwong, Bunchorn Siripongpreeda, Chirayu Auewarakul
Medicine.2021; 100(20): e26065. CrossRef - European Childhood Obesity Risk Evaluation (CORE) index based on perinatal factors and maternal sociodemographic characteristics: the Feel4Diabetes-study
Christina Mavrogianni, George Moschonis, Eva Karaglani, Greet Cardon, Violeta Iotova, Pilar De Miguel-Etayo, Esther M. González-Gil, Κaloyan Tsochev, Tsvetalina Tankova, Imre Rurik, Patrick Timpel, Emese Antal, Stavros Liatis, Konstantinos Makrilakis, Geo
European Journal of Pediatrics.2021; 180(8): 2549. CrossRef - A prognostic nomogram based on competing endogenous RNA network for clear‐cell renal cell carcinoma
Yun Peng, Shangrong Wu, Zihan Xu, Dingkun Hou, Nan Li, Zheyu Zhang, Lili Wang, Haitao Wang
Cancer Medicine.2021; 10(16): 5499. CrossRef - Individual 5-Year Lung Cancer Risk Prediction Model in Korea Using a Nationwide Representative Database
Yohwan Yeo, Dong Wook Shin, Kyungdo Han, Sang Hyun Park, Keun-Hye Jeon, Jungkwon Lee, Junghyun Kim, Aesun Shin
Cancers.2021; 13(14): 3496. CrossRef - Development and Validation of a Deep Learning Based Diabetes Prediction System Using a Nationwide Population-Based Cohort
Sang Youl Rhee, Ji Min Sung, Sunhee Kim, In-Jeong Cho, Sang-Eun Lee, Hyuk-Jae Chang
Diabetes & Metabolism Journal.2021; 45(4): 515. CrossRef - A nomogram for predicting lymph node metastasis in superficial esophageal squamous cell carcinoma
Weifeng Zhang, Han Chen, Guoxin Zhang, Guangfu Jin
The Journal of Biomedical Research.2021; 35(5): 361. CrossRef - Relationship Between Sensibility Tests and Functional Outcomes in Patients With Traumatic Upper Limb Nerve Injuries: A Systematic Review
Liheng Chen, Emmanuel Ogalo, Chloe Haldane, Sean G. Bristol, Michael J. Berger
Archives of Rehabilitation Research and Clinical Translation.2021; 3(4): 100159. CrossRef - Personalized 5-Year Prostate Cancer Risk Prediction Model in Korea Based on Nationwide Representative Data
Yohwan Yeo, Dong Wook Shin, Jungkwon Lee, Kyungdo Han, Sang Hyun Park, Keun Hye Jeon, Jungeun Shin, Aesun Shin, Jinsung Park
Journal of Personalized Medicine.2021; 12(1): 2. CrossRef - Utility of prediction model score: a proposed tool to standardize the performance and generalizability of clinical predictive models based on systematic review
Jeff Ehresman, Daniel Lubelski, Zach Pennington, Bethany Hung, A. Karim Ahmed, Tej D. Azad, Kurt Lehner, James Feghali, Zorica Buser, James Harrop, Jefferson Wilson, Shekar Kurpad, Zoher Ghogawala, Daniel M. Sciubba
Journal of Neurosurgery: Spine.2021; 34(5): 779. CrossRef - Prodromal clinical, demographic, and socio-ecological correlates of asthma in adults: a 10-year statewide big data multi-domain analysis
Jennifer N. Fishe, Jiang Bian, Zhaoyi Chen, Hui Hu, Jae Min, Francois Modave, Mattia Prosperi
Journal of Asthma.2020; 57(11): 1155. CrossRef - Clinical risk prediction with random forests for survival, longitudinal, and multivariate (RF-SLAM) data analysis
Shannon Wongvibulsin, Katherine C. Wu, Scott L. Zeger
BMC Medical Research Methodology.2020;[Epub] CrossRef - Prognostic models for predicting overall survival in metastatic castration-resistant prostate cancer: a systematic review
M. Pinart, F. Kunath, V. Lieb, I. Tsaur, B. Wullich, Stefanie Schmidt
World Journal of Urology.2020; 38(3): 613. CrossRef - Peripheral Nerve Field Stimulation for Chronic Back Pain: Therapy Outcome Predictive Factors
Eric‐Jan van Gorp, Sam Eldabe, Konstantin V. Slavin, Philippe Rigoard, Stefaan Goossens, Dorothee Mielke, Giancarlo Barolat, Christ Declerck, Chris Gilmore, Ismaïl Gültuna, Kris C.P. Vissers, Jennifer Tinsley, Rudolf Likar, Pierre‐Philippe Luyet
Pain Practice.2020; 20(5): 522. CrossRef - Biomarker Score in Risk Prediction: Beyond Scientific Evidence and Statistical Performance
Heejung Bang
Diabetes & Metabolism Journal.2020; 44(2): 245. CrossRef - Variable selection strategies and its importance in clinical prediction modelling
Mohammad Ziaul Islam Chowdhury, Tanvir C Turin
Family Medicine and Community Health.2020; 8(1): e000262. CrossRef - Reply to the Letter to the Editor: Derivation and Internal Validation of a Clinical Prediction Tool to Predict Nonalcoholic Fatty Liver Disease in Patients With Crohn’s Disease
Scott McHenry, Matthew A Ciorba, Parakkal Deepak
Inflammatory Bowel Diseases.2020; 26(6): e46. CrossRef - The application of unsupervised deep learning in predictive models using electronic health records
Lei Wang, Liping Tong, Darcy Davis, Tim Arnold, Tina Esposito
BMC Medical Research Methodology.2020;[Epub] CrossRef - Development and evaluation of an osteoarthritis risk model for integration into primary care health information technology
Jason E. Black, Amanda L. Terry, Daniel J. Lizotte
International Journal of Medical Informatics.2020; 141: 104160. CrossRef - An Individualized Prediction Model for Long-term Lung Function Trajectory and Risk of COPD in the General Population
Wenjia Chen, Don D. Sin, J. Mark FitzGerald, Abdollah Safari, Amin Adibi, Mohsen Sadatsafavi
Chest.2020; 157(3): 547. CrossRef - Optimization of the management of pregnant women at high risk of miscarriage and premature birth
Yu. A. Semenov, V. F. Dolgushina, M. G. Moscvicheva, V. S. Chulkov
Rossiiskii vestnik akushera-ginekologa.2020; 20(1): 54. CrossRef - Developing a triage tool for use in identifying people living with HIV who are at risk for non-retention in HIV care
Merhawi T Gebrezgi, Kristopher P Fennie, Diana M Sheehan, Boubakari Ibrahimou, Sandra G Jones, Petra Brock, Robert A Ladner, Mary Jo Trepka
International Journal of STD & AIDS.2020; 31(3): 244. CrossRef - The impact of age and comorbidity on the postoperative outcomes after emergency surgical management of complicated intra-abdominal infections
Carmen Payá-Llorente, Elías Martínez-López, Juan Carlos Sebastián-Tomás, Sandra Santarrufina-Martínez, Nicola de’Angelis, Aleix Martínez-Pérez
Scientific Reports.2020;[Epub] CrossRef - Precision health through prediction modelling: factors to consider before implementing a prediction model in clinical practice
Mohammad Z. I. Chowdhury, Tanvir C. Turin
Journal of Primary Health Care.2020; 12(1): 3. CrossRef - Demystifying artificial intelligence in pharmacy
Scott D Nelson, Colin G Walsh, Casey A Olsen, Andrew J McLaughlin, Joseph R LeGrand, Nick Schutz, Thomas A Lasko
American Journal of Health-System Pharmacy.2020; 77(19): 1556. CrossRef - Association does not imply prediction: the accuracy of birthweight in predicting child mortality and anthropometric failure
Akshay Swaminathan, Rockli Kim, S.V. Subramanian
Annals of Epidemiology.2020; 50: 7. CrossRef - Who's at Risk? A Prognostic Model for Severity Prediction in Pediatric Acute Pancreatitis
Peter R. Farrell, Lindsey Hornung, Peter Farmer, Angelica W. DesPain, Esther Kim, Ryan Pearman, Beemnet Neway, Ashley Serrette, Sona Sehgal, James E. Heubi, Tom K. Lin, Jaimie D. Nathan, David S. Vitale, Maisam Abu‐El‐Haija
Journal of Pediatric Gastroenterology and Nutrition.2020; 71(4): 536. CrossRef - Biomarkers of Fabry Nephropathy: Review and Future Perspective
Tina Levstek, Bojan Vujkovac, Katarina Trebusak Podkrajsek
Genes.2020; 11(9): 1091. CrossRef - Coledocolitiasis y pancreatitis: las dificultades de la predicción
David Benigno Páramo Hernández
Revista Colombiana de Gastroenterología.2020; 35(3): 266. CrossRef - Consistency of variety of machine learning and statistical models in predicting clinical risks of individual patients: longitudinal cohort study using cardiovascular disease as exemplar
Yan Li, Matthew Sperrin, Darren M Ashcroft, Tjeerd Pieter van Staa
BMJ.2020; : m3919. CrossRef - Outcome prediction with serial neuron-specific enolase and machine learning in anoxic-ischaemic disorders of consciousness
Emily Muller, Jonathan P. Shock, Andreas Bender, Julian Kleeberger, Tobias Högen, Martin Rosenfelder, Bubacarr Bah, Alex Lopez-Rolon
Computers in Biology and Medicine.2019; 107: 145. CrossRef - Machine learning models to predict disease progression among veterans with hepatitis C virus
Monica A. Konerman, Lauren A. Beste, Tony Van, Boang Liu, Xuefei Zhang, Ji Zhu, Sameer D. Saini, Grace L. Su, Brahmajee K. Nallamothu, George N. Ioannou, Akbar K. Waljee, Davide Bacciu
PLOS ONE.2019; 14(1): e0208141. CrossRef - A novel risk calculator to predict outcome after surgery for symptomatic spinal metastases; use of a large prospective patient database to personalise surgical management
David Choi, Menelaos Pavlou, Rumana Omar, Mark Arts, Laurent Balabaud, Jacob Maciej Buchowski, Cody Bunger, Chun Kee Chung, Maarten Hubert Coppes, Bart Depreitere, Michael George Fehlings, Norio Kawahara, Chong-Suh Lee, YeeLing Leung, Juan Antonio Martin-
European Journal of Cancer.2019; 107: 28. CrossRef - Big Data Research in Neuro-Ophthalmology: Promises and Pitfalls
Heather E. Moss, Charlotte E. Joslin, Daniel S. Rubin, Steven Roth
Journal of Neuro-Ophthalmology.2019; 39(4): 480. CrossRef - Nonalcoholic Fatty Liver Disease in Diabetes. Part I: Epidemiology and Diagnosis
Yong-ho Lee, Yongin Cho, Byung-Wan Lee, Cheol-Young Park, Dae Ho Lee, Bong-Soo Cha, Eun-Jung Rhee
Diabetes & Metabolism Journal.2019; 43(1): 31. CrossRef - The use of rigorous methods was strongly warranted among prognostic prediction models for obstetric care
Jing Tan, Yana Qi, Chunrong Liu, Yiquan Xiong, Qiao He, Guiting Zhang, Meng Chen, Guolin He, Wen Wang, Xinghui Liu, Xin Sun
Journal of Clinical Epidemiology.2019; 115: 98. CrossRef - Support Vector Machines and logistic regression to predict temporal artery biopsy outcomes
Edsel Ing, Wanhua Su, Matthias Schonlau, Nurhan Torun
Canadian Journal of Ophthalmology.2019; 54(1): 116. CrossRef - Identifying a risk score for childhood obesity based on predictors identified in pregnant women and 1-year-old infants: An analysis of the data of the Hokkaido Study on Environment and Children’s Health
Yasuaki Saijo, Yoshiya Ito, Eiji Yoshioka, Yukihiro Sato, Machiko Minatoya, Atsuko Araki, Chihiro Miyashita, Reiko Kishi
Clinical Pediatric Endocrinology.2019; 28(3): 81. CrossRef - Development and performance evaluation of the Medicines Optimisation Assessment Tool (MOAT): a prognostic model to target hospital pharmacists’ input to prevent medication-related problems
Cathy Geeson, Li Wei, Bryony Dean Franklin
BMJ Quality & Safety.2019; 28(8): 645. CrossRef - Machine Learning Accurately Predicts Short-Term Outcomes Following Open Reduction and Internal Fixation of Ankle Fractures
Robert K. Merrill, Rocco M. Ferrandino, Ryan Hoffman, Gene W. Shaffer, Anthony Ndu
The Journal of Foot and Ankle Surgery.2019; 58(3): 410. CrossRef - Multidimensional screening for predicting pain problems in adults: a systematic review of screening tools and validation studies
Elke Veirman, Dimitri M. L. Van Ryckeghem, Annick De Paepe, Olivia J. Kirtley, Geert Crombez
PAIN Reports.2019; 4(5): e775. CrossRef - iHealthcare: Predictive Model Analysis Concerning Big Data Applications for Interactive Healthcare Systems †
Md. Ataur Rahman Bhuiyan, Md. Rifat Ullah, Amit Kumar Das
Applied Sciences.2019; 9(16): 3365. CrossRef - Development of personalized mobile assistant for chronic disease patients: diabetes mellitus case study
M.V. Kabyshev, S.V. Kovalchuk
Procedia Computer Science.2019; 156: 123. CrossRef - Validation of childhood asthma predictive tools: A systematic review
Silvia Colicino, Daniel Munblit, Cosetta Minelli, Adnan Custovic, Paul Cullinan
Clinical & Experimental Allergy.2019; 49(4): 410. CrossRef - Predicting intradialytic hypotension using heart rate variability
Samel Park, Wook-Joon Kim, Nam-Jun Cho, Chi-Young Choi, Nam Hun Heo, Hyo-Wook Gil, Eun Young Lee
Scientific Reports.2019;[Epub] CrossRef - Predicting treatment response using pharmacy register in migraine
Thomas Folkmann Hansen, Mona Ameri Chalmer, Thilde Marie Haspang, Lisette Kogelman, Jes Olesen
The Journal of Headache and Pain.2019;[Epub] CrossRef - Development and Evaluation of Electronic Health Record Data-Driven Predictive Models for Pressure Ulcers
Seul Ki Park, Hyeoun-Ae Park, Hee Hwang
Journal of Korean Academy of Nursing.2019; 49(5): 575. CrossRef - Characteristics and outcome of acute heart failure patients according to the severity of peripheral oedema
Ahmad Shoaib, Mamas A. Mamas, Qazi S. Ahmad, Theresa M. McDonagh, Suzanna M.C. Hardman, Muhammad Rashid, Robert Butler, Simon Duckett, Duwarakan Satchithananda, James Nolan, Henry J. Dargie, Andrew L. Clark, John G.F. Cleland
International Journal of Cardiology.2019; 285: 40. CrossRef - Methodological standards for the development and evaluation of clinical prediction rules: a review of the literature
Laura E. Cowley, Daniel M. Farewell, Sabine Maguire, Alison M. Kemp
Diagnostic and Prognostic Research.2019;[Epub] CrossRef - Development and validation of a nomogram to predict the prognosis of patients with squamous cell carcinoma of the bladder
Mei-Di Hu, Si-Hai Chen, Yuan Liu, Ling-Hua Jia
Bioscience Reports.2019;[Epub] CrossRef - Assessing surgical difficulty in locally advanced mid–low rectal cancer: the accuracy of two MRI‐based predictive scores
N. de'Angelis, F. Pigneur, A. Martínez‐Pérez, G. C. Vitali, F. Landi, S. A. Gómez‐Abril, M. Assalino, E. Espin, F. Ris, A. Luciani, F. Brunetti
Colorectal Disease.2019; 21(3): 277. CrossRef - External Validation of START nomogram to predict 3-Month unfavorable outcome in Chinese acute stroke patients
BaiLi Song, XiangLiang Chen, Dan Tang, Mako Ibrahim, YuKai Liu, Linda Nyame, Teng Jiang, Wei Wang, Xiang Li, Chao Sun, Zheng Zhao, Jie Yang, JunShan Zhou, JianJun Zou
Journal of Stroke and Cerebrovascular Diseases.2019; 28(6): 1618. CrossRef - Development and Validation of the Korean Diabetes Risk Score: A 10-Year National Cohort Study
Kyoung Hwa Ha, Yong-ho Lee, Sun Ok Song, Jae-woo Lee, Dong Wook Kim, Kyung-hee Cho, Dae Jung Kim
Diabetes & Metabolism Journal.2018; 42(5): 402. CrossRef - Clinical relevance and validity of tools to predict infant, childhood and adulthood obesity: a systematic review
Oliver J Canfell, Robyn Littlewood, Olivia RL Wright, Jacqueline L Walker
Public Health Nutrition.2018; 21(17): 3135. CrossRef - Letter to Editor
Laura E Cowley, Sabine A Maguire, Daniel M Farewell, Alison M Kemp
Law, Probability and Risk.2018; 17(3): 275. CrossRef - Self‐report assessment of severe periodontitis: Periodontal screening score development
Maria Clotilde Carra, Alice Gueguen, Frédérique Thomas, Bruno Pannier, Giuseppina Caligiuri, Philippe Gabriel Steg, Marie Zins, Philippe Bouchard
Journal of Clinical Periodontology.2018; 45(7): 818. CrossRef - Predictive validity of the CriSTAL tool for short-term mortality in older people presenting at Emergency Departments: a prospective study
Magnolia Cardona, Ebony T. Lewis, Mette R. Kristensen, Helene Skjøt-Arkil, Anette Addy Ekmann, Hanne H. Nygaard, Jonas J. Jensen, Rune O. Jensen, Jonas L. Pedersen, Robin M. Turner, Frances Garden, Hatem Alkhouri, Stephen Asha, John Mackenzie, Margaret Pe
European Geriatric Medicine.2018; 9(6): 891. CrossRef - Prediction of Drug-Related Risks Using Clinical Context Information in Longitudinal Claims Data
Andreas D. Meid, Andreas Groll, Dirk Heider, Sarah Mächler, Jürgen-Bernhard Adler, Christian Günster, Hans-Helmut König, Walter E. Haefeli
Value in Health.2018; 21(12): 1390. CrossRef - Articles inEndocrinology and Metabolismin 2016
Won-Young Lee
Endocrinology and Metabolism.2017; 32(1): 62. CrossRef - Development of Clinical Data Mart of HMG-CoA Reductase Inhibitor for Varied Clinical Research
Hun-Sung Kim, Hyunah Kim, Yoo Jin Jeong, Tong Min Kim, So Jung Yang, Sun Jung Baik, Seung-Hwan Lee, Jae Hyoung Cho, In Young Choi, Kun-Ho Yoon
Endocrinology and Metabolism.2017; 32(1): 90. CrossRef - Response to Comment by Ayubi and Safiri. Insulin Resistance Predicts Cognitive Decline: An 11-Year Follow-up of a Nationally Representative Adult Population Sample. Diabetes Care 2017;40:751–758
Laura L. Ekblad, Juha O. Rinne, Pauli Puukka, Hanna Laine, Satu Ahtiluoto, Raimo Sulkava, Matti Viitanen, Antti Jula
Diabetes Care.2017; 40(9): e136. CrossRef - Encrypted prediction: A hacker's perspective
Tara Karamlou, Daniel A. Velez, John J. Nigro
The Journal of Thoracic and Cardiovascular Surgery.2017; 154(6): 2038. CrossRef - Personalized medicine. Closing the gap between knowledge and clinical practice
Juan-Manuel Anaya, Carolina Duarte-Rey, Juan C. Sarmiento-Monroy, David Bardey, John Castiblanco, Adriana Rojas-Villarraga
Autoimmunity Reviews.2016; 15(8): 833. CrossRef - Comparison of screening scores for diabetes and prediabetes
Eduard Poltavskiy, Dae Jung Kim, Heejung Bang
Diabetes Research and Clinical Practice.2016; 118: 146. CrossRef
PubReader
Cite
XML DownloadHow to Establish Clinical Prediction Models
How to Establish Clinical Prediction Models
| Characteristic | Prevalent/concurrent events | Incident/future events |
|---|---|---|
| Data type | Cross-sectional data | Longitudinal/prospective cohort data |
| Application | Useful for asymptomatic diseases for screening undiagnosed cases (e.g., diabetes, CKD) | Useful for predicting the incidence of diseases (e.g., CVD, stroke, cancer) |
| Aim of the model | Detection | Prevention |
| Simplicity in model and use | More important | Less important |
| Example | Korean Diabetes Score [ | ACC/AHA ASCVD risk equation [ |
| Sensitivity and specificity |
| Discrimination (ROC/AUC) |
| Predictive values: positive, negative |
| Likelihood ratio: positive, negative |
| Accuracy: Youden index, Brier score |
| Number needed to treat or screen |
| Calibration: Calibration plot, Hosmer-Lemeshow test |
| Model determination: R2 |
| Statistical significance: P value (e.g., likelihood ratio test) |
| Magnitude of association, e.g., β coefficient, odds ratio |
| Model quality: AIC/BIC |
| Net reclassification index and integrated discrimination improvement |
| Net benefit |
| Cost-effectiveness |
Table 1 Characteristics of Different Clinical Prediction Models according to Their Purpose
CKD, chronic kidney disease; CVD, cardiovascular disease; ACC/AHA, American College of Cardiology/American Heart Association; ASCVD, atherosclerotic cardiovascular disease.
Table 2 Statistical Measures for Model Evaluation
ROC, receiver operating characteristic; AUC, area under the curve; AIC, Akaike information criterion; BIC, Bayesian information criterion.
