External Validation of clinical prediction models

Validation, particularly external Validation, is a crucial part of developing a predictive model. External Validation is needed to ensure that a prediction model is generalizable to patients other than those in the derivative cohort. External Validation can be done by testing the model’s output in data that isn’t the same as the data used to create the model. As a consequence, it is carried out after the creation of a prediction model.

External Validation

External Validation can take many forms, including Validation in the field such as temporal, geographical and independent Validation. For external validation studies, the sample size calculation estimates based on statistical power considerations have not been extensively investigated. However, in order to achieve adequate model output in the validation set, large sample size is needed to validate the prediction model.

Factors influences and affect external validation data

• The sample size for external validation data for implementing the prediction model is affected by the number of events and predictors.
• According to simulation studies, the prediction model’s external Validation requires a minimum of 100 events and/or non-events. A systematic analysis found that small external validation studies are ineffective and inaccurate.
• Example: Radiology imaging is often treated as effective predictive parameters, and researchers often validate the findings using a clinical prediction model. Every prediction model is based on regression analysis.
• The most common predictive model or the regression model used for the clinical prediction model is linear regression. The dependent variable is continuous, logistic regression model if the dependent variable is binary, and Cox-proportional model if the dependent variable is time-to-event in nature.
•  Al-Ameri et al. (2020) presented a detailed review of clinical prediction models for liver transplantation study.
• Further, Ratna et al. (2020) discussed the clinical prediction model’s quality in vitro fertilization and human reproduction. The Validation of the model was carried out using the re-sampling technique and measured the accuracy using AUC, as shown in figure 1, c-index, and Hosmer-Lemeshow test statistic.

Figure 1: Slope of Calibration plot (Source: Stevens and Poppe (2020))

In addition, Stevens and Poppe (2020) suggested the Cox- calibration slope using the logistic regression model instead of simply using the predictive model’s calibration slope. This suggestion has been made after the scrutiny of around 33 research articles and found that most of the Validation is external Validation and identified the validity using the calibration slope. The sample articles are presented in the following table.

Table1.Stated Interpretation of the “Calibration Slope” Source: Stevens and Poppe (2020)

Arjun et al. (2020) considered the pandemic mortality study of COVID19 and discussed the clinical prediction model’s development and Validation.

Future Scope 

Though much literature suggests several validation techniques for the predictive model, no such proper technique can be suitable for all the clinical datasets. Further, the proper adjustment must be made for the calibration index to validate the prediction model suitable for all clinical datasets.

References:

1. Stevens, R. J. and Poppe, K. K. (2020). Validation of Clinical Prediction Models: What does the “Calibration Slope” Really Measure?. Journal of clinical epidemiology, 118, pp. 93–99.
2. Adibi, A., Sadatsafavi, M., Ioannidis, J. P. A. (2020). Validation and Utility Testing of Clinical Prediction Models: Time to Change the Approach. JAMA. 2020; 324(3):235–236.
3. Ratna, M. B., Bhattacharya, S., Abdulrahim, B. and McLernon, D. L. (2020). A Systematic Review of the Quality of Clinical Prediction Models in Vitro Fertilisation, Human Reproduction, 35(1), pp. 100–116
4. Arjun S Yadaw., Yan-chak Li., Sonali Bose., Ravi Iyengar., Supinda Bunyavanich., Gaurav Pandey. (2020). Clinical Features of COVID-19 Mortality: Development and Validation of a Clinical Prediction Model, The Lancet Digital Health, 2(10), pp. 516-525.
5. Al‐Ameri, A.A.M., Wei, X., Wen, X., Wei, Q., Guo, H., Zheng, S. and Xu, X. (2020), Systematic review: risk prediction models for recurrence of hepatocellular carcinoma after liver transplantation. Transpl Int, 33, pp. 697-712.