Is another validation of a clinical prediction rule necessary? A demonstration of research wastes using recursive cumulative meta-analyses

Problem

Although the number of studies validating clinical prediction rules increased dramatically, they have been unevenly focused on a few prediction rules leaving many prediction rules without any external validation. The objective of this study was to demonstrate research wastes related to conducting many external validation studies of a clinical prediction rule.

Approach

Data from validation studies of Pneumonia Severity Index (PSI) and Alvarado Score included in two published meta-analyses were analyzed. From each validation study, the date of publication, total number of subjects, and number of predicted and observed events were recorded. For each prediction rule, a random-effects cumulative meta-analysis of predictive performance (predicted/observed event ratio) was conducted according to the publication date. Then, the trajectory of previous to current cumulative predictive performance ratio over information step (addition of a new validation study) was graphically assessed. The number of validation studies and participants included in the validation studies were calculated before and after the stability of predictive performance is reached.

Findings

First, 30 validation studies of Pneumonia Severity Index (PSI) which contained 26563 participants were analyzed. After the data from the 12th validation study was added to the recursive cumulative meta-analysis, the trajectory of cumulative predictive performance (predicted/observed event ratio) became stable (sustained less than 5% change). Therefore, 19 (63.3%) validation studies and the data from 17443 (65.7%) participants added little value. Secondly, 34 studies validating Alvarado Score (9778 participants) were assessed. The trajectory of cumulative predictive performance (predicted/observed event ratio) became stable after the data from the 7th validation study was added to the recursive cumulative meta-analysis. 24 (80%) validation studies and data from 8066 (82.5%) participants included in these validation had little value. A recalibration was done in only 1 validation study of PSI.

Consequences

Substantial degree of research wastes were demonstrated in the validation of Pneumonia Severity Index (PSI) and Alvarado Score. Before a validation of a clinical prediction rule is carried out, researchers should carefully consider whether it is truly necessary.