Author(s): Aasthaa Bansal, Nicole Mayer-Hamblett, Christopher H Goss, Lingtak N. Chan and Patrick J. Heagerty
Background: Effective transplantation recommendations in cystic fibrosis (CF) require accurate survival predictions, so that high-risk patients may be prioritized for transplantation. In practice, decisions about transplantation are made dynamically, using routinely updated assessments. We present a novel tool for evaluating risk prediction models that, unlike traditional methods, captures classification accuracy in identifying high-risk patients in a dynamic fashion.
Methods: Predicted risk is used as a score to rank incident deaths versus patients who survive, with the goal of ranking the deaths higher. The mean rank across deaths at a given time measures time-specific predictive accuracy; when assessed over time, it reflects time-varying accuracy.
Results: Applying this approach to CF Registry data on patients followed from 1993-2011 we show that traditional methods do not capture the performance of models used dynamically in the clinical setting. Previously proposed multivariate risk scores perform no better than forced expiratory volume in 1 second as a percentage of predicted normal (FEV1%) alone. Despite its value for survival prediction, FEV1% has a low sensitivity of 45% over time (for fixed specificity of 95%), leaving room for improvement in prediction. Finally, prediction accuracy with annually-updated FEV1% shows minor differences compared to FEV1% updated every 2 years, which may have clinical implications regarding the optimal frequency of updating clinical information.
Conclusions: It is imperative to continue to develop models that accurately predict survival in CF. Our proposed approach can serve as the basis for evaluating the predictive ability of these models by better accounting for their dynamic clinical use.
