Browse the corpus

Clinical prediction models use mathematical functions to combine information from an individual’s characteristics and use the resulting output to estimate the chances that a condition of interest is present (diagnostic models) or will occur in the future (prognostic models).1 These estimates are in turn used to start or withhold treatments, to order new tests, or to educate individuals on their condition’s outlook. Prediction models are increasingly used in clinical medicine. In nephrology, estimators of kidney function are used to predict glomerular filtration rate because the direct measure of kidney function is not feasible in routine settings. In people already diagnosed with chronic kidney disease, prediction models are used to estimate the risk of progression to renal replacement therapy and to predict other outcomes such as mortality or cardiovascular diseases.2

ration rate because the direct measure of kidney function is not feasible in routine settings. In people already diagnosed with chronic kidney disease, prediction models are used to estimate the risk of progression to renal replacement therapy and to predict other outcomes such as mortality or cardiovascular diseases.2 In a linked BMJ paper, Ping Liu and colleagues (doi:10.1136/bmj-2023-078063) report the development and evaluation of KDpredict,3 a suite of new models for predicting future kidney failure and all cause mortality in adults with moderate to severe chronic kidney disease over one to five years. KDpredict was developed using the super learner strategy based on machine learning.4 The super learner is a prediction method designed to find the best combination from a collection of prediction algorithms (also known as learners). Prediction algorithms are ranked according to prediction error (Brier score)—a measure of discrepancy between the expected risk of a particular outcome (from the model derived from a training dataset) and the risk observed in a validation dataset. A lower prediction error indicates a better model performance.

earners). Prediction algorithms are ranked according to prediction error (Brier score)—a measure of discrepancy between the expected risk of a particular outcome (from the model derived from a training dataset) and the risk observed in a validation dataset. A lower prediction error indicates a better model performance. The KDpredict investigators used the discrete super learner approach for evaluation, which selects the best prediction algorithm from a library of prespecified algorithms. They developed KDpredict using data from a cohort of patients in Alberta (Canada) and externally validated the model using patient cohorts from Denmark and Scotland. KDpredict showed that risk of death over five years was higher than the risk of progression to kidney failure in most subgroups with moderate to severe chronic kidney disease at baseline. The model’s performance was excellent when tested in the Danish and Scottish cohorts. Traditionally, the kidney failure risk equation (known as KFRE), also developed in Canada,5 has been the benchmark model for predicting outcomes among people with chronic kidney disease, using the same core predictors as KDpredict. When both models were tested in patient cohorts from Denmark and Scotland, KDpredict performed better than the kidney failure risk equation at predicting end-stage kidney failure.

has been the benchmark model for predicting outcomes among people with chronic kidney disease, using the same core predictors as KDpredict. When both models were tested in patient cohorts from Denmark and Scotland, KDpredict performed better than the kidney failure risk equation at predicting end-stage kidney failure. Beside differences in model development (super learner for KDpredict v standard Cox regressions modelling for kidney failure risk equation), KDpredict and kidney failure risk equation differed in various ways. The population that was used to develop KDpredict was older (median age 77-80 years v 69-70 years) and included people with more advanced chronic kidney disease (estimated glomerular filtration rate <45 mL/min/1.73 m2 v <59 mL/min/1.73 m2). These dissimilarities may account for some of the difference in performance. While both models incorporated the same core variables, KDpredict included substantially more predictors overall. This broader approach makes more efficient use of predictive information, which translates into enhanced predictive performance. This approach can result in models that perform well in the derivation sample, but less well when tested in a different population. Furthermore, the increased complexity of models such as KDpredict means that routine uptake will depend on automated computation of risk. Widespread availability of smart devices and internet connectivity may help to overcome this issue.

ell in the derivation sample, but less well when tested in a different population. Furthermore, the increased complexity of models such as KDpredict means that routine uptake will depend on automated computation of risk. Widespread availability of smart devices and internet connectivity may help to overcome this issue. The KDpredict investigators cite various limitations of the kidney failure risk equation to justify the development of new and more complex models; however, some of these limitations are arguable. Firstly, although the current kidney failure risk equation does not account for competing risks, the investigators of the equation did develop an alternative model using competing risk approach that made no material difference to the original equation's performance.5 Secondly, the need to predict all cause mortality for people with chronic kidney disease (available with KDpredict but not with kidney failure risk equation), is debatable. Cardiovascular disease is the leading driver of mortality among adults with moderate to severe chronic kidney disease so these individuals typically have multiple treatments to reduce cardiovascular disease, irrespective of their absolute cardiovascular disease risk. Therefore, estimating mortality risk is unlikely to significantly affect patients’ clinical management, including preparing them for the possibility of end-stage renal replacement treatment.

typically have multiple treatments to reduce cardiovascular disease, irrespective of their absolute cardiovascular disease risk. Therefore, estimating mortality risk is unlikely to significantly affect patients’ clinical management, including preparing them for the possibility of end-stage renal replacement treatment. The proposed locally optimised decision support with KDpredict that moves away from the current one-size-fits-all approach would require a specific model for each setting. This customisation is not achievable because the data needed to optimise the model locally will not always be available. Furthermore, the opportunity to compare the models’ performance within and across settings will be missed. Additionally, the super learner approach to prediction models requires substantial input from experts with knowledge of the field of interest; knowledge that is generally derived from conventional approaches to prediction. As such, super learner approaches to prediction are better positioned as an extension of, or enhancement to, the conventional one-size-fit-all approach. KDpredict is a potentially useful tool for risk stratification in people with chronic kidney disease, to be integrated into routine care, alongside existing tools such as the kidney failure risk equation. But, further external validation of KDpredict in diverse settings by independent investigators is needed.6 7