Can we distinguish variation in referral thresholds from variation in referral accuracy?

Problem

Variation between primary care practices in referral for possible cancer is widely recognised as a problem. Current reporting does not differentiate between variation in referral threshold (which is largely to do with tolerance of risk and primarily has consequences for resource use) and variation in accurate selection of patients for referral (which may be more of a marker of clinical quality, with consequences for the time to diagnosis). We aimed to develop a method to distinguish between these two sources of variation and applied it to fast-track referrals for possible cancer.

Approach

We analysed data from 5479 English primary care practices over a 5-year period. Publicly available data provide the true positive, false positive and false negative cells of a contingency table. We developed and tested a method for imputing the number of true negatives. From this we estimated sensitivity and specificity for each practice regarding decisions to refer patients on fast-track pathways for possible cancer. We combined these using bivariate meta-analysis and used practices’ 95% confidence regions to characterise outliers in respect of referral threshold and selection accuracy.

Findings

Practices varied more in threshold than accuracy. 2019 practices (36.8%) were outliers in relation to referral threshold compared to 1205 practices (22.0%) in relation to selection accuracy. Practice age profile, cancer incidence, and deprivation showed a modest association with selection accuracy but not with thresholds.

Consequences

1. Existing reporting methods ("detection rate" and "conversion rate") do not permit this description of variation as due to thresholds or accuracy - they should be changed.2. This approach means that practices can be profiled more accurately - in two dimensions rather than one.3. We have used the data to model the effect of changing thresholds without increasing accuracy. If all practices were like the highest quintile of age-standardised referral rate there would be a 3.3% increase in the number of cancers diagnosed via the fast-track pathways but a 37% increase in referrals.