Surgical Performance Data Published
The first set of surgical performance data was published on Friday (28 June 2013) on vascular surgery, as paving the way for the publication of performance statistics for many areas of surgery over the next few weeks and months.
Is this a great step forward for transparency and patient trust in surgeons or does this present a risk of statistical misinterpretation and defensive practice?
Anna Neil comments upon the decision to publish performance data for individual consultant surgeons.
The call for the publication of surgical performance data for individual surgeons was made by Professor Ian Kennedy in 2001 following inquests into child and baby deaths at the Paediatric Cardiac Unit of Bristol Royal Infirmary. The Paediatric Cardiac Unit had been operating on babies for many years despite damning evidence that their surgical outcomes were catastrophically worse than at other specialist units.
The performance data of (adult) cardiac surgeons has been published since 2005. Outcomes have been tracked within the NHS for years in order to improve practice but no data has been made public. This is now to change with data due to be published in Orthopaedic surgery, Bariatic surgery, Colorectal surgery, Urological surgery, thyroid and endocrine surgery, head and neck cancer surgery, interventional cardiology and gastro-intestinal surgery.
Friday’s publication comes in the wake of further scandal and allegation of cover ups within the NHS following the Mid-Staffordshire Inquiry and more recently the accusation of a cover up at the NHS Watchdog – the Care Quality Commission.
Sir David Nicholson conceded this week that there remained a “culture of denial” and “defensiveness” within the NHS.
An improvement in openness and transparency?
Sir Bruce Keogh heralded this as a huge step forward in restoring public trust in the NHS and improving NHS transparency. It is hoped that the published figures will provide the public with the information they need to understand the service they are receiving, the performance levels of that service and the information they need on which to base a truly informed consent to surgery. After years of scandal surrounding lack of transparency and cover up this appears to be the first significant move in the other direction.
It is to be hoped that the NHS will see a culture change toward transparency and candour in relation to its outcomes and performance.
Improvement in Performance?
It is hoped that in publishing data, surgeons and their teams will be encouraged to constantly work on improving their performance. There can be nothing quite like the promise of public scrutiny of your outcomes to motivate you to improve. A surgeon’s performance will be “there to be seen”. It is hoped that it will improve standards and encourage surgeons to strive for excellence.
Reference has been made this week cardiac surgery which has enjoyed significant improvement since the first publication of individual consultant performance data in 2005. This is said to be an example that publishing data has and will lead to improved standards.
What about accuracy and correct interpretation of data?
Ultimately, only 6 of nearly 400 vascular surgeons refused to allow their performance data to be published. Their reasons are set out within the report and are in large part due to concerns over the accuracy of the statistical reporting and potential damage due to misinterpretation of the data.
Did the overwhelming majority take part due to faith in the system or due to knowledge that their names would be printed if they did not?
Whilst the Data report on vascular surgery and the BBC made it clear that none of those who refused to take part had statistics outside the national average, the Telegraph still speculated today that “six surgeons refused, leading to suggestions that their death rates may be even higher”.
What does the data actually tell us?
Data has been published on two different types of vascular operation. For both, the number of operations carried out per surgeon and the mortality rate, as an adjusted percentage, is provided for each and every surgeon. The report points out that the percentage mortality statistic must be interpreted with significant care due to the impact of random variation. Results had to be risk adjusted to go some way to reflect the different risks between procedures and individual patients. The results were then plotted to show whether the results were within the expected range of the national average, bearing in mind the force of random variation.
The report points out that whilst many surgeons were, on the surface of things, outside the national average (either above or below), this was to be expected due to the impact of random variation, and all surgeons fell within the expected range of national average.
What we don’t know is what statistical adjustments have been made, what factors have or have not been taken in to account or the accuracy of the analysis of random variation and it’s impact upon performance as compared with national average. Were all surgeons performing within a reasonable average? What is a reasonable average?
There seem to be a lot of unanswered questions in relation to the presentation of data and interpretation of statistics with considerable risk of mis-interpretation such that poor surgeons may be missed and good surgeons incorrectly labelled as poor.
A league table of surgical mortality?
It has been made clear that the surgical performance data should be used as a guide to surgery within the NHS and to provide the public with an indication of what they can expect of surgery. It is not supposed to be used as a “league table of surgical outcomes” presumably due to the inherent difficulty in interpreting the data outlined above. The vascular data report states in the introduction that it must not be seen as a league table of surgeons’ mortality rates. The report tries to explain the difficulty with interpreting the data. It explains that, effectively, the mortality figures provided must not be taken on face value.
This is all very well and good, but the reality is that most people will take the results at face value. Most people do not understand the innate difficulty with applying and interpreting statistics. The reality is that the data will be seen as a league table of mortality performance. This is potentially dangerous since the reader has no way of understanding the reality behind the data. We do not know what the underlying risk of any particular operation – we do not know the age, weight and co-morbidity of each patient operated upon.
Logic tells us that some things are clear. The less operations a surgeon does, the worse the mortality percentage will be if they have any mortality rate at all. What about competent but new Consultants? What about Consultants in areas of the country where there have been less operations?
The higher the individual patient risk, the worse the mortality percentage will be. The risk of surgery clearly varies from patient to patient. What about the specialist surgeon, top of their field, who take on the most complex and risky operations?
One of the Surgeons who refused to have his data published expressed his concern with interpretation of data and statistics “bad surgeons will not be picked up… and it puts pressure on younger surgeons not to do any difficult surgery”.
There may be some considerable substance to concerns of data misapplication and the fear that this may increase defensive practice.
The Telegraph reported Friday’s publication of data with the headline “Surgeons with high death rates revealed in NHS statistics”. It goes on to name the 5 surgeons with the worst mortality percentage data with no apparent heed to the warning that the data must not be used as a mortality league table and with total disregard to the fact that all surgeons reported fell within the expected range of national average.
If you have any concern in relation to medical treatment or care, please contact our medical negligence team on 0800 316 8892.