King’s Research Portal

Risk-adjusted survival statistics after children ’ s heart surgery are published annually in the United Kingdom. Interpreting these statistics is dif ﬁ cult, and better resources about how to interpret survival data are needed. Here we describe how a multidisciplinary team of mathematicians, psychologists, and a charity worked with parents of heart surgery children and other users to codevelop online resources to present survival outcomes. Early and ongoing involvement of users was crucial and considerably changed the content, scope, and look of the website, and the formal psychology experiments provided deeper insight. The website http:// childrensheartsurgery.info/ was launched in June 2016 to very positive reviews. (

Risk-adjusted survival statistics after children's heart surgery are published annually in the United Kingdom.
Interpreting these statistics is difficult, and better resources about how to interpret survival data are needed.
Here we describe how a multidisciplinary team of mathematicians, psychologists, and a charity worked with parents of heart surgery children and other users to codevelop online resources to present survival outcomes. Early and ongoing involvement of users was crucial and considerably changed the content, scope, and look of the website, and the formal psychology experiments provided deeper insight. The website http:// childrensheartsurgery.info/ was launched in June 2016 to very positive reviews. S ince 2000, all UK pediatric heart surgery centers have contributed data on every cardiac procedure to the National Congenital Heart Disease Audit (NCHDA) [1]. In 2013, the NCHDA began using the Partial Risk Adjustment in Surgery (PRAiS) risk model [2][3][4] to report risk-adjusted 30-day survival outcomes for participating hospitals. That triggered events resulting in the temporary suspension of surgery at one unit in 2013. The attending media scrutiny, impact on families, and public anger [5][6][7][8][9], alongside previous and more recent media coverage on children's heart surgery services [10][11][12][13][14][15], illustrate the need for resources to support appropriate interpretation of outcome data for families, journalists, clinicians, and decision makers. As part of a project to update the PRAiS risk model (see accompanying papers [16,17]), we codeveloped an explanatory website aimed at these diverse public and professional audiences (http:// childrensheartsurgery.info/). It was a multidisciplinary effort and was a transformative and valuable experience for the team (Table 1).
After its launch in June 2016, a Lancet editorial [18] commended the website: "Many more areas of medicine requiring risk communication should take this initiative as a long overdue and most welcome example." In The Annals, Jacobs and colleagues [19] have discussed the drive for public reporting in the United States and highlighted the need for resources "that assist patients and families in correctly interpreting complex data." To encourage and help others to undertake similar ventures, here we describe our approach and the lessons we learned.

Overall Strategy
The final output was aimed at two audiences: (1) older patients and families of children who have had/will have heart surgery; and (2) other interested users, including press officers and policy advisors for medical charities or professional bodies, the media, medical communicators, hospital family liaison services, and patient advocates.
An initial draft of content was followed by a process of iterative web development in response to user feedback. We involved both groups of target users from the outset, convening four 1.5-hour workshops for each group over a year. We aimed for four to six participants per workshop, with none attending more than once to ensure a fresh perspective. Participants received only minimal details about the project and were not required to read anything in advance. Workshops began with a brief background presentation and by establishing appropriate consent and permissions (eg, recording). Next, participants were each given about 15 minutes to explore website material on a laptop (workshop 2 onward). It was made clear to parents that they were free to leave if they did not feel comfortable to continue. There then followed a facilitated discussion, the focus of which varied between workshops as described below. We tested understanding of concepts and plain language explanations in all workshops. We identified where a concept had not been explained clearly and concepts that were especially difficult to understand. Participants were invited to provide feedback remotely on future iterations of the site (all accepted), and we incorporated their feedback on the near-final web material.
At months 10 and months 12 to 15, we also shared web content with the UK Children's Heart Federation, specialists (including three pediatric cardiac surgeons, two cardiologists, two intensivists, and two data experts), and representatives of the NCHDA (including senior clinicians) and incorporated their feedback. Starting 7 months into the project, the psychology team (T.R., E.B.) used a mixed-methods approach to evaluate candidate components for the website. A summary of the strategy is shown in Figure 1.

Workshop Participants
We held four rounds of two workshops, involving 15 participants in the parent workshops and 22 participants in the other workshops. Two team members facilitated each workshop, and two to three others attended as observers.
The feedback from the workshops and psychology experiments fundamentally influenced and changed almost every aspect of the website. Here we concentrate on two major themes: the development of the key messages, and the data display.

Starting Web Material Development
The initial focus was on explaining the key table and graphic in the public NCHDA annual report (Figs 2 and 3) [1]. The vertical axis in Figure 3 is the ratio of actual survival to predicted survival from the PRAiS risk model. If this value is greater (less) than 1, then survival at that hospital was higher (lower) than predicted. The white area gives the 95% prediction interval for this ratio-essentially the range within which we expect hospital outcomes to lie.
Members of the project team (C.P., D.S., M.P., E.J.) discussed potential content of the web material and plans for the first workshops. Fresh to the material, our web programmer (M.P.) suggested that by rotating the chart (Fig 3) through 90 degrees, it could become another column on the corresponding table (Fig 2), which might make the relationship between the table  and the graphic more transparent (Fig 4). We therefore decided to present participants in the first round of workshops with the following: introductory text; NCHDA table of results (Fig 2); NCHDA results graphic ( Fig 3); combined version of the table and graphic (Fig 4); some basic explanatory text for the graphic; and draft answers to example "frequently asked questions" (FAQ).  [1]. Medical center codes are defined in Figure 2. (Green bars ¼ survival much higher than predicted; blue-green ¼ survival higher than predicted; white ¼ survival as predicted; light gray ¼ survival lower than predicted; dark gray ¼ survival much lower than predicted.) S., C.P.) were both familiar with the way NCHDA presented the hospital outcomes (Fig 3). However, the value of the workshops became immediately apparent. Some workshop participants were drawn to the color regions in Figure 3 and interpreted it as a bar chart, which gave the (incorrect) impression that the hospitals on the horizontal axis are ranked by increasing survival. After discussion with participants, we decided to make the prediction interval colored and the outermost areas white, to focus attention on the dot and its predicted range. In both workshops, participants unanimously preferred the combined display (Fig 4), so we adopted it. The workshops demonstrated that ratios are hard to understand and easy to misinterpret. The mathematicians (D.S., C.P.) spent considerable time in both workshops explaining what the ratio of actual to predicted survival represented and why it was used instead of raw survival (we note this ratio is also used in the United States [19]). The ratio is preferred by the NCHDA because every hospital has the same expected value of 1, which gives the graph in Figure 3 a common center line. As discussion progressed, the mathematicians emphasized that hospitals should not be compared directly with each other using their raw survival rates and that the key feature is whether the "dot" on the graph (representing the hospital) is within its predicted range. Discussing this aspect later, we realized that emphasizing that hospitals should only be compared with their own predictions made it confusing to then transform hospital results to a ratio whose main benefit is to allow comparison between hospitals. We also realized that providing the exact predicted survival was inconsistent with emphasizing that the predicted range was the important feature. We therefore decided to present actual survival for each hospital within its predicted range of survival, and not to provide the exact predicted survival rate. PREDICTED SURVIVAL. Predicted survival created much discussion-participants asked when and how predictions were made. For instance, some people assumed that the hospital predicted its survival rates after knowing its actual survival, or that (analogous to predictions of poor performance in sports or education) hospitals with lower predictions were "worse." We had tried to avoid using detail about the risk adjustment method in our explanations, instead using language such as "predicted survival (%) is the percentage of operations where the child would survive at least 30 days after their operation estimated using previous national data about children similar to those cared for at that hospital." That proved unhelpful, because different people interpreted this very differently. Parents in particular wanted to know more about how the survival was predicted. We realized that we would have to explain risk adjustment "up front" and learn how people interpreted the terms "predicted" or "expected." (Green bars ¼ survival much higher than predicted; blue-green ¼ survival higher than predicted; white ¼ survival as predicted; light gray ¼ survival lower than predicted; dark gray ¼ survival much lower than predicted.)

Workshop Round 2, Months 8 to 10
By the second "interested user" workshop, we had developed three draft web pages: an introduction page (Fig 5), a data page (Fig 6), and an FAQ page. The introduction page stated explicitly that the risk adjustment method was objective (based only on patient characteristics), and that raw survival rates should not be compared between hospitals. For the data page, in addition to the changes identified from the first round of workshops, we included additional information available on clicking on a single hospital and by using "hoverovers." SECOND INTERESTED-USER WORKSHOP (MONTH 8). Feedback was generally positive, with participants endorsing the site design, data display, and warnings against comparing hospitals by survival rates, but they suggested that these warnings should be more prominent. Participants were reassured that all hospitals had high survival rates, and they suggested that this be emphasized. Most important, one participant commented that we emphasize that you should not compare hospitals and then present them in a table that invites comparison. The ensuing discussion prompted us to add a data view showing data individually by hospital. SECOND PARENT WORKSHOP (MONTH 10). By this time, we had incorporated much of the feedback from the second interested-user workshop. Participants still wanted us to be clearer about what background information was absolutely essential, and we discussed adding a "key points" section to the introduction.
We created a new "mapped data" page that allowed users to explore hospitals individually (Fig 7). We also provided links to the hospital website and any associated charity. There was still a separate page with the tabled data, as in Figure 6. The parents suggested that adding some explanation, plus links to relevant FAQs, alongside the individual hospital data displays would help.
FIRST SET OF PSYCHOLOGY EXPERIMENTS (MONTHS 10 AND 11). Three mixed-methods experiments explored how people understood and evaluated the prediction intervals and made subsequent judgments about hospitals, comparing the survival-ratio plot used in the National Institute for Cardiovascular Outcomes Research (NICOR) report (Fig 3) [20] against the percentage-survival plot from the website (Fig 6). The findings confirmed our decision to use percentage-survival plots instead of ratio plots (eg, accuracy of understanding for the outcome scale was 71% for the percentage plot compared with 41% for the ratio plot), but highlighted two key messages that we needed to emphasize even more: (1) comparing hospitals' survival rates to other hospitals' survival rates or predicted ranges is inappropriate; and (2) the predicted range is based on an objective statistical formula and only reflects the hospital's case mix. The experiments and workshops also emphasized the importance of consistency in using (or implying) terms such as "luck," "chance," "risk," "uncertainty," and "probability." We decided always to refer to predicted risk as "predicted chance of survival"; the placement of the survival dot in relation to its predicted range as "strong/some/no evidence that chances of survival at that hospital were different to that predicted using the formula"; and inherent uncertainty in outcome as "unforeseeable factors."

Workshop Round 3, Month 11
We focused on updating the site to be intuitive to navigate, to repeat key messages, to provide sufficient detail, and to have a simpler home page (Fig 8). The introduction content was moved to a new page ("What, why, how?") with a key points section at the top (Fig 9). Workshop feedback was now very positive, but nonetheless, participants suggested new content, wording revisions, and layout changes. SECOND SET OF PSYCHOLOGY EXPERIMENTS (MONTH 12). An experiment found that if people viewed hospitals individually rather than alongside other hospitals, their interpretations of a hospital's outcomes seemed more reliant on the prediction interval (and less reliant on other hospitals' outcomes). We therefore promoted the individual hospital view over the combined table/graph. We retained the combined view for transparency and its similarity to the NCHDA report presentation. However, we added text to emphasize that hospitals should not be compared with each other on survival rate. Further details of methods and data for the first two rounds of psychology experiments are currently under review in a separate publication.

Workshop Round 4, Month 13
With the web content and navigation nearly finalized, the project team concentrated on two explanatory animations. We worked with an external animation company but changed their usual development process by allowing time for user feedback at a relatively advanced stage of animation development. In detailed discussions, the parent participants suggested moving key sections from one draft animation (explaining the predicted range) to the other (explaining the data display) and then layering the placement of the animations within the site. That led us to substantially rework the storyboards and scripts.

Finalizing Website Months 14 and 15 and Evaluation
The workshops showed that the key points at the top of the "What/why/how" section were extremely valuable and should be prominent. We therefore illustrated the key points section of "What/why/how" (Fig 10) with images from the animations.
To determine whether the website provided "added value" over and above the NCHDA report [20], a smallscale randomized control trial compared people's ability to answer questions about the audit data and their interpretation when they used either the NCHDA report only, or the website only, or both the report and website together. Results showed that the website improves comprehension and understanding of the data, raising scores for these  . The website offers three particular advantages over the report: it improves comprehension of the data plot; provides clearer information about the audit process triggered for outliers; and explains how often the survival rate will fall outside the predicted range by chance.

Lessons Learned and Summary
We set out to develop material to explain a single graph, but as we engaged with users, they helped us to develop messages about high survival rates, statistical methods, and avoiding inappropriate comparisons. By listening, we clearly communicated these messages and improved our presentation of the data. This early and continuous user engagement was vital, with each workshop improving and shaping the website. The multidisciplinary, multisector nature of our team was crucial to implementing this input (see also online articles from the perspective of a parent participant [21], Sense about Science [22,23], and the mathematicians [24,25]). The website "http:// childrensheartsurgery.info/" was launched on June 21, 2016. It was very well received and quickly endorsed by major stakeholders (charities, clinical specialists, national audit, National Health Service Choices, Royal Colleges, leading medical journals) [18,[26][27][28][29][30]. Building this website took considerable resources, including funding, people, and time. It was extremely helpful to have an external partner guide the stakeholder involvement and act as neutral facilitators at the workshops. For the technical web development, responsiveness and speed in changing the web material was important, requiring a technical strategy for reworking the website outside the norm for website development. Despite allowing ourselves 15 months, we were pressed for time-mainly because we underestimated the demands of implementing multiple iterations of feedback. For difficult topics such as survival outcomes, the parent workshops [21] in particular were humbling and vital for the team to understand the emotional aspect of the data for parents and inform development of sensitive communication. To summarize, there is no substitute for genuine co-production.