Healthcare decision-making is complex, often requiring choices about treatments, technologies, or policies based on limited evidence.
“Structured Expert Elicitation (SEE) has emerged as a powerful approach for addressing these gaps by eliciting expert judgments.”
SEE not only enhances decision quality but also promotes transparency and accountability. In this blog, we will explore the role of SEE in healthcare, its benefits and best practices, as well as drawing insights from recent research by the ISPOR Task Force on SEE and the development of a custom-made protocol for long-term survival outcomes.
Let’s start with understanding the value of SEE in health decision-making.
Decisions in healthcare rely on evidence, which is sometimes insufficient or uncertain. In such cases, SEE can provide credible estimates by quantifying expert knowledge where direct evidence is unavailable or limited.
“For example, expert input might help estimate the likely outcomes of a new drug treatment for a rare disease, the long-term efficacy on a new therapy, or the effectiveness of a diagnostic tool not yet widely used. The elicited quantities could then be used to validate modelled results or incorporated in the analysis as an additional source of information.”
SEE brings rigour to expert judgement by following a structured approach. Rather than relying on informal opinions, SEE uses a systematic method to capture and validate expert knowledge, making it more robust and transparent for use in decision-making.
Five Key SEE Protocols
There are five SEE protocols available, each with unique strengths:
- Sheffield Elicitation Framework (SHELF)1: The SHELF protocol starts with individual elicitation followed by group discussions to reach a consensus. It promotes interaction between expert and peer assessment.
- Cooke’s Classical Method 2: This protocol weights expert judgments based on statistical performance on calibration questions.
- Modified Delphi Method3: Known for its iterative rounds of individual responses and feedback, the Modified Delphi recommends mathematical aggregation after a fixed number of rounds.
- Investigate, Discuss, Estimate, Aggregate (IDEA) Protocol 4: IDEA combines elements of the Delphi and Cooke’s methods, facilitating structured feedback and discussion after individual estimates. The final individual estimates are combined via mathematical aggregation and weights can be incorporated.
- Medical Research Council (MRC) Reference Protocol 5: This protocol defines a reference case for Health Technology Assessment (HTA), which includes individual elicitation with mathematical aggregation using equal weights. If justified, it also allows the use of modified elements of the framework.
Estimating long-term survival when data is lacking is a significant hurdle in HTAs and a common source of uncertainty in decision-making. While expert input for long-term survival estimation has become standard, the methods used to gather these insights can sometimes lead to results viewed as “guesstimates,” which may lower confidence in the values provided. Hence, a custom-made protocol for long-term survival outcomes is needed to overcome these issues.
In line with the ISPOR SEE Task force’s recommendation for further research,6 a custom-made elicitation framework has been designed specifically to enhance long-term survival estimates. This framework offers an interactive, structured approach including a scenario testing as the basis for group discussion to improve the credibility, accuracy, consistency, and transparency of expert judgments of survival quantities, making it a valuable addition to HTA and economic modelling efforts.
To conclude, SEE stands as a vital tool for healthcare decision-making, helping to bridge the evidence gap in areas where data is limited. By following structured methods and drawing on the insights of experts, healthcare providers and policymakers can make better-informed choices. As SEE continues to evolve, it promises to support more robust, evidence-based healthcare that serves patients and society effectively.
Leading methodologist in SEE, Dr Kate Ren (Director Statistics, ConnectHEOR & Senior Research Fellow at the University of Sheffield) will be presenting multiple research projects at ISPOR Europe 2024. If you’re interested in learning more about SEE and the development of a custom-made protocol for long-term survival estimation, we invite you to attend our presentations.
- Beyond “Guesstimates” in Long-Term Survival Extrapolation: Introducing a Comprehensive Step-By-Step Elicitation Framework on Monday, 18 November, from 17:00-18:00 CET.
- Current Landscape and Future Directions of Structured Expert Elicitation in Healthcare Decision Making on Tuesday, 19 November, from 11:45-12:45 CET.
For an in-depth discussion – come to booth #1400.
References:
- Oakley JE, O’Hagan A. SHELF: the Sheffield Elicitation Framework
(version 4). https://shelf.sites.sheffield.ac.uk/ - Cooke, R. M. (1991). Experts in Uncertainty: Opinion and Subjective Probability in Science. Oxford University Press.
- European Food Safety Authority (2014). Guidance on expert knowledge elicitation in food and feed safety risk assessment. EFSA Journal 2014, 12(6):3734, 278 pp. DOI: https://doi.org/10.2903/j.efsa.2014.3734
- Hemming V, Walshe TV, Hanea AM, et al. Eliciting improved quantitative judgements using the IDEA protocol: A case study in natural resource management. PLoS ONE 2018; 13: e0198468. 20180622. DOI: 10.1371/journal.pone.0198468.
- Bojke L, Soares M, Claxton K, et al. Developing a reference protocol for structured expert elicitation in health-care decision-making: a mixed-methods study. Health Technology Assessment (Winchester, England) 2021; 25: 1.
- Soares M, Colson A, et al. Recommendations on the Use of Structured Expert Elicitation Protocols for Healthcare Decision Making: A Good Practices Report of an ISPOR Task Force. Value in Health, Volume 27, Issue 11, 2024, Pages 1469-1478.
