Developing an evidence-based methodological framework to systematically compare HTA coverage decisions: A mixed methods study

Highlights

• We propose, develop and pilot a methodological framework explaining differences in HTA recommendations.

• Mixed methods enabled qualitative and quantitative examination of findings.

• The framework is derived by a case study template and an extensive coding manual.

• The framework captures and compares each step of the appraisal process across settings.

• It's transferability to other disease areas, medicines and countries is ensured by its iterative and flexible nature.

Abstract

Health Technology Assessment (HTA) often results in different coverage recommendations across countries for a same medicine despite similar methodological approaches. This paper develops and pilots a methodological framework that systematically identifies the reasons for these differences using an exploratory sequential mixed methods research design. The study countries were England, Scotland, Sweden and France. The methodological framework was built around three stages of the HTA process: (a) evidence, (b) its interpretation, and (c) its influence on the final recommendation; and was applied to two orphan medicinal products. The criteria accounted for at each stage were qualitatively analyzed through thematic analysis. Piloting the framework for two medicines, eight trials, 43 clinical endpoints and seven economic models were coded 155 times. Eighteen different uncertainties about this evidence were coded 28 times, 56% of which pertained to evidence commonly appraised and 44% to evidence considered by only some agencies. The poor agreement in interpreting this evidence (κ = 0.183) was partly explained by stakeholder input (n_s = 48 times), or by agency-specific risk (n_u = 28 uncertainties) and value preferences (n_oc = 62 “other considerations”), derived through correspondence analysis. Accounting for variability at each stage of the process can be achieved by codifying its existence and quantifying its impact through the application of this framework. The transferability of this framework to other disease areas, medicines and countries is ensured by its iterative and flexible nature, and detailed description.

Introduction

Health technology assessment (HTA) is widely adopted to inform coverage decisions of medicines or health technologies by healthcare systems. It relies on evidence about comparative effectiveness of alternative treatments in a particular clinical setting and aims to ensure that those covered provide value for money (or are cost-effective) [1], ultimately, improving access to medicines. In practice, countries frequently issue different coverage recommendations despite appraising the same body of clinical evidence and using similar methodological approaches. These differences are inevitable due to the complexity of these processes and the context within which they operate, where each country sets its own objectives for conducting HTA reflecting its values, preferences and constraints [2], [3], [4]. Implications include uneven access to these medicines across (often neighbouring) countries, non-optimal use of healthcare resources, and the unpredictability of the pharmaceutical market. Better understanding the application of HTA in different settings and the reasons for diverging recommendations through cross-country learning and sharing of expertise is high on European and supra-national agendas, and may contribute to identify ways to minimize these differences [5], [6] or understand how innovation was rewarded [7], [8]. This is all the more important given the recent appreciation of HTA as a means towards universal healthcare [9] and the commitment of European Member States in implementing cross-border HTA collaboration through the EUnetHTA Joint Action 2.

Nine studies [10], [11], [12], [13], [14], [15], [16], [17], [18] compared HTA coverage recommendations for medicines in more than one country and identified important variations, where agreement ranged from poor to moderate [10], [11], [13]. The countries compared included Canada, Australia, England, Scotland, France, New Zealand, and other European countries. One study concluded that the most common reasons for differing recommendations related to the HTA process and context [10]. Another study highlighted cross-country variations for seventeen of the most expensive medicines, but the extent of, and reasons for these differences were not explored [12]. A more recent study investigated oncologic medicines, where negative recommendations were largely due to the high costs outweighing the marginal benefits [14]. Possible reasons for variations included differences in interpreting the clinical endpoints or in levels of patient input, or issues around appropriate comparators [14]. Another study highlighted differences across therapy areas and countries, suggesting that preferences varied according to the therapy area being appraised [13]. These studies have in common the qualitative approach adopted (retrospective descriptive or cohort analyses) to identify these cross-country variations, highlighting possible reasons for these through single case study analyses. None, however, have attempted to scrutinize these variations and query why they occur in a systematic manner. This is likely due to decision-making processes being complex with many factors being accounted for, which may also be inter-related and thus challenging to compare. Comparing these decision processes systematically could contribute to better understanding the full range of factors accounted for and determining the extent to which they explain differences in coverage recommendations. Doing so would require a methodological approach that decomposes these processes to identify the key drivers contributing to decision-making in a systematic way. While this approach may not necessarily eliminate the variation observed in the criteria used to arrive at decisions, reducing it considerably would also be beneficial.

The aim of this study is to develop and pilot such a methodological framework that allows for a comprehensive and systematic identification and comparison of the key factors that influence coverage decisions in different stages of HTA processes. A better understanding of value assessment processes may help address some of the methodological challenges in conducting HTA and, potentially, minimize cross-country differences when these were a consequence of the review or interpretation of the evidence.

The framework proposed in this study is informed by evidence from medicines with a European Medicines Agency (EMA) orphan medicinal designation [19], which have undergone an HTA in different settings in Europe. Orphan medicinal products are often characterized by significant inequalities in access [20] and are not always cost-effective [21]. In this context, a broader range of factors are likely to be accounted for during the HTA process, which are to be captured by the proposed framework.