To identify, through a Delphi consensus study, the key criteria and categories that define a successful and crisis-resilient healthcare public procurement (HCPP) process across the European Union.
MethodWithin the European project PROCURE, a Real-Time Delphi study was conducted with a multi-stakeholder expert panel (procurers, suppliers, regulators, policymakers, academics and healthcare professionals) from 10 European countries. An initial questionnaire, developed from a structured literature review, was assessed over two online consultation rounds using a six-point Likert scale, followed by a face-to-face meeting to reach final consensus on the wording of the retained criteria and categories.
ResultsThe initial questionnaire (9 criteria, 60 categories) was refined into a final consensus-based framework of 9 criteria and 43 categories. Categories relating to tender, monitoring and control of execution, governance and accountability, and emergency preparedness were ranked as the highest priority. Key insights included the need for a specific, EU-driven legal framework for HCPP, and for standardized metrics and definitions for concepts such as sustainability or monitoring and control of execution.
ConclusionsThis is, to our knowledge, the first pan-European, multi-stakeholder Delphi consensus framework for HCPP, spanning the full procurement lifecycle with dedicated attention to crisis resilience. It provides a validated, prioritized basis for designing, evaluating or improving HCPP processes, and will inform European policy recommendations and country-specific action plans within the PROCURE project.
.
Identificar, mediante un estudio Delphi de consenso, los criterios y las categorías clave que definen un proceso de compra pública sanitaria (CPSS) exitoso y resiliente frente a crisis en la Unión Europea (UE).
MétodoEn el marco del proyecto europeo PROCURE, se realizó un estudio Delphi en tiempo real con un panel de expertos (compradores, proveedores, reguladores, responsables políticos, académicos y profesionales sanitarios) de 10 países europeos. Un cuestionario inicial, basado en una revisión bibliográfica estructurada, se evaluó en dos rondas de consulta online (escala Likert de seis puntos) y un encuentro presencial para consensuar la redacción de los criterios y las categorías retenidos.
ResultadosEl cuestionario inicial (9 criterios, 60 categorías) se refinó hasta un marco final consensuado de 9 criterios y 43 categorías. Las categorías de licitación, seguimiento y control de la ejecución, gobernanza y rendición de cuentas, y preparación ante emergencias, se clasificaron como prioritarias. Entre los principales hallazgos destaca la necesidad de un marco legal específico impulsado por la UE para la CPSS, así como de métricas y definiciones estandarizadas para conceptos como sostenibilidad o seguimiento y control de la ejecución.
ConclusionesHasta donde sabemos, este es el primer marco de consenso Delphi paneuropeo y multidisciplinario para la CPSS, que abarca todo el ciclo de contratación con especial atención a la resiliencia frente a crisis. Ofrece una base validada y priorizada para diseñar, evaluar o mejorar procesos de CPSS, y servirá de punto de partida para las recomendaciones políticas europeas y los planes de acción por Estado miembro de PROCURE.
Public procurement in the healthcare sector faces growing complexity, driven by the increasing diversity of products and services, new technologies, and global supply chains. The COVID-19 pandemic evinced the vulnerabilities of the process, revealing an urgent need to improve the prevailing practices and procedures.
Problems identified during the pandemic included slow response and lack of flexibility, absent stockpiles, supply chain disruptions, and fraud, corruption and e-procurement risks,1,2 underscoring the lack of common good practices and guidelines and the need for a space for experts to exchange opinions and experience.3
Given healthcare's relevance and its position as the second largest government spending area4, improving a process that has such a significant impact in this field, namely public procurement, is considered a priority. With this aim, the European project PROCURE (Public Procurement Assessment in the Healthcare Sector), the project underpinning this study, was conceived. The initial part of the project, The Observational Study,5 highlighted the differences in procurement practices among European countries.
Existing literature on healthcare public procurement (HCPP), however, remains fragmented and largely descriptive. Prior studies have addressed isolated dimensions of the process, such as the degree of centralization of purchasing structures, corruption and transparency risks, or the underuse of innovation-oriented procurement instruments, without offering an integrated, empirically validated framework of the factors that jointly determine procurement success across the diversity of European health systems.6–8 Moreover, most available guidance stems from narrow-scope policy reports4,9, topic-specific narrative reviews,10 or single-country analyses11, and, to date, no study has systematically elicited and validated, through structured multi-stakeholder expert consensus, a comprehensive and prioritized set of criteria applicable at European Union (EU) level. This gap is particularly salient given the persistent heterogeneity in HCPP practices already documented across Member States by the PROCURE Observational Study.5
Against this background, the present study addresses the following research question: which criteria and categories do European multi-stakeholder experts (procurers, suppliers, regulators, policymakers, academics and healthcare professionals) consider essential for a successful and crisis-resilient HCPP process, and how can these be structured into a validated, consensus-based framework applicable across EU Member States? To answer this question, a Delphi consensus methodology was applied, given its recognized value for building structured, unbiased agreement among heterogeneous expert groups on complex, poorly standardized domains.12 More specifically, a real-time Delphi (RTD) approach was used, a validated variant of the classical Delphi method.13,14
To our knowledge, this is the first study to apply this methodology at pan-European level to HCPP, integrating the perspectives of all key stakeholder groups into a single validated and prioritized framework of success criteria, rather than a descriptive account of pandemic-related challenges or individual lessons learned,2 with dedicated attention to procurement resilience during health emergencies.
MethodA RTD approach was used to identify and reach consensus on the key elements for successful HCPP: a variant of the classical Delphi method in which panelists’ responses and group statistics are displayed in real time during online voting, improving efficiency and facilitating agreement across a geographically dispersed, multidisciplinary expert group.13,14 This approach, illustrated in the graphical abstract, combined two structured online consultation rounds using Likert scale15 questionnaires with a final face-to-face meeting to reach semantic consensus.
Phase 1: the initial Delphi questionnaireAn expert librarian conducted a structured literature search on HCPP (2014-2024; five languages; PubMed/MEDLINE, Embase, Web of Science, Scopus and TripDatabase Recolecta), complemented by a parallel search of national and European organizations (OECD, WHO, EU) and further references contributed by PROCURE Work Package leaders; the complete search strategy and document-type criteria are detailed in Supplementary Material (S1 and Table S1).
A multidisciplinary working group was constituted to review and analyze the retrieved documents. Following a two-hour standardization workshop led by an independent methodologist, each document was assigned to a single reviewer, who completed a standardized data-extraction grid summarizing its content and potential contribution to the questionnaire.
The initial items, criteria and domains were not predefined but emerged from this process: extraction-grid findings, together with the stakeholder-mapping matrix from the earlier Observational Study (WP2), were jointly discussed and synthesized on the collaborative platform Miro, triangulating the evidence through three complementary lenses (resilience-oriented, procurement process-lifecycle, and OECD-aligned); the process-lifecycle lens most closely informed the final questionnaire structure (Fig. 1). The resulting draft statements were reviewed by external experts from CORVERS, SILO and the Council of Europe Development Bank.
Phase 2: setting the expert panelRecruitment aimed for a diverse panel, a balanced composition considered essential for robust, unbiased Delphi consensus.16 The Delphi leading team defined seven stakeholders profiles: Procurers, suppliers, healthcare regulators, health policy makers, academic, researchers, healthcare professionals and end-users of healthcare services.
The panel combined participants from within the PROCURE consortium and external experts identified through partners’ professional networks, selected to ensure gender balance, geographical representation and stakeholder diversity (Fig. 2; eligibility requirements in Supplementary Material, S3 and Table S2). No patient representative ultimately participated: patients and patient organizations were invited but either did not respond or could not commit to the full process, including the face-to-face meeting.
There is no international consensus on optimal Delphi panel size17; the ideal size is generally understood to depend on the scope of the problem and available resources.
Phase 3: two online roundsThe online consultation process consisted of two rounds. In the first round (16/09/2024 - 06/10/2024), experts judged the relevance of each category for HCPP using a six-point Likert scale (1=not relevant at all, 6=of most relevance), via the platform ‘SmartDelphi’, designed and customized by Onsanity,18 which displayed each expert's answer alongside the group median in real time, enabling participants to respond in light of the emerging consensus.
The real-time display of group responses during online voting, a feature of the SmartDelphi platform, reflects a deliberate design choice rather than an unintended deviation from classical Delphi principles. This real-time display reflects a deliberate design choice consistent with the RTD variant, introduced to improve efficiency and reduce panel attrition;13,14 methodological comparisons have found that this “feedback effect” does not substantially distort final results relative to conventional multi-round Delphi.14,19 In the present study, this feature was deliberately used to facilitate a more efficient convergence process across a geographically dispersed, multidisciplinary panel, consistent with the RTD rationale.
Item relevance was determined by the median score, and level of agreement by the interquartile range (IQR), following established methodological guidance for Delphi and consensus-panel studies in healthcare.20,21
Given the six-point scale used, IQR thresholds were set more strictly for borderline median scores (4) than for clearly high ones (6), and an IQR of 3 or more (spanning half of the scale's range) was consistently considered indicative of substantial disagreement regardless of median. These thresholds were adapted, with methodological support from the SmartDelphi developers and an in-house statistician, from validated consensus criteria used in prior Delphi/eDelphi studies with comparable Likert scales,22,23 including one from the same research team using a nine-point scale.24
Accordingly, in the first round, three possible scenarios were presented: 1) median score of 6 (high relevance): accepted if IQR was 1-2 (strong agreement); 2) median of 5 (borderline high relevance): accepted as relevant, unless flagged for reassessment when IQR=3 (lack of consensus), or the 4-5 vote proportion exceeded the 5-6 proportion (trend toward lower relevance); and 3) median 4 (borderline low relevance) if the IQR was 0-1, the item was sent for reassessment (relevance unclear) or if the IQR was 2-3, the item was discarded (low relevance and lack of consensus).
A table summarizing the decision rules is presented in Supplementary Material, Table S3.
Items reaching agreement on relevance advanced to a priority-setting stage; those agreed as irrelevant were discarded; unresolved items were carried to the second round. for reassessment.
In the second round (28/10/2024-17/11/2024), experts performed two distinct tasks. The online platform used different colors to help panelists easily identify the status of each item: 1) re-assessment of unresolved categories (a high degree of consensus was reached, IQR 0-2; median ≥5 accepted, median 4 discarded); 2) rating priority for the categories considered relevant in the first round on a 1-6 scale, based on their importance for HCPP (median 5-6=high priority; median 4=medium priority; no category scored below 4).
Phase 4: the face-to-face meetingThe process concluded with a two-day in-person meeting in Zaragoza, Spain (12-13/12/2024), attended by 20 of the 28 experts who completed both online rounds. An internal facilitator with prior Delphi experience led a structured, directive consensus approach, supported by two rapporteurs; representatives from other PROCURE Work Packages attended as non-participating observers. For each criterion and category, panelists reached semantic agreement on final wording, merging, rephrasing or relocating items (additions/deletions were not permitted); a change was adopted with explicit majority approval or, at minimum, no sustained minority objection. Initial and final wording were compared for traceability (Supplementary Material, Table S4).
Ethical considerationsAll experts participating in the Delphi panel signed an informed consent form authorizing the processing of their personal data prior to their participation. Throughout the online consultation rounds, individual responses were displayed to the panel only in aggregate, anonymized form (median and interquartile range of the group), so that no participant's identity or individual answers were disclosed to other panelists at any point in the process. This study did not involve patients or clinical intervention and, as a consensus exercise among professional stakeholders, did not require review by a research ethics committee. The study was conducted in accordance with the data protection and ethical requirements established under the PROCURE project's Grant Agreement with the European Health and Digital Executive Agency (HaDEA) (GA number 101128437).
ResultsInitial Delphi questionnaire on HCPPBased on the initial literature review and expert input, the questionnaire was structured into four main domains, containing a total of 9 criteria and 60 categories (Fig. 1), following the logical lifecycle of HCPP.
Consultation processFigures 3 and 4 show the progression of categories per criterion across both online rounds and the face-to-face meeting (discarded, re-assessed, and considered relevant). The final list of criteria and categories representing the key elements for successful HCPP comprises 9 criteria and 43 categories (Table 1). Prioritization results (Table 2) ranked monitoring and control of execution and governance and accountability as high priority, while initial analysis and sustainability ranked lowest (medium priority).
Final list of criteria and categories that represent the key elements for a successful healthcare public procurement process.
| Delphi on public procurement in the healthcare sector | |||
|---|---|---|---|
| ID | Criteria name | Categories | Priority |
| 1 | Initial analysis | A multiannual strategy for healthcare public procurement is clearly defined and established at member state level in alignment with EU policies and strategies | High |
| There are centralized purchasing strategies developed with input from stakeholders that allow the acquisition of resources to meet the procurement needs | Medium | ||
| The organization has in place updated e-procurement platforms for the public procurement process covering the entire procurement cycle | Medium | ||
| Independent institutions, at European, national and regional level, offer regular information on healthcare public procurement processes | Low | ||
| A healthcare specific EU legal framework for healthcare public procurement has been designed and implemented | High | ||
| 2 | Planning and market consultation | Healthcare public procurement objectives and goals are aligned with institutional (i.e., local, national or international) health strategies and policies | Medium |
| There are available transparent processes and mechanisms to identify the needs, analyze the market and collect information from the suppliers | Medium | ||
| Open market consultation is used by procurers as a tool to obtain information from the market, design and improve tenders | Medium | ||
| All the relevant stakeholders (i.e., procures, users, patients, health professionals, suppliers, etc.) are involved in the healthcare public procurement process | Medium | ||
| 3 | Governance and accountability | There is an appropriate organizational design and budget allocation for the professionalization on public procurement, with political and/or institutional support | Medium |
| Adequate mechanisms and reporting system (confidential, online, secured) in place for prevention, identification, correction, prosecution and communication of misuse of power, fraud, bribery and corruption in healthcare public procurement are set up in the organization | High | ||
| There is an established access to data and public information tools to guarantee transparency across the Procurement process (from planning to the execution of the contract, including evaluation) | Medium | ||
| There is a defined policy to ensure acceptability and use of ICT among stakeholders to enable their involvement in the healthcare public procurement process | Medium | ||
| Organizational structures are clearly defined and described in legal or administrative acts (i.e. governance, roles, responsibility assignments, flows of communication and work and accountabilities) | - | ||
| There is a clear description of the specific role and responsibilities of all stakeholders involved and the relationships among them are well-defined | Medium | ||
| Public procurement processes are integrated into each organizational information systems to monitor and evaluate the process (i.e., accessible channels of communication, specific software, AI, real-time availability…) to guarantee transparency along the processes and to avoid internal fragmentation | - | ||
| A comprehensive operational risk management system as well as mitigation measures are in place | - | ||
| 4 | Tender | Comprehensible and accessible healthcare public procurement processes and documents are set up and properly communicated to the relevant stakeholders to guarantee alignment | High |
| At organizational level, a process to manage penalties and breach of contract has been designed and is applied | Medium | ||
| Most economically advantageous tender criteria (i.e., quality, price, social responsibility, environment, health outcomes, security of supply, etc.) are established in healthcare public procurement | High | ||
| The tenders’ platforms are transformed to be open, accessible and easy-to-use (i.e., TED, government procurement portals, private tender portals, industry-specific platforms, etc.) | High | ||
| 5 | Capacity and resources | Training needs of stakeholders involved are assessed and taken into account in the development of the training programs in healthcare public procurement | Medium |
| Comprehensible and accessible procurement good practices, guidelines, protocols, etc. are developed, updated, efficiently communicated and applied by relevant stakeholders | Medium | ||
| All staff involved in the procurement process possess a wide range of knowledge, skills and competencies for specific tasks in healthcare public procurement | Medium | ||
| There are systems in place to automate the verification of information corresponding to companies (powers of attorney, accounts, social security, treasury, etc.) | |||
| 6 | Monitoring and control of execution | Risk management mechanisms and strategies are in place to prevent or mitigate breach of contract | Medium |
| 7 | Evaluation | Indicators take into account value-based measurements: health outcomes, patient reported outcomes, economic (i.e., budget impacts and efficiency), social, environmental and healthcare professional outcomes | Medium |
| Outcomes assessment has a people-centered approach and is focused on impact (i.e. consequences derived from the healthcare public procurement process) and overall satisfaction of the stakeholders, including users | Medium | ||
| The outcomes and evaluation framework are shared among stakeholders to facilitate improvement through public and open access collaboration, respecting the principles of transparency and equal treatment | - | ||
| 8 | Sustainability | Relevant environmental and social considerations and awarding criteria, aligned with policies and the scope of the contract, are defined on the tender and evaluated along the execution phase, without being an unfair barrier for the competitors | - |
| The healthcare public procurement process can guarantee the viability of the outcomes of the contract in the long term, in a multidimensional approach that integrates environmental, social and economic considerations, promoting sustainable development | Low | ||
| 9 | Emergency situation | The organization establishes clear guidelines and regularly updates a risk management tool (including a contingency plan, modeling to predict) for critical functions | Medium |
| In case of an emergency situation that impacts the health system, an existing emergency committee (with defined roles and functions and executive capacity of decision) is activated | High | ||
| There are clear channels established for internal and external communication, with all stakeholders involved, to facilitate the timely and effective dissemination of information during emergencies | Medium | ||
| Mechanisms such as protocols, guidelines, European and cross-border legal frameworks with third countries, are established to rapidly mobilize and deploy essential resources during emergencies | Medium | ||
| Healthcare systems have the resources to maintain stocks of key supplies and established distribution networks to facilitate timely delivery to affected areas | Medium | ||
| As part of the emergency strategy, there are simplified and fast-track protocols for procurers to approach a wide range of suppliers and manufacturers to deal with stock-outs or other contingencies and ensure quality | Medium | ||
| Policies and procedures are in place to address legal, organizational and ethical considerations related to emergency response, such as patient privacy, consent, and confidentiality | Medium | ||
| A continuous evaluation of emergency actions and efforts taken is done to incorporate lessons learned from all stakeholders, to improve procedures and ensure the organizational resources | Medium | ||
| Inter-organizational dialogues and collaboration networks with suppliers, manufacturers and other stakeholders are in place to ensure equitative distributed access to critical resources in times of crisis | Medium | ||
| Transparency mechanisms are in place throughout the process (ex-pos included) when the procurement procedure is carried out without prior publication for reasons of urgency | High | ||
| Regular training sessions are conducted to prepare personnel to effectively respond to emergencies | Medium | ||
| European and cross-border collaboration strategies are enhanced during non-crisis periods in order to facilitate the acquisition process that benefits the economies of scale when a crisis hits | - | ||
AI: artificial intelligence; ICT: Information and communication technologies; TED: Tenders electronic daily.
Priority distribution of the criteria.
| Priority ranking | Criteria | Mean score |
|---|---|---|
| High priority | Criterion 4. Tender | 5,34 |
| Criterion 6. Monitoring and control of execution | 5,21 | |
| Criterion 3. Governance and Accountability | 5,12 | |
| Criterion 9. Emergency situation | 5,07 | |
| Criterion 7. Evaluation | 5,05 | |
| Criterion 2. Planning and market consultation | 5,03 | |
| Criterion 5. Capacity and resources | 5,00 | |
| Criterion 1. Initial analysis | 4,84 | |
| Medium priority | Criterion 8. Sustainability | 4,36 |
Average of medians obtained on the Likert scale for prioritizing the importance of the different statements for public procurement.
Results showed high agreement among experts: only 10 of 60 categories were deemed not relevant in the first round, though most underwent rephrasing ranging from minor wording adjustments to substantive conceptual revisions (Supplementary Material, Table S4). Main insights per criterion, situated against prior literature, are summarized below:
- •
Criterion 1. Initial analysis, reached consensus with little controversy: experts strongly agreed on the need for a specific, EU-driven legal framework for HCPP, echoing prior calls for regulation distinct from general procurement law,25 but moving beyond earlier policy-level recommendations9 by embedding this need within a broader, empirically validated set of success criteria.
- •
Criterion 2. Planning and market consultation, showed high agreement with few changes, stressing stakeholder involvement, alignment with European strategies, and engagement of high-level decision-makers.
- •
Criterion 3. Governance and accountability, centered on transparency; debate over whether contract performance and evaluation results should be public was resolved by agreeing that transparency should govern all phases, including evaluation. These results, together with Criterion 5, address the integration lacking in prior work on centralization6 and innovation-oriented instruments7 examined in isolation.
- •
Criterion 4. Tender, revealed a recurring tension between supplier and procurer perspectives on “value” (economic vs. patient- and outcome-centered) that shaped much of the face-to-face meeting.
- •
Criterion 5. Capacity and resources, centered on comprehensive training extended to all personnel, dedicated budget allocation, and the role of technological resources (particularly automation and AI) in reducing administrative burden, reflected in the retained category on automated verification of company information.
- •
Criterion 6. Monitoring and control of execution, illustrates that discarded categories can be as informative as retained ones. Four of five initial categories were dropped online (KPI-based monitoring with performance-conditional payments, a multidimensional data collection plan, follow-up reporting during contract execution, and a dedicated monitoring committee) leaving only a single revised category, “risk management mechanisms and strategies”, after experts considered the original wording (“fixing mechanisms and strategies”) imprecise. Despite this attrition, experts strongly reaffirmed the criterion's importance face-to-face, consistent with its high priority ranking (Table 2), some requesting reinstatement of discarded items. This divergence likely reflects practical complexity rather than low relevance: tools like KPI-linked payments presuppose technical and administrative capabilities not uniformly available across institutions,26 compounded by the recognized absence of standardized enforcement mechanisms in HCPP.27 Additionally, effective monitoring is widely regarded as a managerial and governance responsibility rather than a routine operational task;28 the more deliberative, face-to-face format likely allowed experts to reframe the criterion at this institutional level. This is consistent with evidence that monitoring and enforcement mechanisms in public procurement remain weak in practice despite their recognized importance,27,28 and our study adds empirical, multi-stakeholder confirmation of this gap within the healthcare sector.
- •
Criterion 7. Evaluation, was among the most controversial, with debate over whether patient outcomes and economic aspects should carry equal weight; the panel converged on a holistic approach incorporating indicators across the full outcome spectrum, notably proposing a “people-centered” guiding principle. This reflects a broader recognition of the patient perspective that the panel could not directly represent, given the absence of patient participants. The difficulty of recruiting and retaining patients in Delphi and eDelphi consensus exercises is a recognized methodological challenge, particularly in studies addressing structural or administrative processes (such as procurement).29 As a result, the criteria and categories related to patient-centeredness in this study reflect the perspective of professionals advocating on patients’ behalf, rather than patients’ own “lifeworld” experience,30 which may limit the extent to which these criteria fully capture patient priorities.
- •
Criterion 8. Sustainability, was also highly controversial, with three of five categories dropped online; diverging interpretations across environmental, social and economic/institutional dimensions underscored the need for common definitions integrating all three pillars, consistent with prior work showing sustainability in public procurement is understood inconsistently31 and that its integration depends on leadership commitment and strategic alignment.32 Our findings extend this literature by empirically demonstrating, through expert consensus, that this ambiguity remains a live, unresolved tension among HCPP practitioners themselves, not only a theoretical concern.
- •
Criterion 9. Emergency situation, showed swift consensus, likely reflecting shared COVID-19 experience: organizations must be proactively prepared rather than reactive, with a pre-designated emergency committee holding executive authority, and cross-border, public-private and international cooperation, without compromising equal treatment.
These findings reinforce prior calls for strengthened cross-border and public-private cooperation in health emergencies33 and align with COVID-19 lessons on proactive preparedness.2,3 Unlike prior studies documenting individual practitioner learning or diagnosing pandemic-specific failures,2,3 our study translates this experience into a validated, prioritized set of preparedness criteria intended for structural adoption ahead of future crises. These criteria are directly relevant to ongoing EU efforts to strengthen joint procurement and crisis preparedness through HERA,34 offering an expert-endorsed basis for shared standards, evaluation tools, or national action plans.
Taken together, these results address the gap identified in the Introduction: to our knowledge, this is the first study to apply a structured, multi-stakeholder Delphi consensus methodology to derive and prioritize an integrated, pan-European HCPP framework spanning the full procurement lifecycle, providing both a diagnostic and a prescriptive tool that prior, largely descriptive literature has not offered.9–11
Beyond its academic contribution, this framework has direct practical relevance: strategic procurement practices have been shown to reduce costs for medical supplies while maintaining or enhancing quality,35 an objective the identified criteria can directly support.
A comparable Delphi-RAND methodology from the same research team was previously adopted in real-world pilot actions across several European countries,24 illustrating this type of framework's translation potential, including for ongoing EU crisis-preparedness efforts such as HERA. In the specific context of HCPP, these criteria are directly relevant to ongoing EU joint procurement and crisis-preparedness efforts, offering a validated basis for shared standards, evaluation tools, or national action plans.
LimitationsAs a semi-qualitative consensus framework, the Delphi methodology has proven appropriate for capturing diverse perspectives on complex, poorly standardized domains such as HCPP. Nonetheless, several limitations should be considered:
- •
Panel composition and representativeness: despite covering seven stakeholder profiles with diverse professional, gender and geographic composition (Fig. 2), no patient or end-user representative ultimately participated; future applications should actively seek direct patient input. External experts were also primarily identified through PROCURE consortium networks, a snowball-type recruitment common in Delphi studies17 that may have favored more homogeneous perspectives than fully independent sampling.
- •
Participant attrition: panel size decreased from 29 experts in the first online round to 20 at the face-to-face meeting (Fig. 2), within the range reported for multi-phase Delphi studies but possibly reducing perspective diversity at the final consensus stage.
- •
Real-time feedback and potential convergence bias: real-time display of group responses could, in principle, favor premature convergence. However, this reflects a deliberate RTD design choice, and comparative studies have found no substantial distortion of final outcomes relative to conventional Delphi.13,14,19
- •
Consensus threshold definition: although grounded in validated Delphi frameworks, the numerical thresholds were adapted to this study's six-point scale rather than adopted unmodified from a single reference, limiting direct comparability with other Delphi studies.
- •
Facilitation of the face-to-face meeting: the session was facilitated internally rather than by an external, independently trained facilitator, as recommended in RAND/UCLA guidance;20 although limited to semantic consensus (priority ranking having been completed online), this may have introduced moderator bias.
- •
Geographic scope: the panel comprised participants from 10 European countries, reflecting the PROCURE consortium rather than the full diversity of EU healthcare procurement systems; further validation would be needed before generalizing to non-represented countries.
Beyond confirming the relevance of well-established concerns (the need for a specific EU-driven legal framework, and the importance of transparency and stakeholder involvement) this study contributes new, empirically grounded insight into where HCPP stakeholder consensus is strong (governance, tender, capacity-building) and where it remains contested (sustainability, the balance between economic and patient-centered evaluation, monitoring implementation).
This validated, prioritized framework can inform the design or evaluation of HCPP processes at institutional, national and European level, including shared standards and evaluation tools for crisis-resilient procurement relevant to ongoing HERA efforts. Its translation into concrete policy recommendations and Member State action plans is the object of subsequent PROCURE work.
These contributions should be interpreted considering the study's limitations, discussed below, particularly the absence of direct patient representation, participant attrition toward the face-to-face meeting, and the panel's geographic scope (10 European countries), which suggest further validation and contextual adaptation will be needed before generalizing this framework across the full diversity of EU healthcare systems.
Availability of databases and material for replicationAvailability of databases and material for replication: please consult PROCURE for more information.
Currently, there is no consensus on public procurement practices, which vary significantly across European Union countries, nor common, consensus-based guidelines for evaluating an ongoing healthcare procurement process or designing a new one.
What does this study add to the literature?Sharing information and insights among multi-stakeholder experts allowed the identification of the key elements every healthcare public procurement process should have to be successful
What are the implications of the results?Findings offer a guide to evaluate procurement practices, helping leaders track their readiness and identify improvement opportunities, and will inform tailored recommendations per Member State.
Salvador Peiró.
Transparency declarationThe corresponding author, on behalf of the other authors guarantee the accuracy, transparency and honesty of the data and information contained in the study, that no relevant information has been omitted and that all discrepancies between authors have been adequately resolved and described.
Authorship contributionsI. Menéndez Fernández coordinated the study technically, contributed to selecting the Delphi panel candidates, communicated with the selected experts, helped design the online consultation platform, followed up the two online rounds, organized the face-to-face meeting and acted as typer during it, processed the results, and drafted the manuscript. M.J. Vicente Edo, as Delphi expert, designed the study, supervised the elaboration of the initial Delphi questionnaire, contributed to selecting the panel candidates, conducted the face-to-face meeting, processed the results, and reviewed the manuscript. J. López Puy and B. Aguilar Utrilla retrieved and analyzed the literature and drafted the initial Delphi questionnaire; J. López Puy also collaborated in defining the expert panel profile and organized the logistics of the face-to-face meeting, while B. Aguilar Utrilla identified SALUD's experts; both acted as observers during the face-to-face meeting and reviewed the manuscript.
AcknowledgmentsWe appreciate the contribution of María Bezunartea (Navarrabiomed), Carmen de Guerrero (Unizar), Virginia García (CSC) and José María Carrascosa (061) in the elaboration of the initial Delphi questionnaire, and the PROCURE partners who participated in the implementation of the project, especially the coordinators, as well as the Delphi experts who were part of the process.
FundingThe results presented were obtained during the execution of the European project PROCURE (Grant Agreement Number: 101128437), that was co-funded by the European Union, being the European Health and Digital Executive Agency the granting authority under the EU4H Project Grants.
Conflicts of interestNone.


















