The improvement of health in the twenty-first century is inextricably linked to research for health. In response to growing international appeal to address regional health needs, the Pan American Health Organization (PAHO) and its Member States approved the Policy on Research for Health (CD49/10) in 2009. This document represents the flagship regional policy on research for health and outlines how health systems and services in the region can be strengthened through research. It has been implemented by the two components of PAHO —the Member States and the Pan American Sanitary Bureau. The policy contained a specific directive mandating PAHO to report on its implementation, development of subsequent strategies, and action plans targeting its governing bodies. The Americas are the first World Health Organization (WHO) region to issue a regional Policy on Research for Health, which was harmonized with WHO's Strategy on Research for Health, approved in 2010. Attending to the recommendations issued by PAHO's Advisory Committee on Health Research and WHO's Advisory Committee on Health Research, the PAHO Department of Knowledge Management, Bioethics and Research set out to advance the assessment of the implementation of the Policy on Research for Health through the creation of a monitoring and evaluation Scorecard. Indicators relevant to the Policy on Research for Health objectives were mapped from the Compendium of Impact and Outcome Indicators, with new indicators created. A practical framework based on available indicator data was proposed to generate a baseline policy assessment and incorporate a means of incrementally enhancing the measurements. In this case study, we outline the iterations of the PAHO Policy on Research for Health Scorecard, as well as the lessons learned throughout the development process that may be a valuable guide for health research entities monitoring and evaluating the progress of their own policies.
La mejora de la salud en el siglo XXI está inextricablemente ligada a la investigación sanitaria. En respuesta a la llamada internacional creciente, de cara a abordar las necesidades sanitarias regionales, la Organización Panamericana de la Salud (OPS) y sus Estados miembros aprobaron en 2009 la Política sobre Investigación Sanitaria (CD49/10). Este documento constituye la política regional insignia sobre investigación sanitaria, y destaca cómo pueden reforzarse los sistemas y servicios sanitarios en la región a través de la investigación. Ha sido implementado por parte de los dos componentes de la OPS: los Estados miembros y la Agencia Sanitaria Panamericana. La política contenía una directiva específica, que encomendaba a la OPS la realización de un informe sobre su implementación, desarrollo y políticas subsiguientes, al igual que los planes de acción dirigidos a sus órganos directivos. El continente americano es la primera región de la Organización Mundial de la Salud (OMS) que establece una política regional sobre investigación sanitaria, armonizada con la Estrategia sobre investigación sanitaria de la OMS, aprobada en 2010. Atendiendo a las recomendaciones emitidas por el Comité Asesor sobre investigación sanitaria de la OPS y el Comité Asesor sobre investigación sanitaria de la OMS, el Departamento de Gestión del Conocimiento, Bioética e Investigación de la OPS propuso avanzar en la evaluación de la implementación de la política de investigación sanitaria a través de la creación de un cuadro de mando de supervisión y evaluación. Los indicadores relevantes de los objetivos de la política sobre investigación sanitaria fueron pareados a partir de los indicadores de Compendio del Impacto y Resultados, con los nuevos indicadores. Se propuso un marco práctico basado en los datos indicadores disponibles, para generar una evaluación de la política basal e incorporar un medio de mejorar las medidas gradualmente. En este estudio de caso destacamos las iteraciones del cuadro de mando de la política sobre investigación sanitaria de la OPS, así como las lecciones aprendidas a lo largo del proceso de desarrollo, que podrían constituir una guía valiosa para las entidades de investigación sanitaria de cara a supervisar y evaluar el progreso de sus propias políticas.
“Research for health” has been a core activity of the Pan American Health Organization (PAHO) since its establishment in 1902. “Research”, defined as a systematic process of evidence generation in response to a specified hypothesis or challenge, is critical to fulfilling PAHO's role in improving health and health equity in the Americas.1,2 “For health” encapsulates the philosophy that the generation of evidence driving health decision-making is an inter-sectoral, multi-disciplinary activity.2,3 High-quality research for health produces information that can guide PAHO and its Member States in making evidence-informed decision and promoting health equity in the Americas.2
PAHO's Policy on Research for Health (Policy), adopted by its Member States at the 49th Directing Council in 2009, represents an international recognition that research for health is crucial in achieving health equity and represents a core function of the Organization. The Policy is a result of iterative consultations with the World Health Organization (WHO), PAHO Advisory Committee on Health Research (ACHR), as well as external and internal stakeholders, including WHO Collaborating Centers, research institutions, and consumer representatives and others who participated in public consultations during its development process. The Policy also outlines how high-quality research can strengthen health systems and services in the PAHO region through six objectives: (1) promotion of research generation, (2) research governance strengthening, (3) human resource competency development, (4) establishment of effective and strategic alliances, (5) adoption of best research practices, and (6) research finding dissemination and utilization.4
The Policy features a specific directive that mandates PAHO to report to the Governing Bodies on its implementation alongside the development of subsequent strategies and action plans. PAHO reinforced this mandate at its 44th ACHR meeting in 2012 by issuing a recommendation to develop a Scorecard to assess implementation progress of the policy across the Americas.5 The Department of Knowledge Management, Bioethics and Research of PAHO took it upon itself to devise such a tool. In this commentary, we present our experiences and lessons learned in the development of the “Policy on Research for Health Scorecard” (Scorecard) as a tool to monitor the implementation of PAHO's Policy on Research for Health CD 49/10.
Development of the ScorecardA Scorecard leveraging PAHO's existing administrative evaluation tools was designed by the Department of Knowledge Management, Bioethics and Research of the Pan American Health Organization. This tool was derived from a total of three iterations. We initially conducted a survey of 45 managers of the Pan American Sanitary Bureau (PASB), the Secretariat of PAHO, to assess how managers perceive and support the implementation of the Policy. In this survey, 44 percent (20/45) of the PASB managers participated and from their responses, we identified two priority areas of concern: adherence to research registration at PASB and the need for higher compliance with standards and processes for guideline formulation and the review of research protocols. The full results of this survey are reported elsewhere.6 Given the limited response rate of the survey and the need to assess Policy progress in Member States, we opted to pursue a more comprehensive and objective methodology of assessing the Policy implementation.
In the first iteration we identified indicators for each of the recommended actions in the Policy, which resulted in a long list of over 60 questions that would require of workshops in each country to be completed. Recognizing the limited feasibility of the implementation of such workshops and the need for annual maintenance, this option was shelved.
In the second iteration, we reviewed the Policy, the Report of the 45th ACHR of the Pan American Health Organization, the Strategic Plan of the Pan American Health Organization 2014-2019, the Compendium of Impact and Outcome Indicators, and the WHO Strategy on Research for Health. We identified 36 potential indicators to assess adherence and compliance to the Policy.4,7–10 This methodology required workshops in Member States to enable indicator measurement and was deemed highly resource-intensive out of keeping with long-term sustainability.
In preparation for the third and final iteration of the Scorecard, we conducted a scan of PAHO's 2014-2015 Compendium of Impact and Outcome Indicators, as well as the Strategic Plan of the Pan American Health Organization 2014-2019 and found nine indicators that met policy objectives that were being routinely assessed through PAHO's Performance Monitoring and Assessment semi-annual assessments and end-of-biennium reviews.8,9 A review of other repositories of health policy evaluation indicators such as the European Core Health Indicators was conducted, but no further policy indicators that were relevant for CD49/10 were identified.11 By extracting existing indicators in the current internal monitoring system utilized by PAHO, the need for a new data collection methodology was eliminated as proxy indicators in the biannual periodic evaluations were already being conducted within the countries. We accessed the data behind the indicators to populate a preliminary Scorecard based on the existing indicators and proposed novel indicators to address gaps in the coverage of the policy. In leveraging routinely collected data, implementation of the Scorecard did thus not require substantial investment of time or personnel.
Scorecard implementationTo assess of the implementation of PAHO's Policy on Research for Health, we designed an initial Scorecard comprising of 29 indicators (see Appendix 1 online). A summary of the major Scorecard components is featured in Table 1. We divided the indicators into six main categories (quality, governance, human resources, partnerships/alliances, standards, dissemination/impact), reflecting the six core objectives of the Policy. For example, the “Governance” policy objective is captured by indicators assessing countries and territories with functional mechanisms for governance of health research, implementation of regional policy on Research for Health, and implementation of regional knowledge management strategy, amongst others.
PAHO Scorecard to Assess Progress of Policy CD49/10 in Member States.
| Objective 1: Promote the generation of relevant, ethical, high-quality research |
| -Number of regional initiatives or action plans of the Inter-American and United Nations systems dealing with health and development designed or implemented with PAHO support to advance the health priorities of the Region |
| -Number of countries and territories implementing the regional knowledge management strategy for health |
| -Number of countries and territories with functional mechanisms for governance of health research |
| -Number of studies conducted to inform the design of new or improved interventions for reproductive, maternal, newborn, child, adolescent, and adult health |
| -Number of regional initiatives or action plans of the Inter-American and United Nations systems dealing with health and development designed or implemented with PAHO support to advance the health priorities of the Region |
| Objective 2: Strengthen research governance and promote the definition of research agendas |
| -Strengthen its capacity to guide and supervise its research activities and to assess past experience in supporting the development of national health research systems, before generating new solutions for present and future challenges |
| -Assist Member States in developing appropriate research governance structures and in strengthening and maintaining sustained public trust and engagement with research; |
| -Support Member States in developing strategies and action plans to implement and articulate policies for research for health and innovation, as well as in developing strategies and action plans to implement PAHO's Policy on Research for Health |
| -Foster an appreciation, at the political level, of the value of research in accelerating health improvements and development, and seek political commitment to national health research aiming for the allocation of at least 2% of the budgets of ministries of health to research and research capacity strengthening, in order to reach funding levels proposed in WHA resolutions, expert committees, ministerial forums and strategic plans |
| -Help governments increase their capacity to adapt, disseminate, and use knowledge translation tools that facilitate linking research to health care policy and practice and to the assessment and selection of health technologies, essential medicines and devices |
| -Work with Member States to define research agendas at the Regional, sub-regional, and national levels, fomenting country ownership of research agendas |
| -Work with Member States to strengthen research as a public health function, developing agendas for research for health, institutional research capacity, and technical assistance, and support research in public health at the subnational level |
| -Assist Member States in monitoring funding flows for research for health in relation to the needs and expenditures required and, when necessary, advocate for resources to be redirected to priority areas, monitoring progress in reaching recommended milestones |
| Objective 3: Improve competencies of and support for human resources involved in research |
| -Promote the mainstreaming of human resources working in research for health and the integration of global and Regional policies, strategies, and plans of action for human resources in health |
| -Strengthen the capability of its staff to use scientific knowledge and systematic reviews of the literature when they develop technical cooperation and address uncertainties in the face of insufficient research evidence |
| -Work with partners, including but not limited to, health, science and technology, education, development, and legal sectors, and research institutions, to enrich the health sciences curricula; improve competencies in research, monitoring, and evaluation; and engage in capacity building activities to increase health professionals’ capability to understand and use research results and to engage other sectors that influence health care, health systems, and health governance |
| -Assist Member States to evaluate their current and future human resource needs to conduct research for health, to help them develop national policies and long-term plans to educate and retain the necessary number of health researchers with the required skills and capacities, and find constructive approaches that engage expatriate researchers |
| -Help Member States address, through appropriate research and development of strategic incentives, the factors that determine migration and alienation of researchers to promote the development, retention and thriving of productive research groups |
| -Cooperate with Member States to promote gender equity in the composition of research groups and research management structures, and to develop ways to support increasing the number of researchers from under-represented ethnic groups |
| -Support the development of the structures, methods and directives that promote and maintain systematic evidence-informed approaches in the evaluation and selection of health technologies |
| Objective 4: Seek efficiencies and enhanced impact and appropriation of research through effective and strategic alliances, collaboration, and the building of public trust and engagement in research |
| -Engage the private sector with an emphasis on achieving long-term goals and commitments and fostering multi-center collaborations, innovation, and the sharing of ideas and appropriate technology |
| -Facilitate relevant collaboration with the United Nations system, the inter- American system, civil society organizations, development agencies, and other stakeholders |
| -Work with opinion leaders, strategic partners, and governments to mobilize support and resources for research for health |
| -Make more efficient and effective use of its own specialized centers and of WHO collaborative centers |
| -Facilitate communication and coordination between the public health and the industrial sectors to encourage the development of new products and procedures that address relevant priorities |
| -Work in coordination with the education sector, the science and technology sector, independent research centers (non-profit and for-profit), and networks in order to have research groups in Member States have critical skills and sufficient numbers to develop, grow, regenerate, and achieve sustainable progress; |
| -Promote exchange and collaboration within and between countries and sub-regions, with the participation of various complementary disciplines |
| -Promote the identification and implementation of strategic approaches to address the health determinants efficiently and effectively, and facilitating an equitable investment of resources commensurate to the susceptibility to improve the health and well-being of populations by addressing particular determinants |
| Objective 5: Foster best practices and enhanced standards for research |
| -Promote norms and standards that are in line with WHO's Strategy on Research for Health and foster their implementation and compliance with existing research standards |
| -Advocate for research proposals to include plans for the dissemination, translation, implementation of the new knowledge they might generate |
| -Promote civil society's enhanced participation and ownership in research as a true partner in research for health, contributing to the development of research policies, the definition of research agendas, and the development and use of research for health |
| -Help Member States create or access research for health inventories and registers that are comparable and integrated with WHO's International Clinical Trials Registry Platform primary registers, and adopt standard identifiers and data set collections that contribute to international registration efforts and international ethics and publications standards |
| -Promote access and use and further development of helpful organized collections and registries of research synthesis, including systematic reviews, evidence summaries, and policy briefs |
| -Support Member States in the development of strategies and action plans, regulations, and incentives to strengthen adherence to research registration |
| -Work in collaboration with relevant government sectors, the United Nations system, the inter-American system, centers of excellence, collaborating centers, civil society organizations, and other stakeholders to promote the ethical regulation of research for health in humans and the strengthening of ethical review committees and commissions in Member States |
| -Help Member States build effective and efficient tools for determining and assessing the extent to which the research they conduct adheres to international good practice standards, including ethics, safety, and research management standards |
| -Promote the development of validated indicators to assess and monitor the effects of investment in research and scientific production in the Americas (9) and the alignment between research activities and research priorities |
| Objective 6: Promote the dissemination and utilization of research Findings |
| -Continue to encourage open access to scientific literature and to foster novel approaches to copyright and intellectual property that will allow knowledge essential for health, equity, and development to be shared and made widely available |
| -Continue to participate in the international debate on how legal frameworks for intellectual roperty affect research for health, especially the impact on development and equitable access to the benefits research |
| -Promote knowledge sharing among researchers, policy makers, and other users and foster the development and evaluation of new knowledge translation initiatives and tools in the Region |
| -Seek the empowerment and participation of civil society organizations in setting priorities, generating knowledge, and harnessing research evidence |
| -publish relevant research findings, recommendations, and guidelines that emerge from research for health in formats that are most appropriate for the target audience |
| -Work with the media to improve public understanding of the benefits of research for health and to improve scientific literacy of policy makers, health providers, and the public |
| -Work in cooperation with its specialized centers, such as the Latin-American and Caribbean Center on Health Sciences Information (BIREME), to index and organize research evidence in helpful ways and promote the Virtual Health Library model and the indexing, organization, access, and sharing of relevant health information |
| -Promote access to and use of research evidence summaries that integrate results through valid methods, thus facilitating a better understanding of the relevance and effects of interventions and promoting efficiencies in the search and analysis of qualitative and/or quantitative scientific research |
Of the 45 policy sub-objectives, we were able to find 29 data sources within the Compendium of Impact and Outcome Indicators that mapped onto our indicators. Some policy sub-objectives did not have any applicable indicators available. A total of three new indicators/outputs were therefore proposed to be added into the Compendium to fulfill the core of the Policy assessment in future assessments: delivery of research reports by the PASB, budget allocation to research for health, and monitoring of scholars in health research. Our approach was informed by scanning the literature for implementation of other health research policies. The World Health Organization's Monitoring and Evaluation of Mental Health Policies and Plan advised that a combination of quantitative and qualitative data be employed and recommended use of data that may be already available.12
The Research Promotion and Development team of PAHO's Office of Knowledge Management, Bioethics, and Research, piloted the Scorecard in 2015. The purpose of this pilot test was to collect data to evaluate each of the proposed indicators based on three parameters: feasibility, relevance, and convenience. We evaluated feasibility in terms of how easily accessible data was collected for specific indicators. Relevance was assessed based on whether the objectives being captured were reflective of the key components of the Policy. Convenience was determined from the ease and time required for data access.
We populated the Scorecard using previously gathered information from the internal PAHO monitoring system from 2014-2015. We calculated adherence to each objective based on whether data was reported or not. A Member State was labelled to be adherent to an objective if at least 50 percent of sub-objectives were met. Based on our initial population of the framework, we generated percentages of completed reporting per member, as well as a heat map outlining adherence rates. The results of the Scorecard will be used by PAHO initially, and may be subsequently published in the future following collection of feedback and associated Scorecard revisions. Once institutionalized and operationalized, data collected biennially based on the selected indicators may provide relevant stakeholders with an assessment of Policy implementation progress, as well as the factors hampering progress in a timely systemic fashion post-policy adoption.
According to the database maintained on Health Research Web, active policies on research for health have been listed in 16 countries and a region: Antigua and Barbuda (2016), Bahamas (2010), Belize (2014), Bermuda (2014), Brazil (2015), Cayman Islands (2012), Chile (2011), Dominican Republic (2014), El Salvador (2017), Jamaica (2015), Mexico(2013), Panama (2016), Paraguay (2016, State Policy), San Kitts and Nevis (2011), Suriname (2011), United States of America (2013) and 18 CARICOM countries. Active agendas on research have been listed by 8 countries and two sub-regions: Chile (2009), Guatemala (2014), Guyana (2013), Honduras (2015), Mexico (2017) Panama (2016), Peru (2016), and Uruguay (2017). The CARICOM countries share a Caribbean Research Agenda (2011), and the COMISCA Countries share the Health Plan for Central America and the Dominican Republic, and have reactivated their Commission on Health Research in 2017.13 Engagement and commitment in health research is evidently taking place around the region of the Americas, and it is thus important to capture the activity taking place and monitor its development in concordance with Policy on Research for Health. Towards the end of the Scorecard development process, the WHO released a series of indicators and data developed in limited consultation with regional offices as part of the Global Observatory on Health Research and Development.14 These indicators are valuable additions to the literature, and in collaboration with the WHO, PAHO is exploring ways to promote their use and appropriation by Member States into the Scorecard, in keeping with the proposal of the 46th Session of the ACHR to use such complementary data for policy evaluations.15
LimitationsThere were limitations to the development of this Scorecard. Firstly, there is a need for standardization in reporting mannerisms across Member States. Data collection may vary amongst Members, but we proceeded with this approach due to its ability to leverage existing systems and data. Furthermore, some policy sub-objectives are challenging to assess given the difficulty of attaining certain data. We have begun to remedy this by proposing new indicators to add to the Compendium as an initial step, with the intention of eventually building more indicators into the Compendium in order to be able to systematically and routinely collect data from members. Lastly, given the lack of available indicators for all sub-objectives, some areas of policy CD 49/10 are not able to be assessed. We nonetheless believe the development of this preliminary Scorecard is an important step in the assessment of the Policy that will pave the path forward for future systematic and comprehensive policy evaluation and assessment.
RecommendationsOverall, PAHO, Member States, ministries of health, and academic centers will likely benefit from renewing their commitment to the realization of the Policy on Research for Health and reviving their efforts towards bringing about progress in the Americas. Looking to the future, a two-fold approach will be undertaken by the ACHR Secretariat and entities involved in promoting research for health at PAHO. Increased awareness on the Policy on Health for Research and its objectives will be promoted. It is very likely that in a very fast-paced political environment subject to personnel turnover, the policy has fallen off the radar of many PAHO leaders and Member State delegates. Having a continued assessment of progress with the Policy will provide actionable data to guide its implementation. The ACHR Secretariat at PAHO will send out an update on the policy to all governing bodies, as well as suggestions on how to advance and update the policy as the landscape evolves. This Secretariat will also ask country offices to contact actors central to the realization of policy objectives such as academic institutional leaders, health department personnel, as well as science and technology departments to ask them to bolster their initiatives towards strengthening research for health.
Secondly, the entities responsible for promoting and developing research within the Pan American Sanitary Bureau should support country offices to provide details on the indicators within the Scorecard. Increasing the robustness of the Scorecard will include validation of the tool with primary data from the member states. Efforts will be principally targeted towards attaining full reporting across all members, and towards fine-tuning the instruments. Initiatives fulfilling objectives of the Policy may very well be underway in many countries, but without the comprehensive and timely reporting of country offices, PAHO will not be able to assess what areas under its jurisdiction it should continue to support and where significant attention and resources ought to be directed to fulfill the policy mandates. Reinforcement of reporting mandates, consultation with the Governing Bodies over evolving indicators, as well as dedication of appropriate human resources towards the completion of the monitoring and evaluation system is likely to help improve the net quantity and quality of indicator reporting. Uptake from the organizational leadership will be required to champion these endeavors.
In order to institutionalize the Scorecard, support from PAHO's executive administration will be sought out. We will continue to refine the existing indicators and progressively propose the introduction of new indicators that will make the assessment of the policy a corporate product built around the existing administrative tools and systematic across all relevant entities. The Scorecard will be completed on a biennial basis, with milestones documented on a biannual schedule, using data provided by the members. The ACHR will then provide high-level commentary on the results each time the scorecard filled out. The Scorecard will be populated by professionals assessing the Policy implementation within the Secretariat for PASB and the ACHR, and then subsequently circulated for review. Feedback from the ACHR members will be aggregated and then re-directed to the respective members working in coordination with the unit of Program Budget and of Country and Sub-Regional Coordination.
It is important to seek mechanisms to promote harmonization and coordination among WHO regions (e.g. global and regional Advisory Committees on Health Research) that identify strategies seeking the appropriation and participation of Member States and WHO regional offices in the WHO Global Observatory on Research for Health.
ConclusionsThe PAHO Department of Knowledge Management, Bioethics and Research designed a practical way to use available institutional indicators as a means of assessing the implementation of the Policy on Research for Health (CD49/10). Our experiences are likely to be relevant to other WHO regional offices and research entities that are dealing with health and engaged in measuring the progress of their own mandates, policies or strategies and likely to encounter similar journeys in striving for feasible policy monitoring and evaluation. The task of monitoring and evaluating the implementation of a policy is a complex phenomenon yet critical to the policy process. There is a delicate balance between comprehensiveness, accuracy, and feasibility throughout the policy evaluation process that is to be striven for.
Editor in chargeClara Bermúdez-Tamayo.
Authorship contributionsL. G. Cuervo, C. Frankfurter and J. Le contributed to the conception and design of the work. C. Frankfurter and L. G. Cuervo contributed to the data collection and the analysis and interpretation of the data and the writing of the article. C. Frankfurter, J. Le and L. G. Cuervo contributed to the critical review with important intellectual contributions. All authors approved the final version for its publication and agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.
FundingNone.
Conflicts of interestNone. An author (LGC) is a staff member of the Pan American Health Organization. The author alone is responsible for the views expressed in this publication, and they do not necessarily represent the decisions or policies of the Pan American Health Organization.
We would like to thank members of the Advisory Committee on Health Research who invaluably participated in the consultation process on the Scorecard. We would like to thank Dr. John Lavis for his guidance and supervision, Ms. Rachel Hutchinson for her contributions to the earlier versions of the Scorecard, as well as Mr. Rony Maza at the Pan American Health Organization for his support with the Compendium of Impact and Outcome Indicators.
