Implementation fidelity is a key dimension in process evaluation but has been scarcely studied. Reptes is a selective alcohol and cannabis use prevention program performed in vulnerable young people (16-21 years) from different educational and leisure settings. The study aimed to describe the components of this program, its implementation fidelity, and satisfaction among participants and facilitators.
MethodA descriptive cross-sectional study was conducted. Implementation fidelity was assessed in terms of adherence, dose, participant responsiveness, and quality of the intervention through self-reports completed by 15 facilitators and self-administered questionnaires by 99 youth from the 15 groups completing the program.
Results70.7% of the groups performed 30 of the 43 planned activities with high use of various teaching methods, except for Information and Communication Technology resources. Fidelity implementation was above 70%. 2/3 groups completed an acceptable intervention and 1/3 completed a qualified intervention. Satisfaction was higher than 7.
ConclusionsOur results show high implementation fidelity and satisfaction compared with those from similar studies.
La fidelidad en la implementación es una dimensión clave en el proceso de evaluación, pero ha sido escasamente estudiada. Reptes es un programa de prevención selectiva del consumo de alcohol y cánnabis dirigido a jóvenes vulnerables (16-21años) de diferentes espacios educativos y de ocio y tiempo libre. El estudio pretende describir los componentes de este programa, su fidelidad de implementación y la satisfacción de los/las participantes y los/las facilitadores/as.
MétodoSe realizó un estudio descriptivo transversal. La fidelidad de implementación fue valorada en clave de adherencia, dosis, capacidad de respuesta de los/las participantes y calidad de la intervención a través de los informes autocumplimentados por los/las 15 facilitadores/as del programa y los cuestionarios autocompletados por los/las 99 jóvenes de los 15 grupos que completaron el programa.
ResultadosEl 70,7% de los grupos realizaron 30 de las 43 actividades planificadas, con un uso elevado de las estrategias de aprendizaje, excepto para los recursos basados en las tecnologías de la información y la comunicación. La fidelidad de implementación se situó por encima del 70%. Dos tercios de los grupos completaron una intervención aceptable y un tercio completó una intervención cualificada. La satisfacción se situó por encima de 7.
ConclusionesNuestros resultados muestran una alta fidelidad en la implementación y la satisfacción en comparación con estudios similares.
The fidelity of implementation is the dimension of the process evaluation research that has been most investigated in the last decades. It is defined as “the degree to which programs are implemented according to the criteria with which they were designed”.1 It measures how an intervention affects its results.2 Specifically, four basic indicators are defined to measure it: adherence; exposure or dose; quality of implementation and responsiveness (interest or involvement of participants).1–4 Adaptation, the process of implementers bringing changes to the original design of an intervention, is also another component to be explored by implementation fidelity. Pérez et al.5 proposed a theoretical framework to assess implementation fidelity in public health interventions.
The importance of knowing the intervention development process (everything that happens during the implementation of the program in the experimental phase) is today beyond doubt1,3. However, one of the major limitations to know the quality of implementation of the interventions is the scarcity of studies in the literature about the evaluation of the intervention process using rigorous implementation indicators.6 A review4 of 312 publications identified that<10% showed the main core or structural components of the program or the theoretical reference model; and<20% measured the implementation fidelity, apart from the dose or amount of the applied intervention.
The Reptes program for the prevention of alcohol and cannabis use in young people with vulnerability is a selective drug prevention program. Selective drug prevention focuses on vulnerable groups and contexts in relation to the risk of transition to patterns of problematic drug use and to situations of social exclusion.7 In these groups the use of different types of drug use is very common, so the intervention is aimed at reducing the risks of the contexts, the harm associated with drug use and at reinforcing the skills of the young people.7,8 The Reptes program seeks to provide knowledge on the main risk of uncritical cannabis and alcohol consumption; arguing skills regarding decision making on drug use; and to improve social skills to face social pressure to consume drugs.
The Reptes program's design was inspired: 1) in terms of communication styles, in the approach to self-control work through the role playing of assertive behaviors in the face of the consumption of addictive substances from the project Towards No Drug Abuse (TND);9–13 2) regarding the work of emotions, in the approach of identifying the risk associated with personality variables of the project Preventure,14 and the work of examples based on personal narratives to enhance identification with the prevention messages of the Keepin’it REAL program,15–17 and 3) for the resolution of conflicts or problems also in the TND program9 and Keepin’it R.E.A.L.16
The objective of this study is to describe main components, fidelity of implementation and satisfaction with the Reptes program in a sample of vulnerable young 16-21 years old enrolled in social guarantee training environments (oriented to the job market) and in community environments (in civic facilities) of a recreational nature, in Barcelona, during 2017.
MethodStudy design and samplingWe conducted a descriptive cross-sectional process evaluation study of the Reptes program. All centers and organizations (n=15) taking part in the intervention in 2017 participated in the evaluation study. The intervention was implemented by trained facilitators in 17 groups (184 participants). All facilitators and participants were included in the study after their agreement.
The intervention was implemented February-April 2017 among youth (16-21 years old) enrolled in social guarantee programs (SGP) or in neighborhood leisure organizations from vulnerable neighborhoods in Barcelona. SGP are usually offered in secondary schools or educational centers and aim to provide professional training to young people who have not completed their secondary schooling. Neighborhood organizations offer healthy leisure activities to neighborhood residents who might be living in a vulnerable context. Some educational centers and neighborhood organizations were invited in a convenience basis to implement the Reptes program in their centers. As inclusion criterion, people in charge of SGP and neighborhood organizations had to clearly consent to participate. Additionally, facilitators, who were responsible for implementation of the program, were urged to attend a previous 20-hour training program to learn how to deliver the intervention effectively. Finally, 15 centers or organizations agreed to participate.
Description of the interventionReptes is a selective prevention program targeting young people living in a vulnerable context. These young people usually have or have had academic difficulties within the regular education system, are at risk of leaving school early, have started experimental drug use, or live in families at risk. The program aims to provide knowledge on the main risks of uncritical cannabis and alcohol use, to provide decision-making skills on drug use, and to improve social skills to withstand social pressure to take drugs. The Reptes program consists of seven structured sessions and one entertaining closing session consisting of a party. The sessions were weekly 90minutes long. All sessions included 43 activities, imparted by facilitators previously and properly trained. Activities were conducted in classrooms.
The core components of the program were those essential for the successful of the intervention and consisted of the structural and crucial components of the intervention and of the teaching methods through which they should be imparted.18,19 Of the 43 activities included in the intervention, seven were core components. They focused on attitudes, emotions, social skills, decision making or problem solving, which are the main intermediate factors related to substance use. Two more activities were also important for the proper implementation of the intervention. One of these activities was focused on social network assessment of the young's family and friendship and the other one was an individual assessment and debate of the main issues discussed with the program (see Table I in online Appendix). Of note, the program had two different alternatives for some of the activities, namely minimum and maximum activities that were exclusionary. For these activities, the facilitator had to choose whether to implement the minimum or maximum activity depending on the maturity, concentration and cohesion of the group. The contents taught in both types of activities were the same. However, activities differed in the level of complexity of the teaching method applied.
Information sources and study variablesThe information sources were self-report forms and questionnaires. Facilitators completed self-report forms throughout the intervention at the end of each delivered session. These forms recorded process information including the number of activities conducted, the teaching methods applied according to the protocol (oral explanation [OE], working skills and emotions [SE], discussion [D], active teaching and learning [ATL] and information and communications technology [ICT] methods), and satisfaction with the program. The completed forms were collected by the research team at the end of the intervention. Additionally, we administered an ad-hoc questionnaire to all young people participating in the intervention after the implementation of the program (June-July 2017). The questionnaire gathered items on satisfaction with different aspects of the intervention. Table 1 shows the methods used and the indicators obtained for the process evaluation.
Description of indicators and instruments to evaluate the process implementation of the intervention.
| Indicator | Measurement instrument | What does the instrument measure? |
|---|---|---|
| Fidelity of implementation | ||
| Adherence | Self-reported form about sessions accomplished (filled out by facilitators) | The extent to which important activities have been carried out for the program to be successful |
| Dosage | Number of activities completed during the intervention | |
| Responsiveness | Motivation and proactive commitment of the group of young people | |
| Quality | Questionnaire administered immediately after the intervention (POST-1) to youth | The extent to which the intervention has been implemented in accordance with the protocol |
| Compliance with the intervention | ||
| Acceptable implementation | Self-reported form about sessions completed (filled out by facilitators) | The extent to which facilitators have completed all the activities foreseen in the intervention (based on the dosage) |
| Qualified implementation | The extent to which facilitators have completed the main activities in accordance with the program protocol (based on adherence, dosage and quality) | |
| Non-acceptable implementation | Not being neither an acceptable nor a qualified implementation | |
| Satisfaction | ||
| Satisfaction with group dynamics, topics discussed, program materials used and overall satisfaction | Self-reported form about sessions completed (filled out by facilitators) | Satisfaction of young people regarding different aspects of the intervention |
| Recommendation of the program to friends or peers | Participants’ opinions about recommending the program or not to their friends or peers | |
Table created and adapted by the authors on the basis of the one published by Resnicow et al.27
Implementation fidelity was described in terms of adherence, dosage, responsiveness, and quality of the intervention. Adherence is the percentage of core activities implemented (n=7). Dosage is the number of activities implemented, the total number being 43 activities. Responsiveness was determined based on three items related to the degree of agreement reported by facilitators regarding attention, comprehension and control of the group in each session. Participant engagement in the group was good when the facilitator reported fair or total agreement with the three items. Quality was assessed through five items: 1) relationship between facilitator and participants; 2) personal relationship between facilitators and participants; 3) the teacher's ability to motivate participants to engage with activities; 4) the teacher's ability to teach, and 5) the way the educator led the sessions. These items were reported by young participants of the intervention, who scored each item from 0 to 10 points.
For the overall assessment of delivery, we defined a compliance indicator that defined the implementation as non-acceptable, acceptable and qualified, using both quantitative and qualitative criteria. Acceptable implementation was determined by the dosage (quantitative criterion) and was defined as the performance of at least 30 out of the 43 activities of the program. Qualified implementation was determined by the dosage (quantitative criterion) and the implementation of important activities. In this case, maximum activity had to be conducted if applicable (qualitative criterion). Therefore, qualified implementation consisted of: 1) completion of at least 30 activities, and 2) completion of at least six out of nine important activities (seven core activities and the other two activities from sessions 3 and 7 considered as important (see Table I in online Appendix), and having completed maximum activity, if applicable. When the implementation was neither qualified nor acceptable, it was considered as non-acceptable.
To determine participants’ satisfaction with the intervention, the questionnaire gathered information on satisfaction with group dynamics, the topics discussed, the program materials used, and overall satisfaction. Satisfaction was assessed using a 10-point scale. Finally, participants were asked if they would recommend the intervention to their friends and peers (yes/no/I don’t know).
Statistical analysesDescriptive statistics are presented as absolute frequencies and percentages for categorical variables and as means and standard deviation for continuous variables. All items on the degree of agreement were dichotomized into: 1) totally or fairly agree, and 2) neither agree nor disagree, and fairly or totally disagree. All statistical analyses were conducted using Stata v15.
ResultsFifteen out of the initial 17 groups completed the intervention (10 from SGP and 5 from neighborhood organizations [10 different institutions]). To evaluate the implementation process, we obtained responses from 15 facilitators (one from each group completing the intervention) and from 99 participants (response rate 54%).
The mean number of activities undertaken by all groups was 30.4 (out of a total of 43). The percentage of activities accomplished in each session (S) were: 82.2% (S1), 73.3% (S2), 68.9% (S3), 67.8% (S4), 69.5% (S5), 73.3% (S6), 69.3% (S7) and 43.3% (closing session) (data not shown). Various teaching methods were used to carry out the activities. Table 2 shows the number and percentage of groups that used the planned methods for each of the sessions. The mean percentage of planned teaching methods used in all sessions were the following: 100% for oral explanation, 61.9% working on skills and emotions, 83.3% discussion, 78.9% active teaching and learning, and 35% for ICT.
Number and percentage of groups (n=15) where different teaching methods were used, according to the session.
| Session 1 | Session 2 | Session 3 | Session 4 | Session 5 | Session 6 | Session 7 | Closingsession | Totalmean | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| n | % | n | % | n | % | n | % | n | % | n | % | n | % | n | % | n | % | |
| Oral explanation | 15 | 100.0 | 15 | 100.0 | 15 | 100.0 | 15 | 100.0 | 15 | 100.0 | 15 | 100.0 | 15 | 100.0 | N/A | N/A | 15 | 100 |
| Working on skills and emotions | 13 | 86.7 | 11 | 73.3 | N/A | N/A | 6 | 40.0 | 15 | 100.0 | 7 | 46.7 | 2 | 13.3 | 11 | 73.3 | 9.3 | 61.9 |
| Discussion | 13 | 86.7 | 13 | 86.7 | 13 | 86.7 | 12 | 80.0 | N/A | N/A | 14 | 93.3 | 10 | 66.7 | N/A | N/A | 12.5 | 83.3 |
| Active teaching and learning | 14 | 93.3 | 13 | 86.7 | 13 | 86.7 | 15 | 100.0 | N/A | N/A | 8 | 53.3 | 8 | 53.3 | N/A | N/A | 11.8 | 78.9 |
| ICT | 4 | 26.7 | 7 | 46.7 | 4 | 26.7 | 4 | 26.7 | 4 | 26.7 | 5 | 33.3 | 12 | 80.0 | 2 | 13.3 | 5.3 | 35 |
ICT: information and communication technology; N/A: not applicable.
As for facilitators’ perception on the implementation and success of the program in their groups, 73.3% were satisfied with participants’ attention, 77.1% with their comprehension, and 66.7% with control of the group. Overall satisfaction with the intervention was 7.4 out of 10.
Table 3 shows information on the indicators used to assess the fidelity of implementation. The mean adherence to the intervention was 82.9%, indicating that a mean of 5.8 out of seven core activities were completed by the groups. Seven groups achieved an adherence of 100%. The mean number of completed activities was 30.4 (dosage). The mean percentage of facilitators who positively assessed responsiveness was over 65% (73.3% regarding attention, 77.1% comprehension and 66.7% control of the group). There were differences among groups in responsiveness, these differences being more evident in control of the group (degree of agreement below 50% for several sessions in five groups). The quality of implementation was rated with a mean scoring close to 8 points. The most highly rated aspect was the facilitator-participant relationship (8.3 points), followed by the way the educator led the sessions (8.1 points), and the teacher's ability to teach (8.1 points). The implementation of the intervention was at least acceptable (acceptable or qualified) in 66.6% and qualified in 33.3% of the groups.
Fidelity of implementation and compliance with the intervention.
| Group 1 | Group 2 | Group 3 | Group 4 | Group 5 | Group 6 | Group 7 | Group 8 | Group 9 | Group 10 | Group 11 | Group 12 | Group 13 | Group 14 | Group 15 | Total, mean (SD) | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Fidelity of implementation | ||||||||||||||||
| Group | ||||||||||||||||
| Adherencea (%) | 71.4 | 100.0 | 100.0 | 100.0 | 85.7 | 100.0 | 100.0 | 71.4 | 85.7 | 85.7 | 100.0 | 42.9 | 42.9 | 57.1 | 100.0 | 82.9% (20.3%) |
| Dosagesb (number of activities) | 30 | 32 | 33 | 34 | 34 | 34 | 30 | 38 | 28 | 29 | 30 | 20 | 24 | 21 | 39 | 30.4 (5.3) |
| Responsivenessc (% of agreement) | ||||||||||||||||
| Attention of youth | 42.9 | 14.3 | 42.9 | 57.1 | 100.0 | 100.0 | 100.0 | 71.4 | 100.0 | 100.0 | 100.0 | 57.1 | 85.7 | 85.7 | 42.9 | 73.3% (10.9%) |
| Comprehension of youth | 28.6 | 57.1 | 71.4 | 85.7 | 100.0 | 100.0 | 100.0 | 42.9 | 85.7 | 85.7 | 100.0 | 71.4 | 85.7 | 100.0 | 42.9 | 77.1% (8.5%) |
| Control of the group | 28.6 | 28.6 | 28.6 | 42.9 | 85.7 | 85.7 | 85.7 | 85.7 | 85.7 | 85.7 | 100.0 | 42.9 | 71.4 | 71.4 | 71.4 | 66.7% (10.9%) |
| Young participants (n=99) | ||||||||||||||||
| Qualityd(score 0-10) | ||||||||||||||||
| Facilitator-group relationship | 7.7 | 8.7 | 6.7 | 8.3 | 9.2 | 4.9 | 8.7 | 8.5 | 8.3 | 10 | 9.8 | 10 | 10 | 9 | 10 | 8.29 (3.0) |
| Facilitator-young participant personal relationship | 7.7 | 9 | 5.9 | 9.2 | 7.5 | 5 | 7.8 | 9 | 6 | 10 | 10 | 9.5 | 10 | 9 | 10 | 7.88 (3.1) |
| Facilitator's ability to motivate to engage youth in activities | 8.4 | 9 | 6.5 | 7 | 9.3 | 4.7 | 8.4 | 9 | 6.2 | 10 | 9.5 | 9.5 | 10 | 9.3 | 10 | 7.97 (3.1) |
| Facilitator's ability to teach | 7.6 | 9 | 6.2 | 8 | 9.6 | 4.7 | 8.7 | 8.5 | 6.8 | 9.7 | 9.5 | 9.8 | 10 | 9 | 10 | 8.10 (3.1) |
| The way the educator leads the sessions | 7.1 | 9 | 5.9 | 7.5 | 9.5 | 5.6 | 9.1 | 8.5 | 6.7 | 9.8 | 10 | 10 | 10 | 9.3 | 10 | 8.12 (3.1) |
| Compliance with the intervention | ||||||||||||||||
| Kind of implementatione (A/Q/nA) | A | A | A | Q | Q | A | Q | A | nA | nA | Q | nA | nA | nA | Q | A: 33.3Q: 33.3nA: 33.3 |
SD: standard deviation.
Adherence is the percentage of completion of the 7 core activities (A4.3, A4.4, A5.3, A5.4, A5.5, A6.3 y A6.4) within the sessions 4, 5 and 6. We considered the activity was completed when it had been completely carried out independently of whether the minimum or maximum activity was undertaken, if applicable.
The dosage is the total number of activities carried out in all sessions, the maximum number being 43 activities.
Responsiveness was determined on the basis of 3 Likert-scale items related to the degree of agreement reported by facilitators regarding attention, comprehension and control of the group during each session. We calculated the mean percentage of agreement for all sessions. The total value is the mean percentage of agreement for all groups and sessions.
Quality was determined by five items with a 0-10-scoring scale reported by youth. We calculated the mean score for each item.
Acceptable implementation (A) was defined as having completed at least 30 out of the 43 activities of the program; qualified implementation (Q) was defined as having completed at least 30 activities and, at least six out of nine important activities (A3.3, A4.3, A4.4MAX, A5.3, A5.4, A5.5, A6.3MAX, A6.4MAX and A7.4MAX), having carried out the maximum activity, if applicable; non-acceptable implementation (nA): not being neither an acceptable nor a qualified implementation.
Participant satisfaction with the intervention was higher than 7 out of 10 points (7.15 for group dynamics, 7.43 for topics raised and 7.7 for program materials) and overall satisfaction with the intervention was 8.04. Some 62.2% of youth would recommend the intervention to their peers (Table 4).
Satisfaction of youth with the Reptes program.
| Youth from the intervention group (n=98)a | ||||||
|---|---|---|---|---|---|---|
| Total (n=98) | Girls (n=73) | Boys (n=73) | ||||
| Mean | SD | Mean | SD | Mean | SD | |
| Satisfaction (0-10 points) with | ||||||
| Group dynamics | 7.15 | 3.12 | 7.76 | 3.10 | 6.93 | 3.12 |
| Topics discussed in sessions | 7.43 | 2.97 | 8.00 | 2.78 | 7.23 | 3.03 |
| Program materials used | 7.70 | 3.06 | 8.20 | 2.90 | 7.52 | 3.11 |
| Overall satisfaction (0-10 points) | 8.04 | 2.53 | 7.79 | 3.04 | 8.12 | 2.37 |
| n | % | n | % | n | % | |
| Would you recommend the program to a peer? | ||||||
| Yes | 61 | 62.2 | 17 | 68.0 | 44 | 60.3 |
| No | 12 | 12.2 | 3 | 12.0 | 9 | 12.3 |
| Don’t know | 25 | 25.5 | 5 | 20.0 | 20 | 27.4 |
SD: standard deviation.
Process evaluation of the implementation of the Reptes program allowed us to identify high compliance with the intervention and high use of the various teaching methods, except for ICT. The implementation of the proposed activities throughout the sessions was high, except for the closing session, consisting of a party, in which participation was lower than in previous sessions. Implementation fidelity was successful in line with what was found in previous reviews of similar programs.6 The adherence, dosage, responsiveness and quality values were above 70% or 7.5 points out of 10. Interventions were acceptable in two out of three groups and was qualified in one out of three. Young people's satisfaction with the program was high.
In the process evaluation of the Reptes program, the authors learned three important lessons, which may be of use to other planners. First, the Reptes program was designed through a co-production process, based on several pre-existing interventions with proven evidence of effectiveness.9,14,16,20–22 Through a step-by-step process23, it was based on the evidence and consultations with peers and intermediary professionals; pre-existing activities were adapted and new ones were co-produced; and the best ones were prototyped, according to acceptability and fidelity. The “core” of the intervention was concentrated in sessions 4, 5 and 6, and had to be delivered in all groups. More than half of the Reptes groups carried out activities concentrating on skills and emotions and about 80% carried out the interactive procedures typical of conflict resolution. Technologies were not frequently used in delivering the program, possibly due to certain technical problems, such as the lack of electronic devices or good internet connection, or the difficulty of adapting the content of the program to online activities. However, this difficulty was generally overcome with the use of more traditional alternative teaching methods.
Second, the Reptes program aimed for completeness in the measurement of implementation fidelity and a mixed indicator, compliance with the intervention, was created in a way that allowed, through two categories (acceptable and qualified) identification of the level of implementation achieved. Although the sample had some levels of truancy and irregular attendance at leisure and free time activities, average adherence was high, 83%; the average dose was 30.4 activities out of 43; the various student involvement parameters showed values between 66% and 77%; and the quality components were assigned between 7.8 and 8.3 points out of 10. In addition, 2/3 of the groups delivered an acceptable or qualified intervention, and in 1/3 of the groups it was a qualified intervention. This high fidelity of implementation in these groups could point to one of the strengths of the program. High values in these parameters have also been found in the measurement of the fidelity of implementation of other similar programs10,11,22.
Third, the Reptes program also studied participant satisfaction with the program, finding satisfaction levels higher than 7 points out of 10 in several of the aspects analyzed, with an average overall satisfaction with the program of 8 points, and 62% of participants indicated they would recommend the program to their peers. Similar scores were reported in many of the programs used as a reference by the Reptes program,10,11,24 and could in part be due to the high motivation of the facilitators when implementing the program.
Finally, it is necessary to address the controversial aspect about the differences between the implementation of an intervention under experimental conditions and its application under real conditions. In the evaluation phase of the process of the Reptes program, the results of the implementation indicators showed in this study were highly favorable. The main reasons were, probably, the strong motivation of the educators and their good control of the sessions. However, these findings might differ with the dissemination and generalization of the program. Ennett et al.3 studied 10 top-quality programs and found very modest results for adherence (27.7%) and exposure (35.8%), with quality of implementation and involvement values higher than 80% in most programs. These findings confirm that many educators deliver the programs, but not necessarily according to the prescribed protocol.11,17,25 Therefore, tracking the fidelity of the implementation during the implementation of the program in real-world conditions will be necessary.
Limitations and strengthsOne of the main limitations of this study is that the evaluation did not include the groups that left the program before completing the intervention. Likewise, there is no information on all participants in the groups that carried out the intervention, mainly due to data matching problems in the follow-up. This implies an information bias in the evaluation due to the unavailability of data from the participants who could have had the worst scores in the various parameters studied. Another limitation of this study is the high levels of fidelity found in the application of the program, attributable to a large extent to the high motivation of facilitators and their commitment to evaluation. This is a limitation because the dissemination of the program in real-world conditions, where motivation will probably not be as high in all settings, represents a challenge. However, it also represents a strength, since it has allowed comprehensive collection of implementation parameters that otherwise could not have been recorded and analyzed. Other strengths of the program are the scarcity of this type of selective prevention program in the reference environment, the rigor used in the design —based on programs with the strongest evidence in the specialized literature—, and the rigorousness of the data collection, both in the process indicators and in those related to the effect.
To conclude, this study found high compliance with the intervention, quality, and participant satisfaction. These results should be contrasted in the dissemination phase of the program through a follow-up study of the implementation in real-world conditions. The dissemination phase should have less complexity and detail but should identify whether fidelity of implementation and high participant satisfaction are maintained, especially in adherence and exposure, as pointed out by some of the cited authors. Finally, it will be important to analyze whether these favorable results in the process evaluation correspond to good results in the effects generated by the intervention in both intermediate and consumption variables of young people, as it was noticed in the programs that we used as reference.11–13,26
The importance of study the intervention development process of the program is today beyond doubt but the quality of implementation of the intervention is the scarcity of studies in the health literature that describe the intervention process and use rigorous implementation indicators
What does this study add to the literature?This study found high compliance with the intervention, quality, and participant satisfaction of Reptes, a selective prevention program of alcohol and cannabis in youngsters
What are the implications of the results?These results shown the importance of process evaluation. Future research should analyze whether these results correspond to good effects generated by the intervention in youngster consumption.
Databases and other materials are available upon request to the corresponding author.
Editor in chargeJorge Marcos Marcos.
Authorship contributionsF. Sánchez, A. Pérez, O. Juárez, M. Bartroli, N. Vázquez, X. Continente and C. Ariza designed the program and the study. The first idea was from C. Ariza, but them we developed collaborative team meetings to construct the program and study design. L. Muñoz collected data. X. Continente analyzed quantitative data, and N. Vázquez and C. Ariza support him on the interpretation. N. Vázquez, X. Continente and C. Ariza wrote the first version of the paper. C. Ariza focus on the introduction and the discussion sections, X. Continente on methods section and N. Vázquez on abstract, results and references sections, but all of them worked on and reviewed all sections. F. Sánchez, A. Pérez, O. Juárez, M. Bartroli, N. Vázquez, X. Continente, L. Muñoz and C. Ariza reviewed all versions of the paper, provided relevant contributions and approved the final version.
The Reptes Project investigatorsCarles Ariza, Montse Bartroli, Xavier Continente, Olga Juárez, Maria José López, Laura Muñoz, Anna Pérez, Francesca Sánchez-Martínez, Noelia Vázquez and Joan Ramon Villalbí.
AcknowledgmentsWe would like to thank all facilitators and young people for their participation in the study evaluation of the Reptes program. We would also like to thank Laia Clemente, Catalina Londoño and Laura López for their help in recording process data electronically.
Ethics approvalThis study was performed in accordance with the ethical standards of the Parc de Salut Mar Clinical Research Ethics Committee (2015/6416/I) and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.
Consent to participateAll participants were previously informed about the study ant they signed an informed consent to be able to participate in the study and complete questionnaires.
FundingThis study was supported by the Government Delegation for the National Plan on Drugs (Grant 2015/070) (Ministry of Health, Social Services and Equality). The study was also partially supported by the Ministry of Universities and Research of Catalonia (AGAUR) (2017SGR1526).
Conflicts of interestNone.
