Using the Reach, Effectiveness, Adoption, Implementation, Maintenance Framework in the Evaluation of Community-Based Adolescent Care Pilot Program

Posyandu Remaja (Posmaja) is a community-based adolescent care service in Indonesia that comprises a group of adolescents working as volunteers and healthcare workers under the guidance and responsibility of the primary health care and department of health. The study aimed to provide a formative evaluation program of Posmaja using the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework. This study used a qualitative approach and took place in South Tangerang City, Indonesia. The data were collected from July until December 2018. Nineteen people participated in the study. The study used in-depth interviews and focus group discussions that adopted the RE-AIM framework for data collection. Content analysis was used to analyze the collected data. The majority of Posmaja’s participants were male adolescents aged around 15 years. Common themes generated were “adolescent empowerment,” “increasing health knowledge,” and “monitoring health and nutrition” as a result of doing the pilot program. Volunteers and healthcare workers recognized the benefits of Posmaja and thus encouraged the adoption of the program. This awareness, followed by the department of health office’s willingness to fund and adopt the program, was viewed as highly necessary for the program’s continuation.


Introduction
Indonesia is a country that is ripe for taking advantage of such a demographic dividend. 1 However, this advantage could not be utilized if the population lives in unhealthy conditions and practices risky behaviors. 2 Currently, the number and prevalence of risky behaviors are still considerably high among Indonesian adolescents. According to the 2014 Global Youth Tobacco Survey, the number of current cigarette smokers among male and female students aged 13-15 years in Indonesia was 35.3% and 3.4%, respectively. 3 However, the Indonesian Basic Health Survey 2018 also reported that the number of smokers among adolescents increased yearly. 4 A different survey by the Global Student Health Survey showed that 4.4% of the students currently drank alcohol, 8.7% were underweight, 14.8% were overweight, and 1.4% of Indonesians have ever used marijuana. 5 Health care plays an important role in responding to and preventing youth health problems. 6 Therefore, in 2018, Ministry of Health of Republic Indonesian launch -ed a community-based health care program specifically for adolescents called Community-based Adolescent Care, or Posyandu Remaja (Posmaja). Posmaja is a healt hcare service that operates at the community level. It aims to increase access and improve adolescents' health care coverage. Healthcare adopts community empowerment concepts where the healthcare services are provided and performed with the community, by the community, and for the community under the guidance and responsibility of the primary health care (PHC) and department of health. Healthcare also included adolescents between ages of 10 and 19 years old as community health workers who joined voluntarily. Once the adolescents join ed, they were trained to perform the discharging services. Activities at the healthcare center comprised anthropometric measurements (weight and height measurements), blood pressure tests, and hemoglobin (Hb) tests for female adolescents. Other services provided were essential healthcare services like performing youth health assessments by health workers and giving vitamins or iron supplements, referring adolescents to PHC if need ed, and providing health education on particular topics such as reproductive health, harmful effects of substance abuse, youth nutrition, prevention of non-communicable diseases, and violence. 7 In the last trimester of 2018, the Department of Health of South Tangerang City, in cooperation with the Universitas Indonesia, developed a Posmaja pilot program in three subdistricts (Serua, Cempaka Putih, Sawah Baru) across South Tangerang City. This program was implemented for three months from September to November 2018. Before making the service available on a larger scale, it would be worthwhile to evaluate the pilot program's implementation. More so, very few or no other evaluation studies had been made regarding Posmaja. Given the shortcomings of the literature presented, this study aims to provide a formative program evaluation of Posmaja in South Tangerang, Indonesia, using the Reach, Effectiveness, Adoption, Implementation, Maintenance (RE-AIM) framework.
The RE-AIM model was initiated in 1999 as a comprehensive evaluation framework to evaluate public health interventions. 8 This framework has primarily been used in studies that focus on changing individual behaviors. 9 The RE-AIM framework could inspire stakeholders to concentrate on crucial program elements that could improve sustainable adoption and implementation of effective, evidence-based interventions. 9 The model could assess innovations seen at both the individual (e.g., enduser) and organizational (e.g., delivery agents) levels. 10 Even though the model was used in over 400 publications, 11 using such a model for evaluating a health intervention program in Indonesia remains low.

Method
This qualitative study with a case study approach was conducted in South Tangerang City, a city located at the southern border of Jakarta, with approximately 1.6 million people, including 140,000 adolescents. 12 The study data was collected between July and December 2018. The case study approach was chosen to generate an in-depth understanding of an issue in its real-life context. 13 Study participants were chosen using a purposive sampling method whose selection criteria included participants who had an informed knowledge of the study and were actively involved in the study's activities. Nineteen people participated in this research. It involved seven Posmaja volunteers, seven Posmaja service users/visitors, three PHC workers who provided services at Posmaja, and two Department of Health of South Tangerang City staffs res ponsible for the running of Posmaja.
Data were collected through in-depth interviews and focus group discussions. While in-depth interviews were conducted during data collection from Posmaja visitors, focus group discussions were performed during the information-gathering process from Posmaja volunteers and the staff of PHC and department of health. Data was collected using semi-structured interview guidelines, which were adopted from the RE-AIM framework. 14 The interview guidelines contained five major themes, including reach, effectiveness, adoption, implementation, and maintenance. Definitions and a sample of questions are provided in Table 1. The interview process required about 30 minutes to complete, whereas the focus group discussion required one hour and 30 minutes. Two researchers in this study performed data collection. Participants were provided with information regarding the research procedure before data collection, and those who agreed to participate in the study signed a consent form. Anonymity and confidentiality were ensured during the study process, with the ethical clearance of this study being obtained from the Ethics Committee of Health Research of the University of Muhammadiyah Prof. Dr. Hamka No. 03/18.12/033.
Content analysis was used to analyze the data, whereas data organization was performed using Microsoft Excel (Microsoft, Redmond WA, 2010). Initially, a deductive

Reach
The absolute number, proportion, and individual representatives are willing to participate Is the intervention reaching the target population? in a given initiative, intervention, or program.
What is the right strategy to achieve the intervention target? Effectiveness The impact of an intervention on important outcomes, including potential negative effects, Does the intervention accomplish its goals? quality of life, and economic outcomes. Adoption The absolute number, proportion, representative of settings, and intervention agents What characteristics of the Posmaja Program (people who deliver the program) who are willing to initiate a program.
appeals to the organization to adopt the program (Posmaja)? Implementation At the setting level, implementation refers to the intervention agents' fidelity to the various To what extent were the various intervention compoelements of an intervention's protocol, including consistency of delivery as intended and nents delivered as intended (in the protocol)? the time and cost of the intervention. At the individual level, implementation refers to clients' use of intervention strategies.

Maintenance
The extent to which a program or policy becomes institutionalized or part of the routine To what extent were intervention components contiorganizational practices and policies. approach was used to establish preliminary codes based on the RE-AIM framework, and later themes from data were generated using an inductive approach. 15,16 Later, a classification scheme was built based on the interview and focus group discussion questions. The codes were then inserted into the themes according to the data patterns. Multiple coding techniques were used by performing the cross-checking. 17 Data saturation was reached when no new information emerged from the codes. 18

Results
The study dimension involves the number of Posmaja's visitors in the three locations, volunteers' strategy to reach adolescents, and the challenges. Visitors who attended each Posmaja were Serua Posmaja (68 adolescents), Cempaka Putih Posmaja (57 adolescents), and Sawah Baru Posmaja (80 adolescents). Most visitors were aged 15 years (37.1%). The total number of visitors per month in each Posmaja ranged from 57 to 80 visitors (Table 3). Overall, the total number of visitors did not reach the target number, which was initially 30 visitors for each open session.
The involvement of adolescents as volunteers in Posmaja was advantageous for Posmaja to reach their target group since they are also in the adolescent group. Under these conditions, volunteers had known how to communicate and use the communication channel to reach the target group. Posmaja volunteers had thought about strategies that could attract more visitors, such as using the nearest worship places to announce the ongoing session. The teenage volunteers also took advantage of the broadcast message system on WhatsApp and Line app lications to send messages to their friends, encouraging them to join the session. Another strategy used by Posmaja to reach the target group was to hold Posmaja at the center of the community residences, such as the community meeting hall. With these strategies, adolescents could reach Posmaja easily by walking and without having to incur transportation costs. One of the chal-lenges in getting more participants to join was the lack of knowledge by many young people about Posmaja, which contributed to their refusal to participate. The failure to invite more young people was also caused by the volunteers for not knowing the young people who lived around the Posmaja location. This unfamiliarity led to their inability to invite other volunteers to join Posmaja.   Regarding the effectiveness, participants were asked questions about their thoughts on the benefits of the Posmaja program. A visitor admitted that Posmaja could help to spread the knowledge about the detrimental effects of cigarettes, alcohol, and the benefits of healthy foods and physical activities. In another interview, the visitor also admitted that he wanted to reduce his cigarette consumption after joining the session. He also said that he did not intend to drink alcohol, but rather, start doing physical activities.
Health workers delivered health knowledge during health consultation sessions in Posmaja. In the consultation session, health workers provided some health information regarding the several health risk behaviors, their impact on health, and how to prevent them. Health risk behaviors mentioned in the counseling were substance abuse, low physical activity, and unhealthy dietary behavior. Health information was also delivered by the volunteers through board games and card games as media of health education.
Furthermore, health workers admitted that Posmaja could be very beneficial, especially to monitor adolescents' nutrition status and health conditions. In Posmaja, visitors received several services, such as upper arm circumference, body mass index (BMI) calculation, blood pressure test, Hb measurement, and health counseling. "So, for example, we have a problem, we can get guidance (to solve the health problem), then we also learn about health problems… We can also know the condition of our body weight, height, blood pressure". ( The adoption dimension explored how the program added value and benefit to the PHC program and volunteers. The PHC's health workers strongly believed that the activities in Posmaja could help maintain the sustainability of health services for young people, which aligns with the existing PHC's program. PHC organized a regular event in the form of a school visit to promote health information and undertake a health assessment. Therefore, Posmaja is intended to make health services accessible to young people who attend school and all young people, especially those who do not attend school. Similarly, participating volunteers were very satisfied with discharging their services as they learned about adolescent-related health issues.
The key topic of the implementation dimension includes the extent to which Posmaja operated consistently according to its target. At the institutional level, some influencing factors that are believed to affect the implementation process included the volunteers and the health workers who worked at Posmaja, and the individual visitor's willingness to pay a visit to benefit from the services provided.
Posmaja's operation is primarily affected by the volunteers' performance when doing their given roles, since most of them were students who had their own school activities, such as extracurricular activities and other social life events. Unfortunately, volunteers' lack of sufficient knowledge about general health issues also influenced the knowledge transfer process. The other factor affecting the intervention was that health education was used to promote health information. The participants admitted that the media used by the volunteers played a significant role in facilitating their understanding of the delivered message. Thus, the participants claimed that board games were easier to understand than card games, although the lack of knowledge about Posmaja affected many visitors attending Posmaja activities. Some students who attended school on Saturday precluded them from visiting and attending Posmaja activities.
Healthcare workers from PHC and the Department of Health of South Tangerang City showed their readiness to continue the Posmaja program by committing themselves to allocate a budget plan for Posmaja in the Health Operation Cost. In addition, there was a South Tangerang Health Office's direct mandate that instructed the implementation of the Posmaja program in all the PHC in South Tangerang City. As for the challenges, health workers at the PHC found it difficult to maintain a balance between performing their job and organizing Posmaja. The theme focused on was on a personal level, which was consistent with their visits due to the benefit gained by young people who joined the Posmaja program. As for consistency, this study found visitors unable to attend the Posmaja session as scheduled. Some participants acknowledged the need to introduce innovative content, such as doing some exercises, strolling, and other fun activities that would transform Posmaja into a program that would attract a teenage audience. Another participant said that a lack of health awareness among teenagers also contributed to the low number of participants.

Discussion
This study presents a comprehensive evaluation of a community-based health program aimed at improving the health of adolescents using the RE-AIM framework. This study used qualitative data collected from participants, volunteers, and PHC and department of health workers. Participants stated that Posmaja played an important role in health promotion for adolescents. Health promotion for adolescents is necessary and beneficial when their adult health and wellbeing are dependent factors on their adolescent health lifestyle. 19 Since Posmaja aims to empower youth when youth and adults collaborate on this program, positive development may be attained at both the youth and community levels. 20 The results of this study also revealed that Posmaja had a role in monitoring young people's health conditions by identifying risky behaviors and detecting diseases at their earlier stages, thereby becoming a tool to prevent further unhealthy conditions. 21 Posmaja was unpopular among adolescents. So, it was a big challenge to engage more young people to participate. The volunteers were also unable to persuade their peers, which resulted in low participation. Since volunteers had no close relationship with young people in the neighborhood, it was difficult to persuade more young people to join the program. A previous study highlighted the importance of peer groups combined with adequate communication and advocacy skills were the key elements for the success of the intervention to reach specific populations. 22 The lack of sufficient knowledge regarding healthy nutrition, physical activity, and the harm ful effects of substance abuse among youth posed a great challenge in our program. This result was also confirmed by previous studies that found peers working in the field were not fully knowledgeable about their roles and care. 23 Therefore, there was a need for additional training of volunteers regarding several adolescents' health issues to enable the volunteers to educate the service users.
The result of this study found that adolescents had very low motivation for attending healthcare services, which led to their low participation in Posmaja. There was a significant need to raise awareness among adolescents about the significance and the need for health services and motivate them to avail of the services. 24 Volunteers from peer groups played an important role in approaching the target since they represent a source of information regarding many health-related behaviors. 25,26 As reported in the maintenance dimension, PHC had shown a commitment to proceed with the program through budget allocation in their Annual Plans. This program would be implemented once the collaborative program between Universitas Indonesia and the Department of Health of South Tangerang City was completed. Another systematic review study found that program funding was among the top three most significant factors in sustaining community-based health programs. 27 The three locations where the study was performed were typically identified as urban areas; hence, the results might not apply to rural areas. However, the use of a formative approach that utilizes the RE-AIM framework is transferable to many settings. 16 Future evaluations could be done with the program more widely implemented, including qualitative and quantitative measures.

Conclusion
The use of RE-AIM framework in the evaluation of the program suggests that Posmaja is potentially impactful; specifically, for empowering the youth, promoting health knowledge, and for monitoring health conditions among the youth. The volunteers, consisting of a group of young people, play a vital role in the implementation process since they are the ones who promote Posmaja to their fellow peers and persuade them to participate. This is why a better recruitment process of volunteers was needed to meet the expected results. Furthermore, a well-managed training system is also necessary for both volunteers and healthcare workers. Two sets of the most important skills should be included in the training: communication skills and knowledge about different health issues. Furthermore, this program need ed more widescale advertisements to attract visitors and participants.

Ethics Approval and Consent to Participate
Prior to data collection, institutional permissions were obtained from the Department of Health of South Tangerang City and this study was approved by the Ethics Committee of Health Research of the University of Muhammadiyah Prof. Dr. Hamka with number 03/18.12/033. Informed consent from informants was obtained before the data collection process.

Competing Interest
Author declares that there are no significant competing financial, professional, or personal interests that might have affected the performance or presentation of the work described in this manuscript.

Availability of Data and Materials
Due to the nature of this research, participants of this study did not agree for their data to be shared publicly, so supporting data is not available.