Training approach, structure, duration, and competencies

IORT is a learning-by-doing course that uses a deliverable-driven model to teach research skills. Learning-by-doing is a hands-on training approach in which the learner interacts with their environment or project to enable learning [19]. This approach exemplifies ‘doing’ beyond ‘seeing’ and ‘hearing’ during learning. Thus, in IORT, trainees learn research skills by implementing research projects that are of consequence to service delivery and that are publishable. The deliverable-driven model, which breaks down the research process into smaller, manageable components (deliverables), enhances this practical approach. Deliverables help deconstruct the research process to new learners, focusing on smaller milestones that culminate to a research paper. Each milestone has clearly defined outputs and timelines to track learning and progress.

IORT consists of alternating in-person sessions and practicum sessions (Figure 1). During in-person sessions, we implement short lectures on research concepts and skills. We then give trainees time to apply the concepts to their projects. This process reinforces skills acquisition and confidence, and ensures that trainees receive adequate support to effectively implement their research projects [20, 21]. Trainees present their work to their peers for feedback in plenary sessions, strengthening their communication, peer learning, and public engagement skills. Plenary sessions also encourage a culture of critical appraisal of research. The practicum periods follow each in-person session and enable trainees to complete milestones with ongoing mentorship and active engagement from their research authorship team. Trainees who complete each milestone in time for the next in-person session continue participating in the course.

Each course is 8–12 months long. We shifted the frequency and duration of in-person sessions from seven two-day sessions in the first two cohorts to three one-week sessions in the latter courses (Table 1). We found that the short-length high-frequency meetings facilitated stepwise research learning in a context of limited research exposure [17]. However, the frequent meetings disrupted the trainees’ full-time jobs, required more planning, and became cost and time prohibitive when mentors had to travel internationally to facilitate the course [17]. The three one-week sessions structure enabled focused research time with more in-person mentorship and reduced workload in the practicum periods (Figure 1).

IORT covers research competencies in protocol development, data analysis, and manuscript development. Under protocol development, trainees learn how to craft a research question, select appropriate study design and research methods, build a research team, and collect data while observing research ethics in the process. During data analysis, trainees build their skills in data management, data analysis, results interpretation, and presentation. In manuscript development and publication, trainees identify discussion points from their results, develop their manuscript, learn how to choose a journal for publication, and learn how to manage the peer review to publication process (Figure 1).

Intensive hands-on mentorship and follow-up

Mentorship is key for a new researcher’s success. Adequate mentorship is needed for research trainees to produce quality research outputs [22, 23, 24]. Mentorship takes different forms, either hands-off or hands-on [22]. In hands-off mentorship, mentees only receive technical advice from mentors. In hands-on mentorship, mentees receive direct technical assistance in writing their research proposal, analyzing their data, and writing up their results for publication [22]. For the IORT, we provided intensive hands-on mentorship that prioritized in-person one-to-one mentorship during training and practicum periods. We supplemented the in-person mentorship with virtual mentorship through Skype calls and e-mail communication, including feedback on trainees’ research materials.

To avail robust mentorship, we engaged three types of mentors: senior course mentors, junior mentors/research assistants, and project mentors. The senior course mentors had extensive research experience with backgrounds in biostatistics, epidemiology, and public health. They led both course delivery and course mentorship. Each supported a maximum of four research projects. To date, all senior course mentors in IORT have been non-Rwandans who were either Rwanda-based or had lived in Rwanda for at least two years. Notably, local mentors are best suited to ensure training program ownership and increase the sustainability of the investments for national research capacity [23, 24]. This is because local mentors demonstrate the reality of a research career locally and improve the relevance of research projects to local priorities [23, 24]. They also reduce the course delivery costs (for example in regards to international travel costs) that can be prohibitive in nationalizing these training programs [23, 24]. However, deploying Rwandan researchers as senior course mentors remains a challenge, partly because those with the experience to fill this role have lacked the protected time required for the intensive hands-on mentorship.

The junior mentors included a pool of Rwandan and non-Rwandans with research backgrounds ranging from recent undergraduates with some research and statistics experience to global health practitioners with master’s degrees in public health. Each junior mentor supported one to three projects, depending on their availability and experience. Similar to senior course mentors, the junior mentors provide real-time hands-on mentorship during in-person training sessions and the practicum period. Junior mentors also received coaching from the senior mentors to grow their research and mentorship skills. We hope to increase the pool of Rwanda-based mentorship team through nurturing the junior mentors to become senior mentors in future iterations of this course.

We introduced project mentors in the 2015 course due to ethical review delays (delays lasted eight months in our 2013 course) and high review costs (US$1,400 per research proposal). These challenges have also been reported elsewhere in LMICs [25, 26]. Project mentors were local and international principal investigators with active health research projects in Rwanda. They were also affiliated with different programs implemented or supported by PIH/Rwanda. These mentors contributed research topics to the course from their active protocols, in part to fulfill their RMoH obligations in research capacity building [14]. In addition, they covered research project costs such as for ethical reviews and data collection. They also provided mentorship specifically in the practicum periods as their attendance of in-person training sessions was voluntary. Often, the research staff in the project mentors’ research team served as junior mentors for the associated training project. As seen elsewhere, we found a pre-training orientation and post-training reflection for project mentors useful for sharing training objectives, mentorship expectations, tips, and experiences [23].

Our mentorship process respected trainee leadership of the research project and allowed for growth of both trainees and mentors. With each milestone, the trainees led the drafting of the initial research protocol, data analysis, and manuscript writing. Trainees then received and addressed feedback from all mentors, beginning with junior mentors. We used an iterative process for review, within set timelines, until trainees and their mentors agreed that a drafted research document was ready for feedback from the authorship team. Because of the intensive contribution of mentors in the research process, all mentors formed part of the trainee’s authorship team, enhancing both their research careers and leadership skills [27].

Selection of trainees

During course application, trainees stated their research background and interests and provided a letter of support from their employer or supervisor. Because all trainees were full time health or academic professionals, we selected trainees jointly with their institutional leaders. We selected trainees based on three criteria. First, the senior course mentors ranked trainees’ readiness to complete the IORT course based on previous research experiences and training. Second, the project mentors ranked trainees’ existing or potential links to the program or research proposed. Third, the employer or work supervisor ranked applicants from their organization based on their organizational priorities for research training. The contribution of all stakeholders in the trainee selection enhanced stakeholder commitment to the course and secured institutional support for trainees to both attend the training and to lead research within their organizations post-training.

Because of limited funding and mentorship resources, we paired trainees on each research project to train more individuals while maintaining a reasonable number of research projects. Pairing facilitated peer-to-peer learning and engendered research partnerships across institutions. However, training teams are best limited to two individuals per project to enable tangible leadership contributions to the research project. Our trainees published as joint first authors.

Selection of research projects

The selection of research projects in the course shifted from trainees’ self-identified topics in 2015 to trainees selecting topics from a list submitted by the project mentors in later years (Table 1). Self-identified topics enhanced project ownership and ensured that research topics linked closely to trainees’ work. For these reasons, we prefer that trainees identify their own research topics. However, due to the time and cost implication for ethical reviews, we continue to use suggested research topics that are nested within existing approved studies. With this approach, we also shifted from trainees applying to the course with research partners to pairing trainees strategically across institutions and backgrounds. We recommend clustering of research topics under broader themes such as pediatrics, surgery, or non-communicable diseases. This helps trainees to focus on a thematic area of interest. In addition, having at least two projects per theme enhances peer collaboration between trainee teams and promotes focused mentorship as senior course mentors support trainees in their thematic area of interest.

Course evaluation

Trainees in the 2016 cohort completed a course evaluation. We conducted pre-course surveys with the 36 applicants and mid-course surveys with the 15 trainees. All applicants completed a semi-structured questionnaire about their research background and motivation for the course. The trainees completed the mid-course survey using Kirkpatrick’s first level of training evaluation model to assess their reactions to course materials, delivery, and outcome [28].

We analyzed qualitative responses in both evaluations thematically. We used a Likert scale with scores of strongly agree (score = 4), agree (score = 3), disagree (score = 2) and strongly disagree (score = 1) to measure trainee reactions. To measure training pace in the mid-course survey, we applied the scale as too fast (score = 3), just right (score = 2), and too slow (score = 1). We calculated the median Likert score and frequency of respondents with each median score.

Training and mentorship costs

We collected information on annual programmatic training and research project expenditures from research, finance, human resources, and procurement teams. We collected both cash and in-kind expenditures per course, as well as the corresponding number of training participants and research projects. We grouped expenditures into three categories: trainee costs, research project costs, and training delivery and mentorship costs (Table 2). We converted per course costs from Rwandan Francs to United States Dollars using an annual average conversion rate. Cost estimates did not account for inflation rate, which averaged 0.36% over the time period and would have minimal effect on the cost estimates [29]. We then calculated the average cost per cost item (e.g. trainee travels) per course to estimate unit costs. We report the median and range unit costs. We also estimated the median and range costs for a year-long, three one-week sessions course, with 16 trainees paired to lead eight research projects and supported by two senior course mentors and four junior mentors.

Course outputs

Between 2013 and 2017, 132 candidates applied for the IORT courses, 55 (41.7%) of whom were selected based on their research background, experience with chosen research project, and institutional training priorities (Table 3). Of those 55 admitted trainees, 53 (96.4%) completed the training by participating in the training sessions and completing and submitting their research project to a journal for publication. The ratio of female-to-male trainees in the course increased from 1:8 in 2013 to 1:3 in 2017. In the first two courses, our trainees were primarily clinical and program staff from PIH/Rwanda and district level RMoH. In the latter two courses, we admitted trainees from the University of Rwanda – College of Medicine and Health Sciences (33.3%, n = 5 in 2016) and the Rwanda Biomedical Center (20.0%, n = 3 in 2017). The research topics in the courses covered diverse health issues in Rwanda, such as infectious diseases (3 papers), maternal and child health (7 papers), non-communicable diseases (6 papers), surgery (9 papers), and pharmacy (3 papers) (Table 4).

Over the four courses, IORT trainees developed 28 research proposals and manuscripts (Table 3). As of July 2020, 96.4% (n = 27) of these projects were published in peer-reviewed journals, and one (3.6%) was under-development. 12 of the 53 trainees (22.6%) disseminated their research findings in regional or international conferences in Rwanda, and five trainees (9.4%) disseminated their findings at international conferences in Canada and the United States. Three former trainees have since become junior or project mentors in the course, and two junior mentors have become either a project mentor or a senior course mentor. Post-course, eight trainees (15.1%) enrolled in masters-level research training programs.

Course evaluation

The majority of 2016 applicants (N = 36) (69.4%, n = 25) had previously participated in a research project, often as an academic thesis (Table 5). Nearly half (47.2%, n = 17) had completed short research-related courses in epidemiology or biostatistics, and 13.9% (n = 5) had completed PIH/Rwanda’s Introduction to Research Course [16]. By their reports, course applicants’ goals varied from improving their research skills (83.3%, n = 30) to enhancing their ability to identify and close practice gaps (75.0%, n = 27) and improving their ability to critically consume scientific literature (38.9%, n = 14).

In the mid-course evaluation, trainees reported improvement in their research (93.3%, n = 14) and analytical (86.7%, n = 13) skills (Table 5). Most trainees strongly agreed that facilitators were engaging (93.3%, n = 14), and most agreed that the training team was available to support them (66.7%, n = 10). Many trainees (86.7%, n = 13) would recommend the course because IORT builds research skills (93.3, n = 14) covering skills such as critical thinking, writing, research production, mentorship, and leadership. The trainees recommended an increase in in-person and practicum time for data analysis (53.3%, n = 8) to ensure the finalization of results before the manuscript development training.

Course costs (US$)

For individual trainees to complete the course, we spent a median cost of $908 (Range $739–$1,253) per trainee, excluding the cost of laptops which the trainees either owned or borrowed from PIH/Rwanda (Table 6). The median unit cost per research project was $6,653 ($4,023–$12,631) with data collection ($892, $643–$4,528), ethical review ($1,158, $500–$1,450), publication fees ($1,300, $755–$2,100), and international conference travels ($2,645, $1,500–$3,790) as the main cost drivers.

The bulk of total training costs fell under training delivery and mentorship ($63,247, $46,340–$80,226) (Table 6). This was at least three times the trainee costs ($17,982, $14,520–$25,330) and the research project costs ($20,025, $12,626–$52,013). These estimates excluded the costs of international conference travels, publication fees, international travels for mentors, as well as junior mentors’ time which was provided in kind. The total cost for a three one-week IORT courses in Rwanda was $101,254 ($73,486–$157,569) with the above-mentioned exclusions. However, the overall cost without exclusions increased to $171,833 ($118,978–$255,183).

Course completion rates

The high course completion rates in the IORT is comparable to the course completion rates for the SORT IT courses (~89%) [30, 31]. We attribute this success to both the training approach and the intensive, ongoing hands-on mentorship. The learning-by-doing approach and the deliverable-driven model reinforced learning and task orientation for both trainees and mentors [17, 24, 32]. The practical nature of the learning-by-doing approach demonstrated the value of the course to both trainees and their institutions through publications and the development of work-related skills in critical thinking, writing, and analysis. The intensive, ongoing hands-on mentorship strengthened learning outcomes and availed the necessary support for the completion of training milestones, similar to what has been observed in other research capacity building programs [20, 21, 22, 23, 24, 33]. It also reduced the chances of trainees abandoning research projects due to work pressures, especially during the practicum period. Three trainees who changed employers within Rwanda during the course received support from their new employer to complete the course, underscoring trainee commitment and the value of research skills in Rwanda [13].

Gender distribution

The progressively more gender equitable trend in the course reflects programmatic decisions to intentionally select more qualified women in the course. In addition, the proportion of female applicants increased from 4.2% (n = 1) in 2013 to 36.4% (n = 16) in 2017. However, we still had more male than female trainees. This gender imbalance, also observed by other research capacity-building programs, may be attributed partly to time-demanding family obligations that typically fall on women in this context [16, 33, 34, 35]. Of note, nevertheless, the course completion rates were the same for female and male trainees, as has been noted in other training models [16, 36]. To promote gender equity in similar courses, we recommend allocating training spaces for female trainees. Furthermore, implementers should cultivate a family-friendly training environment where trainees can bring their children, as was made available for IORT trainees [33, 36]. Committing to long-term research engagement through sustaining mentorship relationships may increase the number of female trainees in research [33, 36].

Partnership development

Over time, our course increased both institutional and inter-professional partnerships, which may promote future research and build research capacity in Rwanda. In our experience, the diversity in trainees’ professional and institutional backgrounds enhanced knowledge and skills sharing among trainees. It also advanced the formation of a national network of Rwandan researchers. Trainees such as university lecturers have a greater potential for mentoring new students in research skills through their academic positions. We postulate that an annual meeting of a national network of IORT alumni and trainees may sustain these collaborations. Such a meeting should target the growth of additional research leadership skills such as grant writing. We plan to explore this in the future.

Dissemination

There is a growing call for systematic evaluation of the impact of courses like IORT [15, 37]. Individual research capacity building courses often measure research outputs such as publications and short- and long-term assessments of the course’s impact on trainees’ ability to use and transfer research skills [37]. We measured peer-reviewed publications and conference attendance, both of which demonstrate the feasibility of research leadership by first-time researchers in rural and low-income settings. Trainees disseminated their research findings at the three district health facilities where their studies were conducted. This encouraged the uptake of operational research findings by health providers. Strategies to enhance the impact of local dissemination of results such as outlining practice briefs and monitoring the uptake of research results are needed [1]. While we did not measure the link between operational research and policy or practice, similar courses such as SORT IT have demonstrated this relationship [38].

Trainee experiences

Through the surveys, trainees critically reflected on the course, highlighting the short-term impact of the IORT course in improving their critical thinking, writing, and data analysis skills [39]. The availability of the mentorship team with a commitment to providing hands-on intensive mentorship improved learning outcomes. Combined with the first author publications, these are early indications of the impact of the course on trainees’ professional development. Future evaluations need to be more systematic and could assess learning acquisition and behavior change as has been shown in the SORT IT course [28, 30, 36].

Training costs and funding

Few research training programs report their costs. For those that do, the costs vary from US$500 to US$20,000 per research project depending on the competencies covered, length of training, and location of training participants [3, 26]. These costs focus on research project costs, excluding training and mentorship costs. In contrast, our paper describes the cost of delivering an operational research course with details on training, mentorship, and research project costs. These cost estimates show that while intensive hands-on mentorship is crucial for the success of IORT, it is costly. Costs for similar training programs will be program and country-specific. Our estimates provide a rational basis for budgeting and fundraising for other implementers and funders planning for similar trainings.

To fund this course, we leveraged institutional resources and external partnerships. Institutionally, we relied on existing resources of PIH/Rwanda, including training facilities and vehicles for field travels. PIH/Rwanda also funded three of the four courses through operational funds and innovation grants. PIH/Rwanda’s partnership with Harvard Medical School’s (HMS) Global Health Research Core provided in-kind course delivery and mentorship support. HMS funded the time for one senior course mentor for all four courses, another senior course mentor for two courses, and one training coordinator/junior mentor for three courses. We also received a grant through Harvard Global Health Initiative Burke Global Health Fellowship Grant for the 2016 course. Project mentors with their project-affiliated junior mentors also provided in-kind mentorship, estimated at 5% and 10% of their time, respectively. We did not estimate the cost for project mentors given the diversity of these mentors.

Other challenges and limitations

Our reliance on in-kind support for course delivery and mentorship, a key element of the course, affects the sustainability of IORT, especially in a context of declining grant support for international training. This is critical for IORT approaches such as learning-by-doing and intensive hands-on mentorship that are both resource and labor intensive. For example, we fully sponsored trainees, including providing research tools such as analytical software to make the learning-by-doing approach feasible. Our senior mentors experienced a high mentorship workload, which makes senior mentors’ role unsustainable without dedicated funding. We hope that the outcomes of this course encourage funders to invest more in mentorship-oriented capacity-building approaches. We also encourage the government of Rwanda to adopt IORT and dedicate funding for sustaining and scaling the program in Rwanda.

Although a key output for the training is a peer-reviewed publication, publication fees were often prohibitive for the course, as has been observed by others [40]. We prioritized publishing in open access journals that provide publication fee waivers to ensure our training papers were publicly available for all Rwandans and other LMIC users. We also recommend the WHO’s HINARI program, which helped our trainees access articles published in closed-access journals. Similarly, international conference travels were cost-prohibitive. As such, we encouraged trainees to attend local and regional conferences in Africa and to apply for scholarships to attend international conferences.

We only evaluated our course with the 2016 cohort due to challenges with prior planning and recognizing the need for a system of evaluation on a course-by-course basis. Equally, a system for follow-up with the trainees on research engagement post-course, as well as tracking the uptake of research project results on practice and policy, would further demonstrate the value of IORT. This should also measure research capacity outcomes such as changes in publication output post IORT, access to research grants, and growth of trainees and junior mentors to research roles.