Emergency Medicine Training Programs in Low- and Middle-Income Countries: A Systematic Review

Background: Despite the growing interest in the development of emergency care systems and emergency medicine (EM) as a specialty globally, there still exists a significant gap between the need for and the provision of emergency care by specialty trained providers. Many efforts to date to expand the practice of EM have focused on programs developed through partnerships between higher- and lower-resource settings. Objective: To systematically review the literature to evaluate the composition of EM training programs in low- and middle-income countries (LMICs) developed through partnerships. Methods: An electronic search was conducted using four databases for manuscripts on EM training programs – defined as structured education and/or training in the methods, procedures, and techniques of acute or emergency care – developed through partnerships. The search produced 7702 results. Using a priori inclusion and exclusion criteria, 94 manuscripts were included. After scoring these manuscripts, a more in-depth examination of 26 of the high-scoring manuscripts was conducted. Findings: Fifteen highlight programs with a focus on specific EM content (i.e. ultrasound) and 11 cover EM programs with broader scopes. All outline programs with diverse curricula and varied educational and evaluative methods spanning from short courses to full residency programs, and they target learners from medical students and nurses to mid-level providers and physicians. Challenges of EM program development through partnerships include local adaptation of international materials; addressing the local culture(s) of learning, assessment, and practice; evaluation of impact; sustainability; and funding. Conclusions: Overall, this review describes a diverse group of programs that have been or are currently being implemented through partnerships. Additionally, it highlights several areas for program development, including addressing other topic areas within EM beyond trauma and ultrasound and evaluating outcomes beyond the level of the learner. These steps to develop effective programs will further the advancement of EM as a specialty and enhance the development of effective emergency care systems globally.

unique discipline with a well-defined and universal set of technical and cognitive skills -has developed over the last 50 years to lead the provision of clinical care by training not only physicians, but also other healthcare personnel at all levels. Additionally, the specialty of EM also serves as a guide for the overall development of emergency care systems through a focus on core concepts and strategies aimed at reducing morbidity and mortality of emergent conditions and to provide secondary disease prevention [5][6][7][8][9][10].
Although still a relatively young specialty, EM has been expanding rapidly across the globe, and more than 50 national EM organizations are now members of the International Federation for Emergency Medicine (IFEM) [11]. Although some concepts, such as burden of disease and resource availability, are unique to each local context, the specialty of EM provides a unique opportunity where partnerships can develop to share universal content as well as established resources in education and training relevant to high-resource settings, low-resource settings, or both. To date, the development of EM in many countries has occurred in part because of educational partnerships between nations or regions where one partner has more experience with EM as a specialty. Such partnerships can occur regardless of each country's income level, but frequently involve partnerships between a high-income country and a low-or middle-income country (LMIC) [12][13][14][15].
Though programs developed from such educational partnerships are increasingly common, there is little to describe their features. One review describes some of the features of training programs in LMICs, but it does not specifically focus on programs resulting from partnerships [16]. Additionally, there is some guidance in the literature as to how to approach the creation of these training programs as well as the overall development of EM, but this information is now outdated and there is little information specific to training and education [17] In this systematic review, we aim to address this gap by highlighting the scope, content, and features of EM programs developed in LMICs through partnerships.

Methods
An electronic search using PubMed, Web of Science, CINAHL, and EMBASE was conducted for all available publications to the date of the search (30 August 2018) without restriction on the scope of the EM content, the training level of the learners (i.e. medical student, nurse, mid-level, physician), or the length of the program (defined here as short duration -up to one month, medium durationbetween one month and one year, and long durationmore than one year). The search strings may be found in Supplement 1. We have defined an EM training program as "structured education and/or training in the methods, procedures, and techniques of acute or emergency care". Inclusion criteria determined a priori were EM training programs developed in the hospital or clinical setting in an LMIC -as defined by the World Bank at the time of the search -and supported by at least one higher-resource international partnership [18]. In-hospital programs were targeted as, in our experience, partners often seek out institutions given the often more familiar workforce and resources as well as the perception that training is likely to have the highest impact in this setting. Manuscripts focusing only on pre-hospital or non-clinical providers were excluded. Manuscripts with a predominant focus on other specialties (i.e. obstetrics/neonatal care and emergency surgery) were excluded. Only manuscripts in English were considered. Systematic reviews and abstracts/posters were excluded. Two reviewers screened each abstract with conflicts resolved after a third review to determine the included manuscripts.
The included manuscripts were then scored by the reviewers in order to select and more closely examine a high-quality subset. Each manuscript was scored by two reviewers and manuscripts with scores differing by more than three points (n = 13) or that differed between reviewers across the threshold for inclusion (n = 19) were reviewed and scored a third time. Scoring (0-2 points for each category) focused on seven areas: manuscript design, overall quality, overall clarity, ethical considerations, significance, description and/or study of program attributes and/or learner or clinical outcomes, and type of providers (Supplement 2). The methods used for this scoring were adapted from methods established in other reviews in the field of global EM [19]. The reviewers' scores were then averaged for a final score on each manuscript. On review of the average scores, the upper quartile was found to be designated by a score of 12 and above. Manuscripts were then labeled as 'low-scoring' (below 12) or 'high-scoring' (12 or above).
Abstracts were screened using Covidence (Melbourne, Australia). Statistical analyses (Krippendorf's alpha for nominal and for interval data for two or more reviewers) were done using Microsoft Excel and Stata/IC v15.1 (StataCorp; College Station, Texas USA). The study was registered and approved by PROSPERO (Registration: CRD42018100194).

Results
The search produced 7702 results. A total of 2433 duplicates were removed prior to screening, leaving 5269 studies to be screened by two reviewers. From this, 105 manuscripts met inclusion criteria for scoring. Of these, four were found to be in high income countries, six could not be obtained, and one was found to be a duplicate, leaving 94 manuscripts in the final analysis (Figure 1). Programs in 42 countries are represented in these manuscripts (Figure 2).
Of these manuscripts, 30 focus on EM programs with a broad scope, 18 focus on trauma training, 17 on pediatric training, and 16 on ultrasound training. Most programs focus on physician-learners; however, 17 programs train physicians and nurses together, and 15 train multidisciplinary teams (physicians and/or nurses along with other clinical and non-clinical professionals and/or staff). Additionally, eight programs focus solely on nurses and six on medical students. Finally, more than half of the programs are of short duration. Only three are of medium duration, and 15 are of long duration ( Table 1).
The manuscripts were then scored in order to highlight a high-quality subset. When assessing the interrater reliability of the raw scores among the reviewers, the reviewers' scores agreed +/-2 points 74.7% of the time (Krippendorff's alpha 0.732). When assessing the inter-rater reliability to distinguish "low-scoring" from "high-scoring" manuscripts, the reviewers agreed 80.0% of the time (Krippendorff's alpha 0.526). A total 26 manuscripts received an average score of 12 and above.

Broadly-focused EM Training Programs: Program Components
Participants include nurses (two programs), medical students (two programs), physician assistants (one program), physicians (five programs), and nurses and physicians together (one program). All of the manuscripts describe all or a subset of the following components: goals/objectives, program certification/recognition, curriculum outline/ content, methods of assessment, funding, logistics, educational resources, outcomes, and challenges ( Table 4).

Goals and Objectives
Each of the eleven programs conducted formal needs assessments and/or had focus group discussions to identify learning gaps and to determine the content, style, and/or duration of each program, fulfilling the first two steps Kern et al.'s six-step model for curriculum development [46]. These efforts drew upon the strengths and experiences of the instructors while considering the knowledge level of the learners and using familiar teaching styles and/or providing orientation to new instructional methods. One manuscript highlights a pilot study prior to full implementation of a program [35]. Three studies highlight instructor (both international and local) development and orientation prior to commencement of the program [38,41,42].

Certification/Recognition
More than half of the manuscripts discuss alignment with local or national governing bodies, such as the Ministry of Health (MoH) regarding formal recognition, licensing, and/or certification of their respective programs.

Curriculum Outline/Components
The curricula highlighted in these manuscripts span short, foundational programs to medium-length programs with abbreviated, comprehensive EM curricula to full-length residency programs (of at least three years). All offer mixed learning methods ranging from traditional didactic lectures and small group discussions (i.e. journal clubs and case-based sessions) to hands-on practical sessions and workshops, such as simulation and procedural training. Curricula are often organized into blocks or modules by topics or organ systems. Longer programs allocate one to three half-days every one or two weeks for learning sessions in addition to daily bedside clinical teaching.
Volunteer patients, mannequins, laminated algorithms, white boards, and other simulation materials as well as video/online materials, handbooks, and textbooks are highlighted as curriculum tools. Many programs incorporate pediatrics as well. All residency programs list the requirement of off-service rotations, such as surgery, anesthesia, and critical care. Only one manuscript offers extensive online access to their curriculum map and resources [41]. One of the short programs emphasizes the need for refresher courses, particularly when limited curricula are offered and/or baseline knowledge is varied or limited [39]. One manuscript specifically evaluates the acceptability of a novel-to-context learning modality (small group sessions) within the overall program [45].

Methods of Assessment
All programs have a written component of assessment.
These examinations consist of multiple-choice questions, short-answer, and/or essay questions. Many manuscripts highlight programs that also have practical examinations -skills stations, oral and/or case-based examinations as well as objective structured clinical examinations (OSCEs). Longer programs also often track attendance and require case and/or procedure logs with minimum required numbers. Minimum passing requirements for examinations and attendance, when specified, range from 70-80%. There are often end-of rotation evaluations and performance audits, especially for residency programs with off-service rotations. There is usually ongoing informal feedback and mentorship. One program mentions requiring a thesis [44].

Funding
Three of the 11 manuscripts do not discuss how their programs are supported, while the others note a combination of private donors, non-profit organizations, healthcare organizations, and funding from universities in higher-resource settings. Manuscripts that mention their funding sources note a mix of self-funding or subsidized funding for international faculty. Among the larger funding sources listed are the Abbott Fund, USAID, Grand Challenges, and Project HOPE. Salary for learners in residency programs and fees for learners for non-residency programs are less clear -at least one program notes that there are no fees for their learners, and those who had to travel for the program received room and board [36].

Logistics
All of the programs involve international visiting faculty for ongoing full or supervisory support either in person and/or via teleconferencing, messaging, or other methods of remote engagement. Visiting faculty contributions are described as ranging from one week up to multiple years onsite and/or remotely. Longer programs tend to have curricula that repeat one or more times over the course of the program. Train-the-trainer techniques are used in two of the programs [39,43]. Two programs mention their learner-to-instructor ratio for hands-on sessions (a range of 3-6:1) [38,45]. One program is structured to have two identical sessions daily in order to accommodate the learners, who must provide continuous coverage in the clinical setting [35].

Educational Resources
Content for the curricula is drawn from a variety of sources -usually initially from EM content belonging to the high-resource partner. Specific materials and organizations listed among the programs for reference include basic life support (BLS), advanced cardiac life support

Outcomes
Knowledge acquisition, clinical performance, and/or learner confidence/self-assessment are among the outcomes listed for almost all of the programs. All programs conduct immediate assessments; however, many of the programs also assess knowledge retention at a variety of time points: eight weeks, three months, six months, and one year. Additionally, some programs highlight the number of graduates as well as the number of individuals still practicing in EM. None of the program outcomes extend beyond the learner, although future studies of clinical outcomes are mentioned in one manuscript [43].

Challenges
Challenges common to many of the programs include establishing the status and culture of practice of EM in low-resource, novel settings. Additionally, the manuscripts describe how differences in scope, practice, and content between the high-resource and low-resource settings as well as among program sites within low-resource settings require translation and adaptation of any shared educational resources and techniques. Overall resource limitations for learning as well as practice materials are frequently noted. Language barriers (for both instruction and materials) as well as issues with remote connection and use of local and remote technologies are also highlighted. Additionally, lack of continuity and connection between learners and the rotating visiting faculty -who may be present in person for as little as one week -leads to reduced engagement and morale. Decay in knowledge, especially after short programs, is also a concern, with re-assessment between 2-3 months highlighted as potentially an ideal time for re-assessment or refresher training. Difficulties with both learner (e.g. attendance given the requirement of ongoing continuous clinical duties still required during many of the programs) and curriculum flexibility leads to attrition for many programs. There are also ongoing concerns regarding sustainable funding highlighted in several of the manuscripts -for the program, for visiting faculty, as well as for learners/graduates.

Discussion
These manuscripts describe many types of EM programs being implemented via partnerships worldwide to train in-hospital personnel critical for the overall development of emergency care systems -from short, focused courses on topics such as ultrasound training or trauma training to full residency programs. The most established programs are those that are of at least one year in duration and those with formal integration into the healthcare and/or medical education system of the country (e.g. supported by the MoH). These programs can then offer official certification and/or credentialing to ensure that graduates will have both potential positions and financial security as well as a career path as EM develops and the culture of practice changes.
All of the programs involve on-site and/or remote international faculty for varying lengths of time; however, programs in which at least a core subset of faculty are involved in face-to-face teaching through the duration of the program foster stronger partnerships and allow for continuity of evaluation as well as continuous improvement and adaptation of the program. Given the differences in resources and culture of practice, all visiting faculty need at least prior experience in education/training and low-resource environments and/or a comprehensive orientation to the site prior to providing instruction. One manuscript describes the challenges of conflicting expectations between instructors and learners due to the limited preparation and experience by the visiting faculty [38].
Furthermore, the use of international faculty is rarely sustainable, especially when support for these faculty is not available. Programs that involve a train-the-trainer component as well as faculty development are essential, although limitations of these models (especially dilution of instruction) must be kept in mind and addressed using other mechanisms such as refresher courses and mentorship or preceptor programs. Additionally, given the increasing familiarity with and use of technology, non-traditional training mechanisms such as the use of messaging applications and other mobile technology-based training tools should be considered, especially as they can be effective mechanisms to extend training beyond face-to-face interactions and/or to continue training during protracted and/or complex emergencies [47][48][49].
These programs also highlight the inability to simply transfer EM curricula from one setting to another. Some fundamental concepts can be shared, as is evidenced by international guidelines for curricula, and there are several groups that have offered open-access materials that are already adapted to low-resource settings to varying degrees ( Table 5). In order to be effective, however, the development of curricula -whether abbreviated or a full residency program -must be preceded by needs assessments, focus-group discussions, and knowledge of the epidemiology and burden of disease as well as the culture of practice, instruction, learning, and assessment of the implementation site [50][51][52]. Additionally, language barriers must be overcome both in instruction as well as in written and electronic materials [53]. Finally, there must be inter-sectoral engagement and collaboration between the health and education sectors in order to recruit, develop, and sustain a quality workforce that is matched to overall service delivery capacity and population needs [54].
Finally, in order to continue to advance EM training and the overall provision of emergency care, programs must include training in quality improvement (QI) and research, topics not traditionally taught in many medical and nursing schools in low-resource settings. To illustrate this point, less than half of the programs mention a component of QI and/or research, and only two of the manuscripts have a local first author [31,44]. Furthermore, in order to determine which methods of training are most effective, we must move beyond metrics regarding knowledge acquisition and attitude and eventually look to patient-, community-, and population-based outcomes [55]. Only three of the manuscripts evaluate patient and/or clinical outcomes as a result of their training programs [20,26,31].
Despite ongoing work to develop metrics and quality indicators for emergency care and research and to provide guidance in the development of emergency care systems (as evidenced by the WHO's Emergency, Trauma and Acute Care Program), there is still very little research that has focused specifically on the impact of training programs [9,[56][57][58][59][60][61][62][63]. These programs are an integral part of the foundation of a sustainable emergency care system, not only at the facility level, but indirectly within the community and the pre-hospital system as well, therefore, further investigation of and research on these programs is imperative to sustain and to continue to expand EM as a specialty and emergency care systems globally.

Limitations
Overall, we used only four search engines and excluded programs in the gray literature without peer review (i.e. abstracts) -an area in which many more programs are likely highlighted based on an informal review of the excluded abstracts. Also, our results are limited to manuscripts only in English. Additionally, the search terms used to define this rapidly changing and developing field were difficult to specify, leading to one additional manuscript that was identified on informal review that could have been included in this study [64]. Furthermore, partnerships are also not always made explicit in many manuscripts; since our study focused on programs developed through educational partnerships, we may have missed manuscripts in the screening process where this was not clearly defined [65]. This challenge further emphasizes the need for standardization of practice and sharing of knowledge and experience in this field. Finally, our scoring process was developed from tools used in other reviews and was not formally tested with our results; however, it has been used to evaluate literature in field of global EM, and post hoc review demonstrated that the conclusions drawn from the high-scoring manuscripts were representative of the larger group [19]. Despite these limitations, however, this study still identifies a broad range of EM training programs, and it highlights programmatic components essential for effective partnerships and successful implementation.

Conclusions
Training in EM is an essential component to developing strong emergency care systems. This study identifies a broad range of EM training programs developed through partnerships that are representative of the programs that exist globally and highlight programmatic components essential to the development of effective programs. Sharing knowledge and practice across high-and low-resource settings can be an effective mechanism for the global development of programs within a specialty, however significant adaptation to the educational setting as well as true investment and partnership must occur, even for short-term programs. Even when all of the complexities and components are addressed, there are still limitations to this practice. Short-term programs need to expand beyond the few topics that are represented now, and we must invest in comprehensive, long-term emergency care training and professional development programs. Finally, as tools and guidance regarding the development of emergency care systems increase, a strong focus on strengthening educational practices as well as evaluating their impact through research is imperative.

Additional Files
The additional files for this article can be found as follows: •