Study Setting and Context

ABC was implemented between July 2013 and September 2015 in two districts, Kirehe and Southern Kayonza (S. Kayonza), located in Eastern Province of Rwanda. The two districts have been partnered with PIH/IMB and the Rwanda MOH to support health systems since 2005 in all 24 health centers (16 in Kirehe and 8 in S. Kayonza) and 2 district hospitals (1 in each district). The two districts serve a population of more than a half million. In Kirehe, three health centers (Kigarama, Rwantonde, and Mahama) were opened after the launch of ABC in October 2013. In S. Kayonza, one health center (HC) (Rwinkwavu HC) does not provide maternity services so was excluded from some indicators, such as number of pregnant women with antenatal care visits and number of deliveries. During ABC implementation, efforts were made to integrate selected ABC activities, including neonatal care mentorship and peer-to-peer learning, into existing district routine activities. No additional equipment, supply, or training related to neonatal care were given to facilities after the 12-month completion of ABC by PIH/IMB, and all health facilities in the two districts continued to be managed by the MOH.

Study Design

We used a mixed-methods approach with convergent sequential design to study the sustainability of improvements and ABC activities 12 months after the completion of the ABC program (September 2015) and factors related to the success and challenges of sustainability from the perspective of key stakeholders including health providers, mentors, and local leaders. This included a quantitative evaluation using a pre-post design to capture changes in key processes, coverage, and neonatal outcomes from the end of ABC to 12 months post-intervention. We also conducted focus group discussions and in-depth interviews with health workers at the facilities, ABC team members, and local leaders to understand which activities sustained and which ones dropped. We captured their experiences and opinions on neonatal practices, how they changed over time after the ABC implementation phase, and factors they thought were important to increase or threaten sustainability of improvements and best practices.

Data Collection and Sampling

Quantitative: Indicators used for measurement were the same as those used during the ABC program and for the primary evaluation [7]. These were selected based on the globally accepted process and impact outcomes for maternal and newborn health and were aligned with Rwandan priorities and context. Three time periods (three months each) were considered in our analysis: 1) Baseline (pre-ABC), July to September 2013; 2) ABC Endpoint July to September 2015 (9–12 months of ABC interventions); and 3) 9–12 months post-intervention end (July to September 2016).

Data were extracted from the existing national and district data management systems facility (Health Management Information System for Facility Data (HMIS)) and community health worker (CHW) (Système D’Information Sanitaire Communautaire (SIS-Com)). We used patient charts from facilities and records from CHWs to validate data from HMIS and SIS-Com respectively. In case of differences between the two data sources, priority was given to facility and community health records as the primary source. To complement routinely collected facility data, ABC program monitoring data were used for indicators not reported in HMIS or SIS-Com. Facility surveys conducted during the ABC program were also repeated to measure availability of essential equipment and medications.

Qualitative: Four qualitative focus group discussions (FGD) with eight participants each were conducted in Kirehe and S. Kayonza with nurses and midwives providing maternal or neonatal care at health centers and nurses providing maternal or neonatal care at hospitals. Participants were purposely selected based on their experience and active participation in the neonatal and maternity services (at least two years). A facilitator and note-taker were hired and trained to lead the discussions.

Semi-structured individual interviews were conducted with 20 participants, including 2 ABC mentors, 4 MOH mentors, 1 program director, 6 nurses (3 from each district including 2 from health centers and 1 from hospitals), 4 directors of health centers, 1 district hospital director, 1 political leader, and 1 data manager. Recorded interviews occurred at a place that ensured privacy. For both focus group discussions and interviews, questions focused on the following topics: (1) experiences with ABC program; (2) challenges faced during implementation; (3) views of impact on patients; and (4) experiences during the 12 months after implementation. Focus group discussions and interviews were conducted in the local language of Kinyarwanda, were audio-recorded with permission, and were transcribed, then translated into English for data analysis.

Analysis

Quantitative analysis: Monthly data were aggregated for each of the three time periods and analyzed across both districts and individually. Point estimates were calculated for each indicator at each time period, and change that occurred over time was analyzed. Wilcoxon signed rank test was used to test the significance of differences for continuous variables where possible. Improvement was measured from baseline to ABC endpoint, and sustainability was measured from ABC endpoint to 12 months post-ABC completion. Excel and Stata v.14 (College Station, TX: StataCorp LP) were used respectively for data management and analysis. A p-value <0.05 was considered significant.

Qualitative analysis was inductive and employed a content analytic approach [9]. A subset of transcripts was open coded for the purposes of developing a codebook; the resultant codebook was then used to direct code all transcripts. Coded data were examined inductively to derive emergent themes that were developed into conceptual categories that describe either key barriers or facilitators of ABC implementation and sustainability. Each category consisted of a descriptive label, an operational definition, and key illustrative quotes. These initial categories were reviewed and revised to create a set of six final themes that appear in the Results section below.

After independently analyzing the quantitative and qualitative data, content areas represented in both data sets were identified, and all the results were compared, contrasted, and synthesized. The separate results were then interpreted. The themes from focus group discussions and individual interviews were assessed in terms of their ability to explain quantitative results.

Ethics/Institutional Review

All participants for the qualitative assessment provided written consent. The research was reviewed and approved by the Institutional Review Board at Harvard Medical School (HMS) (Ref No IRB16-0584) and the Rwanda National Ethics Committee (Ref. No. 895/RNEC/2016) the National Health Research Committee, a review board of the Rwanda Biomedical Center and both Kirehe and Rwinkwavu District Hospital management teams.

Changes in post-ABC activities and performance

Twelve months following the completion of ABC, Kirehe and Kayonza District health leadership sustained selected key ABC activities although at lower intensity (Table 3). Mentorship was sustained but with a decrease in frequency, from an average of 1 mentoring visit per month in S. Kayonza at the end of ABC to 0.8 visit per month—with a similar drop in Kirehe (0.8 visit per month to 0.4). Peer-to-peer learning through learning sessions as well as regular review of key neonatal data were incorporated into routine monthly coordination meetings. Despite this decrease, the improvements seen in most key quality and coverage indicators during ABC were sustained and/or increased 12 months after the completion of ABC (see Table 2).

The availability of essential medicine and functioning equipment to deliver newborn care remained high in both districts despite no further direct supplies from PIH/IMB. In Kirehe, the availability of essential medications, which had increased during ABC (from 67% to 78%, p = 0.04) kept increasing significantly 12 months after the end of ABC (83%, p = 0.003) (Table 1). The percent of women giving birth at a health facility, already high in both districts at the end of ABC (92.6%), kept increasing significantly 12 months after the end of ABC (95.8%, p = 0.01). Similarly, the percentage of pregnant women who completed four antenatal care (ANC) visits remained stable across the two districts. Aggregated across both districts, improvements were seen in labor management and newborn care after birth as we compare the end of ABC and 12 months later: steroid administration for preterm labor (41.7% to 58%, p = 0.01), systematic monitoring of danger signs for newborn 24 hours after birth (98.7% to 99.3%, p = 0.7). The percentage of babies with immediate skin-to-skin after delivery, already high at the end of ABC (97.4%), reduced slightly (96.2%, p = 0.2).

Similarly, neonatal mortality dropped from 30.1 deaths per 1,000 live births at the beginning of ABC to 19.4 deaths per 1,000 live births. The decrease in mortality was maintained 12 months post-ABC at 19.4 deaths per 1,000 live births (p = 0.7). Building capacity and ownership for QI was also a goal of ABC. At the end of ABC, there were 47 active QI projects towards improving maternal and neonatal care across both districts. While there was a decrease in number, 12 months post-ABC, QI projects were still being implemented. All facilities (100%) in S. Kayonza and 60% in Kirehe were implementing at least one QI project per health center. In addition, a number of key improvements initiated by health center teams during the ABC period continued 12 months later (Table 1).

Qualitative Findings

Six major themes linked to the success or challenges of sustainability of ABC and related improvements were inductively identified. Quotes reflecting each theme are summarized in Table 4.

Leadership “buy-in” and “ownership” (Facilitator): This emerged as one of the powerful factors to facilitate various activities aimed at improving the quality of neonatal care. Participants not only defined leadership as those occupying top positions of political or administrative authority, but also extended the concept of leadership to include any person across the continuum of care whose engagement and enthusiasm pushed forward the neonatal care agenda. Participants noted the presence and active participation of district authorities during learning collaborative sessions and coordination meetings after ABC period.

Young leadership (Facilitator and Challenge): Interviewees described how national, district, and local leaders were ambitious, optimistic, confident, and committed, which was both a facilitator and barrier. Leaders were described as committed and ready to do whatever it takes to reduce neonatal mortality. However, most were young and often lacked managerial and technical experience. This combination of ambition and lack of managerial expertise meant that they were easily pulled in many directions, and participants explained that this led to a loss of focus on neonatal health, which affected the momentum after the ABC period.

Turnover of trained staff (Challenge): The mobility of staff was identified as an important challenge to sustainable improvements. Health care workers who chose to leave did so for multiple reasons. During the implementation phase of ABC, selected nurses and midwives at the hospital and health centers were trained in maternal and neonatal care services. As they became skilled and experienced, they became more attractive to other health facilities, particularly those in urban areas. The departure of the trained staff for other health facilities left their previous facilities with critical staff shortages, which negatively impacted services.

Stuff happens! (Challenge): Both unplanned adverse events and unexpected great opportunities have an impact on neonatal outcomes. Participants described unexpected events that impacted neonatal outcomes both over the course of ABC implementation and during the one year post-intervention period. These events were not anticipated during the design of the ABC program and could not have been predicted, so stakeholders were not always prepared to respond to them. The events included a prolonged drought causing famine in Eastern province, the influx of Burundian refugees in Kirehe District, unintended consequences of new policies from district or central level, such as introduction of an antenatal care fee, which decreased the number of women attending antenatal care services.

Development of self-confidence fostered internally generated solutions (Facilitator): Participants noted the emergence and importance of locally made solutions to improve antenatal care services, delivery management, and post-natal care. Innovative, individualized ideas that could be integrated into ABC practices at each site were generated by health centers and hospital teams with the support of ABC mentors. These internally generated solutions started during the ABC implementation period and continued in the year that followed the end of the program. They took different forms depending on the health facility. These included 1) partnering with community health workers (CHWs) and local leaders to identify pregnant women and encourage them to attend ANC visits or 2) improving internal communication across different services within the health center, 3) motivating women to attend health centers for antenatal care and deliveries by providing non-financial incentives, such as clothes, to the pregnant women who attended the center during the first ANC visit during their first trimester.

Gap between high demand in maternal and neonatal services and adequate human resource (Challenge): Interviewees explained that staff shortages and work overload made it difficult for staff to meet expected goals for newborn care services, a barrier to having achieved and sustained improvements. Community demand steadily increased, and a growing number of women began seeking care at the health centers or hospitals. Almost all pregnant women came to deliver at the health center or hospital. Unfortunately, human resources did not increase to match this demand, which negatively impacted the quality of services provided to pregnant women.