Shun Mabuchi
Why the Study?
Following an exploratory case study in Ondo and Nasarawa states Shun Mabuchi carried out with consultants that described in detail the difference between high performing PHCs and low performing PHCs particularly in management practices at the PHC. Building on the research findings, NSHIP TTL and the NSHIP team plan to carry out a quantitative analysis to understand what specific management practices at PBF PHCs associate with better performance. The Bank through a grant from GAVI contracted Hanovia Medical Limited, a healthcare management consulting firm, to carry out the quantitative data collection to measure management practices at sampled PHCs under PBF in the Adamawa, Nasarawa and Ondo states.
The consulting firm plans to visit the selected PHCs for the study in the three states between 30th March and 27th April, 2016.
The team has arrived Yola today, 29th March 2016, ready to commence the business of the data collection from 54 PBF contracted health facilities across 10 PBF Local Government Areas (Excluding Madagali Local Government due to insecurity).
Terms of Reference
Background
1. Despite rapid economic growth, Nigeria’s health services performance including immunization has been lagging behind. Despite improvements, under-five mortality rate in 2013 remains high at 128 per 1,000 live births, well above the MDG 4 target of 73.9 (Nigerian Demographic and Health Survey: NDHS). DPT3 vaccination coverage and skilled birth attendance are both 38% in 2013, without much progress from 1990 with both 32% coverage (NDHS). Contraceptive prevalence still has been around 10% (NDHS 2013). This is compounded by the severe regional disparity. Northern region is lagged behind South in all indicators, e.g., with under-five mortality rate at 185 and 160 in North West and North East in 2013 respectively. DPT3 vaccination coverage is 62.2% in urban area vs. only 24.9% in rural area in 2013 (NDHS).
2. Critical issues in Nigeria’s health systems include accountability and motivation for results, autonomy of health facilities, and involvement of community, rather than lack of health workers, infrastructure, and drugs. Nigeria has relatively abundant human resources and infrastructure. Health worker per capita is, though poorly distributed, twice the Sub-Saharan Africa (SSA) average. In Nigeria, 67% of population lives within 30 min walk of health center, and 85% within an hour, which is better than most countries in the SSA. There is shortage of drugs, but Service Delivery Indicator (SDI) survey does not show correlation of drug shortage with case load, which suggests that other issues would be bottleneck. Primary health care centers (PHCs) typically have no cash to improve services and outreach to community, and lack accountability and motivation to improve health services. Comprehensive solutions with clear attention to results and accountability are necessary to address these systemic issues at PHCs.
3. Nigeria State Health Investment Project (NSHIP) has been implemented to address these critical health system issues and improve health services performance. NSHIP introduced performance based financing (PBF) at health facilities. PBF provides operational cash for health facilities to improve their services and allocate a part of it (up to 50%) among health workers as performance bonus based on clearly defined quantity and quality indicators (e.g., bonus of US$6.3 for one completely vaccinated child for a facility). This aims to clarify goals and accountability, and enable and motivate health workers to achieve the goals. After 1.5 year pre-pilot in about 30 health facilities, PBF was scaled-up to half of the entire states in Adamawa, Nasarawa, and Ondo states in 2015.
4. The project is showing promising results, but there are large variations between high and low performing PHCs. Service coverage of targeted PHCs improved significantly in all three states. Coverage of completely vaccinated child improved from less than 20% in the target pre-pilot Local Government Areas (LGAs) in the three states to average over 40%. Same level of improvement was observed during the 1.5 years of pre-pilot activities in institutional delivery, new outpatient visits, and total users of modern family planning methods. However, the variation of performance among PHCs, thought it started from very small with everyone having few patients, had increased dramatically over time. Figure 1 shows the large variation in total performance bonus to PHCs as a proxy of consolidated performance. This large variation was also observed in the PHCs that recently introduced PBF.
5. Qualitative case study identified importance of good management at PHCs. In order to understand the key differentiating factors between high and low performing PHCs, the NSHIP task team carried out a qualitative case study in the eight high and low performing PHCs. It found significant differences between high and low-performers in health center management and community support to PHCs. On health center management, staff in high-performing PHCs are fully aware of the target and actual quantities of key indicators for each month, update their monthly target, compare actual performance with targets and previous months, investigate the reasons for performance drop if any, address problems by involving community leaders, and follow up on the results. In contrast, most low performing PHCs just continued to use the targets provided by supervisors at the beginning of the pre-pilot, and none of the staff could explain the target numbers. Only one of the four PHCs explained a specific action to address stagnant performance. High-performers proactively make efforts to motivate staff through multiple approaches, while such efforts were observed in only one low-performer.
6. However, it will be important to understand what management approaches are important for performance, and how they can be strengthened. Following the exploratory qualitative case study and based on the finding that good management is critical, it is important to quantitatively understand what specific management practices are associated with better performance. Also, based on the findings the project needs to define effective management strengthening approaches to improve PHC performance including immunization performance. In this context, as an effort to measure the level of management practices, a management practices scorecard has been developed based on management literature, relevant scorecard (mainly for hospitals in developed countries), and findings from the qualitative cases study.
Objective
This consultancy aims to apply the management practices scorecard to randomly selected PHCs under PBF and provide management practices scores and supporting quantitative and qualitative information. The data will assist the analysis of relationship between management practices scores and performance and support the project team to identify effective management interventions to prioritize.
Activities
(1) Preparation
Initial Preparation
- Review available documents provided from the Bank team on management practices and performance including: (i) a research proposal on management practices scorecard and performance; (ii) draft management practice scorecard; (iii) draft short protocol for management practices scorecard data collection; and (iv) draft report on a ‘Qualitative Study on Key Differentiating Factors for Performance Under Performance Based Financing (PBF)”;
- Discuss with the Principal Investigator (PI) of the research (Shunsuke Mabuchi, Senior Health Specialist, World Bank) categories, interview questions/questionnaire, and scoring criteria of the draft management practices scorecard and propose improvement as needed.
- Develop data collection tools (e.g., questionnaire with breakdown of the scorecard, tablet-based data input and management system) to facilitate the data collection and improve objectivity of scoring;
- Assist the Institutional Review Board (IRB) approval in Nigeria (initial proposal will be developed by the Bank team) including the follow-up with the IRB coordinator;
- Develop and agree with the World Bank task team and National Primary Health Care Development Agency (NPHCDA) and State Primary Health Care Development Agency (SPHCDA) teams on a data collection work plan and timeline and reporting format;
Pre-testing
- Pre-test the scorecard in 3-5 PHCs under PBF that will not be selected for the scorecard assessment to adjust questions and criteria in the scorecard and protocols if needed;
- Report key findings from the pre-testing and improve and finalize the scorecard and data collection tools;
Target selection and training
- Randomly select 110 PHCs based on stratified sampling proportionate to total number of PHCs under PBF for each state (tentative: 54PHCs in Adamawa, 21 in Nasarawa, and 36 in Ondo) to carry out data collection;
- Mobilize teams of data collectors (2 persons per team) who are capable of scoring the management practices of the target PHCs based on pre-defined criteria, an train them on the interviewing, scoring of the scorecard, and recording of data;
(2) Data collection
- Apply management practices scorecard in the 110 PHCs in the three states selected through stratified random sampling based on agreed protocol using agreed and developed tools;
- Take notes responses to selected open-ended questions in the scorecard for the Bank team to be able to understand and validate the reasons for the scores.;
(3) Data clean-up and organization
- Organize the collected management scores in a format for a statistic software (STATA) to analyze results for correlation between management scores and performance, and for validation.
- Organize the qualitative responses/ notes for each open-ended question for each PHC for the Bank team to be able to verify the scores and learn from the responses.
Expected Deliverables
- Data collection work plan;
- STATA-friendly dataset of management practices scores for quantitative analysis. Data should include the following (Sample reporting format is attached in the Annex):
- Management practices scores for each PHC and its detailed breakdown (i.e., score for each item) – score for each rater (of inter-rater reliability analysis), and final score agreed by the two raters for reliability analysis;;
- Supplementary set of qualitative data for selected open-ended questions in the scorecard;
- Note on responses by interviewees and selection of criteria for scoring that a PHC addressed for open-ended questions;
Timeline
- March 15 until June 30, 2016
- Preparation: March-April
- Data collection – March-April
- Data clean up and organization - April
- Any follow up period – April - June