Evaluation Purpose
To evaluate the impact of the CBM ophthalmology training sponsorship program from 2000-2015 on eye health and the effect on strengthening of specialised human resource (Ophalmologists) in the East African region and to inform future eye health program interventions.
Evaluation Type
(e.g. mid term, end of phase)
Impact assessment
Commissioning organisation/contact person
CBM AFE Regional Office
Evaluation Team members (if known)
Primary Methodology
Interviews
Questionnaires
Case studies
Proposed Evaluation Start and End Dates
Start date: 23rd October 2017
End date: 24th November 2017
Anticipated Evaluation Report Release Date
01st December 2017
2. Background of Project
CBM works in line with the overall objectives of ‘VISION 2020: The Right to Sight’, to eliminate the main causes of avoidable blindness by the year 2020 in order to give all people in the world the right to sight. One of the core VISION 2020 programme strategies is Human Resource Development.
The CBM scholarship Programme for Postgraduate Studies in Ophthalmology (P2258) is implemented by the College of Ophthalmology of Eastern Central and Southern Africa (COECSA) who also manages other sponsorships provided by other donors. COECSA is a virtual college that aims to address the shortage of ophthalmologists in Eastern, Central and Southern Africa as well as improve the quality of eye care services in the region. The college was registered in 2012 after a phased merger between Eastern Africa College of Ophthalmologists (EACO) and Ophthalmological Society of Eastern Africa (OSEA). COECSA is in charge of the day to day management of the program as described in the memorandum signed in 2012.
The scholarship Programme is instrumental in the training of Ophthalmologists from all over Africa. So far CBM through its donors have sponsored over seventy students. On completion of their studies, the Ophthalmologists are placed, as per the signed bonding agreement, at VISON 2020 relevant units within the continent. The scholarship is currently offered at University of Nairobi (Kenya), Mbarara University of Science and Technology (Uganda) and KCMC/Tumaini University in Tanzania.
The scholarship programme is funded by CBM Germany with the goal to train high quality ophthalmologists for comprehensive and relevant eye care services. The programme has also collaborated with the German Lions.
The strategies include:
• Provide sponsorship to selected students from across the continent to specialize in ophthalmology; these students are eligibile for the program only if they obtain admission to a training institution; the students are also vetted by CBM for eligibility of sponsorship to ensure that the relevance of the investment will yield the desired benefits.
• Ensure / promote posting of graduates to areas of priority need for quality eye care services, promoting the sustainable development of effective and efficient patient centered service delivery.
• Through this sponsorship programme, the funding has also contributed to the strengthening of COECSA’s coordination- and overall capacity. COECSA is now seen by many as a “model” of local capacity development.
2. Evaluation Objective, Scope and Intended use
The overall objective is to evaluate the impact of the CBM ophthalmology training sponsorship program from 2000-2015 on eye health and the effect on strengthening of specialised human resource (Ophalmologists) within various countries in Africa and to inform future eye health program interventions. The evaluation should also seek to inform the future of the program beyond year 2015.
Specific objectives (use both qualitative and quantitative data)
1. Document the profile of the CBM sponsorship program trained ophthalmologists
a. To describe the general profile of the trained ophthalmologists ( year since training, type of work – e.g. academic, clinical, private practise, in leadership positions at institutional or national program (MoH; policy influencing) levels or working with other NGDOs), and further training since graduation
b. To map the location of the trained ophthalmologists – where are they? Track all posting, positions and locations to assess ‘career development’ patterns. This may include e.g. (and often sequentially) an initial secondment as junior staff (further practicing / learning under supervision; focus on cataract services (at base unit and through outreach) and general ophthalmology; training of other eye health cadres, students and ophthalmologist (at university or through elective term rotations to the unit where he/she is based); subspecialty training and / or other Professional Development activities; leadership role at institution and / or MoH.
c. To determine the proportion of ophthalmologists working for the institution and/or country from which they got the opportunity of training.
d. To describe migration patterns of the CBM trained ophthalmologists
e. To investigate factors why some CBM trained ophthalmologists are working/ not working in the CBM core areas of development
f. To document work outputs of the CBM trained ophthalmologists- numbers of patients/year, numbers of sight restoration surgeries/year, numbers of outreaches/year
g. Analyse the impact CBM support to the trainee has had in terms of books, basic equipment and other resources.
The consultant should ensure that this is detailed to include qualitative answers and numbers as this would provide important information reflecting return on investment.
2. Evaluate the impact the CBM sponsorship program has had on the capacity development of the training institutions
a. To describe how the CBM sponsorship program has supported institutions to build capacity of their training
b. To document direct and indirect support CBM has given to the training institutions: equipment, number of faculty, faculty training,
The key principle categories for evaluating the capacity development achieved would be:
a) the trained Ophthalmologist: outputs (see categories mentioned under 1b), professional development, policy influencing powers etc.
b) the training institution
c) the institution where he/she is / was placed
d) the national program overall (incl MOH)
e) COECSA
3. Evaluate the impact the CBM sponsorship program has had on building capacity of the national blindness prevention programs
a. To analyse and describe how the CBM sponsorship program has supported National programs to deploy and retain ophthalmologists in priority areas.
b. To document the number of ophthalmologists in the respective countries and determine how many of those are CBM sponsored ophthalmologists
c. To document the blindness indicators in the country- cataract surgical rate, cataract surgical coverage, prevalence of blindness
d. What has been CBM’s contribution based on (..C) above
4. Evaluate and describe the terms of the CBM sponsorship program to inform the future program design
a. To evaluate the different components of the CBM sponsorship program- selection process, bonding agreement, current narrative reporting template, cost plan, administration, appointment of a CBM local contact, award review committee, award timelines.
b. To collect experiences of previous and current CBM sponsored students on overall satisfaction of the different components of the program including acceptability of the training by respective countries and suggestions on how this might be improved. Further analyse if the final degree achived by the student is acknowledged in the country of residence particularly if the students start working as ophthalmologists or if they had to be the first years under supervision based on the different requirements by different MOHs in various countries.
c. To collect experiences of CBM grant administrators and institutional heads on managing the sponsorship and suggestions for future improvement.
d. Analyse and report on the evaluation process, bonding agreement (how feasible?) and the narrative reporting of progress
Scope of the evaluation
Through the impact assessment of the sponsorship program, it is expected that the new phase will match objectives of the program, selection criteria and outcomes; career and migration patterns of the trained ophthalmologist and their outputs (this includes post-graduate students being assessed on the impact of them training others and the impact in institutional capacity development – eye clinic, training institutions, National Program Managers, COECSA etc); recommendations for the setup of the new program; assessment of the alignment of the Program with the government policies; document narrative case studies of impact in the communities where they work; and the evaluation of the program by training institutions.
The information from this study will help generate evidence of the impact the CBM sponsorship program has had in addressing blindness related disability in the region. This will form a strong advocacy case for more support and opportunities to scale this to other regions and other eye care related areas such as optometry, ophthalmic clinical officer and ophthalmic nursing. The lessons learnt will be useful for informing the future design of this program.
Target Population of the Evaluation:
The target audience for the sponsorship program are the over 70 students trained in ophthalmology through the training institutions Africa, beneficiary hospitals and where applicable the community benefiting from the services, National Program Managers, Heads of departments/institutions, COECSA officials and CBM funders.
5. Analysis and Evaluation Questions
1. Relevance and quality of design
• To what extent are the objectives of the project still valid?
• Are the activities and outputs of the project consistent with the overall goal and the attainment of its objectives?
• Are the activities and outputs of the project consistent with the intended impacts and effects?
2. Effectiveness
• To what extent were the objectives achieved / are likely to be achieved?
• What were the major factors influencing the achievement or non-achievement of the objectives?
• Have assumptions affected the project; has there been a risk-management in place from the beginning and how effective has this been?
• Assess the quality of the project planning documentation (situation and stakeholder assessment, time frame and milestones, personnel planning).
• Working relationships and communication between stakeholders.
**
1. Efficiency (of Planning and of Implementation)**
• How much did aspects of the project cost to implement? Was this justified? Were activities cost efficient?
• Were funds spent so far appropriate and objectives achieved on time?
• Was the project or programme implemented in the most efficient way compared to alternatives?
• In how far was the strategy of the project approach appropriate?
• Are existing learning processes such as reflection, internal review used sufficiently to keep project plan updated?
4. Impact – Contribution to change
• What changes have been brought about as a result of the scholarship programme in achieving respective national eye health strategic plans?
• What difference has the scholarship programme made to the beneficiaries and individual trainees?
• How many people have benefited? It is important to use disaggregated data for gender, age, type of disability and others as applicable.
• What are the positive changes that had not been anticipated or expected of the scholarship programme?
• Were there also negative effects and consequences of the scholarship programme?
• Is there any plan to use the model of this project or shared best practices of other donors to replicate in other regions or institutions?
6. Methodology
Overall the consultant will use for the assessment the following approaches to the scholarship programme evaluation:
• Brief and debrief from the Regional Office and Member Association;
• Interview COECSA Board and Management Team Members;
• Interview key staff (COECSA, CBM) involved in the management of the scholarship programme
• Interview key external partners such as training institutions, hospitals where the students were deployed, relevant health ministries’ departments, eye health INGOs, communities
• Visit key locations for discussions with doctors, beneficiaries (as applicable).
Overall a mixed methodology will be employed:
The Consultant should obtain a preliminary understanding of the engagement context prior to start by reading the following documents:
• The Partner Agreement signed by CBM East Africa and COECSA
• Standard MOU Agreement
• COECSA Project narrative reports for the programme
• Students regular and Annual Reports for the period under review
• Multi-Year-Plan (MYP) document for the period under evaluation
Quantitative methods
This will be cross sectional survey targeting all CBM sponsorship program ophthalmologists, survey tools will include online surveys, phone administered questionnaire, statistics & narrative reports and self-administered questionnaire depending on the response rate of the participants.
Qualitative methods
Targetting a sample representation of the sponsorship program beneficiary ophthalmologists and department heads of training institutions, heads of eye units in respective health ministry/national eye health coordinators and grant administrators. To conduct in depth key informant interviews to get an understanding on their experiences of the program, successes, challenges and how it can be improved in the future.
Case studies
Target sample of the beneficiary ophthalmologists in the different countries to document personal stories on how the CBM sponsorship program has benefitted them as individuals and how they are using their training to make a difference.
Collect stories from few key representatives of the community on how they have benefited from the services of the CBM sponsored ophthalmologists in the community.
Potentially this information will also be for publications.
7. Limitations
There is no log frame with set indicators to form basis of measuring the project performance, so the evaluation will use generic questions. No data base on the current placement of the beneficiary students – assumption is that they are health in health facilities providing the much needed services.
8. Evaluation Team and Management Responsibilities
Commissioning responsibility
This evaluation has been commissioned by CBM Germany and the CBM Africa East Regional Office, Nairobi.
9. Management of the evaluation and Logistics**
Role of AFE-RO
• To coordinate the process of recruiting a consultant for the evaluation of the scholarship programme and the contracting process.
• Will take the lead role in the entire process of the evaluation
• Ensure the provision of all relevant information available in it’s custody to the consultant that will aid the evaluation process
• Facilitating the payment(s) to the consultant in line with the contractual agreement
• Undertake all the logistics of the exercise including but not limited to flight bookings, in country travels etc.
• Liaison with COECSA in scheduling of interviews
Role of CBM-D
• Funding of the evaluation
• Review of the terms of reference for the evaluation
• Ensure that the evaluation develiverables are in line with the donor requirements and there is an action plan to consider the outcome of the evaluation
Role of COECSA
• Provide contacts of key informants and resource persons including scheduling of interviews
• Provision of information related to eye care health that is appropriate to the evaluation
• Distribution of the TORs to prospective consultants within their network
• Provide to the consultant relevant information about the scholarship programme, that the consultant may need to carry out the evaluation.
Role of Eye Health Advisors
• Technical input into the Terms of Reference and review the findings of the evaluation before the report is finalized
10. Expected Results
The consultant shall be expected to deliver an evaluation report of the scholarship programme, capacity development action plan and recommendations for the scholarship programme.
The report shall be presented in the English language, verdana 12 single spacing, in four bound copies and a soft copy handed over to CBM.
The evaluation exercise is expected to take a maximum of 1 month including the development of a final report.
The inception report prepared by the consultant should have a proposed breakdown indicating how the various tasks will be allocated within the allocated period. The Inception report detailing consultant understanding and interpretation of the TORs will be approximately 3-5 pages.
The report should be presented in the format detailed in Appendix 2 including a detailed work plan and evaluation tools.
Inception Report due by: 30 October 2017
Draft Report due by: 01 December 2017
Finalised Report due by: 11 December 2017
Other expected outputs: Draft of new log frame and a draft new Multi Year Plan.
**11. Duration and Phasing
**The duration and phasing will be determined after a decision is made on the number of students, beneficiaries etc to be interviewed.
12. Requirements of the consultant(s)
Qualifications and Experience
• Sufficient knowledge of relevant laws, regulations and rules with Ministry of Health departments especially across the East Africa Region.
• Fluency in written and spoken English.
• At least 10 years experience in carrying out monitoring, evaluations and learning of a similar nature in the development sector.
• Ability to embrace health systems strengthening through human resource for eye health development.
• Minimum of a masters degree in a relevant field
Because of the extent of the evaluation, the consultant is required to demonstrate experience covering the evaluation of the Medical training programme related aspects as well as the Operational and programmatic Capacity components of the programme too.
How to apply:
Interested consultants should submit the technical and financial proposals to procurement.Nairobi@cbm.orgbefore 9th October 2017. The email applications should clearly mark on the subject- Expression of Interest: Impact Assessment of the Scholarship Program
• The financial proposal must include all costs including professional fees and taxes to conduct the assignment;
• The technical proposal must include the profile of the company; the proposed methodology, work plan; Curriculum vitae for the person (s) proposed for the assignment and a list of similar assignments done with details of the contact person;