Continued Medical Education Theory of change: can its application really make a change?

Evaluation is a systematic and impartial assessment employed by stakeholders mainly to assess the effectiveness and efficiency of a given intervention like an activity, project, programme, strategy, or policy (1). Two key purposes of evaluation are accountability and learning. Accountability includes being accountable for the resources spent: whether resources are being optimally utilized and to what extent the intended results were achieved. Learning component of evaluation provides evidence for evidence-based decision making in the planning process. Evaluators are interested in finding out 'what worked' and 'how it worked' to help decision makers to learn from the best practices of the success stories. However, learning from failures'what failed' and 'why it failed'is also essential to avoid repetition of the same mistake in future (1).


Introduction
Evaluation is a systematic and impartial assessment employed by stakeholders mainly to assess the effectiveness and efficiency of a given intervention like an activity, project, programme, strategy, or policy (1). Two key purposes of evaluation are accountability and learning. Accountability includes being accountable for the resources spent: whether resources are being optimally utilized and to what extent the intended results were achieved. Learning component of evaluation provides evidence for evidence-based decision making in the planning process. Evaluators are interested in finding out 'what worked' and 'how it worked' to help decision makers to learn from the best practices of the success stories. However, learning from failures-'what failed' and 'why it failed'-is also essential to avoid repetition of the same mistake in future (1). Evaluation can be done at different stages of the programme; at the planning stage, in the mid cycle, at the end of the programme or even a few years after the completion of the programme. During the planning stage, ex ante evaluation helps to find whether programmes are based on strong logic. In the middle phase, mid-term evaluation helps to identify whether we are moving in the right direction or if not to take remedial action on time before it is too late (formative evaluation). After completion, outcome and impact evaluations help to quantify the success of the programme (summative evaluation) including intended and unintended benefits (2). Other important considerations an evaluation helps to answer often include cost per beneficiary, whether the programme is worthwhile, how the programme could be improved, and whether there are better alternatives. In addition, whether the programme goals are appropriate and useful, what the unintended benefits or harms are and whether the results are likely to be continued once the programme is concluded will be answered by an evaluation (2). Importance of evaluation is further accentuated in the era of sustainable development goals (SDG). The United Nations 2030 Agenda for Sustainable Development underscores the importance of followup and review mechanisms informed by country-led evaluations as the key to achieve the 17 SDGs at global and national level (3).
In this backdrop, theory of change (ToC) is emerging as a widely used tool for planning and evaluation (4). As the name implies, ToC is the thinking behind how a change (results) will be brought about by a particular intervention by connecting the dots between objectives, strategies, outcomes, and the underlying assumptions (4-5).

Sri Lankan context
The Government of Sri Lanka has adopted the concept of managing for development results in mid 1990s, and as a signatory to the Paris Declaration on Aid Effectiveness is committed to institutionalization of evaluation. Furthermore, the government has recently adopted the National Evaluation Policy. The Department of Project Management and Monitoring under the Ministry of Finance is the national focal point for evaluation of the major developmental interventions. Each government institution has its monitoring and evaluation (M&E) mechanisms to assess their own projects and programmes (6). Nevertheless, the focus is more on monitoring of the physical and financial progress rather than the achievement of results. Majority of the government projects and programmes have never been evaluated. Lack of demand for results and accountability, resources not being allocated for evaluation at the planning stage and inadequate technical capacity to conduct evaluation have been identified as the major factors leading to this situation. Due to the lack of awareness, evaluation is considered as 'policing' and not welcomed by the majority of the programme implementers. It is being conducted in an ad hoc manner, mostly as donor driven once the project or programme is completed. Underutilization of the evaluation findings is another concern in the country context (6).

Theory of change
Predecessor of ToC is the log frame (LF), a linear model that only lists inputs and outputs, without taking the interplay between different elements into account. Furthermore, the LF neither incorporates influences from the changing context nor provides insight into the processes leading to outcomes. Therefore, explaining complex developmental interventions using LF becomes difficult (7). To address these problems, ToC was proposed. Based on the premise of complexity and the unforeseeable nature of social change processes, ToC recognizes the importance of understanding the existence of different perspectives on why and what needs to change and the full analysis of the context and of the assumptions underlying the intervention to achieve the expected change (5,(7)(8).
Developing a ToC follows eight steps and starts with asking 'What is the purpose of ToC?' and 'What is the desired change? (Figure 1). It involves a backwards mapping process starting from the highest results level (programme objective or goal) to strategies exploring underlying assumptions (7,8).

Figure 1: Eight steps in developing a theory of change
For example, the purpose could be to develop a programme to address sexual abuse of children. Therefore, the goal would be to raise children free from sexual abuse. The next consideration is "what is the current situation?" and then "what conditions must be in place to reach the goal? In order to protect children from sexual abuse, children should have the awareness on sexual and reproductive health; the society must recognize rights of children and the government has to put policies in place to ensure children's rights. These prerequisites would then be written in positive language or change language beneath the programme goal and shown as outcomes on the ToC pathway. Pre-conditions to achieve these outcomes will be presented as outputs at the next lower level and the strategies to reach the outputs will be presented immediately below. ToC also outlines the causal linkages in an intervention between the different results levels namely outcomes and impact and shows these changes as a pathway-showing the logical relationship between outcomes in chronological order (8). Once the desired changes at each level are identified, based on this information, indicators for M&E are developed.

Application of ToC
ToC is a management tool to analyze how the intervention will work. Thus, ToC is the "road map" for getting from "here to there" (9). ToC can be used as a strategy development and planning approach, and as a M&E approach (4,8). The multiple uses of ToC are given in Table 1.  The lack of ToC or logically weak ToC hinders developing a robust M&E system. It also makes the programme evaluability at stake. Globally and locally, lessons learnt from diverse programmes including population relocations, health and sanitation points towards the importance of having a strong ToC (10)(11)(12). Services are made available spending scarce resources without achieving the expected results -'Kevin Costner syndrome'-aptly named after the movie 'Field of dreams'-where Kevin Costner plays the role of a farmer who builds a baseball field believing players will come to play once he built it -'if you build it, they will come' (13). Even though Costner realized his dream in the movie, utilization of services once it is made available is far from reality. Human behaviour is a complex phenomenon intertwined with socio-cultural factors -an intervention which worked well in one setting may become an utter failure in another setting.
Therefore, in the real world, the assumption that a particular challenge will be met 'once the facility is built' is not realized in most instances. The underlying change pathway need to be thoroughly analysed using the ToC.

Way forward
Evaluation has been an often completely forgotten or perhaps an afterthought in planning. Sri Lanka is not an exception: the need to strengthen the capacity of government monitoring and evaluation systems to focus on results -outcomes and impacts -beyond the traditional output focus remains a major challenge. If we were to achieve the SDGs by 2030, evaluation must be given its due place without being complacent. An evaluation culture to learn from both success and failures -disseminating evaluation findings to build trust and organizational knowledge, being accountable and transparent to funds being spentfocusing beyond the outputs, thriving to improve results to bring about a change in beneficiaries' livesa quest to perform better to be inculcated at organizational level.
Priorities of the intended beneficiaries need to be identified and a tailor-made intervention to be devised based on a sound ToC. What decision makers perceive as the main problem may be of least concern to the community. Underlying causes to the main problem may be context specific. Perhaps, what works and not works are best known by the community itself. Incorporation of local knowledge into planning can make the difference in the programme results. To achieve the aforementioned, stakeholder engagement from the inception stage is crucial.