Previous - Next
This is the old United Nations University website. Visit the new site at http://unu.edu
David F. Pyle and Mitchel B. Wallerstein
A policy evaluation
methodology
Implication of evaluation
methodology
Reference
Successful evaluation of the policies that facilitate and direct nutrition intervention programmes is a difficult task under optimal conditions. Even when good data exist (or can be mobilized) regarding the clinical dimensions of nutritional status or the economic circumstances of target populations, it still may prove extremely difficult to establish cause and effect relationships. It is perhaps for this reason, and because meaningful policy evaluation often defies quantification, that the policies that underlie nutrition programming have rarely been considered in other than purely instrumental terms (e.g., tons of food aid delivered, number of children fed, etc.), and that what few policy evaluations exist have tended to be descriptive rather than analytical. The result is that programmes undertaken often bear little or no relationship to the original policy conception and/or are continued with little or no accountability for results.
Nutrition policy evaluations are, almost by definition, the opposite of the neat, empirical assessments of nutritional anthropometry or clinical status that are to be found in the public health literature. Policy evaluations are "soft" in the sense that they involve politics and the individual motivations of policy-makers; and experience has shown that these dimensions are difficult to isolate and analyse empirically. These characteristics have been problematic for the scientific community, which is more comfortable dealing with "controlled" - and, therefore, predictable - situations. Assessment of nutrition policy also does not engender enthusiasm or support within the affected bureaucratic and political circles. From the viewpoint of those who have been involved with initiating or implementing policy, it is one thing to determine nutritional impact with the intention of modifying programmes accordingly, but it is quite another to expose to analysis the entire conception or practicability of the undertaking.
For purposes of evaluation, nutrition policies may be disaggregated functionally into three successive, interlocking stages: policy formation, programme development, and programme implementation. Policy formation is a dynamic process wherein the felt needs of various interest groups are reflected in the political arena as officially-sanctioned government goals and objectives. Programme development, in turn, concerns the manner in which policy is interpreted and acted upon to make the stated goals and objectives operational. The final stage of the policy process is programme implementation, which encompasses all of the activities undertaken to fulfill the specific policy mandate in order to deliver services to the intended target group(s).
Clearly, nutrition policy does not always evolve in such a discrete, hierarchical fashion. However, use of this framework permits us to evaluate: (a) stated objectives - i.e. what policy-makers set out to accomplish; (b) programme content- i.e. how the policy was embodied in operational terms; and (c) policy impacts i.e. what the policy actually accomplished. An understanding of all three dimensions is essential, not only in conducting ex post facto assessments of past and present programmes but also in shaping the design of future initiatives.
Nutrition policy evaluations also involve an examination of the degree of "political will or commitment," an oftused but seldom-defined term. As employed here, political will/commitment is intended to mean the capacity and inclination of decision-makers to follow through on rhetorical statements of support, to maintain programme resources in the face of competing demands, and to defend a programme from its critics. To date, little serious effort has been devoted to understanding, much less evaluating, the importance of political will/commitment in terms of programme impact.
Political will/commitment can be measured at each of the stages of nutrition policy. An initial indicator, at the policy formation stage, is the extent to which rhetorical support for a policy becomes embodied as a physical programme, as measured by staffing quotas, office space, and most importantly, funds. Politicians from many nations have expressed rhetorical concern from time to time for problems of hunger and malnutrition, a position certain to enhance an individual's reputation and career at little or no political cost. Yet when it comes to allocating scarce resources, these same people become conservative and equivocal.
The extent of political will/commitment at the programme development stage is indicated by the specific scope and nature of the programmes established, e.g., limited technical approaches versus ambitious social equityoriented programmes. Technical strategies such as food fortification have the advantage of being relatively easy to adopt and implement and of requiring little or no socio-political change. Social equity-oriented programmes, such as multi-sectoral health and nutrition intervention or land reform schemes, will, on the other hand, demand politically- and economically-difficult reallocation of priorities and resources.
At the stage of programme implementation, political will/commitment is indicated by the extent to which programmes are maintained over time. In many situations, it may be far more difficult to defend a programme (and its budget) from its detractors than to gain the "critical mass" of support necessary for initial implementation. This problem is particularly acute in programmes where positive impact data may be delayed (often by many years) for one reason or another, or where a substantial number of similar programmes must all compete within the same resource pool.
The identification and collection of useful and accurate data on social sector programmes such as nutrition have proved to be a substantial obstacles to meaningful policy evaluation in the developing country context. The traditional measurement of per capita income does not, of course, take account of skewed distribution that distorts the true picture of how the poorest segments of the population actually live. A more accurate view of a government's commitment to, and involvement in, issues of social equity must therefore examine not only the pattern of income distribution, but also other indicators such as educational performance and health status.
The most widely-used indicator of social development is the Physical Quality of Life Index, or PQLI (1) which rates countries according to their level of literacy, infant mortality, rate and life expectancy. It is revealing to compare PQLI ratings with per capita incomes. In the early 1970s, for example, countries such as South Africa and Iran had similar per capita income (about US$1,200 per capita), but ranked along with such developing countries as Burma (per capita income of US$105) and India (per capita income of US$133) in terms of PQLI (53 for South Africa, 43 for Iran, 51 for Burma, and 43 for India). The oil states present an even more unbalanced picture. For example, the richest country, the United Arab Emirates, had a per capita income of US$15,368 but a PQLI of 34, ranking 112 out of 150 countries. On the other hand, a poor country like Sri Lanka (per capita income of US$126) had a PQLI of 82, indicating a high level of social development and equity.
What can a policy evaluation derive from analysing a country's PQLI? Several possibilities arise; for example, in a country with a low PQLI and a high per capita income, one explanation is that there is little political support for, or popular interest in, social development. As a result, the status of the lower socio-economic strata is unlikely to improve no matter what outside assistance is provided. An alternative explanation is that the country is taking measures to improve the social development status of the poor, but that results are not yet measurable in terms of the PQLI.
The evaluation of policies and programmes designed specifically to address nutrition as a social development issue will generally require the mobilization of data from a much broader variety of sources than the PQLI. These may include some or all of the following: (a) examination of public documents and news accounts, (b) analysis of data contained in government reports and/or studies by external agencies, (c) interviews with key decision-makers and administrators both past and present, and (d) limited collection of new data through field observations. The difficulty, as we noted at the outset, is that there are few serious and careful nutrition policy evaluations extant with which results can be compared. It shall be our purpose in the remainder of this paper, therefore, to make a first attempt at setting forth such a methodology and at indicating its limitations and implications.
Two basic assumptions underlie the methodology described here. First, it is assumed that the policy evaluations are to be conducted either retrospectively or concurrent with operational programmes, i.e., that the focus is on the origins and ramifications of past and present policies. A second, related, assumption is that the policy evaluations are not intended to be prospective, i.e., that the principal objective is not necessarily the development of new programmatic approaches (although this is certainly one possible outcome) Because the conditions under which policies evolve and are implemented vary substantially, depending on the country context, it is neither possible nor advisable to spell out a "cookbook" approach that can be applied without modification under all circumstances. Rather, we have outlined here a generic methodology that can (and should) be modified and adapted to the specific type of nutrition policy being evaluated.
Our approach has been to set out a series of scales for each of the three policy stages by means of which performance factors can be measured and the impact of a given policy evaluated. When employed successfully, the overall effectiveness of a country's nutrition policy may be determined by comparing actual performance with the potential score. Of course, there is also a problem with the use of scales in attempting to weight the importance of one parameter relative to another; that is, how seriously should a low score on a particular variable be taken? The answer is that such scales can be conceived of most usefully as "gap detectors," indicating which aspects of policy are proceeding well and which are problematic. They do not, however, reveal why the dysfunctions have developed. nor do they suggest what must be done to correct them.
Policy Formation
Evaluating nutrition policy at its earliest. conceptual stage basically involves identifying which agent (or agents) were responsible for proposing and developing a policy, what objectives the proposers were attempting to achieve, and the degree of support or opposition they encountered in the process. Careful analysis of the policy formation stage should reveal the points at which the final prescription diverged from the original conception and/or where the approach adopted was, in restrospect, ill-conceived.
Origin of Policy
The more government has been involved in initiating and formulating a policy, the more committed it is to that policy and its effective implementation. International agencies often dominate the process for several reasons: (a) because they assume they know what needs to be done, (b) because they have the technical expertise to design policies (and later) programmes, and (c) because they wish to further their own objectives. Yet, policies have proven to be more effective when they originate with a broad-based contingent of powerful officials within the country who can influence both the shape and the implementation of the public agenda.
Performance Scale
0 - Exclusively externally initiated/No local input.
1 - Predominantly externally initiated/Very little local input.
2 - Strong external initiative/Minimal local input.
3 - Some internal initiative with much external influence.
4 - Strong internal initiative with moderate external influence.
5 - Primarily internal initiative with external aid as required.
Motivation for Policy
In order for a policy to be successful, it is necessary for there to be a strong commitment to social development, which might be measured in one respect in terms of orientation to "basic needs" programming. Thus, an evaluator can gauge a country's commitment to nutrition policy by the extent of attention to such things as rural health care delivery, housing, and primary education.
Performance Scale
0 - Symbolic social programming ("palliative" approach).
1 - Primarily politically motivated/Little interest in social development.
2 - Politically motivated/Some interest in social development.
3 - Moderate interest in social development.
4 - Substantial concern for social development.
5 - Strong concern for and commitment to social development.
Support for Policy
For a nutrition policy to be effective, it must enjoy widespread support at all levels of government. Narrowlybased support emanating from one or two "charismatic individuals" may succeed in getting a new policy approach off the ground, but it can be reversed quickly. The institutionalization of the support, therefore, is often manifested in a coordinating body (e.g., a nutrition council) that is capable of insuring that requisite actions are carried out by the involved government agencies.
Performance Scale
0 - No support/Externally motivated.
1 - Very narrow support/Dependent on a single "charismatic" individual.
2 - Narrow support/Limited to several actors in one ministry.
3 - Several ministries involved but no formal coordinating body.
4 - Formal coordinating group exists but has little or no power.
5 - Formal coordinating body exists with authority to take action.
Special Interests
The role of special interests in the policy process can range from highly negative to positive. The negative aspect may occur either through acts of commission (active opposition) or omission (failure to support). Professional associations, for example, may lobby or pressure against a programme. In contrast, other actors, while not actively opposing a policy, may not provide the support necessary to permit the achievement of objectives. It is also possible for positive and negative special interests to counterbalance each other.
Performance Scale
0 - Special interests have a strongly negative influence.
1 - Special interests have a mildly negative influence.
2 - Special interests have neutral influence.
3 - Special interests have a mildly positive influence.
4 - Special interests have generally positive influence.
5 - Special interests have strongly positive influence.
Objectives of Policy
Experience demonstrates that effectiveness is increased when objectives are clearly stated and goals/targets are established. This facilitates the practice of management by objective, the identification of priorities, and the establishment of a basis for evaluation and accountability.
Performance Scale
0 - No data/Objectives not stated.
1 - Objectives very vague and general.
2 - Objectives specified but not supported by data.
3 - Objectives generally specified and supported by data.
4 - Clearly specified objectives and data/No goals or targets.
5 - Clearly specified objectives with goals/Targets corresponding to priorities indicated by data.
Programme Development
Evaluating nutrition policy at the level of programme development involves primarily the assessment of the various mechanisms established or employed to translate goal statements into operational programmes. It requires a thorough understanding of both the political and bureaucratic processes as well as the role played (if any) by external agencies as initiators and facilitators.
Policy-Programme Linkages
Policy must be translated into action by means of programmes. Most often an effective nutrition policy will require a number of programmes in order to achieve its objectives. Potential success, therefore, can be identified by determining how closely programmes fulfill the objectives outlined in a policy.
Performance Scale
0 - No programme/No relationship to policy.
1 - Little relationship to policy.
2 - Some relationship to policy.
3 - Close relationship to policy.
4 - Strong relationship to policy.
5 - Fully congruent with policy.
Programme Design
Programmes based on communications with those for whom the benefits are intended and on genuinely perceived felt needs of the target group are more likely to succeed. The centrally planned, top-down approach attempts to apply a limited programmatic conception to large areas with widely varying conditions and problems. Moreover, participation in determining programme content increases target group interest and support. Consultation serves the dual purpose of exposing target group concerns and problems as well as informing people of the availability of new programme services.
Performance Scale
0 - Total lack of awareness of target group needs.
1 - Limited knowledge of target group needs.
2 - Moderate knowledge of target group needs.
3 - Extensive knowledge of target group needs/Limited consultation.
4 - Consultation with target group/But not reflected in programme design.
5 - Extensive consultation with target group resulting in response to felt needs.
Action Agencies
If the lead or "action" ministry in the nutrition programming effort is to carry out the policy with the necessary degree of commitment, it must have an interest in the problem. Assigning nutrition activities to an organization or department that has neither the interest, the knowledge, nor the experience in nutritionrelated programmes (e.g., an agriculture ministry) decreases the potential for impact.
Performance Scale
0 - "Assigned" to agency involuntarily.
1 - Directed by higher authority to undertake/Little relevance to mission.
2 - Directed to undertake/Some interest and relevance to mission.
3 - Agency already active in the field/Limited interest.
4 - Agency active and interested/Pursuit of issue is voluntary.
5 - Voluntary leadership/Pursuit of issues with enthusiasm.
Awareness of Management Issues
To implement nutrition intervention programmes successfully, programme designers must have a deep awareness of the ingredients of successful programming. In this respect, the "form" of the intervention (e.g., supplementary feeding programme, etc.) will only provide superficial guidelines as to how the programme is to be designed and carried out. The organizational and management-related factors necessary to attain programme objectives must also be considered. Among these concerns are: personnel. training, information systems, supervision, and structural constraints.
Performance Scale
0 - No awareness or exposure to organization and management (O & M) issues.
1 - Little awareness or exposure to O & M issues.
2 - Some awareness and exposure to O & M issues.
3 - Significant awareness and exposure to O & M issues.
4 - Process studies conducted/Extensive awareness.
5 - Process studies conducted/Complete awareness.
Budget
Budgetary allocations for nutrition programmes are often taken as a measure of a government's commitment to its stated policy. However, it is difficult to ascertain exactly what budgetary support means in absolute terms. More revealing as a manifestation of government commitment is the extent of funding for preventive versus curative programmes. While ratios may not be totally accurate indicators (since curative care costs more than preventive), the figures provide some sense of where the priorities of the government are focused.
Performance Scales
a) Ratio of rural to urban per capita expenditure on health:
0 - 1:10.
1 - Between 1:10and 1:5.
2 - Between 1:5 and 1:3.
3 - Between 1 :3 and 1 :2.
4 - Between 1:2 and 1:1.
5 - Better than 1:1.
b) Nutrition programmes as a percentage of the total national budget:
0 - Less than 1 per cent.
1 - 1-3 per cent.
2 - 3-5 per cent.
3 - 5-8 per cent.
4 - 8-10 per cent.
5 - Greater than 10 per cent.
Programme Implementation
The third level of nutrition policy evaluation concerns the broad range of implementation activities undertaken to deliver services to the intended target populations. It is frequently at this level, in fact, that nutrition policies break down; that is, policy is formulated and programmes are established but, for a variety of reasons, the planned services are not delivered as originally intended. There are many causes of this system breakdown; they relate mainly to issues of logistics, politics, and economics. Moreover, evaluators must also consider the stage (or age) of programme development, because this is likely to alter the manner in which programme elements fit together and are evaluated.
Infrastructure/Basic Supplies
Little can be expected of a programme that is not supported by the necessary infrastructure (e.g., buildings, staff, supplies, transportation networks, etc.). Deficiencies in professional and pare-professional training and availability, lack of adequate facilities, and shortages of medicine, food and equipment can prevent competent, well-trained workers from implementing a programme as planned.
Performance Scale
0 - Necessary infrastructure non-existent.
1 - Most infrastructure unavailable or under development.
2 - Some infrastructure available/Operational effectiveness doubtful.
3 - Some infrastructure operational/Nothing more under development.
4 - Most infrastructure operational/Remainder under or nearing completion.
5 - All necessary infrastructure available and operational.
Programme Decentralization
Experience demonstrates that most programmes (with the exception of such technical schemes as fortification) operate more effectively if administered within a decentralized framework. This arrangement permits the adaptation of programme interventions to local conditions and circumstances, and it allows for the flexibility to respond to unique demands and situations. This contrasts sharply with the rigid, highly centralized bureaucracy that will not permit any modifications and, hence, loses substantial amounts of potential impact.
Performance Scale
0 - Total centralization.
1 - Some decentralization exists/Little effective impact.
2 - Moderate decentralization/Continued heavy reliance on central government.
3 - Substantial decentralization/Very limited local autonomy.
4 - Substantial decentralization/Substantial autonomy.
5 - Effective decentralization with virtually total autonomy.
Self-Reliance
Successful nutrition schemes must be institutionalized. This is particularly important in programmes that require behavioural change (e.g., nutrition education). Little in the way of long-term results can be expected without the establishment of self-reliant momentum. Indicators of self-reliance include the extent of local support for a programme, the degree of local organization (institutionalization), and the level of local involvement in programme design, implementation and evaluation. More top-down, dependency-creating approaches tend to foster welfare ("let them do it for me") attitudes that reduce likely policy impact.
Performance Scale
0 - Paternalistic/Perpetuates dependency.
1 - Self-reliance espoused/Dependency maintained.
2 - Nominal self-reliance.
3 - Some self-reliance attained.
4 - Significant self-reliance attained.
5 - Total self-reliance attained.
Political Elite Support
Political elites are not only important in the formation of policy but are also vital in the implementation of programmes as well. National and local elected officials can, by actively opposing a programme, decrease dramatically the chances of a policy's achieving results. Similar negative effects result from weak, ineffective support by political elites. It is highly unlikely that a nutrition programme can succeed at the local level without the active support of local political and opinion leaders.
Performance Scale
0 - Political elite actively opposed.
1 - Political elite moderately opposed.
2 - Political elite neutral.
3 - Political elite moderately supportive.
4 - Political elite actively supportive.
5 - Political elite acting as the initiating agent.
Results Orientation
Management by objective is concerned with results as distinguished from other types of management which are concerned chiefly with procedure. Management by objective may be conceptualized as follows:
| Results | Procedure | |
| Objectives | Specific | Vague |
| Target Groups | Priorities | All |
| Indicators of Success | Outputs | Inputs |
| Organization | Team | Specialists |
| Training | Problem-Solving | Routines |
The more results-oriented the programme implementation, the greater the chance for programme impact. While not all the characteristics may be found in a particular programme, the mix of interventions should have the applicable characteristics if the policy is to be judged a success.
Performance Scale
0 - Completely procedure oriented.
1 - Results-oriented in word/Still largely procedureoriented.
2 - Mixed results and procedure orientations.
3 - Modestly results-oriented.
4 - Substantially resultsoriented.
5 - Completely results-oriented.
Targeting
Although impact is the ultimate measure of successful programme implementation, an intermediate indicator of the potential effectiveness of a targeted programme is the ability to reach those for whom the scheme was designed. Delivery of services to the most seriously disadvantaged is notoriously difficult, especially in hardto-reach rural areas. Special efforts must be made to include the politically powerless and unorganized.
Performance Scale
0 - No evidence of successful
service delivery.
1 - Services reaching less than 5 per cent of target group.
2 - Services reaching between 5-25 per cent of target group.
3 - Services reaching 25-50 per cent of target group.
4 - Services reaching 50-75 per cent of target group.
5 - Services effectively reaching more than 75 per cent of target
group.
As noted at the outset, evaluation of nutrition policy does not lend itself readily to empirical analysis, particularly if one wishes to examine the nuances, e.g., why a specific decision was made or why a programme was not carried out. It is entirely possible, in this respect, for a programme to be judged a success on some levels yet a failure in policy terms. How could this occur? For one thing, if the programme as finally implemented bears little or no resemblance to the original policy (or even the programme plan), then even if services were actually delivered the policy itself may be considered non-viable. Alternatively, if the policy called for the creation of elaborate inter-agency coordinating structures as a means of designing and delivering services, the mere creation of such institutional arrangements would not automatically result in a positive evaluation, if little programmatic activity actually occurred. These are merely two of the more obvious examples, but the actual subtleties of a careful and penetrating policy evaluation naturally are a good deal more complex.
What does all of this mean, then, in terms of nutrition programming? Many would argue that the failure of a government or external agency to carry out all of its stated policy objectives is less important than whether it was able ultimately do deliver services to those at nutritional risk. Put more succinctly, this position states that half a loaf is certainly better than nothing at all. This proposition is very difficult to refute, except in those circumstances where it can be demonstrated either that a policy is either misguided conceptually, i.e., that it addresses the wrong problem or adopts an unworkable approach; or that by its implementation, a policy will do "positive harm," i.e., that it will somehow leave the intended target population worse off than before while perhaps benefiting non-needy populations. This has often occurred. for example, in the case of institutional feeding programmes that have reduced the desire of the poor to attain food self-reliance and, on occasion, have even exacerbated pre-existing malnutrition. The interpretation of nutrition policy evaluations must be tempered, therefore, by a strong sense of what is politically, economically, and logistically feasible.
There is, finally, the question of what should be done with the results of such an evaluation. Clearly, whether one is dealing with the results of a programme that has recently been terminated or one that is ongoing, an effective policy evaluation should provide both feedback and information useful for making future decisions. The results of each approach that is undertaken should advance the learning curve for the next programme both within a specific national context and for transfer of concepts elsewhere. To quote George Santayana, "Those who do not learn from history are condemned to repeat it." But often, those who must learn such painful lessons have neither the expertise nor the time to do so, and those who do have these advantages lack the inclination for fear of threatening that which has traditionally been their function.
Thus the importance of undertaking periodic evaluations and acting upon their results must be impressed on key decision-makers at both the national and international levels. Similarly, it is vital that decision-makers understand that the conduct of a frank policy evaluation is not intended to provide a pretext for terminating controversial or costly programmes so much as to offer a diagnostic tool for correcting weaknesses and improving programme performance.
In order to avoid wasting scarce resources and manpower in "reinventing the wheel," evaluation results must not be delivered only to middle-level managers. In the case of externally-aided programmes, additional funding should be made contingent on a candid dialogue regarding the assessed strengths and weaknesses of each programme. Programmes supported entirely by internal funds present, of course, a more difficult challenge. But in an age of mounting LDC debt and constrained budgets, the critical importance of costeffective policy-making and programming increasingly is being recognized by those in positions of authority and influence.