Previous - Next
This is the old United Nations University website. Visit the new site at http://unu.edu
RECOMMENDATIONS FOR FUTURE EVALUATION ACTIVITIES
The difficulties encountered in substantiating impact are indeed discouraging. Regardless of whether they are caused by ineffectual programming, unsound evaluation methods, or simply the problems inherent in performing social science experiments in volatile and complex village settings, the cost of proving the nutritional impact of food aid projects on anything approaching a global basis will be high.
This is amply illustrated by the resources required just to carry out those few sound impact study designs, such as the proposed WFP evaluation in Honduras (8) and the food-for-work study under way in Bangladesh (15). (These costs say nothing of the need to infuse resources into the improvement of the projects themselves.) Expectations from these major research endeavours should be limited, however, given that previous research efforts on food aid projects have yielded equivocal findings. In any event, the cost of applying such methods on a wide scale is prohibitive. One must also question the value of such studies in terms of what is considered the primary pitfall of food aid projects- poor design and breakdown in implementation. Therefore, the question is: Where do we go from here? It is suggested that a two-tier strategy should be adopted. First is to develop built-in evaluation procedures to assist in designing and implementing projects. This will increase the probability of achieving objectives; simultaneously, it will facilitate assessing nutritional impact. Second is to design and undertake a series of small-scale operational research studies to test vital hypotheses rather than try to determine impact per se.
Building Evalution into Standard Operating Procedures
The development of built-in monitoring and evaluation systems should be accorded the highest priority for all food aid projects. The building-in of evaluation procedures begins before operations commence, i.e., when the project is being conceived. This will encourage integrating the collection and interpretation of process and impact data into standard operating procedures.
The attributes of an evaluation system built into the project are described briefly below. The adoption of such a system for all projects is urged.
Building Evaluations In at the Planning Stage
A strategy for impact and process evaluation should be developed concurrently with designing the project. This involves defining the basic components of the monitoring and evaluation systems. According to Miller and Sahn (16) these are:
- a data system that describes the actual variables to be collected, how often, and on what population; these objectively verifiable indicators will include key process and impact elements;
- an analytic methodology that delineates the computational algorithms used to describe and interpret data; this involves defining explicitly the statistics employed to summarize a data set, the nature and methods for aggregating data, and the types of procedures used to interpret the statistics;
- information flow that details the forms to be used and flow of information through the system, including what is passed up to higher levels of management and what information is fed back to functionaries in the field;
- a management structure that outlines explicitly how the information is to be used at each level of management hierarchy, i.e., the range of decisions and types of actions to be taken.
Defining the monitoring and evaluation system as part of the initial project design will yield a variety of dividends. This procedure will encourage improvements in the diagnosis of the nature and extent of the nutrition problems. Encouraging the definition of objectively measurable units (i.e., data elements) for process and impact evaluation will immediately focus attention on objectives and the logical progression of events that will lead to their realization. It will also enable the collection of benchmark (baseline) data on strategic impact indicators. Furthermore, delineating acceptable levels of achievement or changes in these indicators before starting a project will provide quantitative targets to serve as a source of comparison in the future. In this regard, the use of tools such as the logical framework is recommended. However, such tools are often used in a perfunctory manner, They run the risk of becoming an excuse for not thinking through the hard questions. Caution should be taken to avoid "filling in the boxes" at the expense of performing the requisite analysis.
Improving Logistics and Cost-Accounting
It is not unusual for logistical breakdowns to impair food supply lines. Similarly, complementary inputs such as tools (e.g., shovels), equipment (e.g., scales for weighing), and raw materials (cement for road construction, oral rehydration salts for diarrhoeal disease control) are often not delivered to the project site on time, or at all. An information system that carefully monitors the stocks and flows of project inputs serves as an essential ingredient in understanding and explaining impact indicators. For example, consider two villages. In the first, the monitoring of inputs shows logistical breakdowns and impact indicators fail to substantiate nutritional changes in the community; in the second, the information on stocks and flows records few breakdowns and impact indicators show marked improvements. In yet another village, with even greater nutritional improvements, the monitoring system may indicate either that a different set of inputs was provided or that they were applied with different intensity, for example different ration sizes. The availability of these data on inputs represents vital information for constructing a story as to why and how the project succeeded in reaching its goals. The accumulation of knowledge over the extended life of a project will provide a formidable data base not only for assessing impact but for explaining it as well
Monitoring project inputs will also encourage the involvement of local functionaries in rationalizing their flow. This will help guard against characteristic stories such as spoilage of food commodities or pharmaceuticals, workers sitting idle awaiting the delivery of inputs, or the arrival of the monsoon causing ditches to fill up with silt because the delivery of irrigation pipes was late.
Perhaps more important in the context of impact evaluation, a well-developed monitoring system forms the basis for cost-accounting. Experiences in evaluating nutritional impact display a neglect of the relationship between costs and effects. This is especially the case for supplementary feeding programmes. It is stressed that the village-level project managers are not necessarily responsible for the careful quantification of the value of inputs. This task will be beyond their ability and time availability. Inputs should be valued at their opportunity cost. Considered judgements will be required to determine shadow prices and to distinguish which portion of shared costs should be assigned to the project. Nonetheless, simple and well-maintained monthly inventory reports will form the basis for technical experts to assign cost figures to project inputs when doing their analysis.
Achievement of Output Objectives
Accounting for project inputs leads to the next aspect of a built-in monitoring/evaluation system - documenting that the resources were used as planned. Monitoring that inputs were transformed into outputs focuses on whether and what services were actually provided and to whom they were delivered. The type of information included may be, for example, the size and number and composition of food rations delivered; the economic or social characteristics of the individual participants (to assess effectiveness of targeting strategies); the number of miles of roads built; the attendance at the weekly nutrition and health education classes; or the number of new enrollers and drop-outs in the project. In addition, it will often be appropriate to collect key non-project-specific data. These would cover variables exogenous to the project (e.g., staple food prices) that are considered vital to addressing confounding factors for nutritional changes.
Given the historical problems in the operational aspects of food aid projects, not only are data on achievement of outputs necessary to interpret and understand impact indicators, but their collection is commended by the need for project management to have information that can be used to enhance the quality and extent of supervision. This suggests that the selection of output indicators should be largely a function of identifying those variables that are most amenable to change or control by field staff. In addition, the information system, like the project itself, should be viewed as dynamic. It should be revised in accordance with the needs and experiences of field workers and policy-makers
Determining Project Impact
The existence of a monitoring system that substantiates the causal hierarchy of project events (i.e., that inputs were provided, then transformed into outputs) lays the foundation for the evaluation of impact. The basic premise is that a self-evaluation system will be superior to expensive quasi-experimental research. This is supported by a number of compelling arguments.
First is that the steps discussed previously will enhance the probability that projects are well designed and performing efficiently. Those doing data gathering and recording for a project will understand its purpose and be encouraged to interpret and act upon available information.
One of the most difficult aspects of designing development projects is the dynamic nature of the environment in which they are implemented. Characteristically, a standard formula, or at least an inflexible set of instructions, dictates the nature of the services provided. The latitude given to those on site is minimal. Given that the duration of most food aid projects is measured in years, a key ingredient to project success is the ability to make changes in the nature of services provided and level of intensity in application. This involves the ability to respond to factors such as seasonal fluctuations in the food supply or prices, or the evolving context of the project site over time. These types of decisions must be made at the local level rather than in Rome or Washington. The discretion of the local staff based on knowledge garnered through a well-functioning self-evaluation system is therefore indispensable. Just as a field worker must be able to rationalize the stock and flow of project inputs to avoid shortfalls in supplies or wastage, it is important that the monitoring and evaluation system trigger programmatic changes, as appropriate, based on sound and timely data (21).
The data included in a built-in evaluation system may be of a wide variety, ranging from anthropometry to food prices. As an illustration, note that there may be a need to adjust the size of the food ration if no nutritional impact is being observed. The time may also come, according to data collected on the use of oral rehydration solution, to shift emphasis from the efforts to control diarrhoea to promotion of improved hygiene practices to prevent infection. Similarly, the number of work days provided by a FFW project may have to be adjusted based on seasonal fluctuations in labour demand or food availability. Although this may seem difficult, its feasibility has been demonstrated in the context of the Early Warning Information and Intervention System in Indonesia (25). A variety of data collected regularly indicate or predict changing patterns in food availability and consumption that trigger a range of interventions, depending on the severity of the signals from the information system.
Second, a monitoring and evaluation system will afford greater opportunity for outside experts, in conjunction with local field staff, to draw meaningful inferences concerning impact from available data. (As noted previously, the best evaluations performed were those that tapped into an existing information system.) This is attributable to various causes. It is likely that the quality of data will be improved because field workers use them in the context of project management. Longitudinal sequences on routinely gathered data elements will represent an enormously rich data set. Indeed, the number and complexity of impact indicators will have to be limited in order to maintain the manageability of the system. For example, in a supplementary feeding programme, it would indeed be preferable to collect a battery of measurements (e.g., height-for-age and weight-for-height data). Doing so may prove impracticable in a self-evaluation system. It may only be feasible to use a composite indicator such as weight-for-age. The necessary compromise, however, will be more than compensated for by the completeness of coverage and the availability of key process indicators.
A third advantage of a built-in evaluation system relates to the difficulty of controlling for confounding variables (i e., threats to validity). Involving local personnel in the interpretation of data will usually resolve what appear to be surprising or contradictory outcomes of data analysis. External evaluators may be involved in a determination of the changes that have occurred. They may even limit the possible explanations for such changes. In the final analysis, however, on-site personnel must distinguish among the competing explanations for the findings. Only their intimate knowledge of the project will be appropriate for such a task.
It is appropriate to acknowledge the arguments against promoting an internal information system. First is the cost. Undoubtedly a well-functioning information system adds significantly to the cost of the project. Most of the additional costs will be in designing the data systems and training local personnel in their use. In fact, five to ten per cent of project costs may be absorbed by a monitoring and self-evaluation system. However, such systems are considered vital not only to the assessment of project performance but to the achievement of impacts as well. Monitoring and evaluation systems form an integral component in any attempt to improve operational performance. The cost of developing such systems is not only justified but essential.
A second argument against built-in evaluation systems is that project personnel are incapable of collecting and recording the data. The information from the system would then be unreliable. My response is that, if a field worker is unable to measure and record accurately data elements vital for assessing project operations and achievement of objectives, one must question the appropriateness of the entire intervention scheme. Previous experience suggests that it is precisely those operationally superior projects that have well-functioning information systems. Thus, breakdowns in the information system serve to warn country officers or headquarters of potential implementation problems.
The other possible reason for poor data quality is that there will be a great deal of subjectivity and bias in a self-evaluation system. This assumption is not valid, especially when built-in monitoring and evaluation systems are viewed as constructive enterprises by the staff in their operational and management roles.
With a well-functioning monitoring and evaluation system, it will be feasible to adopt techniques such as management by exception. Project sites with abnormally good or bad performance will distinguish themselves. The former can be examined in detail to learn characteristics of successful projects; the latter can become the focus of special correction measures. In the final analysis, the type of built-in information system discussed above will promote adherence to principles such as goal-oriented management and evaluation.
Operational Research
Complementary to promoting built-in monitoring and self-evaluation systems, there is a need to identify and test a variety of underlying assumptions that form the foundation of food aid projects. Despite the enormous opportunity cost of project food aid, in both donor and recipient countries, many of the basic premises upon which projects are based have not been examined. This problem stems from the historical, although changing, perception of food as a surplus commodity; it was simply not appropriate to spend significant human and financial resources to guide its disposal. Now that food aid is recognized as a valuable resource, the time has arrived to do the necessary, although belated, research.
The purpose of operational research, like most research, is not defined in terms of improving the welfare of the study community; it is not even intended primarily to determine whether or not a given project achieved its objectives. Instead, the task is to garner generalizable information that expands the boundaries of knowledge concerning the potential uses of food aid. Interest should be focused on hypothesis-testing in an operational environment. The viability and replicability of a given intervention strategy should be assessed. The insight gained is of primary use for policy formulation at the highest levels of programme responsibility.
Operational research should be directed towards identifying and testing a variety of hypotheses in a select few project sites. The precise questions to be asked should be formulated by policy-makers within WFP and Food-for-Peace, with consultation from appropriate experts. The research should examine the assumptions that link each level of the logical progression of project activities and events that are thought to bring about expected impact.
To amplify, there are many assumptions linking inputs to the achievements of outputs for typical feeding projects. These are relatively straightforward. The verification of these assumptions (as well as that inputs were provided and outputs achieved) is the domain of built-in evaluation systems discussed above. Data collection on these processes should be undertaken at all project sites. In contrast, the achievement of project purposes and goals is based on linking less apparent assumptions. For example, an assumption linking outputs to purposes for a supplementary feeding project is that the ration will not be shared with other siblings or substituted for commodities already provided. For a food-for-work project, an assumption linking outputs to purposes is that increased mobility of labour markets and greater integration of commodity markets that result from infrastructure development will increase employment opportunities for the poor and improve household food security. Verifying these assumptions, which form the theoretical basis for projects, is beyond the scope of a built-in monitoring and evaluation system. Rather, they must be the subject of empirical research performed in an operational setting.
A few important qualifications are required. First is that not all assumptions at the purpose or goal level must be tested. Only strategic questions that remain a source of concern or controversy should be addressed. Second is that, in order to test hypotheses, it will often be necessary to rely on data concerning project impact- i.e., objectively verifiable indicators at the purpose or goal level. Ideally, if a project is functioning well, much of the data should be available as part of a built-in information system. However, special surveys or data collection on impact may be required. It is emphasized that the data requirements for operational research should be clearly defined.
To date, a variety of activities undertaken by Food-for-Peace and WFP, as well as other agencies, fall within or near the boundaries of hypothesis-testing through operational research. There are some excellent examples found in studies now under way. These include determining whether the marginal propensity to consume food out of in-kind income differs from that with wage income (6, 7), the development of a simulation model to test the cost-effectiveness of various targeting strategies (26), examining whether and to what extent random measurement error results in under-reporting of project impact (20), and the proposed testing of the theory underlying the recommendation that the choice of food aid commodities be selected on the basis of maximizing the value of the income transferred to the recipient (27). Similarly, past studies such as Project Poshak, Narangwal, and CARE Phase II and III studies were attempts to examine some fundamental assumptions that form the conceptual foundations for the nutritional impact of food aid projects. In all these cases, decision-makers formulated clear compelling questions. Thereafter, they called on researchers to find answers in an operational environment.
The proposed WFP "in-depth" evaluation studies only partially fall within the domain of the type of operations research discussed above. For example, the well-designed Honduras study (8) is framed around determining the impact of a specific project. Undoubtedly, some generalizable knowledge will be gained. But the motivating question is whether the project improves nutritional status. Although important for justifying budgets, a well-functioning built-in evaluation system would do equally well in answering this question. This is especially so given the differing opinions concerning how well the project itself is operating and the high cost and difficulties of performing social experimentation in the field.
In-depth research should instead focus on examining the linking hypotheses. If these can be substantiated, it will enable one to assume, with confidence, that if the services are delivered as planned, and the achievement of outputs substantiated, there is little question that the project will reach nutritional objectives. The time is right to draw up an operational research agenda, mutually agreed upon by donors and recipients alike. This should give co-ordinated action. Doing so will provide the knowledge to use food aid intelligently. The scope and potential for achieving nutritional impact will become more clear. This, coupled with the development of built-in information systems, should form the organizing theme for moving forward on the issue of evaluating the nutritional impact of food aid projects.
REFERENCES
1. C. Williamson-Gray, "Food Consumption Parameters for Brazil and Their Application to Food Policy," Research Report No. 32 International Food Policy Research Institute, Washington, D.C., 1982).
2. O. K. Knudsen and P L Scandizzo, "The Demand for Calories in Developing Countries," Amer. J. Agric. Econ., Feb. 1982.
3. C. P. Timmer and H. Alderman, "Estimating consumption Parameters for Food Policy Analysis,", Amer. J. Agric. Econ., 61: 5(1919).
4. D. Sahn, "An Analysis of Food Consumption and Expenditure Patterns in Sri Lanka,, (International Food Policy Research Institute, Washington, D.C., and Community Systems Foundation, Ann Arbor, Mich., USA, 1984).
5. S. Kumar, "Impact of Subsidized Rice in Food Consumption and Nutrition in Kerala," Research Report No. 5 (International Food Policy Research Institute, Washington, D.C., 1979).
6. M. P. Demousis, M. W. Harrell, and D. C. Franklin, Income Effect of Title II Commodities in Rural Panama (Sigma One Corporation, Raleigh, N.C., USA, 1982).
7. E. Kennedy, "Determinants of Family and Preschooler Food Consumption," Food and Nutr. Bull., 5 (4): 22 (1983).
8. World Food Programme, Research Proposal: Process and Impact Evaluation of World Food Programme Assisted Project Honduras 2523 (World Food Programme, Rome, 1983).
9. E. B. Hogan, "Scope of Work for a Study of Potential Improvements in the WFP Programme System" (World Food Programme, Rome, 1983).
10. W. D. Drake, J. Gunning, A. Horwitz, R. Miller, H. Rice, and G. Thenabadi, "Nutrition Programs in Sri Lanka Using U.S. Food Aid (An Evaluation of P.L. 480 Title II Programs)" (Community Systems Foundation, Ann Arbor, Mich., USA, 1982).
11. J. W. Gilmore, C. Adelman, A. Meyer, and M. Thorne, "Morocco: Food Aid and Nutrition Education," AID Project Impact Evaluation Report No. 9 (US Agency for International Development, Washington, D.C., 1980).
12. S. Blumenfeld, M. B. Flores, M. Mack, M. Norton, and R. Pooley, "The Impact of P. L. 480 Title II in the Philippines, 1970-1980" (US Agency for International Development, Washington, D.C., 1981).
13. R. C. Weisell, "Study on the Nutritional Status Changes between 1979 and 1981 among 1500 School Children in Mafraq and Madaba (Jordan) and the Possible Impact of WFP School Lunch Food Aid in Mafraq" (Food and Agriculture Organization, Rome, 1983).
14. A. Wijga, "The Nutritional Impact of Food for Work Programmes: Report of a State-of-the-Art Study" (Wageningen, Netherlands, 1983).
15. International Food Policy Research Institute (IFPRI) "Proposal for an In-Depth Evaluation of WFP-Aided Food for Work in Bangladesh Including Impact of CIDA and World Bank Financed Appurtenant Structure" (Bangladesh Institute of Development Studies, Dhaka, Bangladesh, and IFPRI, Washington, D.C., 1980).
16. R. Miller and D. Sahn, "Built-in Evaluation System for Supplementary Feeding Programmes - Why and How," in D, E. Sahn, R. Lockwood, and N. S. Scrimshaw, eds., Methods for the Evaluation of the Impact of Food and Nutrition Programmes, Food and Nutrition Bulletin Supplement 8 (United Nations University, Tokyo, Japan, 1984), p. 265.
17. R. Martorell, "Improved Nutrition and Its Effects on Anthropometric Indicators of Nutritional Status," Nutr. Rep. Internat., 21: 2 (1980).
18. J. P. Habicht, "Some Characteristics of Indicators of Nutritional Status for Use in Screening and Surveillance," Amer J. Clin Nutr, 33: 3 (1980).
19. G. Beaton, "Energy in Human Nutrition," Nutr. Rev., 41: 11 (1983),
20. W. D. Drake, R. Miller, and R. Timmons, "Effect of Classification Errors on Evaluation Outcomes: Nutrition Programs Revisited" (Community Systems Foundation, Ann Arbor, Mich. USA, 1984).
21. W. D. Drake, R. Miller, and M. Humphrey, Analysis of Community Level Nutrition Programs, vol. 1 (Community Systems Foundation, Ann Arbor, Mich., USA, 1980).
22. D. E, Sahn, "An Analysis of the Nutritional Status of Preschool-Age Children in Sri Lanka, 198-81" (Community Systems Foundation, Ann Arbor, Mich., USA, 1983).
23. NCHS Growth Charts (US Public Health Service, Health Resources Administration, Rockville, Md., USA, 1976).
24. D. E. Sahn and P, Pestonk, "A Review of Issues in Nutrition Program Evaluation," AID Evaluation Discussion Paper No. 10 (US Agency for International Development, Washington, D.C., 1981).
25. J. B. Habicht and D. Dapice, "An Indonesian National Nutrition Surveillance Systems" (mimeo., Cornell University, Ithaca, N.Y., USA, 1984).
26. R. J. Timmons, "Targeting: A Means to Better Intervention" (Community Systems Foundation, Ann Arbor, Mich., USA, 1983).
27. S. Reutlinger, "More Effective Aid to the World's Poor and Hungry: A Fresh Look at U.S. Public Law 480, Title II Food Aid," World Bank Discussion Paper No. ARU8 (World Bank, Washington, D.C., 1983).