Contents - Previous - Next

This is the old United Nations University website. Visit the new site at http://unu.edu

Implementing a built-in evaluation system

The previous sections of this report present the arguments, principles and guidelines for built-in evaluation. If the guidelines are followed, we have little doubt that creative programme personnel can formulate internal evaluation systems which will promote iterative experimentation with programme options in the field and, in turn, facilitate the attainment of desired impact. Nevertheless, because built-in evaluation procedures as described above are so rarely applied, we feel it necessary to expand on the specifics of such a system to help others initiate prompt implementation.

Therefore, in this section of this chapter, we will outline one possible starting point for a built-in evaluation system. We emphasize that this is only one of many possible options for internal evaluation systems. Furthermore, we stress that any such system must be contextual. That is, any built-in evaluation system is a function of: (a) the underlying theory or framework which describes how the project intends to achieve impact, (b) the socio-economic environment setting the context of intervention, and (c) the availability of capital and human resources to administer the system. While our example is predicated on the "typical" supplementary feeding programme, it is important to recognize that no actual programme is "typical."

The Hypothetical Feeding Programme

A pre-eminent type of feeding programme. and the focus of this workshop, involves the provision of a food ration to vulnerable groups - mothers, infants and young children. The goal of such a programme is to improve the nutritional status of the participants. The role of the food ration is central in this type of programme, for its direct effect on nutrient intake and/or its indirect effect as an incentive for participation in complementary activities (immunization programmes, educational programmes, and so forth).

For any built-in evaluation system to assess the degree to which the goal of improved nutritional status is attained, it is necessary, as an absolute minimum, to measure nutritional status. Furthermore, because of the central role of food supplementation in the intervention, it is necessary, as an absolute minimum, to monitor the stocks and flows of food commidities. In a "straight" feeding programme (one in which food distribution is the only or primary component), these two indicators - nutritional status for impact and commodity stocks and flows for process - may be all that is needed. Ancillary services would require additional impact and process variables. For example, an immunization programme might require the incidence of selected diseases as an impact indicator and the number of vaccines administered as a process indicator. Similarly, major causes of poor nutrition in the environment, which are exogenous to the project services, might suggest other variables to be included in the system to help distinguish programme effects from the secular changes which take place in villages themselves. For example, in an area where shortfalls in agricultural production lead to nutritional deficiency because of resulting food scarcity and employment cutbacks, it may prove useful to monitor agricultural production or food prices.

In the example being developed in this section, it is assumed that the food distribution is directed at preschoolers and is accompanied by an immunization programme. Furthermore, we assume that the distribution is handled through a medical facility where easily diagnosed diseases are identified (e.g. diarrhoea) and treated using simple techniques (e.g., oral rehydration salts). We will ignore environmental factors. However, the analyst who reaches conclusions on the basis of process and impact indicators should acknowledge the role of those factors when presenting evaluation findings.

The Data Collection: Forms and Procedures

The primary impact indicator for this system must be a measure of nutritional status. With regard to the selection of the best indicators of nutritional impact, other chapters of this publication discuss in detail many of the available options. We will not reiterate the arguments presented in those chapters. For simplicity, it is assumed that anthropometry is selected for our hypothetical feeding programme. We further limit consideration to weight-for-age. The fact that it is the most common indicator available throughout the world today, coupled with the relative ease of measurement protocol and interpretation of the data, commend the use of weight-for-age in a built-in system.

To determine the ratio of a child's weight to an age-specific reference standard, it is necessary periodically to record the weight of each child and his/her age. The most common device for recording such data is some form of growth chart or "road-tohealth" card which features a graphic representation of a child's growth over time. This device is particularly useful if it is used to demonstrate a child's progress to his/her mother. Figure 14.1. is a sample "road-to-health" chart. This particular chart is based on the NCHS-CDC standard (8) and the Gomez classification; that is, Grades I, II and III malnourishement are taken to be 85 per cent, 70 per cent and 60 per cent of the reference standard (9). "The normal line", the 50th percentile of the NCHSCDC standard, is the imaginary curve between the top two curves on the graph. It is the average of the standard for boys and girls as reported by the CDC.

The inclusion of this chart in our example is not to suggest it as a model to be duplicated elsewhere. While we feel the NCHS-CDC standard is applicable in most, if not all countries, the use of a WHO type chart (10) which does not distinguish among Grades I, II and III may be equally appropriate. In either case, the focus of the nutrition education and growth monitoring effort with the individual mother should be in assuring the child gains weight. Nevertheless, the different grades represented in the chart portrayed are useful in the context of examining age-specific changes in the nutritional status of populations through a series of crosssectional surveys. This, combined with the popular use of a Gomez-type classification, make it an appropriate growth chart on which to base our illustrations.

The chart shown in figure 14.1. (see FIG. 14.1. Child's Growth Chart) also contains space for recording additional data about the child. Fixed items (name, sex, date of birth, and an identification number) are recorded on the top of the form. Monthly data on the occurrence of disease is recorded just below with room enough for two entries per month. A record of receipt of ration is kept at the bottom of the weight chart. Below that, a record of immunization history for the child is maintained, as is a record of the administration of special treatment. This type of information is often kept on the back of a growth chart. Doing so allows the vertical axis to be lengthened, thereby easing the task of filling out and interpreting growth curves. Nevertheless, the important point is that the "roadtohealth" chart should be explicitly designed to contain all or as many of the data elements for individual preschool children necessary for the determination of (a) impact on a given child's health and (b) impact on the community as a whole, which may be determined through the process of aggregation of individual charts. Before considering how these community-level indicators are derived, let us look at the basic form for recording process data, the stock and flow form.

Month: _________________________
Village: ________________________
Filed By: _______________________

Number of children served:

_________ x (ration per child in kgs.)
_________ = Food distributed

STOCK BALANCES

IMMUNIZATIONS

Unlike the weight chart, which is child-specific. the Monthly Inventory Report is maintained for the food distribution center. In short, it is a summary of the stock position for all programme inputs, at both the start and the conclusion of a designated reporting period (e.g., one month). In the sample form shown in figure 14.2. (see FIG. 14.2. Monthly Inventory Report), there is an inherent set of checks and balances. Stock positions at the end of each month are determined by subtracting the quantities of each programme input administered from the starting balance (plus any new stock received during the month). Similarly, at the end of each month, the stock position should be "checked" by counting the stocks on-hand. Differences are indicative of losses due to spoilage, theft and/or poor accounting.

The form shown assumes that the managers of the distribution center order additional quantities of inputs, as needed. We are aware that most supplementary feeding programmes do not allow variable shipments of inputs on a monthly basis. However, in the ideal case, such flexibility would enable management to react to the specific problems in their areas by increasing (or decreasing) the quantities of inputs, on hand, as needed rather than curtailing distribution (or dumping valuable commodities) because stocks are not in balance with requirements. This form can serve as an order form for that purpose. The flows are reported as well as the stock positions so that supervisory personnel, at higher levels can determine if commodities (particularly medicines) are actually being used. For example, if the opening and closing balances indicate identical amounts of oral rehydration salts in stock and no new inventory is added. one can guess that the staff at the center is not diagnosing diarrhoeal disease with sufficient care, or not being sufficiently aggressive in encouraging the use of oral rehydration therapy.

To complete the Monthly Inventory Form, it may be necessary to maintain continuous records on some elements in the form throughout the month. To compute the number of children vaccinated against polio in a month. one could page through all the weight charts and count the children who received polio vaccine during that month. However, it would be far easier to use a "work-sheet" (blank copy of the form) to keep a running tally on polio vaccinations administered. Such a tally can be kept by making hachure (or check) marks in the appropriate box whenever a child is vaccinated. Then, at the end of the month, one need only count the marks.

Forms similar to the "road-to-health" chart and the Monthly Inventory Report are fairly common in the field, but by no means universal. Typically, however, the use of the "road-to-health" chart is limited to diagnosis of the individual child while the record of stocks and flows is used to audit past performance of the delivery system. Neither form is used for programme management. By adding some analysis procedures to those carried out already at the distribution center, these data can be made to "come alive."

Consider first the individual records of nutritional status. By combining the nutritional status of individuals into a summary statistic, the status of all participants (comprising a village or other designated target area) can be estimated. A word of warning is in order. If the generation of a summary statistic describing a community's nutritional status is done improperly, the results can be inaccurate and, in some cases, quite misleading. To avoid this, summaries of nutritional status must be agespecific. That is, four-year-olds should be compared only to other four-year-olds, and so forth. Also, in the typical feeding programme, where entry and exit into the programme is an ongoing process, summaries of nutritional status must account for the changing composition of the target population.

The necessity of accounting for age is described more fully elsewhere (11). In brief, in most situations where malnutrition arises due to chronic food shortage and/or environmental hardship, most children pass through a transition period of maximum risk following weaning. In the absence of intervention most children will "score" worst in any computation of their nutritional status during this high risk period (often between 18 and 24 months of age). They will improve naturally as they continue to grow older, passing out of this vulnerable stage of life. Therefore, in the aggregate, two-year-olds often show higher rates of malnutrition that four-yearolds. This natural improvement must be accounted for in any analysis of the change in nutritional status of a community over time.

The entry and exit of participants can cause marked changes in the nutritional status of the community as a whole, if either the drop-outs or the new enrollees, as a group, differ in their nutritional profile from the rest of the community. The extreme example of this is the death of third-degree malnourished programme participants. The number of malnourished children receiving treatment decreases accordingly. Because new enrollees are typically younger than programme "graduates," their nutritional profile is different, due to the aging phenomenon described above. Thus, it is essential to account for these differences in any analysis of the change in nutritional status of a community over time.

There are a variety of statistical techniques which can be applied to account for entry, exit and aging in an analysis of community nutritional status. For its simplicity and practicability in a field setting, we recommend an analysis based on a graphic representation of change over time rather than on a statistical analysis. To illustrate this concept of using the graphic approach in an operational context, we will now describe tables and plots that we refer to as "characteristic curves." One can learn quite a bit about the characteristics of a community from the shape of the curves (11).

Figure 14.3. shows a sample form that may be used in computing the numbers needed to draw the "characteristic curves" depicted in figure 14.4. It is stressed that for each programme the actual forms and field protocol will be different and shall be developed accordingly. In this example the summary data in figure 14.3. are collected every three months. Individuals may be weighed more frequently than prescribed by the aggregation procedure; for example, children may be weighed every month even though the data are aggregated on the Nutritional Status Summary form every three months. Decisions and protocol in this regard are contextual. Just as forms must be tailored to the individual programme, so too should measuring and recording procedures. It is cautioned that any data must be interpreted within the context of changing seasons. Quarterly curves become far more useful in the second and third years of a programme because seasonal trends will become known.

FIG. 14.3. Nutritional Status Summary

VILLAGE:
MONTH:
FILED BY:
CHILDREN WEIGHED IN BOTH THIS AND LAST QUARTER

FIG. 14.4. Sample Characteristic Curves and Data on Malnourished Children (the image)

...and the tables

Quarter ending March 31

Quarter ending June 30

Malnourished = less than 70 per cent of the NCHS-CDC standard.

The data in figure 14.3. consists of the count of children by age group, in each nutritional category. The village level worker may use a blank form filled out by making hachure marks, as for the Monthly Inventory Report, continuously during each weighing session to facilitate preparation of the top two sections of the Nutritional Status Summary form.

In the top section of the form the nutritional status of continuing participants is recorded. For new entrants, a new individual weight chart is prepared initially, since these children were not participating in the previous weighings. The data from these charts are aggregated to fill in the middle section of the Nutritional Status Summary. Drop-outs are those not weighed in this quarter who were weighed in the previous quarter. They consist of graduates from the programme due to age, as well as deaths, out-migrants, and so forth.

Drop-outs cannot be counted using check marks but must be found by paging through all of the weight charts. This can be done by going through some sort of looseleaf notebook which contains all the charts, if they are kept at the center. Alternatively, if charts are kept at home with the mother, a roster of all participants can be maintained with space reserved for a notation that a child participated in any given weighing period. Children not checked in any given period are "drop-outs." Such a roster can be an invaluable aid if a census of all potential participants in a geographical area is made in order to screen out those children not needing help. Also, if no-shows are noted on that roster and reasons for not showing determined through community outreach, it becomes possible to estimate mortality and migration rates.

Depending on the size of the distribution center, it may be necessary to use larger categories than those on the sample form. This can be done, for example, by combining Grade II and Grade III or using age categories spanning nine months or eighteen months. Similarly, it is possible to use smaller categories, e.g., six month age groups. This would be preferable.

In a similar vein, if a Grade I, II, or III classification of malnourished is not used, as would be the case with the WHO growth chart, figure 14.3. may have notations such as "gained weight," "stayed the same," and "lost weight" in lieu of the notations Grade I, II and III. The key to the definition of the categories is their stability. One wants to avoid the situation where a change in the status of only one or two children can markedly alter the percentage of children in any age group declared to be malnourished.

With the data in this table, it is possible actually to draw the "characteristic curves" referred to above. These are graphs with the percentage of malnourished children plotted on the vertical axis and age plotted along the horizontal. Figure 14.4. is a hypothetical example of a set of "characteristic curves" for a group of 200 children who participated in a feeding programme and were weighed in successive quarters. These two curves show an improvement in the nutritional status of children in all age categories, except the youngest. The increase in malnutrition among the youngest should not be surprising because these children are fast approaching the age of maximum risk. Note, the total number of children has decreased by six- the number "graduating" during the three month period. We assumed no new enrolments.

Once again, numerous alternatives to the type of characteristic curve in figure 14.4. should be considered. For example, another possibility would be to use "failure to gain weight," and the corresponding percentage, in place of the concept of malnourished (which, in our example, is defined as less than 70 per cent of the NCHSCDC standard).

The successful application of the concept of the "characteristic curve" requires the selection of an appropriate definition for malnutrition and the selection of the best set of curves to be plotted on a single set of axes. We cannot offer a unique set of guidelines for making these selections. Clearly, various contextual factors must guide the decision-making process for any given programme.

Finally, there are a variety of possibilities for plotting different sets of curves, only one of which is illustrated in figure 14.4. Any, or all, might be suitable for a given analysis. For example, one might wish to plot not only the percentage malnourished at a single point in time, but also the percentage in each grade of malnourishment. In circumstances where there is considerable entry and exit in a programme, one might choose to plot the curves for new enrollees and/or graduates to facilitate comparison of their nutritional profile with that of continuing participants. Over a year's time, one might also plot the curves for all four quarters to detect seasonal trends.

A particularly useful pair of curves are the two curves set a year apart. If twelve month age categories are used, the children in one category during the earlier year are the same cohort (the exact same children) as the children in the next oldest category in the latter year. In this instance, once can see both the progress of a particular group of children and the progress of an age grouping in a single graph. (An additional tool for observing the progress of a fixed group of children is the transition matrix- an array of numbers where each element is a count of the children starting out in one grade of malnutrition [given by the row of the matrix] and "moving" to another grade [marked by the column of the matrix!.)

The form in figure 14.3. contains the raw data for any number of "characteristic curves." One need only compute the relevant percentages and plot the curves. Learning to interpret the curves can be difficult, at first, but interpretation becomes relatively straightforward with experience. One must remember, however, to consider changes in the local economy, the local infrastructure, and such things as the initiation of other social programmes before attributing changes in the nutritional status of the community to the intervention alone.

Use of the data for overall programme management

Thus far, we have considered a village-oriented data system consisting of a record of the stocks and flows of programme inputs and a computation of community-level impacts from data gathered on individual participants. Managers of village-level operations can make use of this data in making decisions concerning their own day-today operations. If the data is passed up to higher levels of management within a large-scale programme, the data can help guide decision-making at those supervisory levels as well. If the summary forms (figures 14.2., 14.3. and 14.4.) are transmitted from the field to management centers, it will be unnecessary to transmit the "road-tohealth" charts on individuals.

One useful way to accomplish higher-level management in the system is through aggregation or consolidation of the data for larger areas or regions. For example, suppose a regional supervisor is responsible for programmes in twenty villages. By using "characteristic curves" made up from the totals derived by adding the numbers on the Nutritional Status Summary forms for all twenty villages, the supervisor can delineate trends in his/her own territory. Of course, the identification of such trends is most useful at a level of management where programming decisions can be made. It does no good for a supervisor to discover that the critical age for preventive feeding is 20 months, if he is forbidden by fiat from above to feed children less than 2 years of age.

A second and, perhaps, more useful way to use data describing the performance of individual distribution centers is to compare the events and trends among those centers to find the exceptional cases. This process was described earlier as the application of the principle of management by exception. If a regional manager recognizes that one village is not using oral rehydration salts (as reported on the stock-flow form) and is showing the smallest nutritional improvement (as shown on the "characteristic curves"), he/she can take steps to correct the procedures in that village or, at least, inquire into the reasons for the lapse in treating diarrhoea at that location. Similarly, consider the situation where progress at one distribution center is extraordinary. If upon further inquiry it is found that the local staff on their own have instituted an educational programme, on the importance of eliminating sharing of the ration within the family, this suggests the need for similar innovative education programmes elsewhere.

There is always some risk involved in providing quantitative information to decision-makers. Poor managers tend to rely solely on the "numbers" rather than on first-hand observation of the processes they control. We are not suggesting that the quantitative information replace sound judgment based on a host of supporting information. In the ideal situation, the analytic results will act as signals which trigger additional in-depth inquiry by good management. Then, the data and the results of such inquiry can be used together to promote better programme operations and design.

Finally, the data base created by the type of system outlined above is ideal for both internal and external evaluation work. Of particular interest is the potential of such a system for promoting analytic responses to evaluation research and/or programme design questions. The possibility for field testing propositions concerning intervention which, heretofore, have been debated in the abstract by the "experts" is unbounded.

An example of such an issue is the ration size for a given programme. By altering the ration size in selected villages, the built-in evaluation system will generate data, automatically, on the impact of the new ration size. Historical trends will exist for the selected villages facilitating longitudinal analyses. Simultaneously, trends in other, similar villages can be mapped in parallel with those in the selected sample. Thus it is possible to set up classical experimental designs without having to initiate special data gathering activities.

References

1. David Sahn, Principles and Guidelines for the Evaluation of P. L. 480 Title II (Food-for-Peace Programmes and Policies), (Community Systems Foundation, Ann Arbor, Michigan, 1981).
2. Roy I, Miller and William D. Drake, "A Revisitation Two Years after Programme End." part 11 of The Promotora Programme in Candelaria (Community Systems Foundation, Ann Arbor, Michigan, 1980),
3. T.D. Cook and D.T, Campbell, Ouasi-Esperimentation (Houghton Mifflin, Boston, 1979).
4. Donald T. Campbell and J.C. Stanley. Experimental and Quasi-Experimental Designs for Research (Rand McNally, Chicago, 1966), pp, 5-6.
5. David E. Sahn and Robert M. Pestronk, "A Review of Issues is Nutrition Programme Evaluation", A,l.D, Programme Evaluation Discussion Paper No. 10, (USAID, Offices of Nutrition and Evaluation, Washington D.C,, 1981), pp, 42-51,
6. Roy I. Miller and David F. Pyle, Toward a Monitoring and Evaluation System for P.L. 480 Title II Maternal Child Health Programmes in India (Community Systems Foundation, Ann Arbor, Michigan, 1981).
7. James E. Austin, "The Perilous Journey of Nutrition Evaluation," American Journal of Clinical Nutrition, 31: 2327 (1978),
8. Peter V.V. Hamill, Terence A, Drizd, Clifford L. Johnson, Robert B, Reed and Alex F, Roche, NCHS Growth Charts, 1976, Monthly vital statistics report, vol, 25 No. 3 supplement (National Center for Health Statistics. US Department of Health, Education and Welfare, Rockville, Maryland, 1976).
9. F, Gomez, C,R. Cravioto and J. Frenk, "Malnutrition in Infancy and Childhood, with Special Reference to Kwashiorkor," Advances in Pediatrics, 7 (1955).
10. WHO, A Growth Chart for International Use and Child Health Care: Guidelines for Primary Health Care Personnel (World Health Organization, Geneva. 1978),
11. William D, Drake, Roy I. Miller and Margaret Humphrey, Final Report Analysis of Community-Level Nutrition Programmes (Community Systems Foundation, Ann Arbor, Michigan, 19801, pp. 94-97

Contents - Previous - Next