Contents - Previous - Next

This is the old United Nations University website. Visit the new site at http://unu.edu

Evaluation

Mason [5], in his chapter "Basic Concepts for Design of Evaluation during Program Implementation," has created a table to show "Appropriate Data Collection and Analysis for Different Decisions" (see Appendix 6). According to Mason, an evaluation should attempt to answer one or more of the following questions (listed in order of difficulty):

1. Is the intervention performing as expected? (Are programme services being delivered to target groups? Is gross outcome acceptable to management? Where should improvement be sought?)
2. Is it worth continuing the project?
3. Should the project be extended to other locations?
4. Is there a causal link to improved nutrition?

The purpose of this evaluation is addressed basically in questions 1 and 2. The management of the Nutrition Center wanted to determine whether the Nutri-Bus programme could give some indications of significant impact even at an early stage of field operations and identify areas for improvement in the various components. The first data were collected in February 1979. At that time barangay nutrition scholars had been trained by the Ministry of Health and NNC and had been fielded for at least six months in all villages in Leyte and Samar which were candidates for Nutri-Bus services (see Appendix 7). The municipal action officers (MAOs) and the city action officer (CAO) in Tacloban City decided which of the BNS villages, accessible by road, would be the first to receive Nutri-Bus services. Their intention was to provide services to the most needy villages first, but the basis for determining need was imprecise.

1979 Sample Selection

All villages accessible by road in Waray-Waray-speaking Leyte and Western Samar were divided into four groups representing different degrees of intervention:

1. VTRL: Nutri-Bus services started between August and October 1978.
2. VTRS: Nutri-Bus services started between November 1978 and January 1979.
3. BNS: villages selected for Nutri-Bus services but not yet reached.
4. Comparison: villages without BNS.

Sample villages from each of the four groups were selected on the basis of criteria meant to control for the factors known to influence nutritional status and nutrition knowledge (see Appendix 8). The comparison villages did not meet the criteria, were slightly better off in socioeconomic status, and did not have BNS. However, we retained the sample for comparison purposes. All villages with lower socio-economic status that met the other criteria had been assigned BNS. Villages were also classified as urban or rural (see Appendix 9). Table 2 shows the intended and actual data collection design. This report will be limited to the analysis of the data for the rural villages only.

Since village populations were relatively small it was decided that it would be more accurate and less costly to weigh all the pre-school children (12 to 66 months) and interview all the mothers of pre-school children rather than sample within the villages. The upper limit of 66 months is dictated by the Harvard standards. Nutri-Pak is targeted for children 12 months and older. As there were no definite boundaries for many of these villages, no maps, no current census data, and no lists of residents, it was impossible to know precisely how close we came to reaching all. Two methods were used to estimate the number of pre-school children in each village:

Table 2. Nutri-Bus impact evaluation data collection design, 1979

a. The objective was to have a minimum of five villages in each treatment condition. The number in parentheses above shows the actual number of villages for which data were analysed. Note that the urban VTRS condition has only one village, so results for that condition are of very limited value.

1. 1975 village population census data x National rate of population growth (4 yrs) x 20 per cent (approx. proportion of pre-schoolers)
2. The village leaders" best estimate.

The data-gatherers, working together with village leaders, did their best to reach all families with pre-school children. If there was no major discrepancy between results of 1 and 2 above, then villages where 80 per cent or more of 1 was reached were retained in the sample. If there was a large discrepancy between 1 and 2, or less than 80 per cent of 1 was achieved, the villages were discarded from the analysis.

1979 Data Collection

Data were collected by four teams working simultaneously in four villages (see Appendix 10 for team composition and equipment used). Team members other than the measurers were recruited in Leyte and were all native speakers of the Waray-Waray dialect, but not known personally in any of the villages; they were registered nurses and schoolteachers. The measurers were NCP staff members, trained and experienced in taking weights and heights. The teams received one week of classroom training and three days of practical training before data collection was begun.

Team members and measuring instruments were rotated so that each contributed an approximately equal number of measurements to each condition. The rotation also attempted to equalize the pairing of individual team members to minimize the chance of pairs developing somewhat different procedures.

The measurements and interviews took place in central locations in each village. Mothers were notified regarding place and time a few days ahead of the weighing by the village headman and BNS, and again on the day of the weighing by the BNS and some other village volunteers. Those who came to the weighing were asked to encourage their neighbours to come. The women registered with the checker and waited to be called by the measurer: after measurement, they were interviewed.

To try to prevent contamination of the interviewing results, each interview area was roped off with cords and stanchions to prevent mothers from listening to each other's answers. When the interview was completed, they were directed out of the chapel away from the waiting mothers and encouraged to go home to avoid conversation with the latter.

Two potential sources of error in weighing are squirming and flailing children and scales not perfectly calibrated to zero. By weighing the children in the arms of a "surrogate mother," and later subtracting her weight, nearly all squirming and flailing was eliminated (weight of mother and child-weight of mother = weight of child). Zero calibration of the scales becomes less critical when the weight is determined by this method rather than by measuring the child's absolute weight.

Another potential source of error in determining weight for age is the accuracy of the birth date. It was found that:

1. most mothers have either a birth certificate or a baptismal certificate (or both) for their children; or
2. church records could be used to Bet accurate birth dates for most of the children whose mothers had lost their documents.

Using this approach, it was possible to get documentation for nearly 88.3 per cent of the children. A significant difference in per cent standard weight for age was found between the documented and undocumented children, the undocumented having higher per cent weight for age. An examination of the other variables collected showed no statistically significant difference between mothers of documented and undocumented children. Based on the assumption that the difference in per cent standard weight for age might be a result of the mothers' faulty memory of birth dates, children for whom documentation of birth dates was not available were excluded from the analysis.

Data were gathered regarding: length of residence in the village; child's name; household location; name of adult accompanying child; date of collection; time of collection; name of checker; sex of child; birth date of child; source of birth-date information; name of measurer; weight of child plus "surrogate mother" (see Appendix 11); weight of "surrogate mother" alone (see Appendix 11); height of children 24 months or older; length of children under 24 months (height and length are not included in the analysis because the data were less reliable than the weight data, and because weight is a more sensitive indicator - see Appendix 11); names of other preschoolers in family being measured; name of interviewer; time of interview; and mother's answers to interviewer's questions.

1979 Results in Rural Villages: Knowledge/Attitude

The data were analysed using the SPSS package. Interviewee responses were analysed and compared across conditions for five key questions regarding child-feeding practices and nutrition services.

These questions (and the titles on the related graphs) are:

1. What do you give for lunch to a child who is two years old? (Complete meals.)
2. What is a good food to give for a snack to a child who is two years old? (Nutri-Pak for snack. )
3. What has (name of BNS) done for you, your family or your barangay? (Specific BNS activities.)
4. Tell me what you know about Nutri-Pak. (Specific Nutri-Pak information.)
5. How many of the Nutri-Bus television shows have you seen? (Reported VTR frequency. )

Fig. 1. Reported VTR frequency by condition, rural Leyte, 1979.

Although our target children were 12 to 66 months of age, answers of mothers of children 0 to 66 months were included in this portion of the analysis. Mothers of infants were considered part of our audience since their infants would soon be "of age." First it is necessary to look at question 5 to see if the answers reflect the assumed differences in the degree of intervention between the VTRL and VTRS conditions. Figure 1 shows a significant difference in intensity of treatment between the VTRL and VTRS conditions, based on the mothers" report of frequency of viewing of the VTR shows. Among mothers in the VTRL condition, 81.1 per cent reported seeing one or more VTR shows as compared to only 47.6 per cent in the VTRS condition. Acceptable answers for the remaining four key questions were decided on the basis of the following:

• Complete meals: mentions at least one food item from each of the following food groups: carbohydrate, fruit or vegetable, protein source.
• Specific Nutri-Pak: gives correct and specific ingredients, method of preparation, effects on children, or nutritional value.
• Nutri-Pak for snack: names Nutri-Pak.
• Specific BNS activities: names correct, specific activities such as: weighs children, sells Nutri-Pak, gathers mothers for VTR, refers sick to rural health unit, holds mothers' classes.

Fig. 2. Percentage of mothers with acceptable response: complete meals by condition, rural Leyte, 1979.

Fig.. 3. Percentage of mothers with acceptable response: specific Nutri-Pak information by condition, rural Leyte, 1979.

Fig. 4. Percentage of mothers with acceptable response: Nutri-Pak for snack by condition, rural Leyte, 1979.

Fig. 5. Percentage of mothers with acceptable response: specific BNS activities by condition, rural Leyte, 1979.

Fig. 6. Percentage of acceptable response by VTR frequency: complete meals, rural Leyte, 1979.

Regarding the acceptability of answers to these four questions, the differences among conditions are significant, with mothers in the VTRL condition having the highest percentage of correct answers (figs. 2, 3, 4, and 5).

If we compare the proportion of mothers with acceptable answers to reported VTR frequency, as in figure 6, we find a significant difference, with zero frequency showing the lowest score and a frequency of four showing the highest score.

1979 Results in Rural Villages: Nutritional Status

Nutritional status was measured by per cent standard weight for age using the Harvard standards with separate standards for male and female. A three-way analysis of variance (nutritional status by sex, condition, and location) showed significant independent effects for sex and condition (VTRL, VTRS, BNS) and a significant interaction between condition and location. The comparison group was not included in this analysis because it was assumed to have a higher socio-economic status.

The results can be seen graphically in figure 7. Of the three intervention groups (VTRL, VTRS, and BNS), VTRL had the highest nutritional status, VTRS had the second highest, and BNS the lowest. (The comparison group is included for completeness.)

Breaking down these means by age group, we see that for VTRS, BNS, and COMP there was a drop in nutritional status from 36 months to 66 months. However, the VTRL group did not show this drop (fig. 8).

To determine whether the higher means indicated less severe malnutrition or simply heavier normals, we also looked at the data using the Gomez classification scheme. The results revealed significant differences between conditions, the VTRL condition showing the most first-degree and least second- and third-degree, and the BNS the least first-degree and most second- and third-degree, with VTRS in the middle (fig. 9).

Fig. 7. Mean nutritional status by condition, rural Leyte, 1979.

Fig. 8. Mean nutritional status/age/condition, rural Leyte, 1979.

Fig.. 9. Gomez classification by condition, rural Leyte, 1979.

1981 Sample Selection

By February 1981, all Waray-Waray-speaking BNS villages in Leyte accessible by road had had Nutri-Bus operations for at least 18 months (see Appendix 12). "No-intervention" matching control villages were not available in Leyte.

An effort was made to find groups based on degree of intervention. All Nutri-Bus villages meeting the same criteria used in 1979 (Appendix 6) were assigned a score based on daily field reports from the Nutri-Bus staff regarding services rendered. Scores were based on number of VTR showings, number and type of supervisory calls to the BNS, and amount of Nutri-Pak supplied, with VTR shows weighted most heavily. Those villages with a score of 60 or above were assigned to the "high-service" group and those with a score of 30 or below to the "lowservice" group. Villages were assigned to six groups representing "high" and "low" service, urban and rural classification (Appendix 9), and locations in the provincial capital area (Tacloban) or the rest of Leyte. Villages that qualified were assigned to complete the design shown in table 3. Again, for this paper we will limit the report and analysis of results to the rural villages.

As in 1979, the objective was to collect data on all the pre-schoolers in each sample village. Using the same methods described earlier to estimate the pre-school population, the percentage actually measured ranged from an average, by condition, of 88.1 per cent to 107.6 per cent of the estimate.

Table 3. Nutri-Bus evaluation data collection design, 1981

1981 Data Collection

Data were collected using the same techniques, types of teams, and equipment described for the 1979 data collection.

1981 Results for Rural Villages

Interviewee responses were analysed and compared across conditions for the same question used in 1979 to determine mothers' report of number of VTR shows attended. To support the assumption that the villages were grouped by degree of intervention, a significant difference would be expected for reported frequency of VTR shows between "high" and "low" service villages. As shown in figure 10, that assumption is not supported by the data.

Fig. 10. Reported VTR frequency by condition, rural Leyte, 1981.

Table 4. Nutri-Bus evaluation data analysis design, 1979 v. 1981

a. Same villages in 1979 and 1981.
b. Some villages the same. some different, in 1979 and 1981, but all selected by the same criteria.

The "high" and "low" service groups were therefore abandoned on the assumption that the field reports were not sufficiently accurate to be used. The data in 1981 were compared to 1979 using the design shown in table 4.

1979 v. 1981 Results for Rural Villages: Knowledge/Attitude

For the rural villages common to both the 1979 and 1981 sample, there was a significant improvement in 1981 in the proportion of mothers describing complete meals to give to a two-year-old child (fig. 11).

Both the "specific Nutri-Pak information" and the "specific BNS activities" questions showed higher scores, but these differences are not significant (figs. 12 and 13). However, the proportion of mothers with acceptable responses has increased to 76.3 and 87.1 per cent, respectively, in 1981. These may be approaching a ceiling that cannot be much improved, considering that the project does not reach all mothers. In any event, the scores show that the 1979 levels of knowledge have been well maintained.

Fig. 11. Percentage of mothers with acceptable response: complete meals by year, common rural villages.

Fig. 12. Percentage of mothers with acceptable response: specific Nutri-Pak information by year, common rural villages.

Fig. 13. Percentage of mothers with acceptable response: specific BNS activities by year, common rural villages.

Fig. 14. Percentage of mothers with acceptable response: Nutri-Pak for snack by year, common rural villages.

The "Nutri-Pak for snack" question showed a decrease in the proportion of mothers with acceptable answers, but the difference is not significant (fig. 14). This is not a surprising finding, since the availability of Nutri-Pak had greatly decreased and Nutri-Pak was not available in most villages in 1981.

1979 v. 1981 Results: Nutritional Status

The mean nutritional status of the pre-school children in the rural areas improved significantly from 1979 to 1981, as measured by per cent standard weight for age, using the Harvard standards (fig. 15).

Looking at the means of the common rural villages individually, it can be seen that all but one improved. The village that did not improve decreased by only one tenth of one percentage point (fig. 16).

Disaggregating the average mean nutritional status of the common rural villages into age groups shows that all ages had higher means (fig. 17).

Applying the Gomez classification to the common rural villages also showed significant improvements, with children moving from second- and third-degree to normal and first-degree categories (fig. 18).

Fig. 15.. Mean nutritional status by year, common rural villages

Fig. 16. Percentage change in mean nutritional status, by common rural villages, 19791981.

Fig. 17. Mean nutritional status/age/year. common rural villages.

Fig.. 18. Gomez classification, common rural villages.

1981 Cost-effectiveness Results

The costs for the Nutri-Bus programme were calculated for the year 1981, as a representative year of operation. All costs associated with a typical year of operation were used.

Costs included were:

• production of communication materials (print, VTR, etc.);
• personnel (including management);
• training of all levels of associated personnel;
• vehicles and equipment maintenance and spare parts;
• operating costs (travel, materials, garaging, local office space, etc.);
• monitoring and evaluation.

Costs not included were: opportunity costs;

• services of consultants from USA;
• use of NCP building and overhead; certain one-time start-up costs;
• services of the BNS village volunteer.

Opportunity costs were not included because we have no way to attach a meaningful value to them and there are no such obvious costs. Including the costs of the US consultants would be misleading since (a) they are being phased out of the programme; (b) the purpose of their inputs was to assist local personnel in the design and start-up of a programme that could be effectively continued without outside consulting support; and (c) they operate as advisors only and do not take the place of any local person - therefore no further costs will be incurred upon their departure.

The building and overheads of NCP exist independent of the Nutri-Bus programme; however, certain management and media production personnel have their offices at NCP. Our best estimates indicate that if a prorated share of these NCP costs were added, the individual cost estimates would increase by less than 10 per cent.

Since our concern here is with projecting cost-effectiveness to assist decisions regarding the future of this programme (as opposed to duplicating the programme from the beginning), certain start-up costs were not included, such as pilot testing and inefficiencies of very early operations. The objective was, as noted earlier, to estimate the costs of a typical year of continuing (and improving) operations.

The factors noted above, which were used for calculating the costs, represent the inputs to the programme. These can be related to several possible levels of output or effectiveness. Zeitlin [4] has listed possible indicators of levels of output in nutrition communication programmes for purposes of a cost-effectiveness analysis. Table 5 is based on her list.

Various estimates of cost-effectiveness can be calculated using the total programme costs as the constant numerator and each of the indicators in table 5, in turn, as the denominator. We have calculated cost-effectiveness based on one indicator at each level of effectiveness shown in table 5. We have combined knowledge, attitude, and reported behaviour change into one indicator - change in what the mother reports that she gives a two-year-old child for lunch.

For ease of calculations and projections, we have calculated the cost-effectiveness on a per Nutri-Bus basis (see Appendix 13 for formulas). The total programme costs in 1981 divided by the number of Nutri-Buses in the same year gave us an annual per bus cost of P140,171 or US$16,888. (All exchange rates given here and later in this section are calculated at the 1981 bank exchange rate of US$1 = P8.30.) On average, a Nutri-Bus serves 90 BNS areas (a BNS area is usually one complete village). So, the average annual cost per BNS area served is P1,557 (US$188).

Table 5. Some possible cost-effectiveness indicators for nutrition communication programme and annual unit costs for Nutri-Bus project

a. Children whose mothers reported giving changed diet.
b. Cost per year over a two-year period for change of second" and third-degree to first-degree and normal.
c. Children whose mothers reported attending VIR.

The main objective of the Nutri-Bus project is to provide both curative and preventive services to reduce malnutrition among pre-school children. According to census figures, the 12to 66-month-old pre-schoolers, who are the ultimate target of the programme, constitute 14 per cent of the total population, or about 140 per BNS area. That gives us an average annual cost per target child in the catchment area of P11.12 (U$S1.34). Or, from another point of view, there are about 100 mothers of pre-school children in each BNS area. The average annual cost per target mother to whom the services are offered is P15.57 (US$1.88).

Of the total target mothers, approximately 66 per cent reported seeing VTR shows in the 1981 data collection. Using this as our definition of target mothers participating in the programme, that gives us an average annual cost of P23.59 (US$2.84) per target mother directly participating or P16.85 (US$2.03) per target child whose mother directly participates.

For the indicators referring to effects on the target groups, our data were collected before and after two years of intervention, so we have calculated on a two-year basis and divided for annual costs (recognizing that effects are probably not linear over the two-year period). Referring to the mothers' report of what they feed for lunch to a two-year-old child and using the 1979 comparison group as the baseline, in 1981 there were 26.8 per cent more mothers reporting complete meals. Assuming this change was brought about by the Nutri-Bus programme, that gives us a cost of P116.19 for two years per additional mother or P58.10 (US$7.00) per mother per year. The costs are P83 for two years or P41.50 (US$5.00) per year for each child whose mother reported giving the improved diet.

And, finally, regarding nutritional status, using the findings of the 1979-1981 study in Leyte as an estimate of project impact, the cost per child "removed" from second- and third-degree malnutrition (using Harvard standards and Gomez classification) over the two-year period is P318 (US$38) or P159 (US$19) per year.

Contents - Previous - Next