Previous - Next
This is the old United Nations University website. Visit the new site at http://unu.edu
DENNIS R. FOOTE
Institute for Communications Research, Stanford University,
Stanford, California, USA
INTRODUCTION
There is a growing interest among developing-country governments in implementing and evaluating purposive communication projects, that is, activities in which communications media and processes serve as a major component of a project designed to enhance education and/or social service delivery. Evaluation of these activities is of paramount importance, because high levels of governmental and individual resources are involved and because the cumulative knowledge about how to design communication interventions effectively is not extensive.
This paper suggests a model for planning evaluations of such projects. The model draws heavily on systems theory in organizing the disparate components into units that can be worked with. The intent of having a comprehensive model is not that every project should be exhaustively evaluated, but that planners of such activities should choose from the comprehensive model those portions that are appropriate to their particular circumstances. A side benefit is that paying attention to all aspects of evaluation during the project planning phase often facilitates more effective and focused efforts. The general outline of the model involves the following stages, although in an ongoing project these stages are not independent or necessarily sequential. Rather, they represent processes that are interrelated and interactive.
Examples from current projects in Honduras and the Gambia in which this methodology is applied will illustrate the discussion. These projects use integrated campaign strategies to introduce and promote changes in behaviours related to diarrhoea, particularly the use of oral rehydration therapy (ORT). The projects, funded by USAID in co-operation with the Ministries of Health in each country, are called Mass Media and Health Practices, or MMHP for short.
One of the major causes of infant mortality in developing countries is diarrhoeal disease. It occurs at significant levels in practically every country, facilitated by undernutrition and poor sanitation. Death is typically caused by dehydration - loss of fluids and electrolytes - before the child's natural defences can defeat the cause of diarrhoea. Less acute consequences malnutrition and waste of human material resources- are more common. In addition to the goal of improving nutritional and sanitary practices in the home for prevention of diarrhoea, an inexpensive oral rehydration therapy (ORT) that treats the dehydration by replacing fluids and electrolytes has recently been developed. Oral rehydration therapy was chosen as the health innovation to be presented in the campaigns in Honduras and the Gambia because its universal relevance meant that successful campaigns in these countries could serve as direct models for efforts in many other countries.
Household-based therapy offers an opportunity for a massive extension of coverage and a significant improvement in outcome statistics. At the same time this raises issues of compliance and the necessity for appropriate and convincing education materials, in addition to changes in the health delivery system.
MODEL FOR EVALUATION PLANNING
This is a need, then, to plan, implement, and evaluate such projects in a way that will lead to understanding and practicality in health campaigns such as those aimed at reducing infant mortality caused by diarrhoea. Taking a systems approach, we can view a campaign intervention as an input-output process altering the state of a system subject to constraints.
The General Model
Figure 1 presents an overview of the evaluation model. Taking a systems perspective, this model shows that before any intervention, there exists: (a) a prior state (of people, their family, their community, the environment, the economy, etc.) that is the baseline to which ongoing and final evaluation measurements are compared; and (b) constraints existing in the system that affect both how the population can interact with the intervention, and how the intervention activities can actually be accessed by the population.
Fig. 1. Overview of the evaluation model.
The left side of figure I shows this prior state, systems constraints, and the intersection of the system constraints and intervention inputs. These three components interrelate and feed into the process component. In the component, the subjects access the inputs within the context of prior states and constraints, and something happens. We hope that the goals of the project are achieved, and indeed the evaluation is focused upon and organized around these substantive outcomes. The evaluation project will attempt to determine and understand which of the substantive outcomes occur. Perhaps more people become informed, change their attitudes, alter their behaviour, become more healthy. Perhaps they only do some of these. Perhaps they do none of them. Perhaps they have indeed changed, but cannot perform the behaviours they would like to. Or, perhaps they behave differently, but the evaluation effort cannot detect this. Again, the system will contain constraints that interact with certain intended outputs from the process component. For example, perhaps a family learns new approaches to hygiene and wants to perform them, but cannot because their personal, cultural, or economic conditions prevent them from buying soap or using suffficiently clean water. Thus system constraints "block" the progression from process to output.
The subsequent condition of the population constitutes a new or post state that must be measured to detect change from the prior state. This post state, which includes outputs from the process components, consists of the information, attitudes, behaviours, and health status of individuals as well as many of the conditions measured in the prior state component. Many of the values of the variables measured may not have changed; some would not be expected to change. And some new aspects of the system may have been introduced, such as new health communication infrastructures or different administrative procedures.
A final note to this overview concerns the MMHP evaluation analyses. Perhaps the most important analytical aspect of an evaluation of an intervention as complex as the MMHP is the need to consider, measure, and assess the effect of the major variables that help explain why certain outputs occurred, or why certain others did not. Thus, the evaluation model takes care to detect and measure constraints as well as the effect of different media, different messages, at different times, in different regions, and with different audiences. It is very important to specify the relative efficacies of implementation activities, relative receptivity of the target audience, cumulative efforts over time, and what constraints and obstacles simply prevented intervention efforts from even reaching the audience or from being utilized in the process component.
Specifying the Prior State
The prior state of the environment, the community, the families, and the individuals, can be broken down into clusters of variables. These variables can be looked at in terms of their content area, or in terms of where they fit in the entire intervention/ evaluation process. For the MMHP evaluation, the clusters of variables by content include:
From these variable clusters, depending on the inputs, target audiences, regions, timing and constraints, particular questionnaire or other data collection instruments can be generated.
Specifying System Constraints
Current development theory and research tells us quite clearly that the socio-cultural/economic structure of the target audience plays a very important role in any social intervention, and particularly in communication interventions, because of the difficulty in translating new information into firmly held behaviours. Other more practical constraints operate as well, such as whether the materials planned by the project are actually broadcast with the planned frequency, or whether the target audience members have access to particular mass media.
The evaluation model shows that it is necessary to detect and measure system constraints that may block or obstruct the progression from inputs to outputs. Major categories of constraints include:
Any one constraint or set of constraints can affect delivery of inputs, access to delivery inputs, or resistance to delivery inputs or can overwhelm the potential effects of delivered inputs. Once the linkages between intervention inputs and potential outputs are specified theoretically (as in the process component) it becomes crucial to specify any linkage obstructions. We need, then, to make a distinction between planned inputs and engaged outputs, and the constraints that lie in between. This process is portrayed in figure 2.
Between the intervention inputs at the bottom of figure 2 and the actual interface (here called "engaged inputs") lie the possible constraints to full delivery of the planned inputs. For example, a series of radio spots with a given frequency of broadcast may be planned, but the broadcasters do not receive the scripts in time or choose not to broadcast all spots according to schedule; the result is termed "real inputs." In the Gambia, 66 per cent of household compounds have at least one radio receiver; in those compounds 75 per cent of the women listen to Radio Gambia, which delivers the MMHP spots. Compare this engaged input to the 3 per cent literacy rate in the Gambia that would prevent any noticeable engagement via print. These real inputs may not be "engaged" because the workers are at work during the broadcast, and miss the "planned inputs." These "engaged inputs" must be considered the true inputs in analysing change and post-state measurements. A detailed accounting of "engagement" would include which engaged input is received at which process step by which individual in which region.
Specification of Outcomes
Measures of success in a health communication project may include a wide variety of outcomes. Typically, a project would hope to detect improvements in cognition, behaviour, and final health status of the target infant population and beneficial changes in the entire medical-social system. The general theory behind such interventions postulates a chain from inputs such as a variety of treatments of infrastructure change, through constraints that may be environmental (season, local conditions), cultural (perspectives on disease), resource-based (water, income, distribution channels), or subject-related (analytical categories, demography, personal differences), to the post state outputs such as attention, cognition, adoption, behaviour, and health status.
Fig. 2. Action of constraints on project inputs.
In the Honduran and Gambian projects, categories of cognitive outcomes include recognition, recall, and knowledge of nutritional and preventive behaviours and ORT treatment messages. Categories of behavioural outcomes include response to the diarrhoeal episode (i.e. administration of ORT, taking child to clinic), infant-feeding practices (breast-feeding and weaning, etc.), water purity (boiling the water) and personal hygiene (hand-washing, hygienic practices). Categories of health outcomes include nutritional status (extent of malnutrition, variety of diet), morbidity (frequency, severity, and duration of diarrhoea and common diseases), and mortality (death rates from diarrhoea or other causes).
Fig. 3. Specification of project outcomes.
Categories of system outcomes include the health system (is ORT accepted and institutionalized? is prevention an accepted medical activity?), communication system (are the messages incorporated in development content? are various media seen as potential development agents?), distribution (do even the most rural areas have access to media, ORT, and health agents?), and training (is preventive and ORT information passed on in local training?).
These categories of outcomes of a MMHP intervention, along with their evaluation specifications, are shown in figure 3.
PROCESS MODEL AT THE INDIVIDUAL LEVEL
The Mass Media and Health Practices project campaigns will attempt to change knowledge, attitudes, and behaviours relating to the prevention and treatment of infant diarrhoea. The intervention is expected to result in changed health practices that, in turn, will lead to changes in health status. The conception of how the intervention is expected to work is diagrammed in figure 4.
Any comprehensive approach to evaluation requires that each successive link be monitored so that useful lessons can be learned from the outcome of this project. In this way one can determine where weak links occur, or where the next step requires particular attention, and use that information to guide the planning and execution of similar projects. To that end, we have elaborated the general structure shown in figure 4 into a more detailed version shown in figure 5.
Fig. 4. Postulated relation between treatment and health change.
We will illustrate this underlying conception of the path from treatment to changed health status using as an example one specific practice the implementation campaign will probably advocate: continued breast-feeding during a diarrhoeal episode. Connections are described as though they were sequential links, although it is realized that this is an oversimplification. Links 1 and 2 concern the existence of the treatment, links 3 to 5 the cognitive and attitudinal changes that are postulated to precede practice change, links 6 and 7 describe adoption of new behaviours and, finally, link 7 is that change in health status expected to result from the treatment.
1. Existence of the Intervention
The first step is to establish the existence of campaign elements related to the topic under consideration. The extent to which the implementation effort is devoted to the topic of breastfeeding will be examined; whether radio programmes or spots concern breast-feeding, to what extent relevant content is incorporated in the printed materials, whether health-care-worker training includes instruction about the topic, and instruction in how to present it to mothers.
2. Exposure to the Intervention
At this level, the potential exposure to the intervention components is assessed, including the number of radio programmes broadcast; times of day of broadcast; listening patterns of target audience; successful physical distribution of printed materials; posting or circulation of printed materials; numbers of client contacts with health-care workers in which continuation of breastfeeding is mentioned; and proportion of time devoted or relative emphasis given to this practice by health-care workers.
Fig. 5. The logical chain from proximal to distal outcomes.
3. Awareness of the Intervention
This step tries to determine whether any of the potential exposures to the intervention actually "got through'' to the target audience. It measures consciousness of actual exposure to intervention components. For example, members of the target audience will be asked to report topics of the campaign. or to recognize theme music or specific intervention messages. Target audience members will also be asked to judge the frequency of relative exposure to specific components and to judge whether that exposure was sufficient.
4. Knowledge of Specific Content
At this stage, amount and accuracy of learning resulting from the actual exposure is measured, using such approaches as: recall of message content, that is, asking the respondent to generate a description of the content of a message rather than merely recognize it: understanding of the content and reasons for adopting the behaviour; accuracy and level of detail of knowledge of the advocated practice; and ability to exercise new skills.
5. Attitude toward Specific Content
At this stage acceptance of or reaction to advocated behaviours is focused on. Typical topics for measurement include: recognition of the current practice as a problem; beliefs about the current practice and the consequences of adopting the advocated practice; level of desire to depart from the status quo; perception of the individual's own position in relation to community norms; perceived conflict of advocated behaviour with other valued beliefs or practices; and attitudes toward message sources.
6. Trial of Advocated Behaviour
This next stage is to discover whether, when confronting an appropriate context, mothers have ever accurately performed the behaviour. For some behaviours this can be measured by observation, but in the case of continued breast-feeding during a diarrhoeal episode, selfreporting, along with adequate description of the context and corroboration by other sources, will be used.
7. Habitual Performance of Advocated Behaviour
This step investigates whether there has been repeated or habitual performance, and the conditions required for maintaining the behaviour will be observed. In the present case, selfreporting will be used, as noted above. Accuracy of performance and rate of display of the behaviour under appropriate circumstances need to be measured. This investigation will attempt to determine the conditions that lead to continued practice of the new behaviour.
8. Change in Health Status
The final stage involves the measurement of whether the changes in health behaviours produce a detectable effect on health status. In the case of continued breast-feeding, a likely effect is a change in nutritional status or growth velocity. For other advocated behaviours, morbidity or mortality measures might be more suitable. (Note that appropriate measures are only being described, and that demonstration of causal relationships is not being suggested.)
Measuring the impact of each of these successive links in the chain will provide information of a detailed nature about exactly where the programme is successful, and what types of remedial action to take to improve the situation. The information obtained in these measurements has a value that extends to other projects as well, because this model is quite general.
The actual structure of the evaluation does not follow the structure of the model; instead, the evaluation structure is dictated by methodological and logistical issues. The model of the process involved is interwoven throughout the evaluation because it serves as the basis for criteria for decision-making among methodological alternatives.
Choices in Research Approaches
The fundamental goals of a health campaign evaluation effort are to:
To ensure adequate control and to assess the validity of results by convergent measurement, each general category of outcomes can be measured using several methods, including questionnaires, structured observations, structured interviews, anthropometric measurements, archival data, ethnographic research, and case-finding/tracer studies. Figure 6 indicates how these methods are applied to the outcome categories in the MMHP evaluation. The characteristics of a variable (in particular, the variance and the rarity of occurrence) determine the frequency with which it should be measured, and the appropriate sample size. For instance, questions about family composition and educational levels need only be asked infrequently, while questions about incidence of acute diarrhoeal disease must be asked quite frequently. Further, consider different analytical foci that can lead to a comprehensive evaluation of health campaigns. Below are six major study groupings used in the MMHP project that differ markedly from one another in magnitude, study population, and measurement requirements.
Fig. 6. Matching of research approaches with categories of dependent variable.
Particular project contexts may lead to rejection of or emphasis on one or more of these studies. For example, archival data analysis is dependent on the existence, validity, and timely availability of relevant data. In Honduras, lack of measurement precision and evidence of marginal returns from changing the media mix or frequency lessened the possibility of a complete cost-effectiveness study.
Control Groups
A major issue in the design of an evaluation plan for an intervention is that of allocating resources between measurements made on the treated population group and measurements made on comparable but non-treated populations. The trade-offs are between precision of measurement, which is increased by concentrating resources on the treated population, and interpretability of results, which is increased by allocating measurement resources to a comparable non-treated population. Because resources for field-work are limited, it is crucial to think through carefully the value to evaluation of mounting data collection efforts in non-treatment areas. To present an example of this process, we will first examine potential sources of data on non-treatment populations that are possibly attainable without conducting a longitudinal survey in a control area.
Comparisons within the Treatment Area
We identified four sources of data within the treatment area (from those mentioned above in the section on research approaches) that can be used for control purposes.
They vary in the quality of the data they provide. We first list them and then, in table 1, assess their usefulness to the project.
Household as Its Own Control
Because we adopted a panel design, we returned to households for repeated measurements. We thus were able to use each household as its own control for many variables.
Making Use of Staged Implementation
If, because of cyclic aspects of system constraints, components of the campaign are introduced in stages in different regions of the treatment area, the study can have non-treated segments of the population within the measurement sample for portions of the intervention period.
Natural Variations in Exposure
Because of the vagaries that can be expected in mounting a complex intervention in a developing country, some components of the campaign will not be implemented as planned. A project can take advantage of these events if they occur in the sites in which data are being collected, and if the monitoring of engaged inputs is satisfactory.
Self-determination of Exposure
Some people will select not to expose themselves to a health campaign, because they do not have access to a radio, because they do not choose to talk to health workers, and so on. Such people are likely to be quite different from those who do accept exposure. None the less, they can be a source of some kinds of information.
We have identified four sources of control information within a treatment area and four sources of information about populations outside a treatment area. Earlier, we identified three types of project outcomes of interest: cognitive outcomes, behavioural outcomes, and health status outcomes. Table 1 summarizes our view of the usefulness of each type of data for variables falling in each of the outcome categories. For example, in the column headed "Staged implementation," we note that there is an "a" in the rows for cognitive and behavioural outcomes, but nothing in the row for health status. Our reasoning is that beliefs, practices, and levels of knowledge can change quickly on exposure to campaign intervention, so repeated measures can capture changes between implementation stages. In contrast, the typical time interval between the onset of data collection and the onset of a later implementation stage is probably too short to provide reliable evidence of control levels of health status variables. As another example, consider the entries in the column labelled "Self-determination of exposure." Those who choose not to be exposed to the campaign are probably too different in their beliefs and practices from those exposed to provide reliable evidence about the first two types of outcomes, "cognitive and behavioural," and thus these cells in the figure are blank. However, they may be a useful source of information about some health status indicators, although such data cannot be expected to be of high quality. We have thus marked this space with a "b."
Table 1. Usefulness of data from various sources for control group comparisons
| Outcome | Within treatment area |
Outside treatment area |
||||||
| Household as own control | Staged implementation | Natural variation in exposure | Self- determination of exposure |
Archival data | Ethnographic studies | Data from other studies | Special studies |
|
| Cognitive | a | a | b | a | b | a | ||
| Behavioural | a | a | b | b | b | b | a | |
| Health status | a | b | b | a | b | b | ||
a. High potential utility.
b. Moderate potential utility.
Comparisons with Non-treated Populations
We can obtain data about people outside the treatment area from several sources. Archival data and ethnographic studies were mentioned previously. Data from other studies are also useful. Other health projects have been and are now functioning in many regions that may provide useful baseline information. For example, the Institute of Nutrition of Central America and Panama has constructed growth curves for Central American children that can be used for comparison and prediction. This project used standardized data available from the Center for Disease Control in Atlanta, Georgia.
Special one-shot studies may assess the level of a belief or practice in a non-treatment area when results in the treatment area are ambiguous.
SUMMARY AND CONCLUSIONS
This paper has presented a general framework for thinking about evaluation design issues, derived from a systems theory approach. It has illustrated the application of that framework with examples drawn from ongoing projects in Honduras and the Gambia.
It is unlikely that any specific project would ever try to utilize every aspect of the framework, since most projects (and to an even greater degree, evaluations) will have only a more limited scope and intent. However, the framework leads one through a planning process that ensures that all aspects will be considered, even though they may be deliberately omitted from any evaluation plan. The key issue is that the decision to omit should be a conscious one.
An important feature of the evaluation framework is the emphasis that it puts on examining structural and resource constraints that exist outside of and prior to any intervention attempt. ID our experience, the most frequent problems associated with the projects stem from inadequate attention to the details of programme implementation in an environment full of unrecognized constraints. If evaluation planning occurs concurrently with the development of an intervention design, likelihood of successful programmes increases, because the constraint issues will be acknowledged from the beginning. This evaluation will contribute to programme success not only in a cumulative way, but also in a formative or process-oriented way.
ACKNOWLEDGEMENTS
The author wishes to acknowledge the support of the Offices of Education and Health in the Bureau of Science and Technology, USAID, under contract AID/DSPE-c-0028, and the contributions of those who have worked on these issues under the contract, particularly Dr. Ron Rice, Dr. Barbara Searle, Dr. Carl Kendall, all Dr. Peter Spain.
