Previous - Next
This is the old United Nations University website. Visit the new site at http://unu.edu
Development of a framework and identification of indicators for evaluation
Stuart Gillespie
Abstract
This paper describes the types of indicators that can be useful in evaluating inputs, activities, and outputs of institutions. While the focus is on food-and-nutrition-related institutions, the recommendations are of general application. Inputs include staff training, sustainable financial support, staff-development planning, physical assets, and technical assistance. Output indicators serve to show how practical, how cost-effective, and how sustainable the intervention is in relation to the existing local problems and the institutional capacity to deal with them. Indicators are also proposed for the evaluation of staff and their activities in management, research, planning, and training.
Introduction
At its meeting in August 1986 the Advisory Group on Nutrition of the ACC/SCN considered the goals of institution building in relation to nutrition and identified a number of structural considerations, constraints, and opportunities. It was noted that any approach to institution building should be sensitive to the needs of the individual country and should start with an analysis of those needs, emphasizing the goal of promoting institutional structures that support operational programmes. A matrix for country-level analysis was constructed that related types of institutions (e.g. planning, research, training) and their needs (e.g. recruitment, management, career development). For each country, priorities of need could be indicated. It was also suggested that, if the agencies can bring together appropriate expertise to develop normative standards that can be used by country programmes, they will have facilitated a development of institutions supporting these programmes in the country-a synergistic approach.
A first step in this process was to gather some information concerning institutional capacities. their relation to operational programme needs, and how they are affected in practice by assistance projects. Therefore a framework for obtaining the necessary information, with specifics of indicators likely to be useful, was put forward to the ACCISCN at its session in February 1990. This paper outlines the proposed framework and information needs.
A distinction should be made between direct assistance for institution building and assistance for interventions that may affect the viability of local institutions in the process. The framework presented here and the indicators derived from it can be applied to institutions that have been implementing programmes financially supported by external agencies.
Types of indicators
Three types of indicators-inputs, activities, and outputs-need to be identified, although the latter are likely to be specific to the particular institutional objectives. The framework presented in figure 1 outlines inputs, activities, and outputs of institutions from which indicators may be selected. The planned output of many institutions is the successful implementation of programmes, and programme evaluations have focused primarily on such outputs. In order to reveal appropriate areas for external assistance that strengthen institutions, indicators used in programme evaluation should include ones that relate to institutional capacity (inputs, people, activities) as well as outputs.
The indicators listed in the sections below are not all-inclusive but illustrate the types-both quantitative and qualitative-that could usefully be employed to differentiate effective from ineffective institutions and to highlight appropriate areas for external assistance. There are indicators for institutional monitoring over time (e.g. the percentage of the budget allocated to staff training) as well as ones relating standards or yardsticks (e.g. training matched to objectives, salaries matched to living costs). Indicators would need to be tried out in operation and the model validated on several country-level examples before being incorporated into systems of project evaluation.
FIG. 1. Framework for assessing institutional capacity
Input indicators
Inputs such as training, staff-development planning, financial support, and technical assistance relate to the primary resource of an institution-its staff. Others include physical assets, for example, buildings, equipment, and transport.
Institutions need to plan for the development of their staff. As well as remunerative salaries (for which sustained financial support will be necessary), staff members need prospects and promotional opportunities. A staff-development plan thus needs to consider staffing structure, including the placement of trainees, and a built-in reward system that promotes confidence development and job satisfaction. Indicators should reflect these requirements. Other indicators that relate to training may include the percentage of the institutional budget allocated to training and the variety and flexibility of training.
Examples of input indicators are:
Activity indicators
Staff and management
Staff
Following on from the considerations of training and staff-development planning-as inputs-staff skills (acquired through training) should relate to the objectives and activities (e.g. research, planning, or management) of particular institutions. External assistance may thus take the form of support for training to provide appropriate technical, managerial, communicative, administrative, and support skills. It should promote independence and sustainability through a continual process of delegating responsibility.
Rewards for individual staff members will include such qualitative factors as prestige, job satisfaction, perceived social worth, peer respect, professional contacts, and promotional opportunities, as well as an adequate salary package. The extent to which such rewards exist for individuals in institutions will need to be elicited through interviews.
Examples of indicators are:
-the relationship between salaries and living costs, -opportunities for promotion,
-the reward system (including factors such as job satisfaction, prestige, and peer respect), -staff flow-recruitment and retention.
Management
Good management is necessary in all types of institutions for the effective functioning of everything else. As well as operational management of supply and logistics. there needs to he skilled management of staff. A good working environment is facilitated by promoting collaboration among members through ensuring proximity, while there should also be scope for interaction with outside counterparts. Leadership should share the means for innovation, project development, and decision making with senior staff.
Examples of indicators are:
Indicators specific to the type of institution
Institutions for research
Research capability is reflected in the ability to conceptualize, plan, decide, operate, monitor, and evaluate; and a continued and fluid interaction should be maintained between basic and applied research. Indicators of this capability will include measures of the collection, classification, analysis, and synthesis of data required to fulfil research goals.
Examples of indicators are:
Institutions for planning
In planning institutions, certain successful approaches have been distilled from evaluations of country experiences with institution building [1]. For example, there are advantages in linking long- to short-term interventions, and unlinked "pure" nutrition projects (e.g. growth monitoring per se) should generally be avoided. Programmes can be planned in a way that reflects success on the implementing institution. The credibility of nutrition institutions has in the past been usefully enhanced in some places by linking them to interventions that the local population understands and may demand-e.g. iodine, vitamin-A-fortification, and anaemia-prevention programmes.
Examples of indicators are:
Institutions for training
The indicators for the activities of training institutions will be similar to the input indicators related to training discussed above. In the former case, future trainers are being trained, and here people are being trained in programme implementation. Both types require a matching of methods with planned outputs, though the outputs will differ, being effective training and successful programme implementation respectively.
Examples of indicators are:
Output indicators
Outputs need first to be related to the objectives of the institution. If the outputs do not approach the objectives, then the people and activities that generate the outputs need to be evaluated, using the indicators chosen, and areas for appropriate external assistance need to be outlined. Implementation of effective nutrition interventions should logically follow from the planning and research steps above. Considerations may include how practical, how cost-effective, and how sustainable the intervention is in relation to the existing local problems and the institutional capacity to deal with them.
Examples of indicators are:
Reference
