Contents - Previous - Next

This is the old United Nations University website. Visit the new site at http://unu.edu

5. Project evaluation

Project management
Type and impact of projects
Selected sectoral reviews

Project management

The high level of aid flowing into the Sahel - $1,500 million in a non drought year, or $40 per capita- suggests that one should be able to begin to see some impact. Effects might be measured in terms of the overall goal of improving the standard of living, the three goals of CILSS, or the design goals of particular projects.

One can also look at the effectiveness of the aid activities, including the selection and implementation of projects as well as the amounts and distribution of benefits. The questions of project formulation and delivery are importan because they directly and indirectly affect the type of project selected and hence its impact on the rural resource base. These interactions, together with the different levels of analysis, mean that there can be nearly as many sets of criteria to evaluate aid programmes as there are types of development projects.

Finally, although this report concentrates on the UN system as a source of aid, and natural resources as a focus of development efforts, there is a great deal of similarity among donors and among sectors. Virtually ali major donors must make a series of decisions regarding the location and objectives of projects and develop mechanism to allow smooth implementation. Projects in the natural resources sector are usually treated in the same way as those in any other sector. While this may not necessarily b' an optimal approach, it does mean that a somewhat more general discussion can have wide applicability.

Perhaps the first prerequisite of a meaningful evaluation is a clear definition of objectives and goals. For example, the World Bank must limit itself to projects that will offer a satisfactory rate of return and thus be able to meet the required repayment schedule; on a macro-scale finances are directed towards the neediest countries, but there is little evidence that on a micro-scale-within a country -they are consistently directed towards the neediest recipients. On the other hand, USAID projects are not necessarily concerned with the rate of return but rather with implementing activities that will become selfsustaining and thereby have a long-term effect; in 1973 USAID was specifically directed by Congress to give more consideration to the rural poor and less attention to capital-intensive or technologically oriented projects (USAID 1979a). Bilateral aid also tends to reflect the ideology of the donor country. Countries must decide, for example, whether to support local entrepreneurs and encourage a market economy or to support state farming enterprises at the expense of individual farmers. Furthermore, bilateral aid usually is an integral part of foreign policy, with the amount and type of aid related to the donor's perception of the receiving country's importance and "friendliness" (see chap. 4).

UNDP's funds, like those of the World Bank, are allocated among countries on the basis of a complex formula of population and poverty. Adherence to this allocation is stricter, but there is not the need to demonstrate a favourable cost-benefit ratio for each project. In fact, the recipient government tends to play the dominant role in determining the use of UNDP funds, as proposals for UNDP funding can only come from the concerned government. Of course, UNDP could refuse a project, at the risk of damaging relations with the host government; alternatively UNDP can urge the government to propose a certain project. In practice UNDP and the various projectidentification missions work closely with the recipient government so as to suggest only projects that will ultimately be acceptable to the recipient government, Usually there is then an internal selection process during which the government, in consultation with UNDP, determines which of the suggested projects will actually be formally proposed. Since UNDP's expenditures are now limited to 55 per cent of the 1982-1986 IPF, this final selection has become more important to agencies competing to have their proposals selected and implemented.

In summary, there is a wide variety in goals and objectives and in the precision to which these are defined. In the Sahel the emergence of bodies such as CILSS and the Club du Sahel has led to unprecedented co-operation among donors and a universal endorsement of certain objectives, but each donor still maintains its own overriding values.

Despite these differences, it should be clear that all major aid organizations do follow the same basic steps in the provision of aid: (1) deciding on the allocation of funds on the macro-scale, either by country or economic sector, (2) staging preliminary talks and visits to identify potential projects that are mutually acceptable, (3) formulating and approving detailed project documents, (4) executing projects, and (5) evaluating completed projects. It may help, therefore, to go through these steps of allocation, project identification, formulation, execution, and evaluation to determine the relative strengths and failings of the varying approaches used by different organizations.

Allocation

Here the term "allocation" means the critical process of deciding who gets what on a global or regional scale. In the case of large international institutions, such as the World Bank and UNDP, complex formulas are used, while political considerations tend to dominate in the case of bilateral agencies. In the smaller UN agencies funds tend to be allocated on an ad hoc basis. As noted earlier, UNDP also operated on an ad hoc basis until it was realized that countries which didn't have the capability to formulate and initiate projects (i.e. the poorest and least-developed) were at a disadvantage compared to countries that had a more developed infrastructure, such as South Korea or Egypt. Thus the Sahelian countries, with the possible exception of Senegal, fared poorly in the aid race of the 1 960s. The project-by-project approach also led to an excess of "salesmanship" by personnel from the various executing agencies as they tried to persuade UNDP and governments of the value of their particular projects (Jackson 1969). Another disadvantage of this approach is the potential for abuse, as powerful people tend to direct projects towards regions or countries where they have close ideological, political, or personal ties.

While the formula approach ensures a fairer distribution of funds on a regional basis, its disadvantage is that governments come to regard their allocation as free funds for use or abuse as they see fit; the money is to be spent in the country in a given period of time no matter whether that is the "best" possible use. UNDP and certain other agencies allocate funds for regional and global projects, but UNDP's Governing Council has put a strict ceiling on the proportion of funds that can be used in this way. Thus UNDP is increasingly seen as a source of resources which governments can then use as a basis for tapping the technical expertise of the UN agencies (ECOSOC 1978) or as a source of counterpart contributions for other (e.g. World Bank) projects.

Project Identification

Once the allocation of resources has been done on either a country-wide or a regional basis, the next step is to identify specific projects. Within UNDP efforts are being made to tie UNDP's programming cycles closely to the respective planning cycles of the host country, and this again helps the recipient country to better guide the project-identification process. CILSS is attempting to do much of the project identification in the Sahel, in general accordance with the guidelines laid down by the Club du Sahel. Of course, many of the sectoral missions sponsored by CILSS include personnel from the major donors (UNDP, World Bank, USAID, etc.), so that the identification of potential projects may be a co-operative venture from the beginning. Nevertheless, it is those countries or organizations that do not have an institutional presence in the Sahel that are most likely to support ClLSS-generated projects. Thus France and UNDP, while actively participating in CILSS, tend to be more independent in identifying projects.

The United Nations Sudano-Sahelian Office is also intimately involved in project identification by working closely with CILSS and by sponsoring missions to the Sahel-often on a multi-donor basis-in order to identify projects for funding. Other agencies, such as the World Bank, sponsor their own missions to a series of countries within a given region, and these typically concentrate on a specific sector of the economy (forestry, industry, agriculture, energy, etc.).

Project Planning

In practical terms, the planning of aid projects is simply a continuation of project identification but on a more formal basis. At least within the UN system this phase is generally initiated by a formal request from a government. The concerned agency then puts together a design team, composed mostly of outside experts and people from the agency's headquarters rather than local people. According to Mason and Asher (1973), FAO, Unesco, and WHO each have units whose primary purpose is to assist governments in project identification, with most of the units'costs being defrayed by the World Bank. UNDP works in much the same way, relying on experts from the different specialized agencies to assist in the formulation of large projects, and then reviewing the proposals at headquarters in New York.

In contrast USAID tends to rely more heavily on consultants, and there has been some criticism of the consultants for setting specifications for their own equipment, recommending that a given project be implemented, and then becoming the project contractor. In other words, their own interests become more important in defining a project than the needs of the local people. Hoben (1979), in his review of livestock projects in Africa, notes that planners are prone to be biased towards the public sector and the formal part of the private sector because these areas are more visible, there are more readily available data on them, and they can be tapped later as sources of government revenue. Similarly, there is often a bias towards certain commodities, such as meat rather than dairy products, hides, and skins, which don't play such a large role in the formal economy. These same factors can largely explain the usual emphasis on cash crops instead of subsistence crops. Hoben and other writers (e.g. Lele 1975) have noted that a variety of political considerations can influence where a project is sited, what its target audience is, and whether funding is available. Because of the limited time available in the field and the usual background of the design teams, physical aspects, such as land-use management and agricultural production, and technological fixes are usually the focus of concern rather than individual subsistence farmers or pastoralists.

If one is concerned with the long-term sustainability of production and minimizing the disruption of ecological systems, the extent to which environmental considerations are taken into account during the planning process is of central importance. A study by the Internationai Institute for Environment and Development pointed out that procedures vary widely (IIED 1978). Of the agencies examined (UNDP, World Bank, regional development banks, European Development Fund), only the World Bank claimed to have a formal procedure for taking into consideration environmental impacts. Specifically it has set up an Office for Environmental and Health Affairs, and the task of a small group within this office is to review projects and, in principle, to force a restructuring where necessary to meet minimal environmental standards. The group is also responsible for preparing materials for use by project off icers to help ensure that proper account is taken of environmental considerations. In reality, the small staff and heavy workload mean that only the more glaring environmental problems are noted in the review process and that project officers have little time to refer to the general guidelines regarding the environmental integrity of the projects they are concerned with (IIED 1978). The most effective way to incorporate environmental considerations is to ensure that the consultants and project staff involved with project planning and execution have the required awareness. This point becomes all the more critical in the case of the African Development Bank, the Arab Bank for Economic Development of Africa, and the European Development Fund, as they do not have any formal structure to review projects for their environmental impact (IIED 1978).

UNDP also lacks a formal structure for environmental review, but since it works through a variety of other agencies, institutionalizing any consideration of environmental impact-other than at the final review stage-will prove difficult. It should be noted that there is considerable variation in project procedures depending on the nature of the specific UNDP-sponsored activity. In UNDP's preinvestment and feasibility studies, environmental considerations are usually an explicit issue, whereas environmental issues may not be a concern in some of the technical cooperation projects, depending on the type of project and the executing agency. UNEP technically has the right to review UNDP projects with regard to environmental concerns, but this is not exercised in practice. UNDP does have basic administrative and referral machinery that could be used to ensure a proper environmental review, but-as in the case of the regional banks-this needs to be institutionalized and made a required step (IIED 1978). At least a formal commitment to these and related goals has been made by UNDP, UNEP, the World Bank, the European Development Fund, and the major regional banks (New Scientist 1980).

Among the bilaterial agencies, USAID has a formal procedure whereby every project must be examined early on and a decision made as to whether it will have a significant environmental impact. If there will be a significant impact, a full assessment must be prepared and approved at the same time as the project review paper is considered (USAID 1976b). In general these environmental assessments (or environmental impact statements) are prepared by outside consulting firms. Given the often limited experience of these firms in developing countries, especially in tropical areas, and the fact that there is no real constituency that reviews the reports critically, they tend to be superficial and based on extremely limited data. Extrapolation from temperate to tropical areas and from the United States to developing countries is often done without any real indication that conditions may be so different as to nullify the results. Furthermore, the preparation of an environmental-impact report in the project-planning stage does not necessarily mean that those concerns will be incorporated in the execution and evaluation stages.

Of course, concern for environmental impact is not limited to donors but can also be expressed by the recipient countries. Ideally it would be the recipient countries who would press upon the donors the need to take these factors into account, as they can draw on past experience and are familiar with the local situation. However, the great lack of trained manpower means that most developing countries, particularly in the Sahel, have much more basic and immediate concerns. It is also true that developing countries tend to be less concerned with environmental matters than the industrialized countries, as the former feel that other needs are much more pressing. Thus the World Bank has met resistance when occasional projects required additional expenditures for environmental safeguards such as pollution-control devices. On the other hand, the I IED report noted that the World Bank has refused to fund environmental safeguards that it deemed "unnecessary or overly sophisticated" (IIED 1978).

The planning of projects that will be supported by loans rather than grants is further complicated by the need to meet certain financial criteria. For natural-resource development projects, any thorough economic analysis is severely hampered by the expression of physical and biological resources in financial terms. These problems then tend to limit the extent to which environmental considerations are taken into account. In the 1978 review of the World Bank's forestry sector, the problem of adequately quantifying the direct and indirect benefits of forestry projects was "the main problem area common to forestry projects in all regions of the Bank" (World Bank 1978a).

A final criticism that has been made by virtually every author who has written about UN development efforts, either with regard to a specific project or on a more general basis, is the extensive delays involved in planning. Jackson (1969) provides the most extensive statistics, noting that the average time between receipt of the official request by UNDP and signature of the plan of operations was 26 months; for certain agencies the average was three years. To improve this rather dismal record, UNDP has raised the ceiling for projects requiring headquarters approval to $400,000. Similarly, the World Food Programme has instituted a "quick-action procedure" to simplify approval at headquarters and thus save time. A review of WFP's experience is instructive, as it shows that 90 per cent of the projects planned under this procedure were approved within 15 weeks (WFP 1978b); in contrast, only 6 per cent of projects following the normal procedures were approved within 15 weeks, while 43 per cent took more than one year. Disadvantages or problems were also noted in nearly half the quick-action projects, which provide an interesting commentary on the difficulties of project execution. Table 27 shows the results of an analysis of difficulties in the implementation of projects approved under WFP's quick-action procedure that might be attributable to the procedure.

The table speaks for itself, but it should be noted that all projects, especially emergency aid activities, are subject to similar failings. In the project reports comments are noted such as "However, the Government distributed the entire 3,000 tons of food to 410,000 beneficiaries instead of 75,000 [beneficiaries] over a period of 30 days instead of 100 days as originally approved," and "It has been reported by the WFP field officer that a very small quantity of food has been distributed in kind, while sales of WFP commodities continued without prior approval of WFP and without notification of the utilization of sales proceeds" (WFP 1977, 1978a).

Of course, other development agencies have similar problems, and it could be argued that delays are inevitable given the complex design procedures. A 1979 report on USAID's Sahel Development Programme noted that 15 to 40 months are required to design each project. Moreover, different phases of the design process are often handled by different teams, and the design may be based on one or more studies done by other consultants on specific issues. Major undertakings, such as dam or airport construction, usually have an initial project for the feasibility phase, another project for the planning phase, and a third project (or series of projects} for the actual construction. Since preliminary studies are usually a separate project, the long lead time can rarely be ascribed to precautionary studies.

TABLE 27. Problems encountered in implementing WFP 'quick-action" projects

Source: WFP 1978b
^a More than one difficulty was identified for some projects.

Project Execution

Once a project document has been signed by the respective government(s) and donorls), the next step is to orchestrate the input of funds, equipment, and personnel as quickly as possible and in the proper sequence. Since in all projects there is some kind of counterpart contribution, these local inputs must be co-ordinated with those from outside. This is an extremely difficult task, and it is a rare project which does not suffer from delays or undergo substantial revision in its execution. Through experience UNDP learned that, if it is to operate in the least-developed countries, it must adjust its requirements for counterpart contributions and provide inputs that in some of the wealthier developing countries would normally be provided by the countries themselves. These liberalization policies began to be implemented in the late 1960s, together with the adoption of country programmes (Inter-agency Consultative Board 1967). The distribution of responsibilities between donors and recipients will remain a matter for ongoing debate, with considerable variation among donors and projects.

With regard to the implementation of development projects, virtually every review in the literature identifies the same basic problems. These might be grouped into categories of (1) delays in obtaining outside experts and equipment, (2) failure of the concerned government to supply a proper counterpart contribution, 131 poor participation by the target population, and 14) lack of infrastructure.

The most comprehensive review of UN development efforts was made by Jackson in 1969. He analysed approximately 250 UNDP projects then under way and found that 50 per cent were behind schedule. Of these, 46 per cent were delayed because of problems with the counterpart contribution (usually personnel), 29 per cent because of delays in the recruitment of qualified outside experts or, less often, equipment delays, 6 per cent because of delays in fellowships, and 25 per cent because of other problems (some projects were delayed for more than one reason) IJackson 1969). Another UNDP study in 1974 noted that 113 projects had been postponed by at least three months from their intended implementation date. A failure to recruit experts in good time accounted for 60 per cent of these delays, and the responsibility for this usually rested with UNDP. Shortage of fellowship candidates, general administrative delays, and project postponements or cancellations each accounted for approximately 5 per cent of the problems. Unspecified problems with the recipient government were cited in about 40 per cent of the 113 cases (UNDP 1975b). Problems with ongoing projects were also reviewed, and it was found that 35 per cent of the difficulties were due to problems with the UN inputs, 53 per cent to problems with government inputs, and 12 per cent to other problems (poor project agreements, war, drought, etc.).

Given these delays, the problems of matching expenditures to the budget for any given year become even more difficult. For example, at the end of 1968 the total difference between planned and actual expenditure for UNDP was $117 million, which exceeded the $112 million actually spent. In 1975-1976 the opposite problem occurred, with UNDP committing more funds than it had available. The resulting deficit of $36 million forced UNDP to cut back severely on its proposed projects. Then, when its financial situation improved in 1977, programme planning and implementation could not respond quickly enough, causing an excess of income over expenditures of $163 million, or 36 per cent (UNDP 1978c). A sophisticated programme-monitoring system has now been set up to help avoid similar oscillations in the future. Although this can do little to prevent any dislocation caused by unanticipated changes in the level of contributions to UNDP, there is usually some advance warning that contributions will not match expectations. The five-year planning cycle (IPF) also provides a longer period during which UNDP can match expenditures with income. For example, over the last three years it has simply progressively lowered the ceiling on IPF expenditures, with the current target being just 55 per cent of the original (and admittedly optimistic) IPF.

Again, the UN is not unique in these problems of implementation. The World Bank forestry sector review, for example, notes that the rate of tree planting consistently falls short of expectations, but does not detail why. In a review of the US Sahel Development Programme several examples of extensive delays in project implementation are given, with the implication that the recruitment and placement of properly qualified experts is a major problem (GAO 1979).

In the early 1970s Lele 11975) reviewed 17 rural develop ment projects in Africa sponsored by the World Bank. Two of these were in Mali, and many of the constraints she identified were similar to those already discussed. In particular she emphasized the need to involve local officials, and that the lack of administrative and service personnel often posed a more severe limitation on the expansion of projects than the availability of funds. She also emphasized the need for projects to be consistent with national policies and the existing administrative structure. More planning was suggested in order to better anticipate the effects of a proposed project, but there is a minor contradiction in that greater flexibility is needed during implementation- at least in the case of pilot projects. However, these pilot projects are generally too resource-intensive to be repeated on a wider scale. Lele concludes that the first step should often be to improve the regional administrative capacity for effective planning and implementation of programmes directed towards only a few productive activities; if efforts are to be oriented towards individual farmers, the local people must be responsible. Either approach implies the presence of an effective infrastructure in communications and transport facilities, administrative centres, etc. Corroboration of Lele's analysis is found in the World Bank's forestry sector review, in which it is noted that "virtually all rural forestry programmes presented to the Bank in FY 78 . . . underestimated the institutional and sociological problems of how to obtain the support of village people for forestry, to secure the land for planting, and to protect young plantations from grazing and fire" (World Bank 1978a).

In addition to the problems discussed above, project implementation in the Sahel must take into account the uncertainties of politics and climate. Civil unrest in Chad has periodically made a shambles of the various aid programmes. For the most part, however, political uncertainties must lie beyond the scope of contingency planning. In contrast, one can at least estimate the chances for a given degree of climatic perturbation. Nevertheless, agricultural development projects seem almost always to assume average or better conditions. Many projects also set specific annual targets with regard to the number of acres to be treated or the number of farmers to be included. Adverse climatic conditions not only disrupt the projected activities but also may make the basis of the project unrealistic or irrelevant. The 1968-1973 drought had this effect, and the World Food Programme reports again provide some interesting case studies.

In the Gambia, for example, one WFP project was to provide food in small quantities in exchange for work on various development projects (road-building, etc.). However, drought in the preceding year had so reduced the harvest that the WFP food was used to feed farmers during the planting season. The shortage of food also resulted in the programme being expanded to include twice the number of beneficiaries but for only half of the planned period. Finally, makeshift means were used for the distribution of food, as the project truck arrived only after the project had finished. In other remarkably candid assessments, WFP noted that in a school feeding programme the poorest children couldn't pay even the minimal amount charged and were thus excluded. In a programme of aid to mothers and pre-school children, regular transport was available to only 6 of the planned 33 centres, unprepared food was distributed instead of fully cooked meals, and because the government could not properly implement the project it was terminated early, having distributed less than 5 per cent of the amount approved to its intended beneficiaries and having suffered 20 per cent losses due to pilferage. In another project the family size turned out to be five rather than the government estimate of eight, so that 50 per cent more man-days of work were available than originally foreseen. In general, WFP has had to demonstrate tremendous flexibility to cope with factors such as the staggered arrival of foodstuffs that were supposed to have been distributed together, the changing needs of the target population (or even a change in the target population), problems of distribution, pilferage' spoilage, and delays in the provision of equipment and funding. (It should be emphasized that these examples are being drawn from WFP not because its projects are less successful but simply because WFP has been more open about its shortcomings than most other multilateral and bilateral agencies.)

Evaluation

Critically important for improving the design and operation of future development projects is the step of evaluation. Most of the larger aid agencies have an evaluation section, but the reports emanating from the UN agencies are generally restricted. On the basis of interviews, experience, and the limited literature available, the problems with evaluation appear to be threefold.

First, an adequate data base is needed. This means that the initial objectives need to have been clearly defined, there should be quantitative data available on the participation in and effects of the project, and the respective reports should have been submitted by the project staff concerned. This last point-the timely submission of final reports-is again a common problem, and a waiting time of two years is not unknown (Jackson 1969; UNDP 1978c; Inter-agency Consultative Board 1967; GAO 1979).

The second requirement is for experienced, qualified evaluators who are familiar with the socio-economic context in which the project took place and are not biased towards the viewpoint of either the donors or the recipients.

The third point is that, once the evaluation is completed, the information needs to be condensed and put into a form useful for project officers, planners, government officials, etc. This last task is made difficult by the need to achieve a balance between generalizing too much, in which case the resulting guidelines are pious and vague, and being too specific, in which case the evaluation will be of limited applicability. Hence it may be best for evaluation offices to organize training sessions on specific topics for project planners and administrators. Some initial efforts in this direction are being made by UNDP's new Centrat Evaluation Office (UNDP 1984g).

Perhaps the biggest problems in carrying out an evaluation result from the various political forces involved in any development project. No agency, whether local, national, or multilateral, is fond of criticism, particularly when it could have adverse effects on future funding and the careers of the personnel involved. Similarly, those conducting the evaluation are often hesitant to offend the host government or the sponsoring agency, for they may be risking their own future involvement. Some agencies, such as the World Bank, have allowed the publication of articles by staff members on a wide variety of potentially sensitive topics, even though many of their in-house reports are restricted. While failing to complete evaluation reports or limiting access to them may be politically desirable in the short term, this means that the same mistakes may well be repeated, thereby making poor use of the available aid funds.

In summary, aid projects-whether multilateral or bilateral -tend to follow the same basic procedures and suffer from the same limitations. Delays in project formulation, project approval, and identification of project staff are nearly unavoidable, particularly in the more complex projects. Problems with co~ordination are inherent in almost any project anywhere in the world. These characteristics, coupled with the climatic and political uncertainty in the Sahel and the severe restrictions in infrastructure and locally available, trained personnel, mean that all aid projects in the Sahel are exercises fraught with uncertainty and frustration. It is, therefore, little wonder that these basic problems tend to overwhelm environmental considerations, especially when the latter are vague, unquantified threats to the long-term sustainability of production. For most aid agencies, day-to-day and year-to~year operations are the only ones that are relevant. Environmental considerations are given lip service, but they rarely stand in the way of a planned development project. It is likely, however, that these environmental factors will determine much of the development work that is to be done one or two decades hence.

Contents - Previous - Next