Previous - Next
This is the old United Nations University website. Visit the new site at http://unu.edu
Improving the use of prior country dietary surveys for comparisons by time and place
Johanna T. Dwyer and William M. Rand
Four questions can be asked on the use of international data sets for assessing diets to improve health: Is it worth while to compare dietary surveys done in different times and places? If the answer is yes, can we compare them with techniques that already exist? This requires that we must examine the general context of dietary studies. Next we ask, what problems are likely to arise? This leads to specifying minimum criteria for worthwhile comparisons. Finally, what are the next steps to overcome problems? Meta-analytic techniques are effective for analysing prior surveys. More sophisticated methods for designing and organizing new studies will permit better comparisons in the future.
Is it worth while to make comparisons between prior dietary surveys collected at various times and places?
It is worth while to make comparisons between prior dietary surveys. It is critically important to compare dietary practices of different populations living in different places at different times. Such studies can be used to determine dietary deficiencies or excesses in specific populations and to clarify associations between dietary practices and health problems. They may also provide information on "natural experiments" relating food intake to health, information that is unlikely to be available otherwise. Comparisons among countries or over time may identify or clarify associations that are not obvious at a single place and time, perhaps because the range of intakes is very narrow or the prevalence of the health problem is very low or high.
Many data exist from individual country surveys. Practical considerations, time, and economics dictate that they be examined repeatedly, since it is impossible to mount new surveys whenever an investigator asks a new question. Each existing survey has to be evaluated individually as to its suitability for the purposes at hand. Some surveys are sufficiently representative and well characterized to make generalizations to larger populations and to provide definitive answers to research questions. Others provide useful information for descriptive purposes, for generating hypotheses, or for planning additional studies.
Can we compare dietary surveys collected at different times and places?
Methods for sampling the population, assessing intakes, converting foods to nutrients, counting disease, defining variables for making comparisons, and analysing and interpreting the associations usually vary from study to study. Every element of variability that is left unstructured and unmeasured endangers comparisons and makes causal inference difficult.
The context of dietary surveys
Comparisons among surveys require careful, explicit examination of the results and how these results were achieved. Surveys differ in their goals, design, organization, conduct, and analysis, and those that are appropriate for one time, place, and purpose often are not appropriate for another. An essential step in comparing them is to scrutinize their results carefully in a more general context. Comparing similar results achieved differently can sometimes lead to insights into the fundamental problems of interest. Assuming that a single cause is responsible can be misleading, clouding our understanding of the underlying phenomena.
A number of different aspects of dietary surveys must be explicitly examined as part of a comparison. Key questions that must be dealt with in evaluating any survey are summarized below.
What was the goal of the survey?
The explicit goal of the survey shapes all its aspects. A survey that is basically economic focuses on where the food came from, how it was obtained, and how much it cost. Health-oriented surveys focus on possible differences in diet existing between sick and healthy respondents. A survey intended to determine the energy intake of a population may be quite different from one that seeks to determine the vitamin A intake of young children. Although one can often predict the comparability of surveys from knowing only their general purpose, more specific knowledge of each study's goals provides additional information necessary to determine it.
Who is doing the eating?
If conclusions are to be drawn or comparisons made, there must be a clear description of the population of interest and how it was sampled to obtain the fundamental units of the survey. Different populations often require different sampling techniques, introducing potential for different sorts of bias. Furthermore, different population samples often require different analytic algorithms for estimating population variables.
The unit chosen for data collection is also critical. The individual, a household, or an entire nation (as in food balance sheet data) are all used in dietary investigations. As the unit becomes bigger, the ability to describe individual food consumption precisely decreases.
A fundamental dilemma is the Heisenberg principle as it applies to food consumption methodology: the better one determines what an individual (or population) is eating, the more localized in time and space, and thus the less useful (generalizable), are the data.
What foods are eaten?
Dietary data must provide accurate (unbiased), precise estimates of what foods are being eaten. For analyses conducted later, meta-data on how the data were gathered are important. The measurement of food consumption is fraught with problems, and much more research is needed on this topic.
Definitions often present problems. Exactly what aspect of consumption is of interest? Is it what is being eaten now, an average over some past period, or intakes many years earlier. The best method to select (e.g. food record versus food frequency questionnaire) depends on the answer to this question.
Estimation is a second problem. Once the objective is defined, how can the measurement of interest and its surrounding variability best be estimated? Any survey needs information on what foods and food combinations are eaten, in what amounts, and how often. Temporal (seasonal) variability and other factors may contribute to comparability problems.
What nutrients are in foods?
Much work has been done lately to enlarge our knowledge of food composition, but much remains to be accomplished [1-3]. At a minimum, we have to develop systems for compiling data on food composition and for keeping track of all the tables that already exist [4, 5].
Before the nutrients in foods can be ascertained, a good nomenclature is needed. This is especially obvious and important when attempts are made to compare diets eaten in different countries [6]. Appropriate time- and culture-specific or country-specific food composition tables are thus required to use with the different food consumption surveys. The latest in the long tradition of efforts to deal with these database problems include those of INFOODS in the 1980s. More recently this work has been carried forward by the participants in the annual nutrient database conferences. Unfortunately, nomenclature problems remain and are multiplying, but efforts to resolve them are less well funded today than they were even a decade ago.
Here are a few examples of the difficulties due to nomenclature. An enormous variability exists in the food combinations that are designated by the names of foods over time and between cultures. A case in point is the milk shake, which varies widely in composition throughout the United States. Similarly, tortillas come in many different varieties in Latin America and the United States; the same food name carries with it very different nutrient values.
The nutrient composition of foods also varies. Similar foods often differ in nutrient composition. These differences may be due to the growing environment (e.g. Florida versus California oranges), because of differences in breed (e.g. the fat content of milk differs between Guernseys and Holsteins), or for other reasons.
Foods change in their nutrient composition over time due to genetic manipulations (e.g. leaner pigs with more separable lean to fat) or technological improvements (e.g. iron-fortified infant formula, calcium-fortified orange juice). Also changes in legal constraints (e.g. labelling laws), hygiene (e.g. less adulteration), and storage (e.g. more use of refrigerators and freezers) may alter nutrient composition.
Definitions of edible portions vary. People differ not only in what foods they call by what specific names but also in what portions they consider edible. For example, many Americans eat tomatoes unpeeled; in other countries they do not. Similar differences may exist in eating or discarding the rind of cheese, beet greens, and so on.
Amounts consumed pose problems. Information on this can often be very fuzzy. For example, eggs come in many sizes, as do meat portions, mangos, and most other foods.
How do individuals use these foods and nutrients?
Much research must be done in this area. The nutrient composition of the diet does not always correlate directly with how nutritious the diet is. For some purposes it is not enough to record foods. Information on the entire diet and patterns of eating are also necessary to identify important interactions between nutrients that affect absorption or metabolism. How individuals absorb and use nutrients (and non-nutrient substances) from foods depends on both the complete diet (what foods are eaten together and when) and individual characteristics (the consumer's health or disease status).
How were the dietary and other data collected ?
Proper interpretation requires contextual information about how the data were collected. Health, demographic, clinical, or economic data must be adequate to explain, correlate, view, or interpret the dietary data.
In a sense, investigators are caught in a vicious cycle. It may be necessary to have certain meta-data, such as demographic or clinical information, to specify how much of the nutrients in a diet are ultimately available to the individual.
Are the data similar enough to make comparisons possible?
This constraint applies to comparisons of intakes collected at different times and places. The components (variables) that are measured must be defined well enough so that they can be compared with each other. The dietary surveys being compared must be somewhat similar. They do not all have to be exactly identical, but they must all collect some comparable information. Also, enough information to permit identification of what is similar and what is not must be available. Otherwise, the risks of incorrect interpretation of dietary intake data and its associations with risk of disease are increased.
What problems are likely to arise in making such comparisons?
The important problems that arise in comparing dietary surveys depend to a large extent on the types of surveys that are being compared. Several different types of situations can make specific comparisons.
Planned comparisons
The simplest situation is when surveys are planned in advance and carried out at a single place and time. The investigator can decide who should be assessed, why, and how, and develop study designs that control possible sources of error in measurement.
Instrument-specific variability in the dietary assessment method is of particular concern in such studies. Many random and systematic differences arise with common methods for dietary surveys of individuals. The pros and cons of each of these methods are well described elsewhere [7-12]. Even after these difficulties have been overcome with careful planning, logistic and economic problems often remain [13].
Prior surveys with unplanned comparisons—same times, investigators, and places
More complexity is introduced when unplanned or poorly planned comparisons are made or post hoc analyses are conducted. When data collected with no purpose or a different purpose in mind are reanalysed, much of the ability to control error is lost. Quasi-experimental designs and other statistical adjustments may be helpful, but they are no substitute for comparisons planned from the beginning.
Planned comparisons—same investigators and time but different places
More complexity is added when the investigations involve different places, such as regions or countries. In large-scale collaborative studies involving investigators at many different places, quality control and quality assurance are even more important than they are in individual studies. Collaborators in some locations may be more capable or careful than others. Subtle differences in the application of the survey method, coding, and other protocols often creep in unless special efforts are taken to avoid them. Including coordinating centres with a specific quality-control role is helpful.
The applicability of a particular type of dietary survey instrument to the respondents may also vary. Response biases may differ. To minimize error, it is vital to have standardized techniques for sampling the populations to be studied and to adopt uniform dietary survey methods and methods for assessing other characteristics of nutrition or health status. Standardized training and continuous monitoring of survey workers ensure that techniques continue to be similar.
Food composition tables and nutrient databases also present particular problems when studies involve many places, since foods are inherently variable in their nutrient composition as well as from place to place. Similar foods produced in different countries may have different nutrient content, and some foods may be unique to certain locales. The differences introduced by failure to standardize nutrient databases may often be as large as the expected intervention-related effects on nutrient intakes.
Comparisons among and between countries over time are often hindered by other practical problems, including expense, methodological differences, and logistic problems that make it difficult to communicate and discuss issues on a regular basis.
Planned comparisons—different times and investigators but same places
When comparisons of data collected at different times are desired, even more formidable problems arise in controlling factors that may obscure true relationships between diet and health. When gaps in time between studies are large, historical memory is likely to be lost because investigators change, and methodological differences may inadvertently creep in.
Changes also occur over time in consumption-related habits (storage and preparation practices, cooking methods, trimming of meat, amount of plate waste, etc.) that may affect intakes of nutrients or other constituents in foods.
Changes in food supply, food composition tables, and nutrient databases must also be expected and accounted for. In addition, alterations occur over time in the health of the population being surveyed and in techniques for collecting health data. Finally, secular trends in other variables (smoking, physical activity) that may modulate diet-health associations are often present.
Unplanned comparisons of prior surveys from different times and places
The situation becomes extremely complex when comparisons using other investigators' data are desired. These data were collected at other times and places, usually for other purposes. The questions of interest are likely very different from those posed by the original workers.
All of the errors already enumerated may be present. Procedures may be incompletely or poorly described, making interpretation difficult. There is the danger of post hoc hypothesis formulation, as well as many other hidden problems in data collection, evaluation, and analysis that can invalidate comparisons.
The threats to validity in making these comparisons are many. Therefore the investigator should approach the task with caution [14]. There is a pressing need to develop techniques for evaluating data of this kind collected in diverse ways with different methods, if valid inferences are to be made from them.
What are the next steps to overcome these problems?
The problems and problem areas that have been outlined above may appear overwhelming. Dietary surveys can be compared; often the potential gain is worth the effort. However, this often requires a great deal of effort for much of the survey data that now exist. The work is risky because disparities between surveys may make valid comparisons few. Future surveys should be designed to be more directly comparable than past efforts have been so that information loss will be minimized.
We recommend two general strategies to accomplish this. First, develop more effective ways to analyse the disparate studies that have been carried out during this century. Second, begin now to develop more sophisticated ways of designing and organizing new studies that will permit better comparisons.
Use modern meta-analytic techniques for analysing existing surveys
Research involving comparisons and secondary analysis of dietary survey data sets requires the same rigour and attention to possible sources of error as does original research. Sophisticated analytic techniques such as meta-analysis can help to make sense out of the disparate dietary survey data that have already been collected.
Meta-analysis provides quantitative methods for combining evidence from different studies in a systematic fashion [15, 16]. It is designed to minimize the degree of disagreement between experts in analyses. Such disagreement may arise either from the methods employed to review the research or from judgement. It helps to eliminate methodological differences that lead to problems in interpreting a body of research studies. Although disagreements may still remain, the meta-analytic techniques help to separate differences due to methods from those due solely to judgement.
Meta-analysis is similar to processes that are involved in other integrative descriptive research [17, 18]. All the relevant studies are gathered together, at least one indicator of the relationship under investigation from each of the studies is developed, and these are then used to compute various statistics and to answer questions. Meta-analysis attempts to bring the same rigour and methodological standards to the synthesis of research that applies in primary research, with the goal that research reviews and other syntheses will become just as replicable as any other type of scientific work. If meta-analysis is used correctly, when conclusions vary from a study, the differences can be traced to explicit analytic choices that can be assessed independently.
Nine suggestions for improving meta-analytic methods are presented in table l as they apply to dietary surveys [19]. All of the points are applicable in considering future directions for research and the analysis of prior country dietary surveys.
Meta-analysis of prior country surveys may help to enlarge our understanding of how and why certain patterns of dietary intakes are related to various health indexes. If the contingencies on which these associations depend can be determined, prediction will improve, better theories may be developed, and ways to achieve greater control over these factors may eventually present themselves. Since prior country surveys are, by definition, historical events for the most part, attempts to explain findings are always context-specific.
Meta-analysis is most commonly used to distil reliable generalizations. Other uses are to estimate the size of various dietary effects on disease, and for explanatory purposes, such as addressing questions of why and under what conditions a given effect is obtained.
Organize surveys differently
The types of dietary surveys that are most likely to be helpful in explanation are those that involve manipulated events, such as before and after the fortification of cereals, the iodization of salt, or the addition of vitamin D to milk. When such information is available on an experimental manipulation, it is likely to be incomplete in that the components of the variable manipulated, the mechanism by which the effect was achieved, or components of the outcome are not all specified. Also, the population may not be representative of the larger group of interest.
Therefore a complete explanation cannot be deduced from such data sets. Usually such field experiments or trials are theory-based, however, and assumptions about the mechanisms by which the effects will come about guide action. The problem is that other mechanisms might also explain the observed relationships equally well.
Some partial explanations of how various factors specific to the persons studied, place, and time might affect results may still be discovered if the many interactions and moderator variables can be sorted out. It may be possible to determine how much and in what directions the associations between the independent variable and the dependent variable are affected or moderated by setting, place, and time. In addition, it may be possible to explain how the sizes of effects noted are related to different but interdependent sources of variation. Since most dietary surveys today are not done in an experimental context, explanation will involve identifying clusters of interrelated characteristics or conditions that appear to account for the variability observed. Paying more attention to experimental design issues at the outset of new studies can help to increase our resulting knowledge.
Begin now to standardize techniques for conducting surveys
Dietary survey methods and data collection need to be standardized. Many years ago the US Department of Defense and the Centers for Disease Control (CDC) developed criteria for standardizing biochemical techniques used in assessing nutrition status. Recently, similar quality control efforts have been undertaken for anthropometric measurements. Now it is time to invest more heavily in dietary survey methodology.
Biochemical and other indexes may be useful in helping to validate methods. Innovative experiments and considerable progress in this direction have been made by the National Center for Health Statistics of the CDC and the Department of Agriculture's Human Nutrition Information Service in the United States. The pioneering work of Woteki in this regard should be noted.* No doubt similar efforts toward standardization also exist in other countries. The International Union of Nutritional Sciences and the United Nations agencies have also made useful strides, many of which were initiated by Dr. Nevin Scrimshaw. However, the pace of progress has been slow. Now the growing interdependence of our countries requires greater speed in integrating our thinking on these and other methodological problems. The advent of interactive, computerized dietary interviewing methods, distributed processing, and computerized dietary assessment software and the growing capacity of computer hardware to store and handle complex and involved data sets greatly simplify and standardize tasks that were previously tedious, time consuming, and subject to error.
TABLE 1. Methods for improving meta-analysis for dietary consumption surveys
| Method | Comments |
| Make better use of existing methodology in research syntheses of prior country surveys. | Recognize that the precision of different measures of effect size are derived from studies with widely varying sample sizes. Regression analyses and unweighted analyses of variance ignore such differences in precision. |
| Have manuals of operation and protocols for analysis of prior country dietary surveys. | Specify precisely what studies are considered relevant. what question is asked, and definitions of constructs or variables; assess data quality for each purpose in a systematic manner; avoid procedural variations; use a written manual of operations (MOP) to ensure uniform procedures; cover data collection, techniques to be used for data evaluation, data analysis, and presentation of results to avoid bias in the MOP; specify deviations when and if they are necessary. |
| Summarize all available research evidence from prior country surveys and health outcomes in the analysis to clarify understandings of these relationships. Identify inadequacies in existing data and suggest where time, money, and effort might best be spent in the future. | For example, in the list of INFID surveys, dietary surveys involving randomized, controlled clinical trials are quite rare. Because controls are lacking, it is difficult to sort out whether health effects are from the setting, the diet. or some other factor. Lack of controlled studies also makes it difficult to estimate diet-related effect sizes. |
| Recognize the limitations of the questions that can be asked from the data and recursive relationships. | Most prior country surveys are observational and non-experimental, so only correlations and not causal inferences can be made. Formulate research questions analysing the data. Post hoc hypotheses are dangerous, because nutrients are often highly intercorrelated. Dietary intakes are often highly associated with income level, education, and health status. However, data may be useful for hypothesis generation and for gathering evidence to support decisions that have already been made. |
| Make sure that the inventory of prior country surveys is as large as possible and that sampling of appropriate studies is done in a workmanlike way. | Assemble an easily accessible repository
of prior country dietary surveys, as has been done in
other fields [20]. Address the issue of the
representativeness of subjects to larger populations. Make comparisons between relevant studies. Sample the studies to be included in a standardized way from those that deal with the phenomenon of interest. Critical data of interest are lacking in many INFID country studies; the reasons may be related to the reason the study was conducted in the first place. Effect sizes may be reported only for significant effects, or may be ignored for other reasons. Include some estimate of relevant effects in meta-analyses and use appropriate statistical techniques to estimate them, even if only the general direction of their effects is possible. The studies in cross-country databases such as the INFID data differ in their characteristics. Handle missing data by having two levels of specificity of some general characteristics that are available in all studies, and additional information available that may be analysed only in some studies. Examine other sources, such as technical reports collected by the United Nations or bilateral agencies, ethnographies, the scientific literature, and new data if it is possible to obtain them. Optimally, members of original survey teams might be consulted for further assistance in making sense out of the data. |
| Keep study quality as high as possible and have uniform quality standards. | Convene an expert group of nutrition scientists to set uniform standards, review prior country surveys, and make independent judgements of their quality. List the various threats to validity and sources of bias to which each study is vulnerable. Since these threats vary in importance from one study to the next, no absolute rating of quality will be possible. Also group studies by the methodology used for dietary surveys, grouping similar studies of similar relative quality together. Exclude flawed studies; in the future devise ways of handling them by correcting their biases. |
| If controls are available. examine variations among both treatments and controls across studies. | Many prior country surveys have no controls. Examine both control and . treatment groups in large-scale randomized clinical trials. |
| Pay attention to variations between studies in effects and the causes of these differences. | Use approaches for studying the variation between studies such as fixed, random, and mixed-effects models. Consult statisticians to make . appropriate choices. |
| Recognize that there may be dependence problems. | Be wary of the many intercorrelations that exist when several dietary characteristics are compared to several health effects. Also, often the same group of investigators has done several of the studies. Use statistical procedures for dealing with these problems. |
Adapted from ref. 19.
Develop archives for prior country surveys and other relevant data sets
A consortium of academic and government investigators should work with international and voluntary agencies to develop and maintain archives of data from dietary surveys. One laudable step in this direction on the country level is the development of archives and public-use tapes for the National Health Examination Survey. Another is INFID (the International Food Intake Directory).
Databases from large-scale clinical trials involving diet that are conducted or funded by government are currently less complete and less readily available for public use than national survey data.
Consideration should also be given to developing means for ensuring their more systematic and rapid inclusion in archives. They should be easily accessible by all responsible investigators, not only those who were involved in the initial studies, and in a timely fashion. Requirements that such data be prepared for submission to the archives in a standardized and well-documented fashion might be included as preconditions on government grants and contracts.
Such measures might include a three- to five-year lag time after the conclusion of the study to give the original investigators time to complete their major papers and safeguard their investment in the research. After that time the data would be more easily accessible and in a more timely fashion to the broader community of scholars than they are today with the net effect that analyses and advances in knowledge might proceed more quickly.
Develop and maintain archives for nutrient databases at various time and places
It is important to have easily accessible and standardized tables of food composition and nutrient data bases for analyses of prior country surveys and other data sets. Those who have constructed the different databases have come to the task with very different assumptions. Before comparisons among studies are possible, these differences must be reconciled.
Converting food intakes into nutrients is a complicated process that is fraught with potential error [21]. The errors introduced by differences between various nutrient databases are considerable, even for comparisons of databases in common use in a single country.
One pressing research need is to develop more complete nutrient databases. Even in highly industrialized countries, knowledge of the amounts of some of the nutrients and other substances known to he present in food and to have potent biologic, effects is imprecise or lacking altogether [22]. 1 many developing countries, even basic staple foods may not have all been analysed.
High-quality food composition data collection involves careful description and analysis of samples, including the name of the food, its origin, the nature of the sample collected, treatment of sample before analysis, analytic methods, and a method of expression of results (e.g., edible portion, as purchased etc.). Even for similar foods there are always differences in the constituents when various tables are compared.
When results from tables used at different times or in different countries are compared, variability is even greater. In part, this is due to the inhere, variability of the foods themselves and their natural biological differences. Analytic techniques may vary, as may manufacturing processes and other preparation methods. For all these reasons it is vital that all tables used be carefully documented.
Many problems must be overcome to standardize databases. It is important to make sure that similar foods are called by standardized names. This is easy to do within a study, but difficult across studies, making comparisons difficult. Another problem is how to handle values, imputed values, and zeros. When data are missing, imputed values, which involve individual judgements, must be used. These issues must be documented. Among the choices are imputations from similar items, from recipes, from food groups, or by educated guesses.
Recipes are another source of differences between databases. There are many different ways of making a cake or any other product. It is important that the finished foods be somehow identified so that it is clear how they are made.
Assemble research groups with the expertise to analyse the data
Seldom can a single individual accomplish all aspects of such an analysis without the consultation or collaboration of experts. To make sense out of different surveys, good judgement, experience, and understanding of methods are critical. Investigators must have mastery over the subject to be studied and basic techniques in the fields of both nutritional and health assessment. Basic research principles must be well understood. Familiarity with the places and times that are being studied is also helpful.
Today our understanding of changes in diet at diverse times or places and how they relate to health outcomes is unfocused. New studies promise to provide better information for future generations of analysts, and they deserve our continuing support. However, even greater methodological standardization is needed.
New studies are expensive, and it will be many years or decades before trends in morbidity and mortality may become apparent. Therefore the use of extant country survey data continues to have much to recommend it. The financial resources for supporting such analyses must be found. We cannot afford to lose all of this valuable material. The good lessons summarized in Guidelines for Use of Dietary Intake Data [23] provide positive steps we can take now to improve analyses.
Partial support for the preparation of this paper was provided from grant MCJ 9120 and MCH training grant 8241 from the Maternal and Child Health Service, US Department of Health and Human Services, and federal funds from the US Department of Agriculture, Agricultural Research Service, under contract 53-3KO6-5-10 to the USDA Human Nutrition Research Center on Ageing at Tufts University.
The authors thank David Rush and Frederick Mostellar for their helpful comments.