Contents - Previous - Next

This is the old United Nations University website. Visit the new site at http://unu.edu

Other considerations

A system for evaluating the quality of published nutrient data: Selenium, a test case

Consideration of food composition variability: What is the variance of the estimate of one-day intakes? Implications for setting priorities

Dietary assessment methods used by the national health and nutrition examination surveys (NHANES)

Systems considerations in the design of INFOODS

A system for evaluating the quality of published nutrient data: Selenium, a test case

Introduction
Background
Procedure
Criteria
Calculation of the mean SE value and confidence code
Results
Discussion
Implications
Acknowledgements
Disclaimer
References

JOANNE M. HOLDEN, ANITA SCHUBERT, WAYNE R. WOLF, and GARY R. BEECHER

Nutrient Composition Laboratory, Beltsville Human Nutrition Research Center,
Beltsville, Maryland, USA

Introduction

Food composition data are used by nutritionists, dietitians, and epidemiologists to assess the adequacy of the diets of population groups, subgroups, and individuals. They are used to determine federal and state government policy regarding food and nutrition programmes and other public health efforts. Data relative to the assessment of nutrients and contaminants such as pesticides are used to formulate the policies of regulatory agencies such as the Food and Drug Administration (FDA). In addition, the food industry uses food composition data to assist in food labelling, quality assurance, and product development. Ideally, data to be used for such varied and far reaching assessment and policy formulation should be accurate and precise by analytical standards, and should represent both the qualitative and quantitative standards and the qualitative and quantitative distribution of nutrients or other components found in the food supply consumed by the group or individuals to be studied. Because data obtained in such a manner that they meet all of these requirements are scarce, critical evaluations of the limitations of available published data are needed.

Food composition data of variable quality and quantity can be collected from many different sources. They can be obtained from manufacturers of food products; these may include individual values for replicates or mean values for specific samples. Data can be taken from product labels. Certain data are obtained directly from the analysis of foods purchased and sampled specifically for the purpose of determining their composition. These studies may or may not be published. Food composition data may also be gathered from the scientific literature as the indirect results of the development of analytical methodology, animal feeding trials, soil treatment trials, and bioavailability studies. While these published studies may accomplish stated objectives, not all the data may be suitable for use in food composition tables and data banks. Since the requirements of data users can be diverse, the suitability of specific data for inclusion in a data base should be evaluated according to objective criteria that are known to users of the data. Some indicator of data quality is necessary for each nutrient in each food to provide the data user with a measure of the reliability and usefulness of specific values in the data base.

The main objective of this work was to develop a set of criteria that could be applied to published analytical data (a subset of available nutrient data) for any specific nutrient or component in foods. Selenium (Se) was selected as a test case for these criteria because of the current high level of interest in this nutrient and the need for accurate assessment of Se intake in a number of ongoing human studies. Furthermore, published Se data have been generated by a limited number of satisfactory analytical methods and constitute a finite data set suitable for testing such a system of criteria.

A secondary objective was to provide analysts with a set of guidelines for designing nutrient composition studies and reporting their results. Such guidelines can also be used by journal editors and their reviewers to evaluate scientific papers submitted for publication and to elicit details of a study important to compilers of nutrient composition data.

Criteria developed specifically for Se were applied to the body of Se data available in the scientific literature. Ratings determined in the process were combined to yield a confidence code (CC) for the mean Se value of each food item reviewed. This paper will describe these criteria and their application, and give several examples. (A table of foods with their Se concentrations, respective CCs, and specific references will be published separately.)

Background

In 1980 the need for objective criteria for evaluating food composition data was recognized by USDA workers at the Nutrient Composition Laboratory of the Agricultural Research Service and the Nutrient Data Research Branch of the Human Nutrition Information Service. Discussions led to the development of data quality criteria which were used to evaluate iron data for publication of the provisional table Iron Content of Food [8].

The various studies for specific foods were evaluated according to criteria in three categories: (a) documentation of analytical method, (b) sample handling and appropriateness of analytical method, and (c) (analytical) quality control. Scores for these criteria led to the assignment of a CC, which appeared in the iron table adjacent to the mean iron concentration. For the first time, users of a nutrient composition table were provided with a measure or degree of confidence in the mean value for that particular food. Asterisks attached to a CC indicated either a limited number of sources or the extent of variability.

Recently, Stewart [26] reiterated the importance of evaluating nutrient data for inclusion in data bases. He recommended several critical activities that contribute to the generation of highquality analytical data on foods, including (a) development of appropriate validated analytical methods, (b) use of sound food-sampling techniques to ensure representativeness of samples, and (c) use of appropriate quality-control systems in conjunction with validated methodologies to ensure the production of validated composition data.

Although we know of no other similar effort in the field of food composition data, some researchers in other fields have been concerned with systems for the evaluation of data quality. W. Mabey et al. [12] have recently published a similar system of criteria to be used for the evaluation of quantitative data to be included in a data base for environmental fate assessment. For each of four major categories, specific check-lists of criteria were presented to permit a thorough evaluation of each datum, resulting in the derivation of a data quality index. This index indicates to the data-base user that an objective evaluation of data quality has been done and should help to educate users and data generators and improve the overall excellence of the data.

Procedure

Fig. 1. Flowchart of evaluation system for published selenium data.

The system of objective criteria described here for the evaluation of published nutrient composition data reflects the basic concepts described by Exler [8] and incorporates the critical aspects cited by Stewart [26]. The system involves several stages (fig. 1): (a) development of categories of criteria to include key issues relative to analytical methodology and sampling; (b) review of literature and collection of those papers that reported the Se content of foods and related methodology; (c) definition of criteria in each category at four rating levels to reflect specific concerns that pertain to the evaluation of literature reporting the Se content of foods; (d) assignment of ratings for each criterion by study and by food item; (e) derivation of the quality index for each study; and (f) derivation of mean value and CC for the Se content in each food by combining data from the acceptable studies.

Five general categories were developed to evaluate the data for each food item: (a) number of samples, (b) analytical method, (c) sampling handling, (d) sampling plan, and (e) analytical quality control. These categories identify the issues essential to any study related to the composition of foods. A rating scale of 0 (unacceptable) to 3 (most desirable) was established. Within each category, the criteria for each rating level were defined specifically for Se. Establishing these criteria demanded knowledge of accepted methodology, sample handling procedures, and quality-control measures specific to this nutrient. In addition, knowledge of statistical methods was required for the sampling plan and number of samples categories. The rating criteria are outlined in table 1 and described in detail below. In general, within each category the level of documentation and appropriateness of procedures are addressed.

An extensive literature search yielded approximately 65 papers (from 33 different journals, reports, proceedings, and books) published after 1960 that report original analytical Se food data. Several references include data from more than one study. Papers published prior to 1960 were collected for historical purposes, but the data were not included in this evaluation because of difficulty in assessing the validity of the methodology used and conceivable lack of relevance of those data to current studies of food consumption due to possible changes in the food product during the last 25 years. The methodology papers referenced in the data articles were also collected. In addition, data from recent FDA Total Diet Study analyses, unpublished at the time of this evaluation [23], were included because this programme is one of the few that has analysed food as eaten, i.e. cooked foods and mixtures. Data from fiscal years 1982/83 and 1983/84 were treated separately, providing two sets of means for each food item analysed. Since the focus of this work was the Se content of foods frequently consumed by Americans, only those studies that analysed foods grown, processed, or sold in the United States and Canada were collected.

As previously mentioned, the data from each study were rated on the criteria on a scale of 0 to 3. In general, O was assigned when information was not adequate to permit evaluation of data for use in food composition data bases, or when certain procedures or practices were inappropriate. A 3 was assigned when procedures were well-documented and appropriately applied.

Criteria

Number of Samples

Statistical rigour requires that the appropriate number of samples for a study be a function of the nutrient variability within the population of each food item [6].

Table 1. Data quality criteria^a

a. See text for complete description of criteria.

Both intrinsic and extrinsic sources of variation affect the levels of Se in foods. However, when evaluating published data, one rarely has access to the magnitude of specific sources of variation for a given food. The standard deviation, when given, is an indication of total variance. Some variability is assessed indirectly by the other categories of criteria and includes: systematic error intrinsic to the analytical methodology or sample handling, variability attributable to differences in brands and varieties of foods analysed, and errors in analytical accuracy and precision in the execution of the analytical method. A statistical formula can be used to estimate appropriate sample size given a mean, standard deviation, and level of acceptable error [6]. However, the error term selected would depend upon the concentration of the nutrient in a given food item and the detection and quantitation limits [11] for the analytical method. Such a judgement could be made given sufficient data for each food. Using a predetermined coefficient of variation of 20 per cent as the limit of acceptable error, we estimated appropriate sample size for a limited number of studies where standard deviations were reported. Our estimate and the actual number of samples analysed in each study were comparable.

In the absence of adequate information for many studies, particularly standard deviation data, we chose to make a somewhat subjective judgement on the sample size limits for each rating: a rating of 1 for one to two samples or when number of samples is not specified; a rating of 2 for three to ten samples; and a rating of 3 for greater than ten samples and inclusion of the standard deviation, standard error, or raw data from which a standard deviation can be calculated. A rating of 0 is not applicable in this category. As documentation improves, it will be possible to evaluate the appropriateness of the number of samples analysed based on statistical considerations.

Analytical Method

Several issues related to the method of analysis are important with regard to ratings on analytical method. Documentation of whatever method is used is primary: suitability of method cannot be determined if no description or reference to details in another paper is included. A rating of 0 is assigned where the method is not described, no reference is given, or a reference is generally unavailable. In some cases the use of "official" methods for the analysis of the specific foods merits a higher score. The use of the official fluorometric method for Se analysis, as published in the handbook of the Association of Official Analytical Chemists (AOAC) [27], is rated a minimum of 2. However, use of an "official" method does not preclude attention to a second important issue: validation of the method.

Validation of the test method for the general matrix (e.g. meat, grain, fat), and preferably for the specific food item in question, is necessary to show that accurate results can be obtained. Use of recovery trials on the same or a similar food is one aspect of method validation. Higher ratings are earned for recovery close to 100 per cent and similarity to the food analysed of the food on which the recovery trials are done. The use of a highly regarded second analytical method on the same or a similar food is another aspect of method validation. In the case of Se, highly regarded analytical methods are the fluorometric method approved by the AOAC, neutron activation analysis, and isotope dilution-mass spectrometry. The analytical results of this second method must be in good agreement with the results of the test method, or a 0 is assigned. Good agreement is defined according to our adaptation of Stewart's general recommendation that the values obtained by comparable methods should be within 10 per cent of each other if a daily intake of the food provides greater than 5 per cent of the US RDA for that nutrient [25]. Since there is no US RDA for Se, we have used 5 per cent of the lower end of the estimated safe and adequate daily dietary intake of 50 200,ug of Se, as recommended by the Food and Nutrition Board of the National Research Council [7].

The analysis of a standard reference material (SRM) by the method in question and comparison of the value obtained with the certified value and the range of estimated uncertainty of the SRM is useful for validation when the matrix and Se concentration of the SRM are similar to that of the food in question [28]. However, the small number of SRMs that are certified for Se has confined this aspect to a rating of 3 for analytical method. As the availability of SRMs with a variety of matrices increases, this aspect of the criterion will be required for ratings below 3.

Finally, evidence that the analysis is carried out above the quantitation limit of the method, as defined by the American Chemical Society Committee on Environmental Improvement [11], is required to assure that the method can determine expected levels of Se in the food to be analysed. For a rating of 3, the quantitation limit must be defined and be below the Se level reported for the food in question.

Other issues with regard to analytical method are of some concern, but were considered secondary to the main points. One such issue is that of the size of the samples that are analysed. Sample size must be adequate relative to the sensitivity of the method so that (a) the concentration is above the quantitation limit, and (b) the sample analysed is representative of the whole food item. This concern was not included in analytical method requirements.

In summary, three concerns must be satisfied for data to be rated 3 in analytical method: (a) a complete description of the method in the same or another accessible publication; (b) validation studies for the food in question, which can consist of either recovery trials with 95 per cent to 105 per cent recovery of Se or comparable results with use of a second method that is highly regarded, as well as use of an appropriate SRM when available; and (c) reported analytical values above the defined quantitation limit of the method.

Sample Handling

How a sample is handled from the time of acquisition to the time of analysis is critical for general nutrient stability. For example, preventing the loss of volatile components is important for maintaining relative nutrient concentration, a factor of importance for Se. Therefore, documentation of sample handling protocol is essential to evaluate data pertaining to the nutrient composition of a food. Lack of documentation of sample handling procedures or use of inappropriate procedures is rated 0. Se contamination of food samples via utensils, cooking ware, grinder, or containers is not a problem, in contrast to the analysis of other inorganic nutrients such as zinc and chromium. However, details of homogenization, temperature control, and other aspects of sample preparation must be known to evaluate the representativeness of a sample aliquot taken from a large batch of prepared material.

Analysis of the edible portion must be reported for a rating of 1 or higher. For example, some canned foods must be drained, raw fruits and vegetables must be peeled or cored, and meats must be boned and trimmed of fat if these foods are generally eaten that way. Thorough homogenization of the food is critical for food items with diverse constituents. Examples of such foods are: breaded and fried fish or poultry, food mixtures, and fruits or vegetables eaten with skin or seeds. Ideally, thoroughness of homogenization is checked by analysing portions from various parts of the final mixture. Additional factors that should be reported for the highest rating are: detailed description of the food, including processing methods (e.g. whether rice was polished, unenriched, or instant); cooking method (if any); general storage conditions, e.g. frozen foods kept frozen, fresh foods analysed soon after pick-up; and measurements of moisture/volatiles content.

Sampling Plan

The sampling plan of the study reflects the representativeness of the samples with regard to the brand or cultivar, method of preparation, and geographic origin of the food. Is the particular food item typical of what many Americans eat? No description of the sampling plan or the use of a non-representative sample is rated 0. This would include a food grown under experimental soil conditions, food grown in someone's home garden, or food prepared in an unusual way. Data from Canadian studies were evaluated because some foods sold in Canada are grown in the United States - e.g. fruits and some grains - and therefore can be representative of what Americans eat. Data for foods grown on Canadian soil but not exported to the United States were given a lower score (0 or 1) for sampling plan since the concern of this work was foods consumed by Americans. The use of popular brands and frequently consumed forms of foods was rated 2. Obtaining representative samples from supermarkets in more than two wellpopulated areas was rated 3. For fresh foods obtained from growers or producers, representativeness of cultivar and geographic source was assessed by referring to Agricultural Statistics [29].

Analytical Quality Control

Information that details acceptable accuracy and precision in the day-to-day execution of an analytical method is necessary for evaluating the quality of nutrient data. Accuracy and precision are each rated as separate aspects of analytical quality control. For each datum, the lower rating of the two aspects determines the rating in this category.

Accuracy is the degree to which an analysed value represents or estimates the "true" value. An investigator must demonstrate that the method is capable of accurately determining the nutrient level in a particular food item; that is, a method must be validated for each general matrix, as described in the analytical method section above. Once the method has been validated, it must be carried out appropriately each time an analysis is performed. Accuracy in the day-to-day use of a method is one of the two elements that must be monitored and reported for a study's data to be rated favourably in analytical quality control.

Day-to-day accuracy is monitored by analysis of a quality-control material that is similar in matrix and nutrient concentration to the test sample. Analysis of such a material should be included with each batch of unknowns or on each day of analysis if several batches are run in a day. Quality-control materials can be SRMs such as those available from the National Bureau of Standards (NBS), which are certified for specific nutrients, or they can be secondary reference materials, i.e. materials developed especially for a study and characterized by one or more methods, including reference methods. NBS SRMs currently available and certified for use in evaluating the accuracy of a method for the determination of Se in foods are: Orchard Leaves 1571 [17], Wheat Flour 1567 [20], Rice Flour 1568 [19], Bovine Liver 1577a [15], Oyster Tissue 1566 [18], and Non-Fat Milk Powder 1549 [16].

Table 2. Accuracy requirements for secondary reference materials

When an NBS SRM is used to judge accuracy, the analysed mean value must fall within the mean plus or minus the estimated uncertainty, as stated in the NBS certificate, for the study to be rated a 3. Use of an SRM with results falling outside the range of estimated uncertainty is rated 0. This apparently stringent rule is based on the sources of SRM values. Many certificates of analysis do not label the uncertainty in statistical terms such as confidence intervals or a certain number of standard deviations. For example, the Rice Flour (no. 1568) certificate states [19]:

The estimated uncertainty is based on judgement and represents an evaluation of the combined effects of method imprecision, possible systematic errors among methods, and material variability for samples 400 mg or more. (No attempt was made to derive exact statistical measures of imprecision because several methods were involved in the determination of the constituents.)

When a secondary quality-control material is used, a reference mean and standard deviation should be obtained by analysis of the material by the same or another laboratory using a reference method. To monitor a method's accuracy, the formulated reference material should be analysed with each batch of unknowns or on each day of analysis, comparing the results to the reference mean and standard deviation as shown in table 2.

The review of Se references carried out for this work revealed that a reference material was frequently analysed at the outset of a study, serving to validate the method before analysis of unknowns begins. Occasionally a mean and standard deviation for a quality-control material were reported, but usually documentation was not sufficient to determine whether the material was analysed with each batch of unknowns. In such cases it was assumed that the investigator used the reference material only to validate the test method rather than to measure day-to-day accuracy. In this instance, an accuracy rating of no higher than 1 was assigned.

The other half of analytical quality control is the level of precision; the aspect of concern here is the amount of variability about the mean value associated with the day-to-day execution of a particular method. The indication of day-to-day variability can be determined only when the analytical method is monitored continuously through the use of a quality-control material similar in matrix to the unknowns to be analysed. Like accuracy, the day-to-day precision of a method is matrix-dependent.

Precision is usually measured by calculating the per cent coefficient of variation (% CV), also known as per cent relative standard deviation (% RSD), from the mean and standard deviation (SD) of several replicates of a sample: % CV or % RSD = SD divided by mean x 100 per cent. The lower the % CV, the more precise the analysis. Our limits for rating % CV are: 5 per cent or less for a 3, 10 per cent or less for a 2, and 15 per cent or less for a 1. The % CV, calculated from replicates analysed within a given laboratory, includes variability attributable to instrument and technician performance and to the method. The precision of a method could be poor in the hands of one investigator and acceptable in the hands of another.

Table 3. Assignment and meaning of confidence codes

When the ratings for accuracy and precision are the same, that rating becomes the analytical quality-control rating. When accuracy and precision ratings differ, the lower rating is assigned for the overall analytical quality-control rating for the study. Data from a study with incomplete documentation are rated no higher than 1 in this category. Lack of any documentation or unacceptable precision or accuracy earns a 0 in analytical quality control.

Calculation of the mean SE value and confidence code

The quality index (QI), a measure of the overall quality of data from a single study, was derived in one of two ways: (a) when analytical method is rated 0 or when three or more categories are rated 0, the QI for that study becomes 0; or (b) when those conditions do not exist, the ratings for each of the five categories are averaged. Thus, the QI can range from 0 to a maximum of 3.0.

The mean Se values reported on a fresh weight basis from those studies which have a QI equal to or greater than 1.0 are averaged together to obtain a mean Se value. Values reported on a dry weight basis cannot be combined with values reported on a fresh weight basis, i.e. foods as eaten, unless moisture levels are included with the dry values to permit calculation back to fresh weight. If no moisture levels are reported, dry values are excluded regardless of their QI.

A confidence code (CC), assigned to the mean Se value for each food, indicates the degree of confidence a user can have in the mean value. It is determined by summing the QIs equal to or greater than 1.0 among the various studies evaluated for a given food item, and then referring to table 3 for the corresponding CC. The basis for the CC is the necessity of confirming the results of one report by other investigators in order to be considered valid. Thus, data from a minimum of three studies with a sum of QIs of 6.2 are required for a mean value to be assigned a CC of a. The cut-off point between b and c of 3.4 was made by dividing approximately equally all the possible sums of QIs, i.e. from 1.0 to 6.0.

Results

Table 4 provides the Se data which were collected and evaluated for three foods: white rolls, whole milk, and canned crab. These foods were selected to show varying levels of Se among foods and within foods, different combinations of ratings, the typical range of QIs, and an example for each CC.

Analyses of white rolls were reported in three references. In reference 22, analyses were carried out on samples of the same product from two different sets of cities. Zeros were assigned in two categories: in analytical quality control because of poor precision in the second set of collections in reference 23, and because of lack of any documentation in references 3 and 22; and in sampling plan because of the Canadian origin of the product in reference 3. Onehalf of the ratings were 2, while 1 and 3 were assigned less frequently. The data from all four studies (three references) were found to be acceptable, and the four Se values were used to derive the mean value of 34 µg/100g, with minimum and maximum values of 21 and 61. The CC of a based on the sum of the QIs, 6.4, indicates that one can use this value with considerable confidence.

Sample analyses of whole milk were presented in 11 references which contained 15 studies. However, data from 11 studies were judged unacceptable based on a 0 rating in analytical method: lack of documentation, lack of validation of the method for milk, or reported Se levels below the stated quantitation limit. Out of the four remaining studies [2, 3, 4, 13], data from two [2, 3] were rated 0 in sampling plan due to their Canadian origin, while all were rated 0 in analytical quality control: references 3 and 4 reported no quality-control measures, while references 2 and 13 did not address both issues of accuracy and precision. Reference 4 does not explicitly report the number of whole milk samples analysed for Se; only the total number of samples is stated (103) which includes three types of milk in addition to whole milk. Although it is likely that a large number per type was analysed, the number of samples category received a 1 due to insufficient documentation. In two studies [2, 3], only one sample (in duplicate) was analysed. The mean Se level in whole milk, based on the data from four acceptable studies, is 1.6 µg Se/100g, with a minimum and maximum of 1.1 and 2.5. The sum of QIs is 5.6, which establishes a CC of b, indicating that a user of this datum can have some confidence in the value.

Canned crab samples were analysed by two sets of workers, with a Se value from only one study [5] receiving an acceptable Ql. The investigators reported no analytical quality-control measures (rated 0), analysed duplicate portions of only one sample (rated 1 in number of samples), and recovered only 80 90 per cent of Se in recovery trials (minimal validation results rated I in analytical method). Thus, the concentration of 22µg Se/lOOg is based on only one sample, with ratings of 0 to 2 in the other four categories. The CC of c makes it clear that more Se data are needed on canned crab.

Table 4. Data quality evaluation for white rolls, whole milk, and canned crab

a. An index > 1.0 is required for inclusion of an individual Sc value in the calculation of the mean.
b Conversion from volume to weight based on item no 01-077. "Whole Milk.'' in Us Department of Agriculture, Agriculture Handbook No. 8-/ (Science and Education

Administration USDA. Washington, D C., 1975).

Contents - Previous - Next