Symposium: Challenges in Targeting Nutrition Programs
Methodological Challenges in Performing Targeting: Assessing Dietary Risk
for WIC Participation and Education 1 Laura E. Cauleld 2 Center for Human Nutrition, The Johns Hopkins University, Bloomberg School of Public Health, Baltimore, MD 21205 ABSTRACT This paper summarizes recent evaluations of indicators of dietary risk used to determine eligibility and nutrition education and counseling within the Special Supplemental Nutrition Program for Women, Infants, and Children (WIC). The concept of dietary risk relates to an inadequate dietary pattern or a failure to adhere to the food guide pyramid and, within WIC, is typically assessed based on an individuals intake from a 24-h recall or FFQ. Available evidence suggests, however, that even with a high-quality technique, substantial error exists in these data at the level of the individual, making the likelihood of misclassication high. Such data, with appropriate statistical procedures, can provide valid information at the group or the population level, and this is a future area for indicator development and incorporation into the management of WIC and other nutrition education programs addressing dietary risk. J. Nutr. 135: 879881, 2005. KEY WORDS: targeting nutrition program diet nutrition risk indicator There is a long history of programs designed to improve the nutrition and the health of vulnerable populations in the United States and throughout the world. Increasingly, there is interest in evaluating the effectiveness of these programs to document impact and to improve performance, as well as pressure to provide evidence for continued or expanded fund- ing. Early evaluations of nutrition programs (1,2) did not demonstrate the impact desired by many in the scientic and programmatic communities, and, since then, a literature has emerged to address many of the methodological issues that may explain less than desired impact. Principal among these issues has been the idea that nutrition programs could be more effective if they were better targeted so that they would reach only those individuals who truly need or would benet from the services offered (3). This can be accomplished if informa- tion is available that adequately distinguishes among individ- uals for this purpose; that is, information that screens individ- uals and correctly classies who should or should not receive program services. Screening for entry into nutrition programs based on income or nutritional risk is common, and, more recently, there have been reviews of the progress made to date on the use of screening indicators for one of the largest nutrition programs in the United States, the Women, Infants, and Children (WIC) 3 Program. The purpose of this paper is to use this information to highlight key methodological issues for effectively targeting programs with nutritional outcomes. The WIC program and the 1996 and 2002 Institute of Medicine reports The Special Supplemental Nutrition Program for WIC began in 1972. The intent of the WIC program is to support and strengthen health care during critical periods of growth and development, to prevent health problems, and to improve health status. To that end, it provides supplemental foods, nutrition education, and health referral services to low-income pregnant or postpartum women, infants, and young children. Currently, it is estimated to serve about 8 million participants per month (4). The WIC program is a grant program for which funding limits are set annually by the U.S. Congress. In addition to categorical and income eligibility criteria, individuals must show evidence of some form of nutrition risk based on either anthropometric, biochemical, medical, or dietary factors. These criteria allow for the prioritization of individuals based on health risk and the potential to benet from the program in the event of limitations in funding. For example, low-income pregnant women and infants with anthropometric or hemato- logical risk (e.g., low weight, low BMI, anemia) are given the highest priority, whereas low-income pregnant or postpartum 1 Presented as part of the symposium Challenges in Targeting Nutrition Programs given at the 2004 Experimental Biology meeting on April 20, 2004, Washington, DC. The symposium was sponsored by the American Society for Nutritional Sciences. The proceedings are published as a supplement to The Journal of Nutrition. This supplement is the responsibility of the Guest Editors to whom the Editor of The Journal of Nutrition has delegated supervision of both technical conformity to the published regulations of The Journal of Nutrition and general oversight of the scientic merit of each article. The opinions expressed in this publication are those of the authors and are not attributable to the sponsors or the publisher, editor, or editorial board of The Journal of Nutrition. The Guest Editors for the symposium publication are Edward A. Frongillo and Jean-Pierre Habicht, Division of Nutritional Sciences, Cornell University. 2 To whomcorrespondence should be addressed. E-mail: lcaule@jhsph.edu. 3 Abbreviatons used: IOM, Institute of Medicine; Se, sensitivity (Se); Sp, specicity (Sp); WIC, Womens Infants and Children. 0022-3166/05 $8.00 2005 American Society for Nutritional Sciences. 879
b y
g u e s t
o n
A p r i l
3 ,
2 0 1 1 j n . n u t r i t i o n . o r g D o w n l o a d e d
f r o m
women and low-income children who are at health risk due to a poor diet receive lower priority for WIC participation. The performance of such a system rests on the quality of the information used to evaluate and classify individual women and children within the prioritization scheme. In the last 10 y, 2 separate Institute of Medicine (IOM) committees have eval- uated the performance of the indicators used for this screening process (5,6). Here our focus is on issues raised in the second report regarding the ability to screen individuals with respect to the criteria, dietary risk, which refers to dietary decien- cies that impair or endanger health, such as inadequate dietary patterns assessed by a 24-h dietary recall, dietary history, or food frequency checklist [Code of Federal Regulations Sub- part C, Section 246.7(e) (iii) (2)]. Operationally, this has been extended to 2 certication criteria, inadequate diet and failure to meet the dietary guidelines. The nature of dietary risk The construct, dietary risk is complex and not open to direct measurement. An individuals risk results from multiple individual decisions made daily over extended periods of time. For example, whether an individuals usual fat intake is low, average, or high depends to a great extent upon the frequency with which they have days of high-fat intake and that on many days, the fat intakes of these individuals will be similar. The nature of the decision process, the choices available, and multiple other inuences lead to the fact that dietary intakes are inherently unstable from day to day within individuals, and, even in circumstances of reduced resources, dietary in- takes are subject to substantial within-subject variability. De- pending on the nutrient, within-subject variability may be 16 times greater than among-subject variability (7,8). Assessing the dietary intakes of children is further complicated, because dietary patterns change as children age, and many individuals (e.g., parents, child, daycare provider, school) usually deter- mine a childs dietary intake. Two distinct approaches have been taken to deal with this variability and to characterize usual intake. First, one can collect multiple days of intake and average the data as an estimate of usual intake; the mean is considered to be an accurate (unbiased) estimate of ones usual intake, with the variance of the data indicating the precision of the derived mean. Second, one can use an FFQ, in which the individual is asked to address the variability by summarizing their usual intake of specic food items, based on their knowledge of their day-to-day dietary choices. It should be noted that one can use either method to quantify usual intakes of foods or food groups to assess dietary patterns or adherence to the food guide pyramid or dietary guidelines; it seems, however, that within- individual variability in food intake is similar or greater that that for nutrient intakes. Because of the inherent difculty in estimating usual di- etary intakes, scientists have also focused attention on the potential of what are called behavioral indicators. Behav- ioral indicators do not measure diet per se but rather depict factors that inuence food choices or dietary intakes (9). Such factors could include consumption of specic foods or groups of foods, specic meal patterns (percent of meals consumed outside the home), health-related behaviors, psychosocial characteristics, family food practices, and ecological factors related to the home, neighborhood, or community. Indicator performance Although one can identify a list of criteria with which to evaluate the usefulness of indicators for screening or other purposes (e.g., feasibility, cultural appropriateness), the hall- mark criteria for indicator evaluation are accuracy and reli- ability (10). Accuracy addresses whether an indicator is really measuring what is intended vis-a`-vis some truth. Reliability refers to the reproducibility of the measure over timethat is, whether repeated measurements provide the same results. Intraindividual variability in dietary intake is real, but its presence acts as a form of random error and diminishes the reliability of the measure of usual dietary intake. In the case of recall data, multiple recalls would provide an accurate estimate of usual intake, and information on the within-subject vari- ance (s w 2 ) or within-subject coefcient of variation (CV w ) can be used in a common statistical formula to calculate the number of days (replicates) needed to attain a desired level of precision of that estimate (7,8). For example, for nutrients with CV w of 3070%, 1540 replicates (days) would be needed to estimate mean intake with a precision of 20% around the true mean. Because the collection of this level of replication may not be feasible, an important judgment must be made with respect to the desired precision with which one would like to make critical decisions, for example, regarding an individuals WIC eligibility. These results highlight the potential advantage of FFQs, if they can provide valid and reliable information on an indi- viduals usual intake. For FFQs, reliability is expressed in terms of the reproducibility of the intake estimates when the FFQ is administered on multiple occasions, and accuracy is evaluated by comparisons of intakes with estimated intakes from multi- ple records or dietary recalls. As reviewed by the IOM (6), reliability coefcients for FFQ generally range from 0.7 to 0.9, and accuracy coefcients generally range from 0.3 to 0.7 across nutrients. An emerging literature provides information on the reli- ability and the accuracy of these behavioral indicators for assessing dietary risk (6). Results vary widely across the indi- cators. For example, the reliability can be near 0.0 for some psychosocial indicators but be moderate (0.50.7) for specic food pattern indicators. The accuracy of these indicators varies as well, but few have accuracy coefcients 0.30. For screening purposes, the WIC professional wishes to estimate and to characterize the individual clients usual nu- trient or food intake and decide whether or not they are at dietary risk (eligible for WIC). Less than perfect measures of dietary risk result in misclassication of individuals regarding their true eligibility. The level of misclassication can be captured by the sensitivity (Se) and the specicity (Sp) of the indicator, with Se referring here to the ability of the indicator to identify those truly at dietary risk, and Sp referring to the ability of the indicator to identify those not at dietary risk. Less than perfect Se means that truly eligible individuals may not be classied as eligible and denied services, whereas less than perfect Sp means that others who are truly not eligible for the services may receive them. Walker and Blettner (11) calcu- lated the probability of misclassication in dietary intake for a given level of error in estimated dietary intake. As shown in Table 1, for accuracy coefcients between 0.3 and 0.7, only 26 to 40% of individuals would classied into the correct quintile of intake. Implications and future directions These results indicate that, even with rigorous measure- ment techniques, there is substantial error in estimates of individual dietary intakes using current methods. As discussed, this error is attributed largely to the complex nature of the dietary behaviors. Unfortunately, it leads to high levels of SYMPOSIUM 880
b y
g u e s t
o n
A p r i l
3 ,
2 0 1 1 j n . n u t r i t i o n . o r g D o w n l o a d e d
f r o m
misclassication of individuals with respect to WIC eligibility. It must also be considered that some errors are more costly than others. Whereas some may focus on the cost of ineligible individuals obtaining services, others would argue that the cost of missing individuals at risk is much higher, particularly in the long run. It is the task of policy makers and the public to decide how much and what type of misclassication error they are willing to tolerate when certifying individuals to receive or not receive federally funded WIC services; however, there is likely consensus that misclassication rates of 50% would not be tolerated. The measures used to assess risk also serve as the foundation for the provision of WIC services, including package tailoring and individualized nutrition education or counseling. Given the level of error in current individual assessment methods for dietary intake, the foundation for effective nutrition education and counseling is not present. New methods or indicators are needed for screening and formulating education and counsel- ing services. This discussion has focused on making inferences about individuals. Indicators can be used, however, to make infer- ences about individuals or groups of individuals. This is an important distinction, because the level of precision (lack of random error) required to make inferences at the individual level is much greater than for making inferences at the pop- ulation level. When making inferences about groups (or at the population level), it is still possible to derive unbiased esti- mates of the underlying phenomena of interest, using well- developed statistical procedures (12,13). For example, recalls or FFQs can be used to characterize the usual dietary intakes or food intake patterns of a target population, such as a WIC clinic area. This is true because even if individual values in the distribution are measured with error or are inherently unstable, these errors tend to cancel each other out, and individual values contribute information on the nature of the population distribution. Thus, with methods currently in use, the WIC program could characterize populations with respect to dietary risk and use the information to provide the context for edu- cation and counseling services. This was recommended by the second IOM committee (6), and, since then, efforts have begun to implement this shift in focus from individual to population. LITERATURE CITED 1. Beaton, G. H. & Ghassemi, H. (1982) Supplementary feeding pro- grams for young children in developing countries. Am. J. Clin. Nutr. 35 (suppl.): 863916. 2. Abrams, B. A. (1993) Preventing low birth weight: does WIC work? A review of evaluations of the special supplemental food program for women, infants and children. Ann. N.Y. Acad. Sci. 678: 306316. 3. Habicht, J. P. & Pelletier, D. L. (1990) The importance of context in choosing nutritional indicators. J. Nutr. 120: 15191524. 4. United States Department of Agriculture [Online]. http://www.fns. usda.gov/pd/ [accessed April 2004]. 5. Institute of Medicine (IOM) (1996) WIC Nutrition Risk Criteria: A Sci- entic Assessment. Food and Nutrition Board (FNB), IOM. National Academy Press, Washington, DC. 6. Institute of Medicine (IOM) (2002) Dietary Risk Assessment in the WIC Program. Food and Nutrition Board (FNB), IOM. National Academy Press, Wash- ington, DC. 7. Willet, W. (1998a) Nature of variation in diet. In: Nutritional Epidemi- ology. Monographs in Epidemiology and Biostatistics, Vol. 15 (Willet, W., ed.), pp. 3349. Oxford University Press, Oxford, UK. 8. Nelson, M., Black, A. E., Morris, J. A. & Cole, T. J. (1989) Between- and within-subject variation in nutrient intake from infancy to old age: estimating the number of days required to rank dietary intakes with desired precision. Am. J. Clin. Nutr. 50: 155167. 9. Baranowski, T. (1996) Psychosocial and sociocultural factors that inuence nutritional behaviors and interventions: Cardiovascular disease. In: Be- yond Nutritional Recommendations: Implementing Science for Healthier Popula- tions (Garza, C., Haas, J., Habicht, J.-P. & Pelletier, D., eds.), pp. 163188. Cornell University, Ithaca, NY. 10. Rothman, K. J. (1986) Modern Epidemiology. Little, Brown, Boston, MA. 11. Walker, A. M. & Blettner, M. (1985) Comparing imperfect measures of exposure. Am. J. Epidemiol. 121: 783790. 12. Traub, R. E. (1994) Reliability for the Social Sciences, Theory and Applications. Sage Publications, Thousand Oaks, CA. 13. Willet, W. (1998b) Corrections for the effects of measurement error, In: Nutritional Epidemiology Monographs in Epidemiology and Biostatistics, Vol. 15 (Willet, W., ed.), pp. 302320. Oxford University Press, Oxford, UK. TABLE 1 Probabilities of correct classication of quintile of dietary intake depending on the level of error in the indicator 1 Correlation coefcient (r) 0.8 0.7 0.6 0.5 0.4 0.3 0.2 Correctly classied (%) 46.7 40.3 35.7 32.1 29.0 26.3 24.0 1 Adapted from Ref. 11. METHODOLOGICAL CHALLENGES IN PERFORMING TARGETING 881
b y
g u e s t
o n
A p r i l
3 ,
2 0 1 1 j n . n u t r i t i o n . o r g D o w n l o a d e d
f r o m
ERRATUM WALTERC. RUSSELL,M. WRIGHTTAYLORANDJAMES V. DERBY,JR. The folie acid requirement of turkey poults on a purified diet. Journal of Nutrition vol. 34, no. 6, December, 1947. Page 632: second line from top of page to be changed to 100 gm of purified diet (1.5 mg per kilo) for optimum growth
b y
g u e s t
o n
A p r i l
3 ,
2 0 1 1 j n . n u t r i t i o n . o r g D o w n l o a d e d
f r o m
ERRATUM Chakrabarty, Krishna, and Gilbert A. Leveille 1968 Influ ence of periodicity of eating on the activity of various enzymes in adipose tissue, liver and muscle of the rat. J. Nutr., 96: 76- 82. In tables 1, 2, and 3, pages 78 and 79, headings immediately above columns of data should read milliunits/mg protein * in stead of units/mg protein l. b y
g u e s t
o n
A p r i l
3 ,
2 0 1 1 j n . n u t r i t i o n . o r g D o w n l o a d e d