Division of Health Interview Statistics

DATA EVALUATION Series 2 Number 54 AND METHODS RESEARCH Quality Control and Measurement of Nonsampling Error in the Health Interview Survey A report describing the selection and training of interviewers, interviewer observation program, a reinterview program, measures of interviewer performance, the editing and coding of questionnaires, a response error study, and an interviewer variability study. The report is based on the Bureau of the Census experiencewith the Health Interview Survey. DHEW U.S. Publication DEPARTMENT OF No. HEALTH, Public Health Health Services National Rockville, Md. (HSM) 73-1328 EDUCATION, AND Service and Mental Health Administration Center for Health Statistics March 1973 WELFARE Vital and Health Statistics-Series 2-No. 54 For sale by the Superintendent of Documents, U.S. Government Printing Offlce, Washington, D. C. 2M02 Price 85 cents domestic postpaid or 60 cents GPO Bookstore NAT ONAL CENTER FOR THEODORE HEALTH D. WOOLSEY, STATIST Director EDWARD B. PERRIN, Ph.D., Deputy Director PHILIP S. LAWRENCE, SC.D., Associate Director OSWALD K. SAGEN, Ph .D., Assistant Director for Health S tatistics Development WALT R. SIMMONS, M.A., Assistant Director for Research and Scientific Development JOHN J. HANLON, M.D., Medical Advisor JAMES E. KELLY, D.D.S., Dental Advisor EDWARD E. MINTY, Executive Officer ALICE HAYWOOD, Information Officer OFFICE OF STATISTICAL M ETHODS MONROE G. SIRKEN, Ph.D., Director E. EARL BRYANT, M.A., Deputy Director DIVISION OF HEALTH INTERVIEW STATISTICS ELIJAH L. WHITE, Director ROBERT R. FUCHSBERG, Deputy Director RONALD W. WILSON, Chief, Analysis and Reports Branch KENNETH W. HAASE, Chief Survey Methods Branch COOPERATION OF THE BUREAU OF THE CENSUS Under the legislation establishing the National Health Survey, the public Health Service is authorized to use, insofar as possible, the services or facilities of other Federal, State, or private agencies. In accordance with specifications established by the National Center for Health Statistics, the Bureau of the Census, under a contractual arrangement, participated in planning the survey and collecting the data. Vital and Health Statistics-Series 2-No. 54 DHEW Publication No. (HSM) 73-1328 Library of Congress Catalog Card Number 72-600133 CS FOREWORD This report is one in a series designed to document the methodology of the Health Interview Survey (HIS) and to investigate the quality of HIS statistics. In previously published reports, the emphasis was on questionnaire development (Series 1, Number 2) and on sample design (Series A, Number 2). Other reports (e.g., Series 2, Numbers 6, 7, 18, 28, and others) present findings of methodological studies that investigated the accuracy of health data collected in household surveys. Specifically, this report deals with the quality-control procedures for the data-collection operations of the Survey. It describes procedures for selecting, training, supervising, and observing interviews and measuring interviewer performance, and for editing and coding questionnaires. It also describes the reinterview program. In this program, a staff of field supervisors and senior interviewers reinterview sub samples of households in the Survey. This report presents estimates of nonsampling error based on the reinterview program and estimates of the interviewer contribution to nonsampling variance based on results of a special study designed for this purpose. These statistics have a dual utility. They are useful in evaluating the quzdity of HIS data and in improving the design of the Survey. Through contractual arrangements with the National Center for Health Statistics, the Bureau of the Census prepares the sample and conducts the field collection process of the Health Interview Survey, and, until 1968, also carried out the data-coding and initial editing procedures. The particular quality-control procedures described in this report are essentially applications of methods that are used by the Census Bureau to monitor the field operations of national household surveys. However, the findings presented in this report relate only to the Health Interview Survey. The work for this report was done under a special contract with the Statistical Research Division, Bureau of the. Census, in close collaboration with the Office of Statistical Methods and the Division of Health Interview Statistics. Elijah L. White Director Division of Health Interview Statistics Monroe G. Sirken Director Office of Statistical Methods ... Ill SYMBOLS Data not available ---------------------------------------- --- Category not applicable ------------------------------- . . . Quantity zero ---------------------------------------------- Quantity more than O but less than 0.05 ----- Figure does not meet standards of reliability or precision ------------------------------ 0.0 * CONTENTS Page Foreword . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ... m . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1 1 2 3 Part L Control of Data Collection and Data Processing Introduction . . . . . . . . . . . . . . . . . . . . Selection and Training of Interviewers . . . . . . Selection . . . . . . . . . . . . . . . . . . . . Initial Training . . . . . . . . . . . . . . . . . Continuing Training . . . . . . . . . . . . . . Observation of Interviewers.. . . . . . . . . . . Introduction . . . . . . . . . . . . . . . . . . Types of Observations . . . . . . . . . . . . . Supervisory Reinterview Program . . . . . . . . . Introduction . .. . . . . . . . . . . . . . . . . Sample Design . . . . . . . . . . . . . . . . . Content of Reinterview . . . . . . . . . . . . . Field Procedures . . . . . . . . . . . . . . . . Quality Control of Interviewers’ Work . . . . . Editing and Coding of Completed Questionnaires . Introduction . . . . . . . . . . . . . . . . . . Regional Office Edit . . . . . . . . . . . . . . Quality Control of Clerical Coding Operations . Central Office Edit.......,.. . . . . . . Measures of Interviewer Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 3 3 3 4 5 6 6 6 8 8 8 8 8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . 16 16 17 17 17 24 24 24 26 28 32 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 Introduction . . . . . . . . . . . . . . Purpose of the Health Interview Survey Brief Description of the Survey . . . Control of the Survey Process . . . . Measurement of Nonsampling Error . . . . . . . . . . . . . . . . . . . . . . . . Part 11. Measurement of Nonsampling Error . . . Introduction . . . . . . . . . . . . . . . . . . Response Errors as Determined by a Reinterview Introduction . . . . . . . . . . . . . . . . HIS Reinterview Survey Results . . . . . . Interviewer Variability Study . . . . . . . . . Introduction . . . . . . . . . . . . . . . . Design of the Interviewer Variance Study . Method of Analysis . . . . . . . . . . . . . Results . . . . . . . . . . . . . . . . . . . Summary . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . Survey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 10 10 11 12 13 13 v CONTENTS–Con. Page Appendix . . . . . . . . . . . . . . 35 Appendix II. Time and Cost Model for HIS Interviewing . . . . . . . . . . General . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Models for HIS Assignments.. . . .. . . . . . . . . . . . . . . . . . . . 36 36 36 Appendix III. Some Theory of Measurement Errors . . . . . . . Some Definitions . . . . . . . . . . . . . . . . . . . . . . . . The Desired Measure or True Value . . . . . . . . . . . . . . . The General Conditions That May Affect the Results of a Survey An Estimate from a Survey (or Trial) Taken Under a Set of General Conditions . . . . . . . . . . . . . . . . . . . . . . The Mean Square Error of an Estimate from a Survey (or Trial) Gross and Net Differences . . . . . . . . . . . . . . . . . . . . Gross and Net Differences as Evidence of Response Variance and Bias . . . . . . . . . . . . . . . . . . . . . . . . . . . . Index of Inconsistency . . . . . . . . . . . . . . . . . . . . . . . . ... . . . . . . 38 38 38 38 . . . . . . . . . . . . . . . . . . 39 39 40 . . . . . . . . . . . . 41 42 . . . . . . . . . . . . . . . . . . . . 43 (NHS-HIS-406) . . . . . . . . . . . 45 . . . . . . . . . 48 Appendix Appendix I. Formula IV. for Computing Least-Squares Solution V. HIS Observation Appendix VI. HIS Reconciliation Appendix VII. Summary Appendix VIII. Production Report Report Error Rate Form (NHS-HIS-R-IX-T) of NHS-HIS Reinterview Guide for NHS (11-102C) . . . . . . . . . . . . (NHS-HIS-R-401) . 50 . . . . . . . . . . . . 52 QUALITY CONTROL AND MEASUREMENT OF NO NSAMPLING IN THE HEALTH ERROR INTERVIEW David A. Koons, Statistical Research Division, SURVEY U.S. Bureau of the Census INTRODUCTION This report presents a summary of procedures used in the Health Interview Survey (HIS) to control the quality of the data collection and data processing operations. It also provides some results of measurements related to the quality of HIS statistics. PURPOSE OF THE HEALTH INTERVIEW SURVEY The Health Interview Survey is an integral part of the program of the National Center for Health Statistics (NCHS). This program is designed to provide continuing statistical measurements of the extent of disease, disability, and other health characteristics of the population. The legislation authorizing the HIS, The National Health Survey Act,l contains the following provisions: (b) It is, therefore, the purpose of this Act to provide (1) for a continuing survey and special studies to secure on a non-compulsory basis accurate and current statistical information on the amount, dutribution, and effects of illness and disability in the United States and the services received for or because of such conditions: and (2) for studying methods and survey techniques for securing such statistical information, with a view toward their continuing improvement. This provision for methodological research has strongly influenced the NCHS program. Since the HIS began, emphasis has been placed on improving statistical output rather than on continuity and comparability of estimates. Changes to improve the methods and procedures used in the survey have been made since it began in 1957. BRIEF DESCRIPTION OF THE SURVEY A description of the HIS is necessary to understand the quality control procedures used during the collection and processing of data. The Health Interview Survey uses a questionnaire to obtain information on injuries, acute illnesses, chronic conditions, impairments, utilization of medical services, and other health topics, in addition to information about personal and demographic characteristics. The findings from the survey are tabulated for the Nation as a whole and published by NCHS. Separate reports are issued which cover one or more of the specific topics. The population covered by the sample for the Health Interview Survey is the civilian, noninstitutional population of the United States living at the time of the interview. Persons in long-stay hospitals, nursing and convalescent homes, and so forth are excluded from the universe to be sampled. The sampling plan of the survey follows a multistage probability design which permits a continuous sampl~mg of the civilian population of the United States. The first stage of this design consists of drawing a sample of 357 from about 1,900 U.S. geographic divisions called primary sampling units (PSU). A PSU is a county, a group of contiguous counties, or a standard metropolitan statistical area (SMSA). Within PSU’S, ultimate stage units called segments, selected from clusters of 18 neighboring households or addresses, are defined so that each one contains an average of six households. (In July 1968, the average segment size 1 changed from nine to six households.) Two general types of segments are used: (1) area segments, which are defined geographical y, and (2) other segments, which are defined from a list of addresses from the 1960 Decennial Census and a current Survey of Construction. Prior to interviewing in area segments, interviewers make a list of the addresses of all households or dwelling units in the selected segments. Wherever possible, the visit of the interviewer is preceded by a letter from the Director of the U.S. Bureau of the Census announcing that an interviewer may be expected to visit and setting forth the general purposes of the survey. The confidential treatment that will be accorded any information given is emphasized. As a general rule any adult member, 19 years of age and over, of a family may be interviewed concerning the characteristics of all the members of the family. Persons in the household who are not related to the head of the household are expected to answer all questions about themselves. Exceptions are made for persons who are not competent to answer for themselves. Persons aged 17-18 may respond for themselves, while persons under 17 must be responded for by an adult. The sample is evenly distributed throughout the year, so that interviews are conducted in approximately 800 households each week. Since household members interviewed each week are a random sample of the population, samples for successive weeks can be combined into larger samples. Thus the design permits both continuous measurement of characteristics of high incidence or prevalence in the population and, through the larger consolidated samples, more detailed analysis of less common characteristics and smaller categories. This continuous collection of information has administrative, operational, and technical advantages since it permits field work to be handled by an experienced, stable staff. In addition, this design eliminates biases due to the seasonal nature of certain conditions or the occurrences of short-run epidemics. Approximately 100 interviewers, about half of whom work each week, are used in the HIS. Each interviewer is assigned an average of three segments (about 18 households) as a week’s 2 work. (As of July 1968, an average interviewer assignment changed from two nine-household segments to three six-household segments.) The interviewers, as well as the entire field staff for the HIS, are employees of the U.S. Bureau of the Census. Specifications for the survey are established by the NCHS. In accordance ‘with these specifications, the U.S. Bureau of the Census selects the sample, conducts the field interviewing as an agent of the Center, and checks questionnaire entries. Data preparation, consisting of the preliminary editing and the coding of questionnaires, is carried out by the NCHS. Further editing and preparation of tabulations is done by NCHS using electronic computers. The Bureau of the Census has 12 regional offices located in 12 major cities where supervisors of the HIS are stationed. Each supervisor spends a great deal of time visiting the approximately 30 PSU’S in his region in which the interviewing is carried out. Since there are three to four PSU’S per interviewer, many of the interviewers are also required to do a considerable amount of travel. CONTROL OF THE SURVEY PROCESS The quality control program for the HIS has two purposes: to minimize errors in the survey results and to provide data to evaluate the extent of bias caused by interviewers and respondents. Nonsampling errors can occur at any stage of a survey. They may result from the improper statement of the objectives, from faulty concepts, or from improper definition of the population to be studied. They may arise during the sample selection, during the conduct of an interview, or during the processing of the data, e.g., coding, editing, or tabulating. Both the field and office quality control programs of the HIS strive to minimize these errors and to maintain the quality of the interviewing and of the editing, coding, and other data-processing operations. The quality control activities in the field are process controls rather than product controls. That “is, very little work in the field is done over again because it does not meet quality control standards. To control errors contributed by the program seeks to identify interviewers, interviewers whose work is defective in particular areas so that remedial action can be taken to improve future work. Such remedial action generally takes the form of retraining the interviewers on those aspects of the survey in which their performance was poor. Sometimes, however, interviewers cannot be helped by retraining and must be replaced. Some intemiewers resign when they dkcover through the field quality control program that their performance is not up to par. The turnover, inchxling interviewers whose performance is satisfactory but who resign for personal and other reasons, is about 15 to 25 persons per year among the approximately 100 interviewers assigned to the HIS. However, about 40 percent of the interviewers have been with the survey for at least 5 years. MEASUREMENT OF NONSAMPLING ERROR Nonsarnpling errors that arise during the interview may have as their source the respondent, the interviewer, or the questionnaire. They may result from such causes as respondent memory lapse, the misunderstanding of a question, improperly omitted questions, or incomplete answers. PART 1. CONTROL OF DATA Two of the many attempts that have been made to measure the nonsamplin~ error in the HIS are discussed in this report: One attempt has been the systematic supervisory reinterview which consists of reinterviews conducted by the field supervisory staff and senior interviewers at a subsample of households included in the sumey. The resuIts of the reinterview survey are compared on a case-by-. case basis with the results of the original survey. Data from these comparisons are presented as net and gross differences. Net differences are differences between the statistics produced from regular HIS interviews and the statistics produced from the reinterviews. Gross differences are disagreements in individual classifications made by the interviewers and the reinterviewers. A more detailed discussion of these measures is included in this report in the section part II Response Errors as Determined by a Reintemiew Survey. Another approach has been an interviewer variability study in which interviewer assignments were randomked to obtain estimates of between-iiterviewer variance. (See Interviewer Variability Study in this report for a more detailed discussion.) COLLECTION INTRODUCTION Quality control is commonly achieved by measuring performance and setting standards. Illustrations of performance measures are noninterview rates, item-response rates, editing-failure rates, and error rates in clerical and cardpunching work. Such measures are frequently the basis for applying formal quality controls in the conduct of surveys. These quality controls are specified in terms of minimum performance standards that maintain the quality of work in various operations and thus contribute to the accuracy of survey results. This part of the report describes the controls imposed on the collection and processing of HIS data. Quality control measures are applied at five different stages in the HIS: (1) interviewer selection, (2) training of interviewers, (3) obser- AND DATA PROCESSING vation of interviewers, (4) supervisory reinterof the completed and (5) editing view, questionnaires. In the HIS, about 30 percent of the total field budget goes into quality control. Table 1 provides a distribution of the costs incurred by activity for calendar year 1968. SELECTION AND TRAINING INTERVIEWERS OF Selection HIS interviewers are selected with great care. Because of the potentially delicate nature of an HIS interview, candidates must have not only the necessary qualifications for handling the interview questionnaire but also unusual tact and sensitivity. 3 Table 1. Percent distribution of field costs by detail expense item: Detail expense Total Observation 0fficework3 Another . 100.0 .. 63.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .- 4.1 Groupandhometraining. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 5.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 8.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .:::::........”. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ‘ Includes overhead. 2 Includes a check for completeness 3 Includes preparation of reinterview of coverage. assignments. Survey, Total item . . . . . . . . . . . . . . . . . . . . . . . . ........c.m-,.-. training ReinterviewZ Interview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Interviewing Initial Heelth 12.1 2.2 1968 7 Salaries Travel 3theri 47.6 27.7 = 24.7 29.5 20.8 13.3 1,2 2.0 0,,8 2.3 1,,7 1.3 3.3 2.7 2.3 1,8 1.4 zyxwvutsrqponmlk 4.6 Ji2 7.5 1.2 0.3 0.6 4 Less than 0.05 percent. EmploymentasHIS interviewers islimitedto women. The typical respondent, a housewife, is generally thought to be more willing to reveal complete health information to another woman than to a man. In addition, employment of women with formal nursing or medical training as HIS interviewers is discouraged because interviewers with such training may tend to diagnose or interpret rather than merely record the information obtained from respondents during the interview. Interviewers must pass atest that measures reading comprehension, arithmetic ability, and map-reading ability. An elaborate program of training and observation early inan interviewer’s career also appears to aid in the selection process. The very low level of refusal, about 1 percent ofhouseholds contacted, andof complaints received seems to show that the interviewers who are finally selected appear to be doing a good job of gaining public cooperation. Initial Training The initial training consists of five separate stages: preclassroom training, classroom training, postclassroom training, on-the-job training, and editing of questionnaires by the supervisor. Preclassroom training. –Preclassroom training is designed to familiarize the new interviewers with both the purpose, scope, and general uses of the HIS and the interviewing materials and the interviewing job. 4 The usual method of .Dresentirw such trainhw is through self-study materials. The interview; is given a self-study package to complete before reporting for classroom training. The contents include administrative materials, a copy of the HIS questionnaire, an interviewer’s manual with instructions to read certain sections, and copies of the letters that the respondents receive. Occasionally, in order to become better acquainted with the survey, the new interviewer also spends 1 day observing an experienced interviewer. Classroom training and practice interviewing. —Classroom training consists of 5 days of instruction, which covers the interviewer’s manual, the questionnaire and related forms, and interviewing techniques. This training is usually conducted in one of the 12 Census Regional Offices, permitting the trainee to become acquainted with some of the regional office staff members and with general office procedures. The classroom portion of the initial training combines formal classroom training with mock interviewing. Mock or hypothetical interviews are created from situations the interviewer may face. The formal classroom training is primarily for teaching survey concepts. Several training techniques besides mock interviewing are employed in the classroom. They include lectures by the trainer, reading portions of the interviewer’s manual, answering questions, participating in group discussions, and completing written exercises. Practice field interviewing gives the trainee a chance to apply her knowledge of the survey materials to actual interviewing situations. This serves to familiarize her as emly as possibIe with the work she will be doing as well as to stimulate her Iearning of the concepts and techniques by using them under actual conditions. An observer accompanies the interviewer during practice interviewing assignments. He coaches her on how to handle difficult situations and explains interviewing techniques. During the interview, the observer does not interrupt the trainee unless she becomes very confused. It may occasionally be necessary for the observer to conduct the first interview so the interviewer can get an idea of how the interview should be conducted. The observer uses an observation report to record all pertinent details of each interview, including any errors the interviewer makes. After they leave the household, the observer discusses with the interviewer the points that he has marked on his report and gives her some hints on how to improve her interviewing techniques or to solve problems that arose during the interview. He encourages her to look up the solutions to problems in the interviewer’s manual. In addition to evaluating the interviewer’s technical understanding of the rules and definitions that apply to subject matter, the observer checks the interviewer’s performance in the following specified areas: 1. 2. 3. 4. 5. Introduction at the doorstep Use of identification card Explanation of survey Getting settled in the household Interviewer’s ability to maintain a businesslike but friendly attitude with the household members 6. Ability to handle unusual or difficuh situations 7. Adeptness with forms (i.e., following skip patterns, probing where answer is incomplete, asking questions as worded, recording answers as instructed) 8. Dress and posture Postclassroom training. –Postclassroom training is designed to familiarize the interviewer with rules and procedures that, although important, are not as frequently used as those covered during the classroom training. The fiist postclassroom assignment is completed after classroom training and before the first interviewing assignment. It consists of reviewing classroom topics, completing a lesson that describes the persons to be included in the survey along with a description of housing and sample units, and reading a discussion of administrative forms the interviewer will use. The second postclassroom training assignment is completed at home by the interviewer between her first and second interviewing assignments. This assignment is designed to help the interviewer understand sample unit coverage in axea segments, the use and background of’ the address Iists taken from the 1960 Decennial Census, and the procedures to be followed at special dwelling places, e.g., motels or convents. The third postclassroom training assignment is completed at home by the interviewer immediately before her first listing assignment. A list of all housing units in each area segment must be compiled before any addresses can be selected for interviewing in these segments. The interviewer travels around the segment and records the addresses or other description of all places where people live or might live within the segment. On-the-job training. –On-the-job training is conducted by the supervisor-trainer during the interviewer’s first two interviewing assignments and her first listing assignment. This type of training is usually referred to as initial observation and is discussed more fuIly in the section “Observation of Interviewers.” Edit of questionnaires. –All work of new interviewers is edited by the field supervisor. This includes questionnaires for about 70 households from about four assignments conducted over an 8-week period. A complete check is made of the questionnaire, errors are identified and tzdlied, and the number and description of the errors is given to the interviewer. Continuing Training Several different kinds of continuing training are used. 5 Group training. –Group training is used for experienced interviewers and normally takes are brought place twice a year. Interviewers together in the 12 Regional Office cities. This provides an opportunity for the widely scattered interviewers to meet one another, exchange views, and receive formal training. There is one session in December or January of each year to learn the new questionnaire for the next calendar year, and a midyear refresher session usually held in June or July. Home study and exercises. –There is a formal home-study program for which the interviewers are paid. Home-study assignments, which generally take an average of 3 hours to complete, are made four times a year. These assignments cover various aspects of the interviewer’s job, such as the order in which the questions should be asked, when a question is necessary, etc. They also emphasize the importance of the survey and of the interviewer’s vital role in its operation. Feedback of errors. –Supervisors in the regional offices edit 1 week’s assignment per interviewer per quarter. Additional editing is done as needed, i.e., when previous editing results, observation, or reinterview indicate any consistent type or pattern of omissions or inconsistencies. On the average, about one-sixth of the completed questionnaires are edited each quarter in the regional offices. Certain categories of errors are identified and tallied, and the errors are recorded on forms that are forwarded to the interviewer immediately upon completion of the editing. For example, the interviewer may be required to give an explanation of each error or to make a written reference to the part of the interviewer’s manual that describes the correct procedure. A second edit is carried out at an early stage of the data-processing operation in Washington, D.C. Here the data from all questionnaires are examined. This edit takes place, however, weeks or even months after the questionnaires have been filled out by the interviewers. Although the immediate feedback that is provided by the field edit is lost, the degree of uniformity in the detection of errors made by interviewers becomes much greater in the central office edit. This procedure therefore provides a better basis for the numerical error scores that 6 are an important part of each interviewer’s performance record. Other training. –Informal training takes place every time supervisors and interviewers get together in connection with the quality control programs discussed in the next two sections. In addition to formal training and the informal meetings with supervisors, interviewers are encouraged to use referral sheets for describing problems. They can mail these sheets directly to the regional offices where solutions for their problems can be quickly determined and mailed back to them. OBSERVATION OF INTERVIEWERS Introduction An important part of the quality control program for the HIS is the observation of interviewers. Either the HIS Program Supervisor, Alternate Supervisor, or Senior Interviewer observes in each regional office. An interviewer is observed in a group of households in her assignment. The observation program contributes to on-the-job training as well as evaluation of the interviewer’s performance. The main focus of the observation is to see how the interviewers conduct themselves in obtaining information in their assigned households. The type of controls possible through the observation program depends a great deal on the ability of the supervisor to detect inadequacies and correct them. An HIS observation report (appendix V) is used as a guide while observing interviews. On it are recorded the observer’s of the interviewer’s performance. impression Some of the items are entered on a person-byperson basis, and some relate to the day’s work. A copy is placed in the interviewer’s performance file in the regional office, and the original is sent to the U.S. Bureau of the Census. Types of Observations Observations are classed as initial, systematic, and special needs. Although the same procedure is followed for all kinds of observations, they have different purposes. Initial observations.-The purpose of the initial observations is to give new interviewers on-the-job training to correct weaknesses at the beginning of their interviewing career. Initial observations are conducted for each new interviewer for 2 full days on her first interviewing assignment, for 1 full day on her second interviewing assignment, and for part of a day on her first listing assignment. The new interviewer begins her day by locating the segment in which she will work. The observer allows the interviewer to find the segment without guidance, unless she becomes completely confused in reading the maps. If this occurs, he assists her in finding the first household. On interviewing assignments, the interviewer introduces herself, and the observer merely observes unless the interviewer asks for help or makes errors. If the interviewer is having serious difficulties, the observer then assists by conducting as much of the interview as needed to show the interviewer the proper procedures. At the end of the interview, the observer reviews with the interviewer any general problems that have arisen, such as misapplication of definitions or poor interviewing techniques. He discusses his notes with her immediately after leaving the interviewed household and before going to the next. He also reviews (edits) the questionnaire for completeness. At the end of the observation, he may also review the interviewer’s time and mileage records and discuss he noted for special any general points attention. Systematic observations. –Most observations are regularly scheduled visits by the supervisor. These are called systematic observations and are designed to serve three broad purposes: to give on-the-job training in areas where specific weaknesses are observed; to allow each interviewer a regular opportunity to discuss her work with her supervisor, to make suggestions, and to communicate in general with the regional office; and to provide information for evaluation of the overall quality of the interviewing in the HIS. Systematic observation assignments are made by the regional offices. One-half of the experienced interviewers are observed each quarter. A systematic observation is made of newly trained interviewers in the quarter following is contheir initial training. No observation ducted when assignments are aIso scheduled for reinterview. Before conducting a systematic observation, the observer reviews records of the interviewer’s past performance. In addition, he edits recent examples of her listing of households, reviews the office copy of the report of her last observation, and, in general, tries to determine which points should be observed most closeIy. interobservations. —S ome Special-needs viewers need more contact with their supervisor than is provided by the systematic observations, and the regional offices need the flexibility of being able to give additional training to interit is needed. Special-needs viewers when observations are used for this additional training. A special-needs observation is usually made for an interviewer whose work is rejected in reinterview. Rejection in reinterview is based on the number of differences between the original interviewer’s resuhs and the reinterviewer’s results. An interviewer’s work is rejected if the number of differences is in excess of specified limits given in a table of acceptability. (See the next section for further discussion of the reinterview program.) The special-needs observation is scheduled for the interviewer’s first assignment following the reinterview. A special-needs observation may also be made for an interviewer whose work falls below certain minimum performance standards such as the following: poor production, e.g., too few completed interviews per day or too much travel time; a high nonintexview rate; an excessive number of recording errors and omissions on the questionnaire filled out by the interviewer; and poor performance on recent observation. The purpose of the special-needs observation is retraining. If, in the judgment of the supervisor, an interviewer needs retraining after falhg below the minimum standard for some aspect of her work, an observation is conducted. Sometimes a seemingly poor performance can be explained,for example, a high noninterview rate in the summer, and no retraining is necessary. Except in the case of rejection in reinterview, the decision to conduct a special-needs observation is made entirely in the ;egional offices. The procedure for the special-needs observation is the same as that for the systematic 7 observation, but with special attention given to the aspect of the interviewer’s work that needs improvement. SUPERVISORY REINTERVIEW PROGRAM Introduction The major purpose of the program of supervisory reintemiews is to control quality. The program provides a process control on the work of individual interviewers so that interviewers with high levels of error can be identified and remedial action taken. The remedial action is intended to improve the quality of the individual interviewer’s work through retraining, observation, and discussion of errors with the interviewer, as necessary. The reinterview also serves as a periodic check on interviewers to see that assignments are carried out as instructed. This process also provides assessments of the reliability and accuracy of the HIS because the quality control technique employed in this program requires that the supervisors fill out that can then be complete questionnaires matched with the questionnaires filled out by the interviewers. Since these questionnaires constitute, by design at least, a probability sample of all HIS interviews, estimates bearing on the reliability and accuracy of HIS statistics can be made. Sample Design Reinterview assigriments are made on the basis of interviewer workload, that is, interviewers with larger workloads have more reinterviews in a given year. A reinterview assignment consists of one weekly work assignment, and there is an average of three reinterview assignments per year per interviewer. Originally, one reinterview assignment was carried out for each interviewer per quarter. In January 1963, the program was reduced to three assignments per year per interviewer. On July 1, 1965, the selection method was changed to reflect the variability of interviewer workload. An additional reinterview assignment is made for interviewers rejected in the previous quarter. The number of interviewers for which this is 8 done is limited to not more than 10 percent of the total number of interviewers. Reinterview assignments are evenly divided among the weeks of the quarter, and there is only one reinterview assignment in a regional office area in any given week. A reinterview of 12 of the 18 households in a typical interview assignment is conducted with one person reinterviewed in each of the 12 households selected. The reintemiew sample is divided into two parts. The sample of households selected for reinterview is subdivided into an 8 O-percent subsample and a 20-percent subsample. In the 80-percent subsarnple of households, the supervisor carries out a reconciliation of reinterview results with the results of the original interview. No reconciliation is carried out for persons in households designated for the 20-percent subsarnple. The division of the reinterview sample into an 8 O-percent subsample and a 2 O-percent subsarnple began in January 1959. Before that, reconciliation was carried out for the entire reinterview sample. Content of Reinterview The first part of the reinterview is a coverage check to see if all household members have been properly incIudcd in the survey. The second part of the reinterview deals with the reporting of personal and health characteristics. In general, the reinterview covers all questions relating to the reporting of health conditions and their characteristics originally included in the first interview. Supplements, such as hospital insurance, eyeglasses , or hearing aids, are not usually included in the reinterview. Field Procedures At the beginning of each quarter, regional supervisors are told which weeks will have a reinterview assignment. One week before interview week, the y are told which assignments are to be reinterviewed and given instructions for selecting the subsample of households designated for reinterview and the sample persons within reinterview households. One person per household is randomly selected for th(: part of the reinterview that covers personal and health characteristics. The reinterviewer is instructed not to look at the original interview results before reinterview. The HIS reconciliation questionnaire (appendix VI) containing the transcribed information from the original interview is given to him in a sezded envelope. He does not open this envelope until he completes We reinterview. For the 20-percent subsample of households for which reconciliation is not carried out, the original questionnaires are not transcribed. For these households, the note “Omit Content Reconciliation” is placed on the reconciliation questionnaire inside a sealed envelope. Thus the supervisor is not supposed to know in advance the households where he will not do reconciliation. The data from the 2 O-percent group are used to test the extent to which accessibility of original responses to the reinterviewed has apparently affected reinterview results. The reinterview fieldwork incIudes verifying the originzd interviewer’s work in the listing of addresses in area segments, checking househoId composition in the sample households, and reinterviewing one person in each reinterview sample household. The reinterview is scheduled for the week following the original interview and must be completed no later than 2 weeks after the date of the original interview. Since the questions on the HIS schedule refer to specific time periods, such as “last week or the week be fore,” “a year ago /’ and “past 12 months,” the reinterviewer, in asking these questions, must be certain to get information for the same time period used by the original interviewer. In order to do this, the reinterviewer must specify the exact dates of the reference period used in the original interview. The reintemiewer makes a personal visit to each household selected for reinterview. The questions relating to coverage of persons within the household may be asked of any eligibIe respondent. For the health information for adults, the most acceptable respondent is the person who provided the data in the original interview. If he is not available, however, the sample person may be interviewed. (Before July 1, 1965, the only acceptable respondent was the sample person himself.) Information for children is obtained from parents or an adult responsible for the child’s care. Responses are entered on the reintemiew questionnaire and changes are not made after this part of the reinterview is completed. (Before January 1967 , supervisors used a special questionnaire containing only the subjects covered in the reinterview. Since then supervisors use a blank HIS questionnaire for recording the reinterview restits.) In 80 percent of the reinterviews, differences in responses from the two interviews are reconciled immediately after completion of the reinterview. For personal characteristics, the reinterviewed transcribes the information he has obtained to the reconciliation questionnaire. He then compares these answers with the original responses and reconciles any that are different. Next, he compares the responses to the health questions on the reinterview questionnaire with the reconciliation questionniare, which contains the information from the orighud interview. If the reinterviewer finds that differences exist, he attempts to determine from the respondent the proper response and any possibIe reasons for differences. The reconciliation questionnaire provides space for recording reasons given by the respondent for differences between the original interview and the reintemiew on the reporting of ibesses and other health conditions and hospitzdizations. Table 2 shows the number of persons reinterviewed in fiscal years 1959 through 1967. Table 2. Number terviewed: and percent Health of persons Interview Survey, Number Fiscal year of interviewed and rein- fiscal years 1959-67 Number of completed completed interviews reinterviews (persons) (persons)’ 1959 . . . . . . . . . . . . 126,841 3,478 2.7 1960 . . . . . . . . . . . . 118,068 3,061 2.6 1961 . . . . . . . . . . . . 112,086 3,206 2.9 1962 . . . . . . . . . . . . 118,432 2J338 2.4 1963 . . . . . . . . . . . . 138,055 2~96 2.2 1864 . . . . . . . . . . . . 128,801 2,391 1.8 1965 . . . . . . . . . . . . 138,152 2,081 1.5 1966 . . . . . . . . . . . . 138,486 2,053 1.5 1967 . . . . . . . . . . . . 133,916 1333 1.4 lThe decline result of a cutback in the number of completed in the reintewiew reintewiews is the sample size. 9 Quality Control of Interviewers’ Work One purpose of the supervisory reinterview program is quality control. Specifically, the program is designed to check on coverage and content errors. Errors in coverage of the population can occur because of incorrect listing of addresses in sample segments, failure to conduct interviews at the correct addresses, and incorrect application of definitions of housing unit and household member. Content errors are errors in the data obtained by the interviewer concerning personal and health characteristics of members of the sample household. Through the reconciliation of original interview and reinterview results, the supervisor tries to obtain, the best answers to the HIS questions. A second purpose of the reinterview program, that of obtaining measures of nonsampling errors and biases, is discussed in part II of this report. After a reinterview assignment has been completed, the reinterviewer completes a summary report of the HIS reinterview (appendix VII) showing the number of differences for five categories of the interviewer’s work: listing; household composition; personal characteristics; characteristics of conditions and hospitalizations; and number of conditions, hospitalization, and injuries. Tolerance limits a~e established for each category. The interviewer’s work is required to meet the standards for each category separately. The tolerance limits are listed in a table of acceptability, which shows for each category separately the number of differences that are acceptable for a particular sample size. Only cases where the respondent was the same on both interviews are used in the table of acceptability. The acceptance numbers are set so that a difference rate at a 5-percent level will be accepted 95 percent of the time. An interviewer’s work is rejected if the number of differences in any classification is in excess of the numbers given in the table of acceptability. An analysis of reintewiew assignments was carried out for the period July 1, 1962, through June 30, 1967. During this time, 1,554 original interview assignments were reinterviewed. Rejections were noted in 115 assignments in one or more categories. These 10 115 assignments grounds: were rejected Category of rejection All categories of rejection Listing on the following Number . . . . . . . . . . . . . . 139 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 Houshold composition . . . . . . . . . . . . . . . . . . . . 7 Personal characteristics . . . . . . . . . . . . . . . . . . . . 31 Characteristics Number of conditions of conditions and hospitalizations and hospitalizations . . . 30 . . . . . . . 53 Of the 115 assignments that were rejected, 95 were rejected on one category, 16 were rejected on two categories, and 4 were rejected on three categories. No assignments were rejected on more than three categories. Supervisors in the regional offices initiate retraining, observations, etc. of interviewers whose work is rejected. A report of actions tziken is made to headquarters in Washington, D.C. Table 3 presents the actions taken for those interviewers whose assignments were rejected in reintemiew by reason for rejection for the period July 1, 1962, through June 30, 1967. EDITING COMPLETED AND CODING OF QUESTIONNAIRES Introduction When completed assignments are received in the regional offices from interviewers, the questionnaires are edited for consistency and completeness. A systematic edit is carried out for a specified sample of assignments. The results of this edit are sent to the interviewers with identification of errors and specific references to sections of the interviewer’s manual to review. Further editing is carried out during coding and processing operations at headquarters. The NCHS assumed responsibility for the coding and data preparation in 1968. A new questionnaire format and new coding procedures were adopted at that time. The coding and quality control procedures described in this report are those used by the Bureau of the Census before 1968. About 800 household questionnaires are received each week for processing. After the r Table 3. Percent distribution of actions taken in cases of interviewer rejection July 1, 1962-June 30, 1967 by nature of rejection: Health Interview survey, I Category Action taken after rejection Listing Of rejection Household Personal composition characteristics Tables Health conditions Percent distribution . . . . . . . . . . . . . . . . . . . . . . . 100.0 100.0 Retrained . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27.6 14.3 3.2 3.3 3.8 Observed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33.3 14.3 64.5 40.0 41.5 Total rejections 100.0 100.0 100.0 . . . . . . . . . . . . . . . . . . . . . . . 5.6 14.3 12.9 30.0 15.1 . . . . . . . . . . . . . . . . . . . . . . . . . . 22.2 14.3 9.7 3.3 9.4 Resigned . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 0.0 0.0 0.0 0.0 1.9 Dismissed . . . . . . , . . . . . . . . . . . . . . . . . . . . . . . 0.0 0.0 0.0 3.3 Retrained and observed Errors discussed No action necessary . . . . . . . . . . . . . . . . . . . . . . . . 0.0 28.6 6.5 10.0 Action . . . . . . . . . . . . . . . . . . . . . . . . 11.1 14.3 3.2 10.0 not reported ] The control supervisor of the may interviewer. decide For that no actions example, are necessary one confused if, in his judgment, respondent may contribute the differences are due to factors al I the differences because & 7.5 beyond the he misunderstood questions. questionnaires are checked in, they are grouped into work units of approximately 25 questionnaires each. The questionnaires are assigned in work units to clerks who check the questionnaires for completeness, assign codes to the information on the questionnaires, and transcribe aIl of the information on the questionnaires to punch card transcription sheets. Questionnaires go. through nonmedical coding and medical coding operations. Nonmedical coding assigns codes to the demographic items and items related to health conditions. Medical coding, which is a more complicated operation, assigns detailed diagnostic codes to the illnesses, injuries, and h“ospitalizations reported on the questionnaires. Diagnostic codes are assigned, with some modification, according to International Classification of Diseases (ICD). To control the level of errors in these coding operations, specifllc quality control procedures are followed. These procedures are discussed in the section on quality control of clerical coding operations. Regional Office Edit Specifications office edit are for carrying out a regional sent to each office at the beginning of a year. These specifications define the minimum editing that must be done. Additional editing is carried out on the basis of need, i.e., if previous edit results, observation results, or interview resuIts show errors such as omissions and inconsistencies. The specifications for editing the work of experienced interviewers generally provide for more editing at the beginning of the year when new items are added to the questionnaire. As interviewers become more experienced with new items, the amount of editing is. reduced. However, the first four assignments of new interviewers are always edited. There are two types of edits performed in the regional offices, diagnostic and nondiagnostic. The diagnostic edit must be done by the HIS supervisor. Errors are assigned for missing or inadequate entries for ilhesses, injuries, or hospitalizations. The nondiagnostic edit can be done by a qualified clerk and consists of identifying omissions and incorrect entries in identification and control items on the questionnaire. The results of the diagnostic and nondiagnostic editing are sent to the interviewers and provide some immediate feedback on errors. A copy of the results is retained in the regional 11 office for comparison with the more intensive edit and identification of errors made during the central office coding and processing operations. Quality Control of Clerical Coding Operations There have been a number of changes over the in the processing of HIS questionyears naires. From the beginning of the survey in 1957 until November 1965, processing consisted of transcribing and coding information from questionnaires to document-sensing cards from which IBM punchcards were mechanically prepared. The assignment of diagnostic codes to illnesses, injuries, and hospitalizations was independently verified on a 100-percent basis. Two coders independently assigned diagnostic codes on the information in the questionnaire. These codes were compared, and differences were resolved by a supervisor. The coding and transcription of nonmedical entries was completely verified by a second coder’s examining the entries on the document-sensing card to see if they had been correctly transcribed by the first coder. In November 1965, a new schedule format was introduced into the survey. Entries on this schedule could be read directly by machine, thus bypassing a large amount of clerical transcription. In addition some of the codes for nonmedical items, such as age, were entered on the schedule by interviewers and required no further coding. However, a substantial amount of editing and coding was still required, particularly for diagnostic entries that had to be medically coded. At the time this new schedule was introduced, it was decided that sample verification to control the quality of coding would provide a better use of resources than 100percent verification, particularly for medical coding. Consequently, starting in November 1965, a sample verification plan was introduced into the medical coding operations of the survey. The plan provides for two stages of control in the medical coding operation: a training and qualification period during which the coders’ work is independently verified 100 percent, and a posttraining period during which the coder’s work is independently verified on a 10-percent sample 12 basis. All errors detected corrected. during verification are During the qualification period the new coder codes to a work sheet. Then the coding is done over again by a qualified coder independently on an HIS schedule. A comparison clerk matches the medical codes entered on the worksheet with the medical codes on the corresponding schedules. Differences in medical codes are reviewed by an adjudicator. The adjudicator assigns an error if, in his judgment, the original coder assigned the wrong code. If, however, the differences in codes are a matter of coder judgment, a decision concerning the proper code is made, but an error is not assigned. In order to qualify for sample verification, a coder must code four consecutive work units out of a maximum of eight with an error rate of 4 percent or less for each work unit. If a coder fails to qualify within the first sequence of eight work units coded, a second sequence of eight for quahfication is started. A coder has a maximum of three sequences in which to qualify. If a coder fails to qualify in the third sequence of eight work units, he is not considered for sampIe verification. Once a coder has qualified for sample verification, his work continues to be verified on a 10-percent sample basis. A record of verification is maintained for’ each medical coder. When the cumulative number of verified codes reaches 45, a decision is made to determine if the coder’s work is still acceptable. If a coder’s work is rejected three or more times in 10 decisions, he must requalify for sample verification. During the requalification period his work is verified on a 100-percent basis. If the coder fails to requalify, he is no longer considered for sample verification. Additional changes have been made in the format of the questionnaire since the sample However, verification plan was introduced. essentially the same verification procedures continue to be used, i.e., independent verification of medical coding on a sample basis and 100percent dependent verification of nonmedical coding. Records for the period April 1, 1967, to March 31, 1968, show an average error rate of about 2 percent in assignment of medical codes for experienced coders on sample verification. For the same period, the nonmedical error rate was also about 2 percent. coding Central Office Edit As part of the coding operations described above, a comprehensive review of each questionnaire is carried out to identify omissions, inadequate entries, and inconsistencies. Error codes are entered on the processing record so that the number and identification of errors can be tabulated and summarized for each interviewer. In addition, specific descriptions of errors are provided on a separate document. These descriptions identify the number and type of errors for different sections of the questionnaire, for example, person page, condition page, hospitalization page, etc. These errors are divided into two main categories: diagnostic errors on conditions and hospitalizations, and nondiagnostic errors. The forms used to record the number and types of errors are the same used in the regional office edit. Weekly summaries of diagnostic errors are sent to regional offices, and quarterly summaries of both diagnostic and nondiagnostic errors are sent to the regional offices which in turn notify individual interviewers. The quarterly summaries also form the basis for computing an interviewer error rate, which is one of the measures used to evaluate interviewer performance as described in the next section. Additional editing is done on the computer, which performs a series of adequacy and consistency edits. Individual records with errors are identified, the original questionnaires are located, and corrections made, as necessary, to the records. MEASURES OF INTERVIEWER PERFORMANCE In the preceding sections of this report the activities for controlling the quality of survey results have been presented. Results from these quality control activities are combined with other data to provide an overall evaluation of interviewer performance. The measurement of interviewer performance in the HIS is a combination of subjective ratings by supervisors and quantitative measures based on an examination of an interviewer’s completed work. Minimum standards of performance on the quantitative measures are set up. A cumulative record of performance for each interviewer is maintained in the regional office. If, at any time, this record indicates that an interviewer’s work has fallen below the minimum standard, corrective action is taken. This corrective action may consist of retraining, observation, or, in some cases, replacement of the interviewer. In practice the evaluation of interviewer performance is based on the pattern of performance over time and on different aspects of the interviewing job rather than performance on any single aspect. In the HIS, three quantitative measures of performance are computed on a continuing basis. They are the error score, the noninterview rate, and the production ratio. The error score is computed as folIows: (number of errors)/(total conditions + total accidents + total hospitalizations). (See appendix I.) The numerator is the number of errors identified during processing. Errors include omitted entries, missed conditions, missed hospitalizations, and diagnostic errors. Missed conditions and missed hospitalizations are those identified in the early or probing section of the interview but not followed up for additional information in the latter section of the interview. Diagnostic errors occur when the interviewer faik to record sufficient information to allow a medical coder to assign diagnostic codes. The noninterview rate is computed as follows: (number of noninterview households)/(number of interviewed households + number of noninterview households). The nonintefiew households are households eligible to be included in the HIS, but for which no interview was conducted. Included as noninterviews are those the interviewer has reported as “refusals,” “no one at home,” “ temporarily absent,” etc. The production ratio is measured as follows: (estimated time based on production standards)/ (actual payroll time charged by the interviewer). The numerator is estimated from a mathematical equation (appendix II) that takes into account such things as the average time per household, the number of assigned households, and the distance to area of assignment from interviewer’s home. 13 In addition to quantitative measures used to evaluate interviewer performance, the results of supervisory reinterviews and observations are used as much as possible. Also, if specific individual interviewer errors are discovered during the processing operation at the central office, they are noted and forwarded to the regional supervisor. He in turn informs the interviewer of these errors and suggests means of eliminating them. After a probationary period of 6 months, each interviewer receives a report on her performance over the past quarter. The report contains both a descriptive rating and a numerical score. The descriptive ratings are “Excelor lent,” “ Satisfactory, “ “Needs Improvement:’ If an interviewer receives a “Unsatisfactory.” rating of “Needs Improvement” or “Unsatisfactory,” she will receive a warning notice Table 4. Average interviewer Survey quarter performance unless, in the judgment of her supervisor, there are extenuating circumstances. In addition to the quantitative measures used in evaluating individuzd interviewer performance, other measures are used to provide an overall summary of performance. These include number of conditions per person,a number of missed conditions and hospitalizations, number of diagnostic errors, and proportion of reinterview assignments accepted. (See “Regional Office Edit” for a discussion of diagnostic errors.) Tables 4 and 5 show the average rates for HIS interviewers over a 4-year period for five of these rates and over a 7%-year period for three of these rates. The variation from quarter to quarter is small; the only apparent trend being in the number of conditions per person, wlhich has steadily increased throughout the survey. Table 6 shows average rates for interviewers on various measures by survey quarters: Average Average number of Number error rate missed conditions of inter- (percent) per 100 persons viewers 3ate N’ Rate N Health Interview Survey, January 1962-June 1965 Average number of missed hospitalizations per 1,000 persons Rate N Ratio N N Proportion 1962: Jan.-Mar. . . . . . . . . 107 4.6 105 .10 105 .64 105 1.09 100 .95 94 Apr.June . . . . . . . 128 5.8 128 .25 127 .83 127 .91 114 .95 95 July -Sept. . . . . . . . 120 I 0.0 119 .30 119 .45 118 .89 110 .95 92 . . . . . . . . 117 8.3 116 .38 116 .68 115 .80 103 .91 94 Oct.-Dee. 1963: Jan.-Mar. . . . . . . . . 112 5.8 112 .32 112 .49 112 .99 91 .94 62 Apr.-June . . . . . . . 104 5.9 103 .23 103 .64 103 .97 87 .95 66 July -Sept. . . . . . . . 99 5.0 98 .09 98 .24 98 1.01 91 .85 61 . . . . . . . . 104 6.9 104 .31 104 .45 104 .97 98 .80, 70 Oct.-Dee. 1964: . . . . . . . . 98 7.3 98 .40 98 .42 98 1.06 92 .89 61 Apr.-June . . . . . . . 98 6.9 98 .33 98 .39 98 1.03 94 .95 64 July -Sept. . . . . . . . 102 6.8 101 .28 101 101 1.00 100 48 . . . . . . . . 101 6.9 101 .42 100 .42 100 1.02 106 .92! .89 . . . . . . . . 100 7.8 100 .16 100 .66 100 1.05 96 .90 68 . . . . . . . 97 6.4 97 .12 97 .58 97 1.03 96 .92 61 Jan.-Mar. Oct.-Dee. 24.46 65 1965: Jan.-Mar. Apr.-June IN ‘This = number of interviewers included in the computations. high rate is due to one interviewer who interviewed viewers work, the measure is .54. aBecause of the differences from one assignment to anothez in the population covered, the number of conditions per person is not used as a performance measure for indkidual int ewiewers. 14 10 persons but had 4 missed hospitalizations. Excluding the inter- Table 5. Average interviewer performance on various measures by survey Health quarters: Interview Survey, January 1958- June 1965 Survey quarter Average Average number Number non interview of conditions of inter- rate (percent: per person Average number of diagnostic errors per 100 conditions viewers Rate N’ N Rate Rate N 1958: 5.6 102 102 104 4.6 Jan.-Mar . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 4.8 Apr.-June . . . . . . . . . . . . . . . . . . . . . . . . . . 103 4.3 July-Sept . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 Oct.-Dee . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 102 .94 102 4.8 .92 102 4.6 102 .84 104 5.1 104 1Oe .90 106 4.2 106 1959: Jan.-Mar.2 . . . . . . . . . . . . . . . . . . . . . . . . . . 87 Apr.-June . . . . . . . . . . . . . . . . . . . . . . . . . . 92 4.4 84 .91 84 4.4 84 July-Sept . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 7.4 93 .86 92 4.2 91 Oct.-Dee . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 4.7 85 .80 84 3.7 84 1960: Jan.-Mar . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 4.0 82 .87 82 2.4 82 Apr.-June . . . . . . . . . . . . . . . . . . . . . . . . . . 93 3.4 92 .93 93 2.1 93 July-Sept . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 6.2 92 .95 92 2.1 92 Oct.-Dee . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 3.8 98 .94 98 1.4 98 1961: Jan.-Mar . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 4.7 103 1.00 99 2.3 99 Apr.June . . . . . . . . . . . . . . . . . . . . . . . . . . 108 4.3 108 .97 105 2.7 105 July-Sept . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 5.7 108 .88 108 3.3 109 Oct.-Dee . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 4.5 108 1.00 109 3.8 108 1862: Jan.-Mar . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 4.7 105 1.07 105 3.4 105 Apr.-June . . . . . . . . . . . . . . . . . . . . . . . . . . 128 4.5 128 1.02 128 5.0 128 July-Sept . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 7.0 119 1.01 119 6.0 119 Oct.-Dee . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 4.9 116 1.04 116 5.9 116 1863: Jan.-Mar . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 3.6 112 1.08 112 4.2 112 Apr.-June . . . . . . . . . . . . . . . . . . . . . . . . . . 104 4.3 103 1 .!34 103 4.1 103 . . . . . . . . . . . . . . . . . . . . . . . . . . 99 5.3 99 1.03 99 3.0 99 Oct.-Dee . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 3.5 104 1.06 104 3.6 104 98 July+ept . 1964: Jan.-Mar . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 3.7, 98 1.08 88 3.9 Apr.-June . . . . . . . . . . . . . . . . . . . . . . . . . . 98 3.9 98 1.06 88 3.6 98 July-Sept . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 4.8 101 1 .m 101 5.3 101 Oct.-Dee . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 3.6 101 1.12 100 4.0 100 100 4.0 100 1.12 100 4.1 100 97 4.2 97 1.14 97 3.7 97 1865: Jan.-Mar . . . . . . . . . . . . . . . . . . . . . . . . . . . . Apr.-June . . . . . . . . . . . . . . . . . . . . . . . . . . J ‘N= number of interviewers 2 No information available included in thecomputetions. for this quarter. 15 Table G. Amm@inmwieMr petiormance onvarious measurS bynumber afquartom of experience — Number of qua rt e rs of e x pw ie n c o ~umber of interviewers — Akarage error’ rate [percent) Average nonint e rvie w ra t e (pe rc m t ) N’ — Rate 97 12.3 48 94 90 82 77 75 12.9 9.4 7.5 6.0 5.3 5.7 46 46 41 38 38 38 5.2 4.1 4.4 4.3 6.0 4.0 4.4 33 63 63 4.1 4.4 4.6 60 58 58 4.9 4.4 4.4 57 56 55 4.9 3.7 4.2 4.4 3.9 4.5 4.7 3.5 3.4 4.0 4.3 4.4 32 34 39 41 56 55 47 48 44 41 38 37 36 34 30 4.4 4.3 4.2 3.8 3.5 — 28 28 24 17 15 — 48 48 44 41 38 37 36 34 30 29 28 24 17 15 of conditions per person — Rate 74 67 Average number 33 34 N Rate Rate N 97 1.01 91 89 80 62 73 74 .96 .95 1.00 1.06 1.02 1.01 89 81 62 73 74 3.5 3.8 3.3 67 63 63 .99 1.01 1.02 67 63 63 3.7 3.1 4.2 60 58 58 1.02 1.01 1.05 2.9 4.2 3.8 4.1 3.2 4.1 3.6 4.0 3.9 3.7 3,5 57 56 55 3.0 2.8 6.3 2.3 3.7 3.2 — 18efore survey quarter July.Sept. 1961, the definition 47 43 44 41 33 37 % 34 30 26 28 24 17 15 Average number of dkgnostic errors per 1W conditions N 95 89 67 63 63 60 58 58 2.4 2.3 2.4 2.2 2.4 2.6 .87 1.01 1.05 1.08 1.05 1.05 1.03 1.05 1.05 1.10 1.06 57 66 55 47 48 44 41 38 31 36 34 2.7 2.5 2.9 2.6 2.6 2.6 2.8 2.2 2.0 1.9 2.2 1.05 30 28 28 24 17 15 2.2 2,2 2.3 2.1 2.3 2.2 1.12 1.06 1.14 1.21 1.27 of m isse d re ndit ions 89 81 52 73 74 pe r 1 0 0 pe rsons Rate 6.9 5.9 4.7 4.1 3.8 2.8 3.1 95 89 Average number .40 .26 .21 .26 .20 ,14 .10 .12 .10 .12 .15 .16 Awrage number of missed hospitalizations par 1,000 persons N N .71 41 .79 19 46 47 42 40 53 40 40 .76 .90 .95 1.04 1.05 .99 1.00 42 42 37 33 34 42 .83 .69 35 35 29 23 29 1.00 3s 38 1.05 1.07 .62 .= 3s 38 40 1.05 1.06 1.10 37 33 37 36 35 41 .93 1.00 1.00 .49 .11 .20 .40 1.13 .00 .37 .28 .13 .26 .19 43 48 48 47 48 44 41 38 37 36 34 42 46 46 ,91 .94 1.00 .23 .07 .59 .42 .00 .00 m 28 28 24 17 15 1.05 1.10 1.05 1.10 1.10 1.03 1.01 1.03 1.07 1.10 1.08 1.02 47 48 44 40 2$ 37 36 34 30 .85 .95 .93 .87 ,96 ,96 .86 .95 32 36 34 41 38 37 30 30 23 25 22 20 1.05 1.06 1.05 1.07 1.12 — 28 28 23 17 15 — 1.00 1.00 .88 .s0 1.00 19 18 17 10 10 .28 .15 44 48 & 47 48 44 41 38 37 38 34 30 26 26 24 17 16 .26 .20 .22 .03 .03 28 28 24 17 15 N 48 39 35 40 38 39 42 a ssignm e nt s a c c e pt e d — N — .76 1.51 .30 .78 1.51 1.00 .49 .14 .19 .16 .12 .16 .03 .14 .17 .28 .15 .24 .24 .24 ra t io Average proport ion of re int e w ie w Rate 45 48 48 44 40 40 40 w 58 58 57 58 55 47 48 44 41 28 37 26 34 30 Average produc t ion .28 .12 — ,83 “91 “97 .94 .86 .89 ,88 31 29 24 36 29 23 31 of error rate was actually an omission rate. Therefore rates for quarters of experience 1-14 exclude quarters before July- Sept. 1961. ‘N = number of interviewers included i“ comp”tmicms. NOTE: Restricted to HIS interviewers employed during the period Apr.-June 1965. by the number of quarters of experience in the HIS. As expected, the more experienced integviewers have a better performance, on the average, than new interviewers. From this table it appears that about eight quarters, or 2 years, of experience are necessary for interviewers to PART Il. MEASUREMENT INTRODUCTION A sample survey must take into account nonsampling errors and methods of control as well as sampling errors. The allocation of resources between control of nonsampling errors and increase in sample size is a c“om~licated question, an answer to which is not attempted here. 16 achieve maximum performance in terms of error rates, nonintewiew rates, and production. However, the data in table 6 do not represent a pure learning curve since the same interviewers are not included in all quarters. OF NO NSAMPLING ERROR Estimates of total sampling variance for important statistics can be made more or less routinely. The estimation of response variances and variances contributed by other aspects of the survey process, e.g., editing and coding, is more difficult. Particularly difficult is the estimation of biases in the measurement process. This part of the report describes two programs carried out in connection with the HIS to obtain estimates of nonsampling error, including bias. One is the supervisory reinterview program described in part I, which, in addition to serving as a field quality control device, provides overall estimates of response variance and bias. The second is a study designed to measure the interviewer contribution to the variance of estimates from the survey. RESPONSE ERRORS AS DETERMINED BY A REINTERVIEW SURVEY Introduction In the HIS, information is obtained by personal inquiry or a self-administered questionnaire on age, number of chronic conditions, number of hospital episodes, dlability, and other characteristics. The set of measurements or observations recorded in the collection operation ordinarily is examined for internal consistency and acceptability, certain “corrections” are made, and some of the entries coded to identify them in a classification system. Results are then summarized into totals, averages, correlations, or other statistical measures. Taken together, the collection and processing operations constitute the measurement process and are the source of any measurement errors. The interpretation of reinterview survey results or comparisons of results from a survey with case-by-case matched responses or measurements from some other source has been the subject of much research and study.z’9 Some theory of measurement errors that may help @ the interpretations of the results of two sets of measurements i,s presented in appendix III. The first set of measurements is obtained by the regular survey procedures. The second set is obtained from reinterviews or through matching of survey results , unit by unit, with records providing information similar to that obtained in the survey. HIS Reinterview Survey Results The use of the reinterview program as a device for evaluating the. reliability and accuracy of statistics of the HIS is a byproduct use. There are two important. respects in which the supervisory reinterviews do not meet the standards that are imposed for the original interviews. First, the supervisory reinterviews are by and large conducted by men, whereas it is a requirement that HIS interviewers be women. Second, there is a longer time interval between the and occurrence of heahh-related reporting events in the reinterviews than in the original interviews. The reinterviews occur at least a week later than the original interviews. There are, however, some offsetting factors. For 80 percent of the households selected for reinterview, the reinterviewer has the benefit of the results from the original interview. Where differences exist, the reinterviewer is to determine the proper answer and also possible reasons for the difference. It seems reasonable that in general better responses would be obtained from reconciliation of two interviews than from a single interview. The original survey data can be compared with reinterview data under three procedures. Procedure 1.–The reinterview in 20 percent of the households in the reintefiew sample is conducted without the results of the original interview being available to the reinterviewer. No reconciliation of resuIts is carried out. Procedure H. –The results of the original interview are available to the reinterviewer for 80 percent of the households in the reinterview the reintewiewer is not to sample. However, examine the results of the original interview until after a reinterview has been completed. Procedure II is a comparison of the results of the original interview with the reinterview before any reconciliation of responses in the two interviews is carried out. If the reintemiewer follows instructions, this comparison is the same one as procedure 1. Procedure III. –After conducting the reinterview in 80 percent of the households, the reintemiewer compares the responses obtained in the two interviews. Where differences exist, the reinterviewed tries, with the help of the respondent, to decide upon the proper response. Results of this reconciliation are compared with original results under procedure III. Summary measures. –To analyze the data obtained from a case-by-case comparison of an 17 original and a reintemiew survey, certain summary measures should be defined. Table 7 compares the results of an original survey with a reinterview survey. The total number of differences affecting the tabulated figure for a given class is equal to the number of cases included in that class in the original survey but not in the reinterview survey plus the n“umber of cases included in the reinterview survey but not in the original survey. This sum is called the gross difference for the class in question. In terms of table 7, b + c is the gross difference, and (b+ c)/rz is the gross difference rate. The net difference of the tabulated figure for a given class is the difference between the total for the class obtained in the reinterview and the original surveys. The gross difference usually includes differences in both directions that partly or substantially offset eaeh other. The net difference is the nonoffsetting part of the gross difference. In table 7, the net difference is b - c, and (b - c)/n is the net difference rate. Net differences. –Table 8 summarizes the net difference rates for procedure III for 7%+year averages. Since procedure III provided an opportunity for reconciliation of differences, the estimated net differences obtained from it are regarded as the best estimates of bias that the supervisory reinterview program can provide. Except for persons with one or more chronic conditions, the net difference rates would be regarded as small by almost any standard. However, as indicated in the table, all of the net difference rates are statistically significant; i.e., significantly greater than zero. Many of the results from the HIS are published as rates per person. Table 9 presents rates Table 7. General representation of results of original from the reinterview survey for the original interview and for the reinterview after reconciliation. The percent net differences shown in the table can be considered as an estimate of the relative bias of the original survey results. According to the reintemiew, chronic conditions tend to be underreported by about 24 percent. Disability days are underreported by about 13 to 18 percent. Table 9 shows that hospital episodes and hospital days are better reported. Estimates made from 1959-61 reinterview survey results show percent net differences of about 8 percent for hospital episodes and about 5 percent for hospital days. Gross differences and the index of inconsistency. —Gross differences are differences in individual classifications between the original interview and the reinterview. As discussed in appendix III the gross difference rate can be used to estimate the simple response variance of the original survey estimates, that is, the basic trial-to-trial variability in survey responses. (Appendix III also shows the derivation of an index of inconsistency based on the gross difference rate. This index provides a measure of the unreliability or inconsistency of classification and is defined as the ratio of the simple response variance to the total variance.) Reinterview without reconciliation (procedure I) provides the best estimate of simple response variance. However, the data from procedure I were tabulated only for fiscal years 1959-61. The gross difference rates and indexes of inconsistency shown in table 10 are based on data from procedure III after reconciliation. Table 11 shows consistent declines in moving from procedure I to procedure II to procedure and reinterview surveys for identical persons Results of original survey Results of reinterview Number having the characteristic Number nothaving Total 18 survey . . . . . . . . . . . . . . . . . . , . . . . . . . . . the characteristic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Number Number having not having the charac- the charac- teristic teristic a b c Total a+b C+cl zyxwvutsrqponmlkjihgfedcbaZYXW d a+c b+d n=a+b+c+rj Table 8. Estimated proportions, net difference rates, and standard error of net difference rate for procedure 1II after reconciliation for a 7Kyear quarterly average Survey Percant in class on original interview item, persons with: Oneormore chronic conditions . . . . . . One or more hospital episodes in past 12 months . . . . . . . . . . . . . . . One or more restricted activity days in pest 2 weeks . . . . . . . . . . . . . . . . . Ona or more bed days in past 2 waeks . . . One or more time-lost days in past 2 weeks Percent in class on reinterview Estimatad standard error of average Net difference rate (percent) net difference Underestimate rate Overestimate . . . . . . . . 42.3 49.2 -7.0 0.2 0.4 . . . . . . . . 9.3 10.0 -r).16 0.1 0.1 . . . . . . . . . . . . . . . . . . . . . . . . 10.6 5.6 3.4 12.1 6.4 4.2 -1.5 -0.8 -0.8 0.1 0.1 0.1 0.2 0.2 0.2 NOTE: Includas fiscal years 1959, 1960, 1961, 1963, 1964, 1965, one-half of fiscal year 1966 and all of fiscal year 1967. A fiscal year runs from July 1 to June 30. Table 9. Estimatad annual rates per 100 parsons, original interview and reinterview, reconciliation and percent net difference after recon- interview III after Percent net iifferencei Reinterview Original Characteristic for procedura ciliation Rateper100persons Chronic conditionsz Restricted activity Bed days’ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hospitalepisodes3 Hospitaldaysa . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ‘Original -reintewieWx reinterview 27%-year averages. 33-year averages. Table IO. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tima-lostdays2 Estimated proportions, gross difference Survey item,persons rates, andindexes ofinconsistency quartarly average forprocedurelll with: class on original interview Oneormorechronicconditions . . . . . . . . . . . . . . . . . . . . . . . One or more hospital episodes inpast 12months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . One or more restricted activity days in past 2weeks Oneormore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . time-lost -23.6 -13.3 -14.4 -18.3 -7.5 -5.1 ,OO. Percent in Oneormorebeddaysin 107.4 1,596.7 544.5 351.2 10.7 99.7 82.0 1,383.6 466.3 287.1 9.9 94.6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . days2 past2weeks days inpast2 . . . . . . . . . . . . . . . . . . . weeks . . . . . . . . . . . . . . . . . 42.3 after reconciliation ParCent in class on reinterview Index of Gross difference x 100 49.2 10.0 0. 9 10.6 12.1 6.4 4.2 3.4 3.4 inconsistency rate X 100 9.3 5.6 fora7Z-year NOTE: Includes fiscal years 1959, 1960, 1961, 1963, 1964, 1965, Ona-half of fiscal year 1966, andalIof year runs from Julyl to June30. I 5.5 17.2 2.3 17.5 1.6 23.4 fiscal year 1967. A fiscal 19 Table Il. Estimated indexes of inconsistency bythree procedures, 3-year averages, fiscal years l959%l =— Procedura I (no recon- Survey item, persons with one or more: ciliation) Chronic conditions activity Baddays inpast2 daysinpast2weeks Hospital daysinpast weeks III (after reconciliation) 22.2 7.6 7.0 6.0 . . . . . . . . . . . . . . . . . . . . . . . . . 44.5 28.6 18.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.1 26.6 15.8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37.6 32.9 21.4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.8 19.5 19.1 2weeks weeks of self-respondents with proxy respondents. —In the original interview for adults, the health questions are asked of the person himself if he is home at the time of the interview. If he is not at home, a related adult may provide the information. The person who is not present at the time of the interview is referred to as a proxy respondent since the information on such a person is obtained by proxy. However , in reinterviews all adults are self-respondents (in all reinterviews conducted during fiscal years 1959-67). Table 13 shows 20 %ocedure 30.9 III in the estimated indexes of inconsistency. If procedures I and II were carried out as specified, then the expected difference between the indexes of inconsistency would be zero. The estimated indexes for procedure H, however, are about 20 percent smaller than the indexes for procedure I. The differences between procedures 11 and III are in the direction that would be expected: reconciliation reduces the gross difference rate. Thus, the estimates in table 10 are an understatement of the gross differences that would occur if repetitions of the HIS were carried out without reconciliation. Some values of indexes of inconsistency for demographic items computed from other studies are provided in table 12 by size classes and compared with indexes for health items computed from the reinterview program of the HIS. The index of inconsistency for hospital episodes in the past 12 months is in the same size class as the simpler demographic items such as sex, color, and age. Time-lost days compare with the more difficult items to measure such as income and educational attainment. Comparison (before reconciliation) . . . . . . . . . . . . . . . . . . . . . . . . . . months months daysinpast2 Time-lost II . . . . . . . . . . . . . . . . . . . . . . .. . . . inpast12 Hospital episodes inpast12 Restricted Procedure 17.4 3-year averages of proportions, net difference rates and ind-exes of-inc-onsistency for six survey items as reported by self-respondents and proxy respondents in the original survey. The data are based on results after reconciliation (procedure III). Comparison of self-respondents and proxy respondents are limited by the fact that the selection of the respondent on the original interview is not a random selection. Thus part of the differences in IeveI may be attributed to inherent differences between respondents who are available to report for themselves and respondents who are not available at the time of interview and whose health conditions are reported by another member of the household. For four of the six items, the reconciliation tends to bring the proportion in the class for closer to that of selfproxy respondents respondents, that is, the net difference rates are greater for proxy respondents. The estimated prevalence rates for chronic from the reinterview conditions per person survey are presented in table 14 for selfrespondents and proxy respondents. In attempting to estimate what effect the respondent has, certain assumptions were made about the differences. Specifically an assumption was made that for self-respondents the net difference between the original rate and the reinterview rate can be considered as the difference due to the second interview. For proxy respondents, the net difference consists of second-interview differences and differences due to the use of a proxy respondent and that these differences are additive. Furthermore, an assumption was made that the differences due to the second interview are the same for self-respondents and proxy Table 12. Comparison of estimated indexes of inconsistency for Health Interview Survey items with d nographicitems Current 1960 Size of index (X 100) Population Decennial Census Evaluation Suwey Program’ Reinterviews HIS Reinterviews 1961%62 <lo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sex Employed Color In labor force Hospital episodes Age 11-20 Labor force . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Unemployed Mobility Hospital days Chronic conditions Bed days Restricted >20 Educational . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . attai nment Time-lost activity days days Income ‘ U.S. Bureau of the Census: of Data on Popu Iation E~aluation Characteristics and Research as Measured Program by Reinterviews. of the U.S. Censuses of Population Series ER 60-No. and Housing, 4. Washington. U.S. Accuracy 1960: Government Printing 0ffice,1964. 2 U.S. Bureau of the Census: Paper No. 19. Washington. The Current U.S. Government Population Printing Survey Office, Reinterview Program January 1961 through December 1966. Tech. 1966. respondents.’The last column of table 14 shows the net difference expressed as a proportionof the reinterview estimate, and the difference between these relative net differences is a measure of the effect of proxy respondents, that is, that about 16 percent of the relative net difference for persons reported for by proxy respondents can be attributed to the use of a proxy respondent. Effect of nonreporting on estimates of magnitude. –Differences in reporting of the number of conditions, days, episodes, and so forth between the original interview and the reinterview are classified as folIows: differences due to a change in the number of conditions, days, episodes, etc., reported on the two interviews or as differences due to a report of no conditions, days, episodes, etc., on one interview and a report of one or more conditions, days, episodes, etc., on the other interview (table 15). The change from a report of none on one interview to a report of one or more conditions, days, episodes, etc., on the other interview has a relatively small effect on estimates of proportions, but with the exception of hospital days in the past 2 weeks it has a major impact on estimates of magnitude. This section presents estimates of the part of the net and gross differences for estimates of magnitude that can be accounted for by a change from a report of none on one interview to a report of one or more on the other interview. Table 16 shows estimates of the components of the total net difference and the ratio of each component to the total net difference. Table 16 shows that for hospital episodes, restricted activity days, bed days, and time-lost days, the net increase on reintemiew for estimates of magnitude is principally due to the change from a report of none on the original interview to a report one or more on reinterview. For hospital days the number of days accounted for by changes from a report of none on the original interview to a report of one or more on reintemiew is about the same as the number of days involved in changes of a report of one or more on the original interview to a report of none on reinterview and thus canceI for estimates of net difference. For chronic conditions, the changes in the number of condltions reported for cases which are one or more on both interviews are close to the number of conditions accounted for by changes of a report of none on the original to a report of one or more on reinterview and increase the volume of conditions reported on reinterview. Changes 21 Table 13. Estimated proport~ons, interview net and gross difference compared with proxy rates, and indexes respondents, procedure of inconsistency Persons interview, and respondent Average in class Survey item, subject and respondent on original by subject on original I I 1, 3-year averages, fiscal years 1959-61 persons with: Original Average net Reinter- of difference rate view Index gross difference Inconsistency rate One or more chronic conditions: Adultrself-respondent Adult, proxy . . . . . . . . . . . . . . . . . . . . . 59.4 64.8 -5.4 6.1 13,1 . . . . . . . . . . . . . . . . . . . . 47.4 58.5 -11.1 14.9 30.3 12.0 6.3 respondent (self-respondent - proxy respondent) . . . . . . . . . . . . 5.7 One or more hospital episodes in past 12 months: Adult, self-respondent Adult, proxy . . . . . . . . . . . . . . . . . . . . . 13.2 13.5 -0.3 0.6 2.4 . . . . . . . . . . . . . . . . . . . . 9.0 10.0 -1.0 1.3 7.3 4.2 3.5 respondent (self-respondent - proxy respondent) activity days in past 2 weeks: One or more restricted Adult, self-respondent Adult, proxy 0.7 . . . . . . . . . . . . . . . . . . . . . 13.5 14.3 -0.8 3.5 14.8 . . . . . . . . . . . . . . . . . . . . 8.2 11.0 -2.8 5.0 28.9 5.3 3.3 . . . . . . . . . . . . . . . . 6.7 7.1 -0.4 1,7 13.4 . . . . . . . . . . . . . . . . . . . . 4.7 5.5 -0.8 2.0 21.1 2,0 1.6 respondent (self-respondent . . . . . . . , . . . . - proxy respondent) . . . . . . . . . . . . 2.0 One or more bed days in past 2 weeks: Adult, self-respondent Adult, proxy .,... respondent (self-respondent One or more time-lost - proxy , . . . . . . . . . . . 0.4 days in past 2 weeks: Adult, self-respondent Adult, proxy . . . . . . . .. . . . . . . . . . . . . . 3.1 3.5 0.4 0.8 11.8 . . . . . . . . . . . . . . . . . . . . 4.3 4.7 –0.4 2.4 28.2 respondent (self-respondent respondent) - proxy respondent) . . . . . . . . . . . . -1.2 -1.2 One or more hospital days in past 2 weeks: Adult, self-respondent Adult, proxy Table . . . . . . . . . . . . . . . . . . . . . 0.7 0.7 0.3 21.6 . . . . . . . . . . . . . . . . . . . . 0.4 0.4 0.2 23.9 0.3 0.3 respondent (self-respondent - proxy 14. prevalence Estimated reinterview respondent) . . . . . . . . . . . . rate of chronic conditions per person by subject and respondent estimates for a sample of identical persons, reconciled reinterviews, m Chronic Subject Adult, self -respondent Adult, proxy Estimated NOTE: 22 interview - proxy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . respondent sampling error of difference A similar comparison The magnitude on original . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . respondent Self-respondent and respondent on original interview, original and fiscal years 1959-61 conditions per person (12 months) Net cliff erence (a) - (b) (a) - (b) (b) 1.26 1.52 -.26 -.17 .84 1.25 -.41 -.33 . . . . . . . . . . . . . . . . . . . . . . .16 . . . . . . . . . . . . . . . . . . . . . . . .02 for a different time period of effect that the type of respondent and including has is about the same. all conditions, acute and chronic, presentad in table 15. Table 15. Estimated rate of conditions per person by subject and respondent for a sample of identical persons, reconciled on original reinterviews, interview, I m Subject and respondent at original interview Adult, self-raspondent Adult, proxy Salf-respondent Estimated Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . respondent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - proxy 16. Estimated total and reinterview estimates Conditions Pw person Net (a) - (b) — (b) difference (a) - (b) 1.61 1.74 -.13 -.07 1.00 1.31 -.31 -.24 . . . . . . . . . . . . . . . . . . . . . . .17 . . . . . . . . . . . . . . . . . . . . . . . .01 respondent sampling error of difference original fiscal years 1963-67 net difference between original reconciled interview results and reinterview reinterviews, results and components of net difference, fiscal years 1959-51 Ratio of components Components to total of total net difference net difference T ot a l net difference Survey item in reporting in reporting from presence on original Chronic conditions past 12 months Hospital in reporting magnitude reinterview both interviews (4) (3) (2) when present on to presence on to absence of ufagnitude 0,1 iifferance difference ~ I(2) I - I(3) I (1) (1) (5) (6) in . . . . . . . -12,541 964 -26,700 -15,123 .43 .57 .85 .15 .16 .83 .70 .30 .94 .06 episodes in past 12 months Hospital from absence on original on reinterview (1) Due to change Due to change Due to change (x-y)’ . . . . . . . -881 220 -969 584 -552 -132 days in past 2 weeks Restricted . . . . . . . . . . . . activity 185 . . . . . . . -13,808 7,244 -17,003 . . . . . . . . . . . . -3,592 1,533 -4s1 -3,361 1,327 -4,684 inpast2weeks 153 days -4,149 Bed days in past 2weeks Time-lost -204 days in past 2weeks . . . . . . . . . . . . *x is the estimate ‘Algebraically of magnitude this component from the original interview, and y is the estimate is [column (2) + column (1)] - [column (3)1. -4 of magnitude 1.00 (’) from the reinterview. A minus sign indicates a net increase on reinterview. s Less than 0.005. NOTE: components The tabulations for hospital of the net difference a change of class interval days, restricted were estimated activity days, bed days and time-lost by using midpoints of the 2day intervals. days are in terms of 2day intervals. The Changes in tasponse that did not result in have no effect on the estimates. from a report of one or more on the original to a report of none on reinterview do not have much effect on the estimates of chronic conditions. A large proportion of the gross differences in estimates of magnitude is accounted for by changes from a report of none to a report of one or more in both directions. The following index is an estimate of the proportion of the total response variance that is accounted for by cases which either change from a report of none on 23 the original interview to a report of one or more on reinterview or from a report of one or more on the original interview to a report of none on reintemiew: & +~foy index of net shift is simply the ratio of the net difference rate to the percent in class on reinterview. INTERVIEWER VARIABILITY STUDY izyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA LO(X - 0)’ (o- 3))2 Introduction y=lzyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA .=1 = Ww The joint effects of sampling and nonsampIing errors determine the accuracy of survey results. The mathematical model of response errors in surveys presented in appendix III shows how the mean square error of a statistic is divided into its where x is the value on the original interview, y various components: sampling variance, response is the value on reinterview,zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA fxo is the number of variance, interaction, and square of bias. persons with x value on original and O on The response variance can be further divided reinterview, ~. is the number of persons with O into simple response variance and correlated on original an 2 y value on reinterview, and ~XY is response variance. In the preceding section, the number of persons with x value on ongmal estimates were presented of the simple response interview and y value on reintenriew. variance and of response bias as measured by Table 17 shows that except for chronic reinterviews. The usual estimates of sampling conditions most of the response variance for variance include the simple response variance estimates of magnitude can be accounted for by and possibly a small part of the correlated changes in the reporting of the presence or response variance due to field interviewers. absence of the characteristic. However, the major part of the interviewer Tables 18-22 present estimates by year of the contribution to response variability is not insummary measures discussed in previous seccluded in the estimates of sampling variance. tions. All of these results are for procedure III, This section describes an interviewer variance that is, a comparison of the original interview study designed to measure the contributions of with reinterview after reconciliation. The yearly, interviewers to the vanabilit y of health statistics. estimates are subject to large sampling errors Significant between-interviewer variance in since the reinterview sample in any one year is the reporting of health data has been observed in relatively small. One additional summary measa number of studies. Data from other studies ure, the index of net shift, is presented. The also indicate that interviewer effects may operate differently for different statistics. Table 17. in Proportion reporting of total of presence cedure I I 1, reconciled response variance or atxence due to changes of characteristic results G Characteristic . Chronic conditions Hospital episodes in past 12 months in past 12 months Hospital days inpast2 Restricted Beddays activity weeks icpa$t2weeks NOTE: . . . . . . . . . . .41 , . . . . . . . . . . .92 , . . . . . . . . , . . . . . .96 days in past 2 weeks Hospital . . . . . . . . . .88 . . . . . . . . . . . . . . . . . .80 days, restricted time-lost days for the tabulations Midpoints of the 2day intervals values. Changes in response which class interval 24 activity days, bed days, and are in terms of 2-day intervals. were used for the x and y did not result in a change of have no effect on the estimates. Design of the Interviewer Variance Study pro- The interviewer variance study was conducted over the 4-year period 1960-63. For the first 2 years of the study, randomization of interviewer assignments was carried out in eight large SMSA’S where there were two or more HIS interviewers. The study included 10 SMSA’S during the second 2-year period. The assignments (within each pair or triplet) of interviewers within an SMSA were randomized in an interpenetrated design so that each interviewer of a pair would have produced results with the same expected value if there were no betweeninterviewer variabilityy. Table 18. Estimated proportions, net difference rates, indexes of net shift, gross difference rates, and indexes of inconsistency for procedure III after reconciliation for parsons with one or more chronic conditions in the past 12 months, by year > I Percent in I Fiscal year I class on original intarview 38.5 42.5 41.7 1959 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1960 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1961 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1962 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . NA2 1963 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1964 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1965 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.5 42.8 43.7 48.5 46.3 18663 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1867 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Net difference in class on reinterview Parcant 45.2 48.3 49.2 NA 48.2 50.5 51.3 58.2 51.9 Net rate X 100 -6.8 -6.7 -7.5 NA -6.7 -7.7 -7.6 -7.7 -5.5 rate Percent in class on reinterview Index of net shift 1 x 100 -15.0 -13.8 -15.3 NA -13.8 -15.2 -14.8 -13.8 -10.7 Gross difference rate X 100 1 Index of inconsistency x 100 8.6 7.9 8.9 NA 8.3 17.9 16.2 18.1 NA 16.8 19.3 17.6 17.9 13.4 9.5 8,7 8.9 6.7 z Not available. ‘2 quarters only (July 1, 1965 -Dec. 1965). Table 19. Estimated proportions, net difference rates, indexes of net shift, gross difference rates, and indexes of inconsistency for procedure III after reconciliation for persons with one or more hmpital episodes in the pest 12 months, by year Percent in class on original interview Fiscal year 1959 1860 1961 1962 1963 1964 1885 1966Z 1867 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Net difference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4 9.4 8.0 NA 9.6 11.1 11.9 7.3 8.7 Percent in class on reinterview 8.6 10.1 9.1 NA 10.9 11.8 12.1 7.5 9.1 Net difference rate X 100 -0.1 -0.7 -1.1 NA -1.4 -0.7 -0.2 -0.1 -0.4 Index of Gross net shifti difference rateX 100 x Iw -1.2 -6.9 -12.1 NA -12.8 -5.9 -1.7 -1.3 -4.8 0.9 0.9 1.2 NA 1.5 0.8 0.4 0.1 1.1 Index of inconsistency x 100 5.5 5.2 7.4 NA 8.3 3.6 1.6 1.0 6.9 ;ate Percent in class on reintewiew 22 quarters only (Jul.{ 1, 1965-Dec. 1965). For the first 2 years of the study, calendar years 1960 and 1961, the data cover theworkin 25 interviewer assignment areas. Six of the eight SMSA’S had a pair of interviewers, Los Angeles had atriplet, and the New York SMSA hadfive pairs ofinterviewers. For the second 2 years, calendar years 1962 and 1963, the data cover the work in 30 interviewer assignment areas for two quarters and the work in 28 interviewer assignment areas for the remaining six quarters. (The HIS sample was redesignedin 1962, and one interviewer pair was dropped in the New York SMSA.) Assignments in Chicago and Los Angeles were randomized among three interviewers during the second2-year period. During the 4-year period, a total of 6,415 segments of six to nine households were in- 25 Table 20. Estimated procedure proportions, net difference III after reconciliation rates, indexes of net shift, gross diffarance for persons with one or mora restricted Percent in class on Fiscal yaar original interview Percent in Net class on reinterview difference 1959 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.4 11.9 1960 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.5 12.8 1961 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.4 14.1 1962 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1963 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1964 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1965 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . NA 9.5 10.9 11.1 6.4 10.9 NA 10.0 11.9 12.3 7.5 13.6 1966’ . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1967 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Net difference rate Percent in class on reinterview 22 quarters only (July 1, 1965-Dec. Percent Raason for exclusion of total segments Total assigned segments not used in Segments completed . . . . . . . . . . . . . . . . . . by “other Segmants where it could not be determined complated the assignments 18.8 places” . . . 1.2 2.0 of Analysis For this analysis, ratios of two variables were used at all times for two reasons. First, most HIS data are presented as rates orproportions of the total population; and, second, this method minimizes the effect of the variability in size of interviewer assignment on the analysis. The starting point for the analysis was a computer tape containing quarterly totals for 26 I ndax of nat shift] x 100 -11.8 -17.9 -12.1 NA -5,0 -9.2 -10.6 –14.7 -19.9 T Gross difference rateX 100 4.0 4.4 3.3 NA 2.5 2.3 3.0 1.8 4.5 Index of inconsistency x 100 1 20.6 22.0 14.6 NA 14.2 11.1 15.0 13.9 23.1 each of 84 health and demomaphic character. istics for each of the intem~ewer assignment areas. The mathematical model usedto compute the total variance is basically acomparison of results between a pair of interviewers, extended to the whole study area. The task was to estimate the expected value of the difference between two interviewers’ findings for a specified period of time. In the model, the total relvariance for the study area is (1) who . . . . . . . . . . . “special dwelling 12.4 3.1 The remaining 5,211 segments used in the analysis contained about 34,000 households with 110,000 persons. Method -1.4 -2.3 -1.7 NA -0.5 -1.1 -1.3 -1.1 -2.7 for than assigned interviewers” . . . . . . . . . . . . . . . . . . Sagments in non-self-representing PSU’S . . . . . . Sagments containing rate X 100 rates, and indexes of inconsistency days in the past 2 waeks, by year 1965). eluded in the assignments. Of these segments, 1,204 were excluded from the analvsis for the . followingreasons: analysis activity where r= is the ratio computed from a random half of the interviewers and r~ is the ratio computed from the other random half and the expected value is taken over all possible halfsamples. An estimate of this expected value was made by assigning each interviewer in a pair to a or b at random and then averaging over all pairs of interviewers.b Different permutations of the pairs give other estimates. Twenty-five permutations were used to give the estimator: bStrictly speaking, it is interviewer assignment area instead of interviewer. When there was a change in interviewers due to resignation, illness, or other administrative reason other than a temporary substitution, the work of the replacement interviewer was treated es part of the same interviewer assignment area. Table proportions, net difference rates, indexes of net shift, gross difference rates, and indexes of inconsistency for procedure III after reconciliation for persons with one or more bed days in the past 2 weeks, by year 21. Estimated Fiscal yeac 1959 1960 1961 1962 1863 1964 1965 1966= 1967 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Net difference rate Percent in class on reinterview 22 quarters only (July 1, 1965 -Dec. 1965). Table 22. Percent in class on original interview Percent in class on reinterview difference rateX 100 5.6 5.5 5.4 NA 5.3 6.0 5.1 4.5 6.8 6.1 6.4 6.3 NA 6.0 6.4 6.0 5.0 8.8 -0.5 -0.9 -0.9 NA -0.6 -0.5 -1.0 -0.5 -1.9 L class on original interview Fiscal year 1959 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1960 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1961 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1962 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1963 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1964 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1966 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1967 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 -8.2 -14.1 -14.3 NA -10.0 -7.8 -16.7 -10.0 -21.6 3.8 2.1 1.7 NA 2.1 1.6 1.2 1.7 3.4 15.8 18.3 I 5a NA 19.9 13.8 12.7 22.7 25.6 -1- Estimated proportions, net difference rates, indexes of net shift, gross difference rates, and indexes of inconsistency for procedure III after reconciliation for persons with one or more time loss days in the pest 2 weeks, by year I Percent in 19662 Net Net difference 4.2 3.7 3.3 NA 3.1 2.6 3.4 3.0 3.6 in class on reinterview Net difference rate X 100 Index of netshifti x 100 difference rate X 100 Index of inconsistency x 100 4.9 4.9 3.8 NA 3.8 3.1 4.2 3.1 5.2 -0.7 -1.2 -0.5 NA -0.7 -0.5 -0.8 -0.1 -1.6 -14.3 -24.5 -13.2 NA -18.4 -16.1 -19.0 -3.2 -30.8 1.4 1.7 2.0 NA 1.1 1.6 1.2 0.6 2.6 19.3 24.1 29.5 NA 15.8 31.6 15.0 10.7 37.2 Percent Gross rate Percent in class on reinterview 22 quarters only (July 1, 1965-Dec. 1965). 25 where j denotes the permutation and r the estimated ratio computed from the work of all interviewers. This estimator has two components that must be identified separately :the between-interviewer relvariance and a sampling relvariance arising from the fact that each interviewer of a pair interviewed a separate random sample of households. Estimate~of V; were mad: for asingle quarter of data, two quarters combined, three quarters, up to an estimate based on eight quarters of data. In the analysis, the results were treated as if they were two independent studies of 2 years each. One reason for treating the results as two observations was to minimize the effect of interviewer turnover. Another reason was strictly practical; 2 years of data could rehandled more easily than 4years. If it is assumed that the sampling variance is a function of sample size but that the estimate of the between-interviewer variance is a function only of the number of interviewers, then (3) A where ~ is the estimat~ of the betweeninterviewer relvariance and WE/n is the estimate of the sampling relvariance for a single quarter divided by the number of quarters used in the estimate. The parameters of this function were estimated in terms of V$ by the method of least squares (appendix IV). This least-s~uares s~lution was then used to determine ~ and V:. Figure 1 sh~ws an example of the expected behavior of ~~ as the sample size increases. As the number of quarters included in the estimate increases, the sampling component of the variance decreases and the estimate of total variance approaches the between-interviewer variance asymptotically. The assumptions of equation 3 are undoubtedly not fully warranted. There is evidence from other studies conducted by the Bureau of the Census that the response variance cannot be viewed as a constant even if, as is not the case in the present study, the interviewers did not change over the 2-year period. To illustrate further that the assumptions are not completely true, it can be shown that the response variance is the sum of two terms. The first is a simple response variance, which expresses the variability in repeated measures (interviews) on the same persons. This quantity varies inversely with the sample size and thus * ‘* 2 VT i+ +F -)F +F --! . . . -. . -. 1 . . . . . -------- 2 3 ------- 4 NUMBER Figure 1. Expected 3: . . . ------ 5 6 * -1e . . . . . . . _. . 7 - -- 4; 8 OF QUARTERS reduction in I/; as sample size increases. depends on the number of quarters of data. In the estimation scheme used in this study, the simple response variance has been treated as part of the sampling varim~ce and subtracted out. Thus, the estimator ~ is somewhat of an underestimate. The second term is the product of two factors: the simple response variance and a correlation expressing the extent to which each interviewer tends to introduce her own systemerrors in her assignment (see atic response appendix III). An assumption of this study is that this correlation is a constant over the 2-year period, although there is evidence from other programs that the correlation may decline with an increase in experience because of training, increased proficiency, and the increasing heterogeneity in the population included in the assignment. Results Tables 23 and 24 contain the results of this study. These results are presented in two tables, one for each 2-year period. Each table shows results for 67 items. The estimated relvariances shown in the tables relate to annual estimates prepared from the work of approximately one-fourth of the ‘inter-’ viewer staff. For some items the estimates of between-interviewer relvariances are negative. These are items which presumably have a very low interviewer variability. The sample size is too smzdl to place much reliance on the specific estimates of the relvariances; however, the last column of the tables is probably sufficiently reliable to provide a general ranking of the characteristics. This last column expresses the between-interviewer relvariance as a proportion of the total relvariance. Discussion of the results shown in tables 23 and 24 is based on this last column. The results shown in the tables indicate that the reporting of chronic conditions and activity restriction associated with such conditions have the highest ratios of between-interviewer variance to total variance. In addition, as might be expected, reporting of income also has a fairIy high ratio. As has been observed in other studies of between-interviewer variance, reported in papers Table 23. Intewiewer variance study of estimates of cwnponents of relvarianm for annual estimntas of selectad characteristics based on 26 interviewer eight SMSAW assignment areas in zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLK United States, 1 S%0-61 — Denominator Numerator Health characteristics (Magnitude (total number of the following] Ratio Betwen- l-otalra!variance ~2 T Interviewcr Sampling relvariawe relvariance zyxwvutsrqponmlkjihgfedcbaZYXW g ~: items): Conditions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chronic conditions for females Chronic conditions formales . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.563 .769 .00395 .00463 Males .612 Persons .00026 .00041 .00434 .00339 .00442 .0Q370 .CK)W’f .934 .915 .664 .115 .00476 .oo4rM .00072 .846 Chronic conditions .633 .01328 .01045 .00283 .7s7 Persons Persons .502 .115 .W411 .00285 .00296 .03202 .00115 .oom3 .720 .706 Persons .024 .00661 .0Q604 .043256 .702 Accidents 1.182 .01269 .W642 .C0447 .663 Acute conditions 3.119 .00273 AX1135 .0013s .494 Acute conditions for males 2.696 .136 .00524 .00623 .W244 .00261 .465 .227 4.246 .0026s .0Q675 .00%1 .00115 .00243 .00462 .00153 .0C332 .439 .430 .217 .00359 .00135 .00224 .376 Househol& Females Chronic conditions with 1 or more ted days in last 12 months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Restricted activity days for chronic renditions inlast2w3ek5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . R=tdctWatiivi~&Winlsti2Mh . . . . . . . . . . . . . . . . . . Acute conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chronic conditions with 1 or more bed days in last 2weeks Disability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . days in last 2 weeks fmm all =idents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Restricted activity dcys in last 2 weeks for acute conditions Restricted . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . cctivity days in I&ct 2 weeks for ccute conditions formdac. . . . . . . . . . . . . . . . . . . . . . . . . . Bed days for chronic conditions in k.t 2 weeks . . . . . . . . . . . . . Bedd.cys inlast2vmks . . . . . . . . . . . . . . . . . . . . . . . . . Bed days for chronic conditions in last 12 months . . . . . . . . . . . Days lost from school or work in Iaat 2 weeks . . . . . . . . . . . . . . Persons Persons Chronic conditions Persons currently fEBOl15 D8yslostfrom Baddaysfor workinlast2weeks . . . . . . . . . . . . . . . . . . . acute conditions inlast2weeks Dayslostfrorn school inlact2weeks Hospital disclmges inlastl Hospitalizations . . . . . . . . . . . . . . . . . . . month inlast12months Hospital dayainlast 12months . . . . . . . . . . . . . . . a@d 5-16 Persons currently .423 employed and YCWS emplcyed Acute conditions .206 .00517 1.261 .00203 .0013s -.00007 .00365 .IX121 o .259 -.036 Persons aged 6-16 years .23s .01041 -.00113 . . . . . . . . . . . . . . . . . . . . Persons .103 .0C049 -.00007 .01154 .W056 -.1 2a . . . . . . . . . . . . . . . . . . . . Households .333 .00045 -.OQO11 .CCI056 -.266 10.732 .00236 -.00133 .W169 -.563 10.456 .00183 -.0+31 70 .00353 -.926 .6% . . . . . . . . . . . . . . . . . . . . . . Hospitalization -.109 in last 12 months Hcspital days for ail discharged in last 12months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hcspital discharges in lest 12 months Health characteristic (Attributa items): Persons with 1 or mom chronic conditions . . . . . . . . . . . . . . . . Per30ns .369 .00196 .00174 .00022 . . . . . . . . . . . . . . . . . . . . Pa fiOns .450 .00165 .W146 .wcr19 .666 Males .359 .00230 .C4MS3 .00037 .636 Malmwithl ormoreconditions . . . . . . . . . . . . . . . . . . . . . Perxons limited in kind or amctmt of activities . . . . . . . . . . . . . . Males Persons .426 .066 .00166 .00263 .00165 .W152 .00033 .W132 .632 .536 Acute conditions rnadically attended . . . . . . . . . . . . . . . . . . . New cases ccute respimto~ conditions, 1 w more Acute conditions .731 .00031 .000Q3 JYJ022 .2a3 Persons . 026 .00424 .W1C8 .00317 .2M .147 .01540 .00369 .01171 .240 .s46 .018 .0U360 -.00107 .00233 -.-1 Persons withlor Maleawith baddays moreconditions lormore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hospitalizations formales chronic conditions for tonsillectomy inlsst12months or adanoidectomy . . . . . . . . . . . . . . . . . . . . . . Hospitalizations for operations, males Hospitalizations for operations on the female genital system inlast12 months . . . . . . . . . . . . . . . . . . . . . . . Hospitalization for cfmations, femdas, exclusive of delivery Persons unable to carry on major activity .%ciceconomic Fmilieswith income >$6000 Families withincome <$20W P.waons who are not employed . . . . . . . . . . . . . . . . . . . . . . .00467 .00313 -.296 -.W6 Families .521 .00115 .m77 .00a29 .665 . . . . . . . . . . . . . . . . . Persons .606 .0@39S .0GQ66 .000S3 .664 . . . . . . . . . . . . . . . . . . . . . . Families .117 .00371 .00127 .00244 .342 Persons aged 17 years and over .075 .00133 .0003S .mlcm .246 Persons aged 17 years and over .561 .OC@X .Coxrl .00LW5 .116 Persons .065 ,00356 .LIJ031 .~326 .066 Femalas aged 17 years and over .330 .00040 .00042 -.069 and not keeping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Employadpmons . . . . . . . . . . . . . . . . . . . . . . . . . . . . PersOns in families with inc0rne>$2000 Employed Persons characterictica: PersOns in families withinc0nw>$5000 house . . . . . . . . . . . . . . . . . femalec . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ~ -.WJO03 29 Table 23. Interviewer variance study of estimates of componentsaf re[variance for annual estimates of selected characteristics eight SMSA’s: United States, 1660.61-Con, based on 25interviewer Total rel(total number of the following) ?atio variance y Betweennterviewer relvariance +2 I assiqnmentareasin Sampling ,elvariance ~; Nonresponse Items: Persons with amount ofeducationu”known . Heads of households with amount of education unknown . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Families with unknown family income Conditions from amidems, u“knmvn . . . . . , . . . . . if m~t~r WMcIe Conditions from accidcmts, u“k”cwv” lEatiO” Demographic . . . , , . , . . . . . . characteristics (includhg . . . . . . . . . . . , . . . . . . . . . . . . . . Perscms aged 17 years and over .021 .01961 .01535 .00426 HcmsehOl& .026 .02727 .02064 .00663 .757 Families .065 .01275 .00798 .626 Accidents .003 .37186 .34642 ,00477 JJ~E44 Accidents .002 .53790 .58952 Chronic conditions .016 .239 .00883 .00133 .00545 .00036 .00337 Acute conditions .00098 .618 .267 Chronic conditions .049 .00251 .00050 .00201 .200 Acute conditions Femalesr aged 17 years and over .102 .00520 .00068 .00452 .131 .643 .00013 .CK)ool .00011 .108 Persons .007 .03511 .80200 .03311 .057 -.05162 .783 .932 1.096 health characteristics by age groups): Chronic conditions with 1 or more &d days, perscms aged5.14 years . . . . . . . . . . . . . . . . . . . . . . . . . . . . Acute conditions, persons aged 25.44 years . . . . . . . . . . . . . . . Chronic conditions with 1 or more bed days, persons aged2544 years . . . . . . . . . . . . . . . . . . . . . . . . . . . . Acute conditions, persons aged 15-24 years Females, married, spouse present Perscms ofather , . . . . . . . . . . . . . . . . . . . . . . . . . . . races . . . . . . . . . . . . . . . . . . . . . . . . . . . Chronic renditions agedunder . . . . with 1 or more bed days, persons 5 years . . . . . . . . . . . . . . . . . . . . . . . . . . . Perscmsof Negro race . . . . . . . . . . . . . . . . . . . . . . . . . . . .00076 ,01310 .055 Chronic conditions .007 .01386 Persons .106 .01216 -.00011 .01227 -.009 .00175 -.050 Acute conditions .241 .00167 -.00008 Chronic conditions .028 .00376 -.00024 .00400 -.064 Acute conditions Persons aged 17 years and over .075 .465 .00756 -.00066 .00822 -.o87 .00005 -.00001 .00006 -.164 Persons .020 .00161 -.00033 .00194 -,203 Persons .960 .00000 -.00000 .00000 -.214 Persons aged 17 years and over .535 .00003 -.00001 .00004 -.222 . . . . . . . . . Acute conditions .170 .00235 -.00074 .00309 -.316 Acute conditions, persons aged 5 years and over . . . . . . . . . . . . . Persons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Acute conditions .173 .00282 -.00107 .00389 -.380 .OQoos -.WO05 .00013 -.677 -.cf1430 .01058 -.683 Acute conditions, persons aged 5.14 years . . . . . . . . . . . . . . . Chronic conditions with 1 or more &d days, persons 65years And over . . . . . . . . . . . . . . . . . . . . . . . . . . . Acute conditions, persons aged 65 years a“d over Males aged17 years Andover.. Persons agedunderl Persons agedl year yearand . . . . . . . . . . . . . . . . . . . . . . . . . . . . . over.... . . . . . . . . . . . . . . . . . . . Females aged17yearsandowr. Acute conditions, . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . persons aged 45.64 years . . . . Households 3.167 Chronic conditions with 1 or more bed days, persons aged15-24 Chronic conditions .011 .00629 Chronic conditions .053 .00130 -.00092 .W223 -.706 . . . . . . . . . . . . . . . . . . . . Persons .667 .00005 -J33004 .moo9 -.646 . . . . . . . . . . . . . . . . . . . . . . . Persons .333 .00020 -.00018 .00036 -S96 — years . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chronic conditions with 1 or more bed days, persons aged45+4year5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . Permnsa gedl?yearsa ndover. Persons agedunder17 years . by Eckler and Hurwitz,l O and Hurley, Jabine, p.nd Larson,l 1 there are considerable betweeninterviewer variances in nonresponse rates. Note, for example, that the ratio of betweeninterviewer variance to the total variance is about .8 for the number of persons with eciucat.ioz urdmown. It is also of interest to note that the demographic differentials in morbidity rates (distributions of acute and chronic conditions by age gyoups) are not subject to any significant between-interviewer variance. 30 The estimates of interviewer variability from the two periods (1960-61 and 1962-63) differ considerably for some items. The material for the first 2 years was investigated to determine the cause of the higher estimates of interviewer variability for these items. During the 11960-61 period, one interviewer of the 25 in the study contributed a disproportionate amount to the estimates of between-interviewer variability. However, the response variance study of the 1960 censusl 2 demonstrated that the distribution of individual interviewer contributions to Table 24. Interviewer variance study of ettimates of components of relvariance for annual estimates of selected characteristics 10 SMSA%: Numeramr Denominator (total number of the following) Health characteristic (Magnitude based on 2S interviewer assignment areas in United States, 1962.S3 [total num be r of the follom”ng} Total rel Ratio variance 62 T Betwaen- %mpling intwdewm relvariance relvarianm $2 zyxwvutsrqponmlkjihgfedcbaZYX E q items): COnditiOns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chronic conditions for femal~ . . . . . . . . . . . . . . . ., . . . . Households . 000S4 .Cn1024 .00039 Females .848 .00117 .GO06B .0Q048 ,5% Chronicconditionsformales Males .667 .Oom .00029 .00064 .312 Persons .132 .00142 .mw .00077 .455 .0QS39 .00250 .770 .00204 .00110 . . . . . . . . . . . . . . . . . . . . . . Chronic conditions with 1 ormorebed days in last 12mon%hs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Restricted activity days for chronic conditions in last 2weeks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chronic conditions .6s3 .010s9 . . . . . . . . . . . . . . . . . Pemons .610 .0+)314 . . . . . . . . . . . . . . . . . . . . . . . . . . . . Persons .120 .00054 Restrictedactivitydaysinlast2weeks Acuteconditions 2.85c Chronic conditions with 1 ormorebed .00066 -.W012 ,3s4 .649 -.230 days in last 2weeks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Disabilitydaysinlast2weeksfromall accidems . . . . . . . . . . . . PertOns Accidents .028 .01029 .CQ734 .00295 .713 1.022 .0W80 .IX1353 .00327 .519 Acute conditiom 2.812 .00118 .00015 .00103 .126 Acute conditions for males 2.716 .00232 .00042 .00190 .1s0 .172 .01242 .00552 .00s90 .445 .06C65 .00180 Restricted activity days in last 2 weeks for acute c ondit ions . . . . . . . . . , . . . . . . . . . . . . . . . . . . . . Restricted activity days in last 2 weeks for acute conditions for males . . . . . . . . . . . . . . . . . . . . . . . . . Beddays forchronic conditions inlast2weeks Beddays inlast2weeks Beddays forchronic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . conditimw inlast12 months Days lost from school orworkinlast2waeks . . . . . . . . . . . . . . . . . . . . . . . PemOns Persons Chronic conditions Persons currently employed and persons aged 6-16 years Dayslost fromwork inlast2weeks Beddays foracute conditions Dayslostfrom schmlinlest2 Hmpital discharges inlast12 Hc=spitalizations inlast12 Hospital daysinlast . . . . . . . . . . . . . . . . . . inlast2v.eeks . . . . . . . . . . . . . . , weeks . . . . . . . . . . . . . . . . . . . months . . . . . . . . . . . . . . . . . . . months . . . . . . . . . . . . . . . . . . . , 12 months”.. . . . . . . . . . . . . . . . . . . . . Hospital days foralldischarges inlast12 months . . . . . . . . . . . . . Penons currently employed Acute conditions Pemons aged 6.16 yeart Persons Households Hospitalizations Persons withl ormorechronic .00245 .00273 .231 .00219 .00Q05 .00214 .217 .00342 .00012 .00330 .036 .00172 .W1004 .00168 .023 1.= .240 .00712 .071 .382 .00102 .00261 10.s45 1.3s4 .C0279 -.0000s .265 -.021 .021 JX1981 -.37s .00034 .Wol 7 .000S8 .00044 .332 .281 .00297 .00004 .00292 .014 .C0766 .C0327 .0@138 .428 -.00270 in Iatt 12 months Hospital discharges in last 12 months Health characteristics (Attribute .2s3 4.677 items): . . . . . . . . . . . . . . . . Persons .4W .0CQ34 .W016 .00018 .475 . . . . . . . . . . . . . . . . . . . . Persons .46s .OCHX% .ml 1 .00014 .446 Males with 1 ormore chronic conditions . . . . . . . . . . . . . . . . . Males with lormoreconditions . . . . . . . . . . . . . . . . . . . . . Males .3s7 .00046 .Wol 7 .0002s .262 Males .444 .00U% .m13 .CQ023 .354 Persons limited inkindor Pertons .06s .00211 .00131 .00060 .621 Acute conditions .76s .0Ca315 .CQOcE .GQolo .35s .030 .00272 .W016 .W266 .058 .149 .01129 -.00173 .01302 -,153 -.GQ020 .00255 .Cm250 -.0s4 .00026 .00026 .280 .044 Persons withlor conditions more conditions amount of activities . . . . . . . . . . . . . Acute conditions m%ficallyattended . . . . . . . . . . . . . . . . . . . Newcases acute respiratory conditions, l or more bed day s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hospitalizations malasin for tonsillectomy last12 months... Persons or adenoidectomy, . . . . . . . . . . . . . . . . . . . . . Hospitalizations for operation, males Hospitalizations foroperations, system inlast12 months.. female genital . . . . . . . . . . . . . . . . . . . . . Hospitalizations for operations, females, exclusive of del ivery Persons unable tocar~on Socioeconomic major activity . . . . . . . . . . . . . . . . . .347 .00235 Persons .018 .002!33 .@3003 .012 characteristics: . . . . . . . . . . . . . . . . . . . . . . Families .573 .0D035 .C4)oo9 Persons in families with income >$50W . . . . . . . . . . . . . . . . . Families withincome <$20GQ . . . . . . . . . . . . . . . . . . . . . . Families withincome >$5000 Persons .657 .115 .W27 .CW68 .Oooo1 .0010s .WY160 .4m Persons whoarenotemploy edandnot keeping house . . . . . . . . . . Employed persons . . . . . . . . . . . . . . . . . . . . . . . . . . . . PersOns in famities with inc0me<$20W . . . . . . . . . . . . . . . . . Persons aged 17 years and rover Persons ag?d 17 years and over .07S .E60 .000s5 .00004 .0cw3 .Occm .00W2 .00004 .122 .00272 .224 Female .064 .324 .00078 Employ ed females .00015 .CO025 -.721 . . . . . . . . . . . . . . . . . . . . . . . . . . . . Families Persons aged 17 years and over .00351 -.00011 .269 31 ‘ Table 24. Intewiewer variance study of estimates of components of relvariance for annual estimates of selected characteristics I08M8A’s: based on 28intewiewer United States, 1962-63-Con. Numerator Denominator (total number of the following) (total numb?r of thO following) assignment areasin — Total Ratio rel- variance 92 T . Botweeninterviewer Sampling relvariance ;2 E —— fiz J~z T relvariance zyxwvutsrqponmlkjihgfedcbaZ $/2 Nonresponse Items Persons with amount of education unknown . . . . Persons aged 17 years and over .016 .02338 .01950 Households .019 .00549 .802 .055 .02767 .01244 .02218 Families .003s0 .711 .00388 .834 Heads of households with amount of education unknown ..,...... Families with unknown . . . . . . . . . . . . . . . . . . . . . family income . , . . . . . . . . . . . . . . . . . . . Accidents .008 .08156 .00884 -.00321 . . . . . . . . . . . Accidents .004 .18992 .0s211 .10781 .432 Chronic conditions with 1 or more bed days, persons aged5.14 years... . . . . . . . . . . . . . . . . . . . . . . . . . Chronic conditions .017 .00401 Acute conditions .246 .00109 .00040 .00023 .003B1 Acute conditions, .100 .208 Chronic conditions .052 .00168 .00015 .Ocll 53 Acute conditions .123 .00207 -.0+3053 .00260 Females, aged 17 years and over .633 .00011 .00002 .00009 .208 Persons .011 .C6787 .01118 .05669 .165 Chronic conditions Persons .008 .118 .00896 Persons of Negro race . . . . . . . . . . . . . . . . . . . . . . . . . . . .01480 .00543 .00937 Acute conditions, Acute conditions .231 .00134 .00016 .00118 Chronic conditions .029 .00279 Acute conditions Persons agad 17 years and over .C64 .0C630 .00076 .00555 .120 .466 .00002 .00000 .00002 .103 Per30ns .020 .00210 .00051 .00159 .241 . . . . . . . . . . . . . . . . . . . Persons .960 .00000 ,00000 .00000 .275 . . . . . . . . . . . . . . . . . . . Persons aged 17 years and over .534 .00002 .00000 .00002 .152 Acute conditions, persons aged 45.64 years . . . . . . . . . . . . . . Acute conditions, persons aged under 5 years . . . . . . . . . Acute conditions Acute conditions .171 .166 ,00252 .00053 .00025 .00199 .00245 .210 Persons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chronic conditions with 1 or more bed days, persons Households 3.231 .0WC6 -.00001 .00007 .093 -.224 Conditions from accidents, unknown if motor vehicle Conditions from accidents, unknown location . . . .0s477 -.039 Demographic characteristics (including health characteristics by age groups): persons aged 2544 years . . . . . . . . . . . . . .00086 Chronic conditions with 1 or more bed days, persons aged2544 years . . . . . . . . . . . . . . . . . . . . . . . . . . . . Acute conditions, persons aged 15-24 years Females, married, spouse present . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Persons of other races . . . . . . . . . . . . . . . . . . . . . . . . . . . ,091 -.257 Chronic conditions with 1 m more bed days, persons agedunder 5years . . . . . . . . . . . . . . . . . . . . . . . . . . . persons aged 5.14 years . . . . . . . . . . . . . . . Chronic conditions with 1 or more &d days, persons aged65 years And over... . . . . . . . . . . . . . . . . . . . . . Acute conditions, persons aged 65 years and over . . . . . . . Males aged 17 years And over.... Persons agedunderl Persons agedlyear year . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . And over.... Females aged 17yearsand . . . over... -.00112 .01043 .00391 -.164 .367 .118 -.404 aged 15-24 years . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chronic conditions .014 .00587 -.00009 .00596 -.015 Chronic conditions with 1 or more bed days, persons aged45-64 years . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chronic conditions .052 .00119 -.00004 .IY3123 -.037 Persons aged17yearsandover Persons agedunder17years Persons Persons .657 .342 .00004 .00015 -.00000 -.00001 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . the between-interviewer variance is highly skewed. For example, only about 5 percent of the Census enumerator pairs produced high estimates of response variance for four or more of six nonresponse items. SUMMARY In addition to the programs described in this report, three other approaches have been employed in efforts to assess the reliability and acc;ra~y of the statistics Produced by the HIS. The “first is estimates “of accuracy through record checks. Comparison of survey responses 32 .00270 -.00147 .00004 -.00015 -.024 -.038 with records have been made in a number of special studies designed to assess the accuracy of reporting chronic conditions, frequency of hospitalizations, and frequency of visits to doctors.1 3‘18 The samples have been limited to persons whose names appear on designated groups of records, e.g., patrons of the Health Insurance Plan of New York City and patients at designated hospitals in Detroit. The check starts from the records and goes back to a set of interviews. A second approach is comparison of the statistics of th; ‘HIS with statis~ics from other of the internal consissources and examination tency and reasonableness of the HIS statistics. This is a continuing activity. The re;~ts, however, have not been published. experimental Third, studies have been designed, in effect, to measure the difference in accuracy between the HIS interviews as conducted and other alternative data collection techniques. The criterion of the more, the better has been explicit in this type of study. That is, a procedure that gives higher estimates of morbidity, hospitalizations, etc., is regarded as having produced more accurate statistics than the one with which it is being compared. Thus the differences between the estimates obtained by the HIS procedure and the estimates obtained by alternative procedures that give higher estimates are regarded as lower bound estimates of the biases of the HIS procedure. None of the methods, singly or in combination, has as yet produced an estimate of the total mean-square error of any HIS statistic. This is a task of formidable proportions that prob- ably has not been accomplished for any statistical program. Such an estimate requires not only the assessment of the accuracy of reporting by the respondent but also the assessment of the effect on statistics of such factors as errors of coverage, nonresponse, recording, coding, and other processing errors. The attainable goals of a program of measurement of the reliability and accuracy must, for the foreseeable future, be regarded as rather modest ones. The chief benefit to be hoped for is that of providing a basis for detecting and correcting shortcomings in the data-collection and data-processing programs. The second goal is the rather vague one of increasing the awareness by the user of the Imitations of the statistics. In this way, informed judgments, rather than estimates, of the orders of magnitudes of total mean-square errors can be made at least for some of the HIS statistics. It is the purpose of the research on measurement of error to improve the quality of these judgments. REFERENCES lNational Health Survey Act, Public Law 652, Chapter 510, 84th Congress, 2d Session, S. 3076. 2Hansen, Morns H., Hurwitz, William N., and Bershad, Max A.: Measurement errors in censuses and surveys. BulLInst. Internat.Statist. 38(2):359-74, 1961. 3Hansen, Morris H., Hurwitz, William N., and Pritzker, Leon: The estimation and interpretation of gross differences and the simple response variance. Contributions to Statistics Presented to Professor P. C. Mahakmobis on the Occasim of His 70th Birthday, pp. 111-36. Calcutta, Indm. Pergamon Press, 1965. 4U.S. Bureau of the Census: The current population survey reinterview program, some notes and discussion. Tech. Paper No. 6. Washington. U.S. Government Printing Office, 1963. 5U.S. Bureau of the Census: Evaluation and research program of the U.S. censuses of population and housing, 1960: Accuracy of data on population characteristics as measured by reinterviews. Series ER60-No. 4. Washington. U.S. Government Printing Office, 1964. 6U.S. Bureau of the Census: Evaluation and research program of the U.S. censuses of population and housing, 1960: Accuracy of data on population characteristics as measured by CPS-census match. Series ER60-No. 5. Washington. U.S. Government Printing Office, 1964. 7U.S. Bureau of the Census: Evaluation and research program of the U.S. censuses of population and housing, 1960: The employer record check. Series ER60-No. 6. Washington. U.S. Government Printing Office, 1965. ‘U.S. Bureau of tbe Census: The cursent population survey reinterview program, January 1961 through December 1966. Tech. Paper No. 19. Washington. U.S. Government Printing Office, 1968. 9B~m, B~b~: Recent res~rch in reinteMew procedures. ].Arn.Statist.A. 63(321):41-63, 1968. 10Eckler, A. Ross, and Hurwitz, Wdliam N.: Response variance and biases in censuses and sueys. Bull. Znst.Internat. Stat&t. 36(2): 12-35, 1958. 1 lHurley, R., Jabme, T., and Larson, D.: Evaluation studks of the 1959 census of agriculture. Proc.Soc.Statist. Sec. Am. Statist. A. Paper presented at 122nd Annual Meeting, 1962, pp. 91-103. 12U.S. Bureau of the Census: Evrduation and research program of the U.S. censuses of population and housing, 1960: Effects of interviewers and crew leaders. Series ER60-No. 7. Washington. U.S. Government Printing Office, 1968. 13National Center for Hesdth Statistics: Reporting of hospitalization in the Health Interview Swey. Vital and Health Statktics. PHS Pub. No. 1000-Senes 2-No. 6. Public Health Service. Washington. U.S. Government Priiting Office, July “ 1965. 14Natio~ Center for Health Statistics: Health interview responses compared with medical records. Vital and Health Statistics. PHS Pub. No. 1000-Senes 2-No. 7. Public Health Service. Washington. U.S. Government Printing Office, July 1965. 33 15Natiomd Center for Health Statistics: Interview data on chronic conditions compared with information derived from medical records. Vital and Health Statistics. PHS Pub. No. 1000-Series 2-No. 23. Public Health Service. Washington. U.S. Government Printing Office, May 1967. 16National Center for Health Statistics: Reporting health events in household interviews: Effects of reinforcement, question length, and reinterviews. Vital and Health Statistics. Series 2-No. 45. DHEW Pub. No. (HSM) 72-1028. Washington. U.S. Government Printing Office, Mar. 1972. 34 17Nationaf Center for Health Statistics: Reporting health events in household interviews: Effects of an extensive questionnaire and a diary procedure. Vital and Health Statistics. Series 2-No. 49. DHEW Pub. No. (HSM) 72-1049. Washington. U.S. Government Printing Office, Apr. 1972. 18Natio~ cent= for He~t,h Statistics: Effect of some experimental interviewing techniques on reporting in the Health Interview Survey. Vital and Health Statkt;cs. PHS Pub. No. 1000-Series 2-No. 41. Public Health Service. Washkgton. U.S. Government Printing Office, May 1971. APPEND~X FORMULA Error rate = (A+ B+ C+... FOR I COMPUTING +G)XIOO TC+T’+TH ERROR E F where G A B C D is omissions from table I of naire, is omissions from table II of naire. is omissions from table A of naire, is missed conditions from 6-12 multiplied by 4, questionquestion- TC TA TH RATE is missed conditions from table II multiplied by 4, is missed hospitalizations mukiplied by 3, is diagnostic errors (inconsistencies or other omissions) multiplied by 2, is total conditions, is total accidents, and is total hospitalizations. questionquestions The weights assigned to the types reflect the seriousness of the errors. of error 35 APPENDIX TIME AND COST MODEL GENERAL Interviewer assignments in the HIS are classified as either resident or nonresident assignNonresident assignments require the ments. interviewer’s staying away from home one or more nights; resident assignments do not. For convenience, separate models were developed for resident and nonresident assignments. MODELS FOR HIS ASSIGNMENTS II FOR HIS S2 dl d2 d3 r~ 72 73 INTERVIEWING is number of list segments; is average distance between segments; is average distance from home to a segment; is average distance traveled within a segment (including alI visits); is travel speed between segments (minutes per mile); is travel speed from home to segments; and is travel speed within segments. The time T required for a resident HIS is listing and/or interviewing, as~ignment, expressed as Values of the parameters in both models are identical. Some of these values depend on the particular assignment: n, S1 and S2 ~~Other values are estimated from accumulated data and T = ntl +(XISl +S2-h2)d1T1 are assumed to be constants in the model: Al, + 2A2d2r2 +S1d3r3 +zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA S2t2 “ d3, rl, r2, r3, tl, t2. These estimates are prepared separately for five subuniverses that are For nonresident assignments, defined by degree of urbanization and popula- T= ntl +(XISl + S1d3r3 +S2-2)d1r1 + S2t2 + 2d2r2 where T n t~ t~ i; A2 S1 36 is total time in minutes for an interview assignment; is number of interviewed households; is time per completed interview, including interview waiting, homework, time for nontelephone, and interviews; is time per segment for listing; is average number of visits per interview segment; is number of days on which travel is required; is number of interview segments; tion density. The remaining values, except for X2, are functions of the home address of a particuku interviewer and the location of the PSU’S where she works. The number of days on which travel is required A2 applies to resident assignments only and is a function of other terms in the equation. To use the preceding equations for each interview assignment becomes very cumbersome Some simplification is and time consuming. needed so that the clerical computations can be handled routinely. When the expression for A2 is substituted in the resident model and terms are collected, the equation reduces to T = (CA1)S1 + (CA2)7Z+ (C43)S2 where CA is the computation allowance, so that CA ~ = ?tIdlrl CA2 = CA3 = +d3r3 + 2d2r2d3r3 _ dl 280 Y1 d3r3 280 2d2r2t1 _ dlrl tl —+tl 280 280 2d2r2 t2 dl rl tl -—+d1r1+t2. 280 280 The nonresident model reduces to T = (CA1)S1 + (CA2)n + (CA3)S2 + CA4 where CA1 = Xldlrl CA2 = tl + d3r3 CA4 = 2d2r2 - 2d1rl . If the subuniverse parameters Al, d3, T1, 72, and r3 are known (they are actually estimated from data), the computation allowances (CA) are functions of dl and d2 only. Since an interviewer may treat a PSU assignment as resident one time and nonresident another, both sets of computation allowances are computed for each interviewer. The dl and d2 values are flexible and can be changed when circumstances warrant, such as the interviewer moves or gets a new assignment, a replacement interviewer is hired, etc. Regional offices compute the computation allowances and keep a cumulative record (appendix VIII) of assignments for each interviewer. 8 37 APPENDIX SOME THEORY OF MEASUREMENT SOME DEFINITIONS The term “survey “ is used to refer either to surveys. In complete censuses or sample considering measurement errors we shall regard a survey as being conceptually repeatable, that is, repetitions relate to the same point in time so that carrying through the operation once does not influence results obtained through repetitions. The particular data obtained in a survey are the result of one trial. This concept provides the basis for defining variance and bias due to response, processing, or other sources of measurement errors. Such a postulate can reasonably approximate actual conditions for a single survey regarded as a sample of one from such a set of surveys, even though in practice independent repetitions of a survey may be impracticable or impossible. THE DESIRED MEASURE OR TRUE VALUE We conceive of some desired measure or goal to be estimated from a survey. For simplicity, the assumption is made that the desired or true value to be measured is represented as a proportion of the population having a specified characteristic. Although ordinarily there will be many such values to be estimated from a survey, one will be considered. Thus, it is assumed that the population consists of N persons, each of whom can be regarded as having the value of 1 if the person has one or more chronic conditions (or has some specified characteristic) or as having the value of O if the person does not have one or more chronic conditions (or does not have the specified characteristic). The desired or true cThe discussion in this appendix is based on material in references 2 and 4. 38 Ill ERRORSC proportion of persons having the characteristic is said to be estimated, even for a complete survey of the population under consideration, because only observations or responses, which are subject to errors, can be recorded. THE GENERAL CONDITIONS THAT MAY AFFECT THE RESULTS OF A SURVEY Measurement errors have many different causes and depend on the general conditions under which a survey is taken. Some of these general conditions may be beyond the :ontrol or specification of the survey designer as, for example, the general political, economic, and social situation at the time of the survey. Uncontrolled conditions also include many temporary chance situations appearing at the time a response is obtained. Some conditions can be controlled to influence the quality of survey results in the sense that various aspects of the conduct of the survey are specified. These specifications are typically made in the effort to insure adequate quality and include questionnaire design and survey procedures, personnel qualifications, pay system and rates, training, operating methods, inspection, and controls in the survey. Such conditions, which may be only partially subject to the sponsor’s control, are usually indicated in the form of fixed rules under which the survey is to be taken,, Other controllable conditions that may be varied by design, or may be regarded as varying between the conceived repetitions of the survey, are the particular choice of interviewers and other personnel chosen to do various aspects of the work, the specific assignments, and other similar variable factors. Actually, a survey sponsor is unable to specify all of the factors, controlled or uncontrolled, that may affect the survey results. He can introduce certain chance factors explicitly or implicitly, he can impose certain specifications or conditions, but he must accept the effects of other uncontrolled factors. AN ESTIMATE FROM A SURVEY (OR TRIAL) TAKEN UNDER A SET OF GENERAL CONDITIONS For simplicity, it is assumed that the survey is either a complete census or a sample in which all units have been given an equal probability of selection, but without, at this time, any other restriction on the sample design. In accordance with the definitions in the previous sections, this particular survey is regarded as one trial, i.e., one survey from among the possible repetitions of the survey under the same general conditions. An observation on a person or other unit in the survey has the value 1 if the unit is assigned to a particular class under consideration or the value O otherwise. A repetition of the survey on the same or different units would constitute a second trial. The generaI conditions, both controlled and uncontrolled, under which the sample has been taken will have an effect on the observations made in a trial. In repetitions of the survey it is assumed that the response actually observed for any individual in the survey can be regarded as having been drawn by a random process from the possible answers he might have given under the same general conditions. (In practice a survey cannot be repeated independently under the same general conditions because respondents have been exposed to the original survey and because of other reasons. However, the initial survey can be properIy regarded as a sample of one from a set of independent replications.) Thus, we are dealing with a random variable xjt, whose value is 1, if element j is classified as having some characteristic on trial t of a survey (x may denote the class “one or more chronic conditions”) or O, ifj is not so classified. The estimate obtained from a survey, i.e., a trial, is the proportion classified as having the specified characteristic in triaI t of a survey of n elements: For example, in measuring the proportion of persons with one or more chronic conditions, x = 1 in a particukr survey (or trial) if the person is classified as having one or more chronic conditions; otherwise x = O. Then Pt ‘+’Lxjt j is the estimated or more chronic proportion conditions. of persons with one THE MEAN SQUARE ERROR OF AN ESTIMATE FROM A SURVEY (OR TRIAL) Continuing the illustration, pt is the survey estimate of the proportion of the population with one or more chronic conditions and ~ is the true proportion in the population. While we generally cannot determine in practice the true value U. for any person, we can postulate that the goa i of the survey is the true value of the proportion of the population h~ving one or more chronic conditions. Thus, U, the desired true proportion, is estimated by the statistics actually obtained in the survey where (1) Forpt, the mean square error (MSE) is MSE@t) = E@t - ~z (2) where the expected value is taken over all trials. The mean square error can be divided into its two main components: MSE@t) = E@t - P)2 + (P- D)2 (3) where P is the average of the estimates P ~ taken over all trials and over all possible samples. The fiist term in equation 3 is the total variance of pt and the second term is the square of the bias of pt. In practice, we are not able to measure the bias, P - ~, but sometimes we can define and estimate useful approximations to it. For example, a superior procedure or measurement may 39 be identified as a standard. If s represents the value obtained from such a superior measurement, then (pt - s) may be used as an approximate estimate of bias. The total variance of pt can be divided into the response variance and the sampling variance: MSE(pt ) = Response variance+ + interaction Expressing terms of sampling variance + square of bias, the response variance response deviations of Pt in where djt = (Xjt - ~j) is the deviation of the response recorded for person j on trial t from the average value of the responses for person j over all trials, the response variance can be expressed as d Sij = ~ ?2[ 1 + Pdjtdkt(rt - 1)1 where Od, 2 the simple response variance, is the basic trial-to-trial variability in response averaged over all persons. The correlation term pdj~k~ reflects the effect of correlated errors introduced into the survey process by interviewers, supervisors, coders, and by persons engaged in other operations. If the intraclass correlation among response deviations is zero, the total response variance of pt is l/n o~. On the other hand if the product (n - l)p is large, the total response variance may be large even if the simple response variance is relatively small. Thus, this model of errors in surveys permits the partitioning of the MSE (pt ) into a set of components. These various components may be estimated by means of special surveys and experiments. GROSS AND NET DIFFERENCES In comparing the case-by-case results of two sets of measurements, the total number of differences affecting the tabulated figure for any given class of a population is equal to the number of cases included in that class in the first trial but not in the second trial, plus the number of those included in that class in the second trial but not in the first trial. This sum may be termed the gross difference for the population in question. 40 The net difference of the tabulated figure for the Riven class is the difference between the total-for the class obtained in the first and the second trials. Usually the gross difference will include differences in both directions, partly or substantially offsetting, and the net difference is the nonoffsetting part of the gross difference. For example, suppose that the survey identifies each person as having or not having one or more chronic conditions, and that a total of n persons have been sampled with equal probability and included in both a first and second trial. Table I shows that a of the individuals were classified as having one or more chronic conditions in both the first and second trials, a + c were classified as having one or more chronic conditions in the first trial, and a + b in the second trial. The gross difference in the classification is b + c and the net difference is (a+c)-(a+b)=c-b. Now let Xjt represent the result recorded for a particular person in the first trial and Xjt J the observation recorded for that same person” in the second trial. Furthermore, ~jt is assigned the vahe 1 if the person is recorded as having a particular characteristic, and O otherwise, and similarly for xjt t. Then the response difference for a particular person in the two surveys is represented as e. = Xjt - xjt t. The sum o~ the values of ~j over the n observations is the net difference between the two results. Table 1. General representation of results of two sets of measure- ments on identical persons Results of second trial k=F5Y=Q, Xjt, = I la Ibl a+b If the individual response differences were uncorrelated from one unit to another, the estimated standard error of the net difference rate would be and n n n _= is the net difference rate. Similarly, b + c is the gross difference and is equal to~ ej2. This follows since e2 = 1 when- ever the response obtained in the first and second trials are different, that is, (O - 1)2 = 1. The gross difference rate is ~_(b+c) .— . n GROSS AND NET DIFFERENCES AS EVIDENCE OF RESPONSE VARIANCE AND BIAS The estimated variance of response difference is .2 e = b+~ ~(ej-~)2 ‘—n-1 n-l the individual (.-b)* n(n-1) where ‘e J s: —. n In practice, the individual response differences will not be independent from one unit to another but will tend to be positively correlated. Under these circumstances, ~ gives a lower bound for SF. Given certain conditions, an overestimate of Sz can also be obtained. These conditions, would be met if, for example, a survey is repeated over time or over different areas or population groups and if the reinterview survey is conducted on different units in each of these repetitions of the survey. This is the situation for the HIS, which is taken each week. The HIS reinterview survey is taken on a distinct set of HIS households each week and summarized quarterly. From the group of quarterly repetitions an overestimate can be obtained of the standard error of the average net difference. (It will be an overestimate of the standard error of the difference obtained by repetitions of the two surveys taken for different samples but with the same personnel.) Thus, from the HIS reinterview survey, results of both net differences and net difference rates are obtained for each m quarters. If ZU is the net difference rate for quarter u, and nu is the number of persons in the reinterview sample in that quarter, then for the m quarters involved, the av>rage net difference rate can be expressed as m n n Often Z* is smalI enough that $ will be very nearly equal to g, and it is then convenient to use the gross difference rate g as the measure of the variance of the response differences. In any event, g is the mean squared difference for the original and reinterview survey results and provides a useful measure of the consistency or reliability y of the measurement process. It can be an exceedingly useful measure of reliability of response with a weI1-designed evaluation study or reintemiew survey. ~ . z%% n where n = Enu is the total number of reinterviews over the m quarters. Then will be an overstatement of Sz, the estimated standard error of the average difference rate. 41 Thus an overestimate and an underestimate of Sz can be obtained. If these are not too different, they yield a measure of the standard error of the average net difference between the original and reinterview survey results. INDEX OF INCONSISTENCY An index of the reliability of measurement; called the index of inconsistency, can be constructed using the gross difference rate. The index of inconsistency is the ratio of the simple by g/2, to the response variance, estimated maximum value it could take on, estimated by the binomiaI variance p(l - p). In terms of the table, let p ~ = (a + c)/n; that is, P is the proportion, based on the original survey, of the population in the specified class and P2 = (a + b )/n is the proportion based on the reinterview survey. Then, i!? f= Pl(wh)+fm-pd” The estimated maximum value for the gross difference rate between the survey and reinterview is pl (1 - pl) +p2(l - p2). This maximum value is obtained on the assumption that the survey and reinterview were conducted independently or that the results are positively 42 correlated to the extent that they were not A second assumption conducted independently. is that the reinterview is a repetition of the survey process and the expected value obtained Under these assumptions, in the survey. 91(1 ‘Pi) +Pz(l - Pz) is very nearly equal to 2P ( 1 -p) where p is the average proportion in the original survey and reinterview having the specified characteristic. The index of inconsistency lies between O and 1 if the assumptions given above hold. However, the estimator can be greater th~ 1. A simple interpretation of 1 follows, Assume that a sample of n elements is drawn with equal probability and with replacement. Also, assume that the between-element covariance of response deviations is zero; that is, that the quality of response of one person is independent of the quality of response for any other person. Then the total variance defined in the first term of equation 3 of the statistic pt reduces to the sum of the simple response vzuiance and the single random sampling variance. The simple response variance is equal to or less than P( 1- p). As the measurement of the specified characteristic becomes less reliable, but remains unbiased, the simple response variance increases and the sampling variance decreases; the total variance remains constant. A high index of inconsistency is associated with a high level of response error. APPENDIX IV LEAST-SQUARES SOLUTION The variance model used in the study is (1) In this model, the interviewer variance ~ is assumed to be dependent only on the number of interviewers, which remained constant over the study period. On the other hand, the sampling variance term V~/n is a function of sample size. In equation 1, n is the number of quarters of data used to compute the individual estimates of total variance. The data used are sets of estimates ~ for ~=1 , . ...8. Each V: is an average of the estimates based on n quarters. The @n are estimates of Vf(n) for n = 1, . ...8. The model given above represents the functional relationship existing between the variables HT, ~, and V; as n varies. The problem, then, is to estimate the parameters of this function so that the estimates of ~ and V’ can be identified separately. There are several ways of estimating these parameters. The ;al is to estimate @~ so that the estimates of # T(n)-2 are as close as possible to the observed values Vn. The method used was that of least squares. The criterion for the least-squares estimate is to make the sum of the squ~es of the difference between_the estimate of V; ~ ) and the observed values ~ as SmalI as possib I e, that is, to minimize the value of @, where I I I I I I 1234567 MEASURE I I 8 OF SAMPLE SIZE (NUMBER OF QUARTERS) As required by the model, a hyperbolic function. rather than a strakht line. was fitted to the ~ata~ No boundaries w~re placed on the value of ~; it can become negative as n + 00. Using the method of least squares, the partial derivatives of equation 1 are taken (3) If then These differences follows: can be seen graphically as $$=2W+~-+(’)=0 ‘5) 43 and the two partial simdified to vi Equation 527_ = 1.655 @E and fi can be (2-718)2 ~. 5 becomes (7) Equation = derivatives 8 ‘@- 0.562 ~~: (9) and 6 becomes (8) = 0.125 ~ ~n -0.340 ti; . (lo) Since By substituting and 44 in the value for ~~ , These estimates of ~ and @E for o~e quarter were used to compute the values of V; for one to eight quarters. The values shown in tables 23 and 24 are estimates for a sample covering four quarters. APPENDIX HIS FORM OBSERVATION t11$406 uS. (a .7 .4 9 1 REPORT DEPART M EN T BU REAU V OF COM M ERCE OF THE 1. CEN SU S HEALTH INTERVIEW SURVEY OBSERVATION REPORT 4. Date observed 5. ~r:: observed (NHS-HIS-406) Regional 3. Name 2. M u Office of interviewer ~Code I 6. Date of last observation To I I 7. Type of observation = m Systematic Reinterview n n rej ection 8. Reference uotesfor special notes from last observation, m Initial - First assignment Initial - Second assignment attention etc. -e.g., retiterviewresu1&, Other - .Specify= ~ Smple I ; B- Error rate Type A rate Production I B- 9. Segment coverage Item T {:CB S# nnm : and type IE xplain I Type A or callback Interviewed Tall of house 1 .Ids observed each “No” Did interviewer answer correctly below Use maps, locate segments, B Check area segment boundaries? Ca nva ss D Fill area segment listing a re a se gm e nt s Yes and look for concealed Determine Fill Cols. 8, 9, and 13 of B segment address G Fill extension L each point Yes No lN /A when required? lists? in B segment? for !he Asking probe questions lN/A units? — 10. GENERAL PERFORMANCE Evaluate NTA w sheets? E sheets Yes locate sample addresses? F “year built” No lN/A - A c TA B E;ec$ entire day’s Explain 2c%& observation below Needs im- fJnsat?spr.avemcnt fa c t oiy tfot applicable when needed and only when needed 2. Neutral probing 3. Allowine respondent 4. Maintaining 5. Listening reasonable a business-like carefully time for answeriniz auestions attitude and rapport with respondent to respondent 6. Accurately recording respondent’s answers and completing all required entries on questionnaire (Evdute this item on the basis of respondent’s answers ~fter the interview.) 7. Accepting suggestions and your edit of the questionnaires and criticisms 8. Applying housing unit definition 9. Listing 10. Planning ASK INTER I s * ha re ❑y and interviewing within special dwelling places itineray VIEWER pa rt ic ula r DURING THE DAY: pm i of t he proc .dur. y.. fe e l unsure a bout or w ould lik e t o hove c ove re d by furt he r t m ining? I Remarka 45 ast name 1 erson No. 2 3 4 5 6 7 Handled correctly: Xf’’No,” explain 8 ame ,ge ROBE AGES Person Question Yes No Yes No I ONDlION AGE(S) Person Condition Yes No Condition Person I I _J None I I I IOSPI. ‘AL ‘AGE(S) Person Page Yes No Yes No Yes No 1 2 _J None I 3 fK;:R ~ Person Yes No Person Yes No SljPpLE. MENT Person None ‘ERSON ‘AGE(S) m None ‘1 HOUSE::KJ; 2 3 (Es year bul“ft and coverage items) 4 5 6 OMMENTS (including 46 edit) Question RESU LT S Dve ra ll e va lua t ion o OF OBSERV AT I ON on a ll pha se s of w ork n Excellent Satisfactory n n Needs improvement Unsatisfactory Comments on general performance Recommendations for next observation 3BSER VER: Note any area m clarification. )bserver’s ofthe questionnaire or interviewer’s instructions which ia your view require modification Date signature , }~~ & To bezyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA COWAM {fobsw vm r mcemrrtwdk Wlwr{o#* probrMen# w Offrer aAIa ‘- , .= 2omments of the Regional Director legional Director’s signature Date 47 APPENDIX HIS RECONCILIATION B. A. RECi)NCILIATiON SECTION FOR BEO DAYS, RESTRICTED ACTIVITY DAYS, ANO TIME LOST OAYS IN PAST TWO WEEKS The original qucstionnoire showed , . . and I have . . . which is the correct information? Original .%. Bed days 5.. None ANb HOSP ITALI ZATIONS REPORTED CONDITIONS (hospitalizations) conditions D.te .1 ho.pitalizatlo,’ Condition Nor.. n None Dale ,1 ho.ptt.liz.lio. ! Ex+mti.am Explmn.tlcm o IN ON E INTERVIEW which were reported at one time but not at I Cut down dr,ym D None Work d.ys Sf. WO,k daya m N... n No.. Non. 59. School days Soh.aol d.y. RECONCILIATION m n NOW SECTION FOR DIFFERENCES Sustained D WITHIN MATCHED Deleted CONDITIONS 1. Person No. Malcb ❑ t The origins I qutsti$mnai - 2 1 Rcinterviowaondition No. !.la:ch FOR out more about Bed dmp Cut dews day. C. SECTION In finding (N HS-HIS-R-lX-T) Ccr.d it ion R.conclliatlm ❑ FORM RECONCILIATION We arc interested VI Sustained fJ Deleted . . . and 1 hove . . . . which is the correct i re showed 3 Rekmavlmv condition No, 1. Pa,ac.a No. MmLch R=hterview 4 M.tch condition No., 1. Person No. R.intmvimv aondilion No. 1. Pera.ar, No. I Original R.concl I Origin.al Ii.tlc.n Original R.em.ili.N.n 2. 2. D D Ye. No l_JYe. a 30. Yes D No OYOS i_J No Ye. d 30, 3b. Yes n D No Ye. a No m Ye. l_J No 3.. $ I I R.ceneiliotlm I 2. n ON. 2-3. 3b. Origiml R..oncili.tion 2. n N. I 3b. 2b. I 3.. 3c. %. 3d. 3d. 24 3*. 3*. 2., 3.. I Oriqinal R*co.ci 90. 0 Ye. ON. 9b. a Yes D 10. No I Days lioNon O Y.. ON. D Y,. a Original No D.y. a Ye. a I!u222” !‘y- ‘N” 9b. D Yes I_J No 25d. hlonth(.): Year(s) L, ,,0.. 48 M.anihls) i Yom(s) ! ..s..,,. ”., x.’, (..,.,>, ‘we 25b., D Ye. 25.. Cured Under c.anlrol Other - SPOCNYF El DY., n I Day. !20” ‘yes I 1. No 0 Ye. n No ❑ Yes lg No O:YC. m No 9.s. ❑ 9b. a Yea O No I_JYe* a No 9b. OYC. Day. 10. I D.T= Ye, R*conclliotion a No UN. Days a Yes m No n Y,. a No Days !oye”‘N” 120” ““B ‘N” I‘yes‘N” Im” “e’ ‘N” I““” ‘No No O Yes 8 Cmod Under control Dther - SPaCllY~ O 25b. No I 25d, Mcmlh(d ; Ycnr(.4 90. I 10. n.Y. Oriolnol ec.a. cillatim Otlginal ~No 3*, I R.cc.ncl No Non 90. I 10. 3d. a Ye. N C.rcd Under control Other - s.uoclfy~ 2s.. a I ❑ Yes ON. 25b, 25.. B Cured Cnder con,rol O,ha - s#ec1fy7 I I Month(s) : Yew(.) NO 25d. Month(s); Yc.r(s) O Yes la Cured Under c.ntr.l Dther - S@fy7 a No a Y.S El Cared Undmr 00.uo1 Other - SL.eclfyj? ON. I Mmlh(s) ; Y..,(,) 25d, Mo,:h(s) : Yem(s) Month(e) I Ye.r(8) - BUT NOT IN BOTH INTERVIEWS mother. Con you think of any ●xplanation for . . . not hovinq been reported }n the (ari~inol D.!. C.ndiliom .1 b.spit.li..ticm n interview today)? Dmte .1 ho.pit.Nz.LIem S,xpl.n.tion E.xp18s.tI.. > interview/our Camditi.. m Su.tslmed n Deleted Condltic.m Due of ho.piuNz.ti.m Explu.tien s“a.ined m . i_J %stained Deleted u Deleted nfonnatlon? 6 s ?lelnterview ccmditlon No. M.tch Orlgln.1 1. P.m.. Mmtch Rcintavlew ONO a ❑ Yes No 2“0 1. Person No. Mntch Fleintetvlew ca.dith R.c..cili.tion ~No Y.* 8 7 condition No. Odgi”d Rmcortclllmtio. 2. UY.S No. O Yes Original ON. ‘U Ye* N.. Malcb No. o Yes 3.. 30. 35 3b. 2b. 3b. 3b. 3c. 3.. 3.. 3C. 3d. I 3d. 13d, Orldnd 3d, a Ye. DNo ~YG. ON. 9b. D Y.. mNo ‘a D D.p I Ye. No a Ye. ~ No a Yes m No 9.. 0 0 Ye. u N. m Yes a NO 9b. mYe. I Month(.) j Ycu(el 1 n.p 10. 1 I I 25d. Month(s); Year(.) 9.. Dsys —— 25d. Month(.); Yc.is) I M.mh(.) i Year(s) NO. No ~ L Person N.. R.cottci[ia*ic.n n ❑ Yes No i. I ‘“” Odqi”d Reconciliafie. 9b. 10. Day. I I 3“” Oric.1..l R.conclllmtlon 9.. 10. , I 3*’ 3.. c.mditi.. Oflgln.1 ‘ore. ON. 30. t Fleknewi.w Re..nciIiaNon I UN. I. Pc-o. Ye. R...ncil o No D.p D No 9.. n a N. ?b. i_JYu 10. Dv. u I I I I 25d. Momth(,) ; Year(.) Yes ‘I_J Yes a ON. I Original lotion Yes R..onciliotjon a D.,. I I Month(.) i Y..r(.) 8 No i_JNo u Y.. a No a Ye. a No D.y. 1 I ! &!. M,mb(s) ; Ye,r(d Month(.) , Y-d.; i-i u, . ...!... 49 Vll APPENDIX SUMMARY FORM t ’4 ii5 -H 1 5 -i? -4 0 1 [s-5 .6 6 ) OF NHS-HIS REPORT U .S. DEPART M EN T OF SUREAU COM M ERC OF THE CENSUS 1 . I nt e rvie w e r’s 3. Reintervi SUMMARY REPORT OF NHS-HIS REINTERVIEW Se c t ion Reinterview Number Type (1) (2) rotal this reinterview Previous :umulative total 4ew :umulative f Exclude dwelling I - RE NTERVIEW ‘m e ewer’s 5. Reinterview name I Code I I I In Part segment A - Area Number Number of sample units before rein terview Listed before rein terview f (3) Segments date Part B Segment of units B Check Deleted Correct (4) (5) (6) (7) Incorrect I I4 Office Job t it le Program Supervisor Psu i . Altemaie Senior l.terviewer Od-ter 6. Sample Se c t ion II HOUSEHOLD COMPOSITION CHECK Number Number of wrong households interviewed (8) . of persons Segment No. Household serial No. Before reinter“iew Added Del eted (1) (2) (3) (4) (5) (9) I .,, total dwelling places in NTA segments and in ‘“large” ,, special of differences Explain in sections name I and II section and segment, numbers of added or deleted persons. changes in classification of added or deleted units, and to segment Describe type of error If column (8) is of “year built” here OISO.) I I Totol I I I I this Previous 50 2 , Regional P art c Extension sheet entries Added No. I COVERAGE CHECK OF HOUSING AND OTHER UNITS units in special places. reference to serial numbers :hecked. I Telephone 1 ,, ‘Give md / Code 1 m n m If the most recent listing for an NTA segment w a s performed by another person, enter in ““Explanation of Differences” and prepare separate form NHS-HI%R-40 I reporting columns (4), (5), and (6) data for the segment. explanation (N HS-HIS-R-401) reinterview cumulative I total I --u u a e c non Pa rt A Se gm e nt N o. Dlff e re n t re sponde nt H ouse hold s=ria l N o. N u$ fbe r (2 ) (3 ) Cha ra c t e rist ic s Sa m e re sponde nt Dlffe ren ces pe rsons (1 ) Pe rsona l (4 ) N uo~be r pe rsons (5 ) Diffe r- Ill - LUN IEN I LtltLK I Pa rt B - Cha ra c t e rist ic s Wit hin Condit ions a nd H ospit a liza t ions Diffe re nt Sa m e re sponde nt re sponde nt I N um be r of * Q* Di f f=r. =“==* (7 ) (8 ) N um be r o .:=;: *f :;f& - ‘“==s (6 ) (9 ) (lo) Pa rt C - N um be r of Condit ions rnd H ospit a liza t ions Diffe re nt Sa m e re sponde nt re spondm t 1 Ac c e pt Ac c e pt Diffe rDiffe ra bilit y a bilit y e nc e s 3 e nc e s 3 ba se 2 ba se 2 (11) ( 12) (1 3 ) (1 4 ) T ot a l t his int e rvie w ROV iOus c um ulot iva * t a l N e w c um ula t ive t ot a l 1 I nc lude s a dult s re sponding / for c hildre n / unde r 19. Plus sust a ine d c ondit ions zm d hospit a liza t ions a (@ c he c k c ondit ions ?m d hospit a liza t ions plus num be r of c ondit ions a nd hospit a liza t ions a dde d a nd sust a ine d t o re c onc ilia t ion form . 3 N um be r of origina l c ondit ions a nd hospit a liza t ions de le t e d from re c onc ilia t ion form plus num be r of c ondit ions a nd hospit a liza t ions a dde d a “d sust a ine d t o re m nc ilia t ion form . t I (I/) * De finit ions c he c k s: Ea c h de c k re pre se nt s one c ondit ion or hospit a liza t ion re port e d on bot h int e rvie w s. I Sust a ine d c o”dit io”s or hospit a liza t ion a re ~o=e w hic h a re re port e d only c m o“e int e rvie w ~d re t a ine d a ft e r re c o”c i I ia t io”. I Section IV - ACTION TAKEN N ot e : T his se c t ion m ust be fille d if t he int e rvie w e r ha s be e n re ie c t e d in a “y of t he pre c e ding =e c t ions. Ex pla in a c t ion t a k e n \or pla nne d t o re t ra in int e rvie w e r if he r w ork ha s be e ” re je c t e d (R). (If more space is re quire d, use oddit io”c d she e +.) id V PERFORMANCE e c t ion De c ision section A R - Accept Re je c t 2 a I-A l-c II II-A II-B Ii-c] UScor.w.oc I 51 VI I APPENDIX PRODUCTION p#ll. GUIDE U.S. DEPARTMENT 102C BU REAU FOR OF COMMERCE OF THW INTERVIEW SEGMENTS wEE K (.) Psu (b) NUMBER (d HOUSEHOLDS INTERVIEWED c: (d) 1. REG1ONAL (11-102C) 2. SAMPLE OFFICE c EN SU S B4. INTERVIEWER 3. INTERVIEWER PRODUCTION GUIOE FOR NHS Lye: NHS LIST SUPPLEMENT SEGMENTS NUMBER y NUMBER =: (e) (0 (s) (h) CA 4 (i) NVMBER o) THIS CUMULATIVE ASSIGNMENT ALLOWED MINUTES ALLOWED MINUTES cd (1) CODE PAYROLL MINUTES AM;::::: PAYfiOL1. MINUTES (0) (n) (m) PRODUG TlON RAT I O (P) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 I I 15 REMARKS I ‘7 I CHIEF, 52 FIELD DIVIS!OH COPY :U SCOM M .OC 21760 P-68 INSTRUCTIONS FOR COMPLETING FORM 11-102C GENERAL Maintain a Form 11-102C for each interviewer on a three month (NHS Sample) basis. At the end of the three month period send the yellow copy to Chief, Field Division, retain the original in your files and start a new 11-102C for each interviewer. One line of the 11-102C should be completed view and/or listing assi~ment. for each inter- Columns (a)-(p) Enter the week in column(a) Post the computation (f), (h), and (i), using and the PSUin column (b). allowances Production to columns (d), Standards Memo- randum No. 5 (Formerly knowledge (from payroll assignment was overnight GAM No. 70) and your records) of whether the or non-overnight. Enter the workload associated columns (c), (e), and (g). with each alIowance in Enter the number of current supplements (if any) compieted in co Iumn (j), and the allowance per unit, as given in Operations Memorandums, in column (k). Multiply each allowance by its workIoad and add the results to obtain the totaI allowance. (Be sure to add in CA 4.) Enter the payroll Enter this 1 1 -1 OZ C in column (1). minutes in column (m). Revise the cumulative (n), (o), and (p). FORM total production ratio in columns U SC (1 0 -7 -6 3 ) OM hl-DC 21759 P.6 s 53 * u. s. OCXWNMENT p~G OFFICE :1973 5 E,.2 1 2 1 A.8 f VITAL AND HEALTH Originally Public STATISTICS Health Service PUBLICATION Publication No. SERIES 1000 and collection procedures. — Reports which describe the general programs of the National Center for Health Statistics and its offices and divisions. data collection methods used, definitions, and other material necessary for under standing the data. Series 1. Programs Series 2. Data evaluation and methods research. —Studies of new statistical methodology including: experimental tests of new survey methods, studies of vital statistics collection methods, new analytical techniques, objective evaluations of reliability of collected data, contributions to statistical theory. Series studies .—Reports presenting carrying the analysis further 3. Artalvtical statistics, analytical or interpretive studies based on vital and health than the expository types of reports in the other series. Series 4. Documents and committee reports. — Final reports of major committees concerned with vital and health statistics, and documents such as recommended model vitai registration laws and revised birth and death certificates. Se7ies 10. Data jkom. the Health Interview .%rrvev. —Statistics on illness, accidental injuries, disability, use of hospital, medical, dental, and other services, and other health-related topics, based on data collected in a continuing national household interview survey. Series 11, Data Jrom the Health Exmnination Survey. —Data from direct examination, testing, and measurement of national samples of the civilian, noninstitutional population provide the basis for two types of reports: (1) estimates of ,the, medically defined prevalence of specific diseases in the United States and the distributions of the population with respect to physical, physiologi~l, and psychological characteristics; and (2) analysis of relationships among the various measurements without reference to an explicit finite universe of persons. of Series 12. D@Z from the Institutional Population Surveys —Statistics relating to the health characteristics persons in institutions, and their medical, nursing, and personal care received, based on national samples of establishments providing these services and samples of the residents or patients. relating to disci:;rged patients Series 13. Data from the Hospital Discharge Survey. —Statistics hospitals, based on a sample of patient records in a national sample ~f hospitals. in short-stay Series 14. Data on health resources: manpower and facilities. —Statistics on the numbers, geographic distribution, and characteristics of health resources including physicians, dentists, nurses, other health occupations, hospitals, nursing homes, and outpatient facilities. on mortality other than as included in regular annual or Series 20. Data on mortality .—Various statistics montruy reports —special analyses by cause of death, age, and other demographic variables, also geographic and time series analyses. on natality, marriage, and divorce S6nies 21. Data on rtatality, mawiage, and divorce. —Various statistics other than as included in regular annual or monthly reports-special analyses by demographic variables, also geographic S~”es 22. Data from the National and deaths not available records, including such last year of life, medical * Fbr a list of titles of reports and time series analyses, studies of fertility. Natdity and Mcrrtality Surveys. — Statistics on characteristics of births from the vital records, based on sample surveys stemrrdng from these topics as mortality by soci~onomic class, hospital experience in the care during pregnancy, health insurance coverage, etc. published in these series, write to: Off ice of Information National Center for Health Stat istics Public Health Service, HSMHA Rockville, Md. 20852 /