Introductionto Meta-Analysis

01 02 03 04 05 06 07 Introduction to Meta-Analysis 08 09 10 Michael Borenstein Biostat, Inc, New Jersey, USA. 11 12 13 14 Larry V. Hedges Northwestern University, Evanston, USA. 15 16 17 Julian P.T. Higgins 18 MRC, Cambridge, UK. 19 20 21 Hannah R. Rothstein Baruch College, New York, USA. 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 A John Wiley and Sons, Ltd., Publication 24th January 2009 07:50 Wiley/ITMA Page iii ffirs 01 02 03 04 05 06 07 08 This edition first published 2009 Ó 2009 John Wiley & Sons, Ltd Registered office John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ, United Kingdom For details of our global editorial offices, for customer services and for information about how to apply for permission to reuse the copyright material in this book please see our website at www.wiley.com. The right of the author to be identified as the author of this work has been asserted in accordance with the Copyright, Designs and Patents Act 1988. 09 10 11 12 13 All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, except as permitted by the UK Copyright, Designs and Patents Act 1988, without the prior permission of the publisher. Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books. 14 15 16 17 18 19 Designations used by companies to distinguish their products are often claimed as trademarks. All brand names and product names used in this book are trade names, service marks, trademarks or registered trademarks of their respective owners. The publisher is not associated with any product or vendor mentioned in this book. This publication is designed to provide accurate and authoritative information in regard to the subject matter covered. It is sold on the understanding that the publisher is not engaged in rendering professional services. If professional advice or other expert assistance is required, the services of a competent professional should be sought. 20 Library of Congress Cataloguing-in-Publication Data 21 22 23 24 25 26 27 Introduction to meta-analysis / Michael Borenstein . . . [et al.]. p. ; cm. Includes bibliographical references and index. ISBN 978-0-470-05724-7 (cloth) 1. Meta-analysis. I. Borenstein, Michael. [DNLM: 1. Meta-Analysis as Topic. WA 950 I614 2009]. R853.M48I58 2009 610.72—dc22 28 2008043732 29 30 31 32 33 A catalogue record for this book is available from the British Library. ISBN: 978-0-470-05724-7 Set in 10.5/13pt Times by Integra Software Services Pvt. Ltd, Pondicherry, India Printed in the UK by TJ International, Padstow, Cornwall 34 35 36 37 38 39 40 41 42 43 24th January 2009 07:50 Wiley/ITMA Page iv ffirs 01 Contents 02 03 04 05 06 07 08 09 10 11 12 13 List of Tables List of Figures Acknowledgements Preface Web site xiii xv xix xxi xxix 14 15 PART 1: INTRODUCTION 16 17 1 HOW A META-ANALYSIS WORKS Introduction Individual studies The summary effect Heterogeneity of effect sizes Summary points 2 WHY PERFORM A META-ANALYSIS Introduction The streptokinase meta-analysis Statistical significance Clinical importance of the effect Consistency of effects Summary points 18 19 20 21 22 23 24 25 26 27 28 29 3 3 3 5 6 7 9 9 10 11 12 12 14 30 31 32 33 PART 2: EFFECT SIZE AND PRECISION 3 OVERVIEW Treatment effects and effect sizes Parameters and estimates Outline of effect size computations 17 17 18 19 4 EFFECT SIZES BASED ON MEANS Introduction Raw (unstandardized) mean difference D Standardized mean difference, d and g Response ratios Summary points 21 21 21 25 30 32 34 35 36 37 38 39 40 41 42 43 11th February 2009 06:11 Wiley/ITMA Page v ftoc vi 01 Contents 5 EFFECT SIZES BASED ON BINARY DATA (2  2 TABLES) Introduction Risk ratio Odds ratio Risk difference Choosing an effect size index Summary points 33 33 34 36 37 38 39 6 EFFECT SIZES BASED ON CORRELATIONS Introduction Computing r Other approaches Summary points 41 41 41 43 43 7 CONVERTING AMONG EFFECT SIZES Introduction Converting from the log odds ratio to d Converting from d to the log odds ratio Converting from r to d Converting from d to r Summary points 45 45 47 47 48 48 49 8 FACTORS THAT AFFECT PRECISION Introduction Factors that affect precision Sample size Study design Summary points 51 51 52 52 53 55 9 CONCLUDING REMARKS 57 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 PART 3: FIXED-EFFECT VERSUS RANDOM-EFFECTS MODELS 34 35 36 37 10 OVERVIEW Introduction Nomenclature 61 61 62 11 FIXED-EFFECT MODEL Introduction The true effect size Impact of sampling error 63 63 63 63 38 39 40 41 42 43 11th February 2009 06:11 Wiley/ITMA Page vi ftoc Contents 01 02 vii Performing a fixed-effect meta-analysis Summary points 65 67 03 04 05 06 07 08 09 12 RANDOM-EFFECTS MODEL Introduction The true effect sizes Impact of sampling error Performing a random-effects meta-analysis Summary points 69 69 69 70 72 74 13 FIXED-EFFECT VERSUS RANDOM-EFFECTS MODELS Introduction Definition of a summary effect Estimating the summary effect Extreme effect size in a large study or a small study Confidence interval The null hypothesis Which model should we use? Model should not be based on the test for heterogeneity Concluding remarks Summary points 77 77 77 78 79 80 83 83 84 85 85 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 14 WORKED EXAMPLES (PART 1) Introduction Worked example for continuous data (Part 1) Worked example for binary data (Part 1) Worked example for correlational data (Part 1) Summary points 87 87 87 92 97 102 29 30 31 PART 4: HETEROGENEITY 32 33 34 35 36 15 OVERVIEW Introduction Nomenclature Worked examples 105 105 106 106 16 IDENTIFYING AND QUANTIFYING HETEROGENEITY Introduction Isolating the variation in true effects Computing Q Estimating  2 The I 2 statistic 107 107 107 109 114 117 37 38 39 40 41 42 43 11th February 2009 06:11 Wiley/ITMA Page vii ftoc viii 01 02 03 04 Contents Comparing the measures of heterogeneity Confidence intervals for  2 Confidence intervals (or uncertainty intervals) for I 2 Summary points 119 122 124 125 05 06 07 08 09 10 11 12 17 PREDICTION INTERVALS Introduction Prediction intervals in primary studies Prediction intervals in meta-analysis Confidence intervals and prediction intervals Comparing the confidence interval with the prediction interval Summary points 127 127 127 129 131 132 133 18 WORKED EXAMPLES (PART 2) Introduction Worked example for continuous data (Part 2) Worked example for binary data (Part 2) Worked example for correlational data (Part 2) Summary points 135 135 135 139 143 147 19 SUBGROUP ANALYSES Introduction Fixed-effect model within subgroups Computational models Random effects with separate estimates of  2 Random effects with pooled estimate of  2 The proportion of variance explained Mixed-effects model Obtaining an overall effect in the presence of subgroups Summary points 149 149 151 161 164 171 179 183 184 186 20 META-REGRESSION Introduction Fixed-effect model Fixed or random effects for unexplained heterogeneity Random-effects model Summary points 187 187 188 193 196 203 21 NOTES ON SUBGROUP ANALYSES AND META-REGRESSION Introduction Computational model Multiple comparisons Software Analyses of subgroups and regression analyses are observational 205 205 205 208 209 209 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 11th February 2009 06:11 Wiley/ITMA Page viii ftoc Contents 01 02 ix Statistical power for subgroup analyses and meta-regression Summary points 210 211 03 04 PART 5: COMPLEX DATA STRUCTURES 05 06 22 OVERVIEW 215 23 INDEPENDENT SUBGROUPS WITHIN A STUDY Introduction Combining across subgroups Comparing subgroups Summary points 217 217 218 222 223 24 MULTIPLE OUTCOMES OR TIME-POINTS WITHIN A STUDY Introduction Combining across outcomes or time-points Comparing outcomes or time-points within a study Summary points 225 225 226 233 238 25 MULTIPLE COMPARISONS WITHIN A STUDY Introduction Combining across multiple comparisons within a study Differences between treatments Summary points 239 239 239 240 241 26 NOTES ON COMPLEX DATA STRUCTURES Introduction Summary effect Differences in effect 243 243 243 244 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 PART 6: OTHER ISSUES 27 OVERVIEW 249 28 VOTE COUNTING – A NEW NAME FOR AN OLD PROBLEM Introduction Why vote counting is wrong Vote counting is a pervasive problem Summary points 251 251 252 253 255 29 POWER ANALYSIS FOR META-ANALYSIS Introduction A conceptual approach In context When to use power analysis 257 257 257 261 262 33 34 35 36 37 38 39 40 41 42 43 11th February 2009 06:11 Wiley/ITMA Page ix ftoc x 01 02 03 04 05 Contents Planning for precision rather than for power Power analysis in primary studies Power analysis for meta-analysis Power analysis for a test of homogeneity Summary points 263 263 267 272 275 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 30 PUBLICATION BIAS Introduction The problem of missing studies Methods for addressing bias Illustrative example The model Getting a sense of the data Is there evidence of any bias? Is the entire effect an artifact of bias? How much of an impact might the bias have? Summary of the findings for the illustrative example Some important caveats Small-study effects Concluding remarks Summary points 277 277 278 280 281 281 281 283 284 286 289 290 291 291 291 22 23 PART 7: ISSUES RELATED TO EFFECT SIZE 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 31 OVERVIEW 295 32 EFFECT SIZES RATHER THAN p -VALUES Introduction Relationship between p-values and effect sizes The distinction is important The p-value is often misinterpreted Narrative reviews vs. meta-analyses Summary points 297 297 297 299 300 301 302 33 SIMPSON’S PARADOX Introduction Circumcision and risk of HIV infection An example of the paradox Summary points 303 303 303 305 308 34 GENERALITY OF THE BASIC INVERSE-VARIANCE METHOD Introduction Other effect sizes Other methods for estimating effect sizes Individual participant data meta-analyses 311 311 312 315 316 39 40 41 42 43 11th February 2009 06:11 Wiley/ITMA Page x ftoc Contents 01 02 xi Bayesian approaches Summary points 318 319 03 04 05 PART 8: FURTHER METHODS 35 OVERVIEW 323 36 META-ANALYSIS METHODS BASED ON DIRECTION AND p -VALUES Introduction Vote counting The sign test Combining p-values Summary points 325 325 325 325 326 330 37 FURTHER METHODS FOR DICHOTOMOUS DATA Introduction Mantel-Haenszel method One-step (Peto) formula for odds ratio Summary points 331 331 331 336 339 38 PSYCHOMETRIC META-ANALYSIS Introduction The attenuating effects of artifacts Meta-analysis methods Example of psychometric meta-analysis Comparison of artifact correction with meta-regression Sources of information about artifact values How heterogeneity is assessed Reporting in psychometric meta-analysis Concluding remarks Summary points 341 341 342 344 346 348 349 349 350 351 351 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 PART 9: META-ANALYSIS IN CONTEXT 39 OVERVIEW 355 40 WHEN DOES IT MAKE SENSE TO PERFORM A META-ANALYSIS? Introduction Are the studies similar enough to combine? Can I combine studies with different designs? How many studies are enough to carry out a meta-analysis? Summary points 357 357 358 359 363 364 41 REPORTING THE RESULTS OF A META-ANALYSIS Introduction The computational model 365 365 366 34 35 36 37 38 39 40 41 42 43 11th February 2009 06:11 Wiley/ITMA Page xi ftoc xii 01 02 03 Contents Forest plots Sensitivity analysis Summary points 366 368 369 04 05 06 07 08 42 CUMULATIVE META-ANALYSIS Introduction Why perform a cumulative meta-analysis? Summary points 371 371 373 376 43 CRITICISMS OF META-ANALYSIS Introduction One number cannot summarize a research field The file drawer problem invalidates meta-analysis Mixing apples and oranges Garbage in, garbage out Important studies are ignored Meta-analysis can disagree with randomized trials Meta-analyses are performed poorly Is a narrative review better? Concluding remarks Summary points 377 377 378 378 379 380 381 381 384 385 386 386 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 PART 10: RESOURCES AND SOFTWARE 24 25 26 27 28 29 30 31 32 44 SOFTWARE Introduction The software Three examples of meta-analysis software Comprehensive Meta-Analysis (CMA) 2.0 RevMan 5.0 Stata macros with Stata 10.0 Summary points 391 391 392 393 395 398 400 403 45 BOOKS, WEB SITES AND PROFESSIONAL ORGANIZATIONS Books on systematic review methods Books on meta-analysis Web sites 405 405 405 406 REFERENCES 409 INDEX 415 33 34 35 36 37 38 39 40 41 42 43 11th February 2009 06:11 Wiley/ITMA Page xii ftoc