The Delphi Technique: Methodological Considerations and the Need for Reporting Guidelines in Medical Journals
Background. The Delphi technique has been increasingly used in healthcare research over the past decade. Study aim. To present the development workflow and methodological issues encountered while designing a Delphi study targeting family care practitioners in northeast Romania. Method. Extraction from existing literature and analysis of several reported factors which may decrease the internal validity of a Delphi study and therefore, the overall study reliability: expert selection criteria and process, expert retention, dropout management, types of questionnaires, questionnaire delivery, data analysis in qualitative and qualitative rounds and construct of Likert questionnaires. Result. Based on the existing literature data and taking into account the study objectives we propose a design allowing a decrease of the systematic bias and an improvement of the study findings: purposive stratified sampling, financial incentives, dropouts’ inclusion in the analysis, dual administration of questionnaires, open-ended questionnaire for round 1, Likert forms with 7 ascending, fully labeled, horizontal response options including a midpoint. Conclusion. Careful consideration should be given to all variables that may influence the internal validity or induce a consensus bias in healthcare studies using the Delphi technique. A more detailed and standardized reporting is needed for Delphi studies published in biomedical journals and we propose a checklist to be used by readers and editors.
Delphi Technique, Health Care, Delphi Reporting, Expert Panel, Likert Questionnaire, Qualitative Research
[1]
Pana M. România după 25 de ani (I) / Evoluția reală a PIB: Creșterea economică și deceniul pierdut, 2014. (http://cursdeguvernare.ro/romania-dupa-25-de-ani-i-evolutia-reala-a-pib-cresterea-economica-si-deceniul-pierdut.html).
[2]
Pana M. Sănătatea 2014 și banii ei: Bugetul scade, cheltuielile cresc. Subvențiile urcă la 0,7% din PIB, 2014. (http://cursdeguvernare.ro/sanatatea-2014-si-banii-ei-bugetul-scade-cheltuielile-cresc-subventiile-urca-la-07-din-pib.html).
[3]
Health at a Glance: Europe 2012: OECD Publishing, 2012. (Health at a Glance: Europe). ISBN: 9789264183605.
[4]
Eurostat. GDP per capita in PPS: Eurostat, 2014. (http://ec.europa.eu/eurostat/tgm/table.do?tab=table&init=1&language=en&pcode=tec00114&plugin=1).
[5]
Osan A., Lungu D. Astărăstoae: Deficitul de medici din România a ajuns în jur de 40-42 la sută. Bucuresti: Mediafax, 2014. Available from: http://www.mediafax.ro/social/astarastoae-deficitul-de-medici-din-romania-a-ajuns-in-jur-de-40-42-la-suta-13215107.
[6]
Mullan F. The metrics of the physician brain drain. N. Engl. J. Med. 2005; 353: 1810–8.
[7]
Institutul National de Statistica. Populaţia după domiciliu la 1 ianuarie 2015 a scăzut cu 0,3% faţă de 1 ianuarie 2014. Institutul National de Statistica. Bucuresti, 2015. Available from: http://www.insse.ro/cms/files/statistici/comunicate/com_anuale/populatie/PopDom2015r.pdf.
[8]
Delbecq AL, Van de Ven, Andrew H, Gustafson DH. Group techniques for program planning: A guide to nominal group and Delphi processes: Scott, Foresman Glenview, IL, 1975. ISBN: 0673075915.
[9]
Norman Dalkey. An experimental application of the Delphi method to the use of experts. Management Science 1963; 9: 458–67.
[10]
Linstone HA, ed. The Delphi Method. Techniques and Applications. 2nd ed., 2002. 1 vol.
[11]
Fink A, Kosecoff J, Chassin M, Brook RH. Consensus methods: characteristics and guidelines for use. Am J Public Health 1984; 74: 979–83.
[12]
Keeney S, Hasson F, McKenna H. The Delphi Technique in Nursing and Health Research. 1st ed.: Wiley-Blackwell, 2011. 1 vol. ISBN: 978-1-4051-8754-1.
[13]
Okoli C, Pawlowski S. The Delphi method as a research tool: an example, design considerations and applications. Information and Management 2004; 42: 15–29.
[14]
Baker J, Lovell K, Harris N. How expert are the experts? An exploration of the concept of 'expert' within Delphi panel techniques. Nurse Res 2006; 14: 59–70.
[15]
Bolger F, Wright G. Improving the Delphi process, Lessons from social psychological research. Technological Forecasting and Social Change 2011; 78: 1500–13.
[16]
Akins RB, Tolson H, Cole BR. Stability of response characteristics of a Delphi panel: application of bootstrap data expansion. BMC Med Res Methodol 2005; 5: 37.
[17]
Chia-Chien Hsu & Brian A. Sandford. Minimizing Non-Response in The Delphi Process: How to Respond to Non-Response. Practical Assessment, Research & Evaluation 2007; (12)17.
[18]
VanGeest JB, Johnson TP, Welch VL. Methodologies for improving response rates in surveys of physicians: a systematic review. Eval Health Prof 2007; 30: 303–21.
[19]
Edwards PJ, Roberts I, Clarke MJ, et al. Methods to increase response to postal and electronic questionnaires. Cochrane Database Syst Rev 2009; (3): MR000008.
[20]
Flanigan TS, McFarlane E, Cook S, eds. Conducting survey research among physicians and other medical professionals: A review of current literature. In: Proceedings of the Survey Research Methods Section, American Statistical Association. 2008. p. 4136-47.
[21]
Thorpe C, Ryan B, McLean SL, et al. How to obtain excellent response rates when surveying physicians. Fam Pract 2009; 26: 65–8.
[22]
Martins Y, Lederman RI, Lowenstein CL, et al. Increasing response rates from physicians in oncology research: a structured literature review and data from a recent physician survey. British Journal of Cancer 2012; 106: 1021–6.
[23]
Frewer LJ, Fischer A, Wentholt M, et al. The use of Delphi methodology in agrifood policy development, Some lessons learned. Technological Forecasting and Social Change 2011; 78: 1514–25.
[24]
Schmidt RC. Managing Delphi surveys using nonparametric statistical techniques. Decision Sciences 1997; 28: 763–74.
[25]
Franklin KK, Hart JK. Idea Generation and Exploration, Benefits and Limitations of the Policy Delphi Research Method. Innov High Educ 2006; 31: 237–46.
[26]
Bowling A. Mode of questionnaire administration can have serious effects on data quality. J Public Health (Oxf) 2005; 27: 281–91.
[27]
Braun V, Clarke V. Using thematic analysis in psychology. Qualitative research in psychology 2006; 3: 77–101.
[28]
Pope C, Ziebland S, Mays N. Qualitative research in health care: analysing qualitative data. BMJ: British Medical Journal 2000; 320: 114.
[29]
Jamieson S. Likert scales: how to (ab) use them. Med Educ 2004; 38: 1217–8.
[30]
Carifio J, Perla RJ. Ten Common Misunderstandings, Misconceptions, Persistent Myths and Urban Legends about Likert Scales and Likert Response Formats and their Antidotes. J. of Social Sciences 2007; 3: 106–16.
[31]
Kampen J, Swyngedouw M. The ordinal controversy revisited. Quality and quantity 2000; 34: 87–102.
[32]
Norman G. Likert scales, levels of measurement and the "laws" of statistics. Adv Health Sci Educ Theory Pract 2010; 15: 625–32.
[33]
Sullivan GM, Artino AR. Analyzing and interpreting data from likert-type scales. J Grad Med Educ 2013; 5: 541–2.
[34]
Stevens S. S. On the Theory of Scales of Measurement. Science 1946; 103: 677–80.
[35]
Kuzon WM, Urbanchek MG, McCabe S. The seven deadly sins of statistical analysis. Ann Plast Surg 1996; 37: 265–72.
[36]
van Vaerenbergh Y, Thomas TD. Response Styles in Survey Research, A Literature Review of Antecedents, Consequences, and Remedies. International Journal of Public Opinion Research 2013; 25: 195–217.
[37]
Cicchetti DV, Shoinralter D, Tyrer PJ. The effect of number of rating scale categories on levels of interrater reliability: A Monte Carlo investigation. Applied Psychological Measurement 1985; 9: 31–6.
[38]
Carolyn C. Preston & Andrew M. Colman. Optimal number of response categories in rating scales: reliability, validity, discriminating power, and respondent preferences. Acta Psychologica 2000; 104: 1-15.
[39]
Miller GA. The magical number seven plus or minus two: some limits on our capacity for processing information. Psychol Rev 1956; 63: 81–97.
[40]
Matell MS, Jacoby J. Is There an Optimal Number of Alternatives for Likert Scale Items?, Study I: Reliability and Validity. Educational and Psychological Measurement 1971; 31: 657–74.
[41]
Weng L-J. Impact of the Number of Response Categories and Anchor Labels on Coefficient Alpha and Test-Retest Reliability. Educational and Psychological Measurement 2004; 64: 956–72.
[42]
Lozano LM, García-Cueto E, Muñiz J. Effect of the Number of Response Categories on the Reliability and Validity of Rating Scales. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences 2008; 4: 73–9.
[43]
Finstad K. Response Interpolation and Scale Sensitivity: Evidence Against 5-Point Scales. Journal of Usability Studies 2010; 5: 104–10.
[44]
Weijters B, Cabooter E, Schillewaert N. The effect of rating scale format on response styles, The number of response categories and response category labels. International Journal of Research in Marketing 2010; 27: 236–47.
[45]
Matell MS, Jacoby J. Is there an optimal number of alternatives for Likert-scale items?, Effects of testing time and scale properties. Journal of Applied Psychology 1972; 56: 506–9.
[46]
Presser S, Schuman H. The measurement of a middle position in attitude surveys. Public Opinion Quarterly 1980; 44: 70–85.
[47]
Garland R. The Mid-Point on a Rating Scale: Is it Desirable? Marketing Bulletin 1991; (2): 66–70.
[48]
Clark LA, Watson D. Constructing validity: Basic issues in objective scale development. Psychological assessment 1995; 7: 309.
[49]
Krosnick JA, Holbrook AL, Berent MK, et al. The impact of" no opinion" response options on data quality: Non-attitude reduction or an invitation to satisfice? Public Opinion Quarterly 2002; 66: 371–403.
[50]
Nowlis SM, Kahn BE, Dhar R. Coping with ambivalence: The effect of removing a neutral option on consumer attitude and preference judgments. Journal of Consumer Research 2002; 29: 319–34.
[51]
Kulas JT, Stachowski AA. Middle category endorsement in odd-numbered Likert response scales, Associated item characteristics, cognitive demands, and preferred meanings. Journal of Research in Personality 2009; 43: 489–93.
[52]
Tourangeau R, Rasinski KA. Cognitive processes underlying context effects in attitude measurement. Psychological bulletin 1988; 103: 299.
[53]
Kulas JT, Stachowski AA. Respondent rationale for neither agreeing nor disagreeing, Person and item contributors to middle category endorsement intent on Likert personality indicators. Journal of Research in Personality 2013; 47: 254–62.
[54]
Friedman HH, Leefer JR. Label versus position in rating scales. Journal of the Academy of Marketing Science 1981; 9: 88–92.
[55]
Chan JC. Response-order effect in Likert-type scales. ERIC Clearinghouse, 1990.
[56]
Friedman HH, Herskovitz PJ, Pollack S, eds. The biasing effects of scale-checking styles on response to a Likert scale. In: Proceedings of the American statistical association annual conference: survey research methods, 1994(vol. 792).
[57]
Albanese M, Prucha C, Barnet JH, Gjerde CL. The effect of right or left placement of the positive response on Likert-type scales used by medical students for rating instruction. Acad Med 1997; 72: 627–30.
[58]
Albanese M, Prucha C, Barnet JH. Labeling each response option and the direction of the positive options impacts student course ratings. Academic Medicine 1997; 72: S4-S6.
[59]
Barnette JJ. Likert Response Alternative Direction: SA to SD or SD to SA: Does It Make a Difference? Annual Meeting of the American Educational Research Association (Montreal, Quebec, Canada, April 19-23) 1999.
[60]
Barnette JJ. Effects of stem and Likert response option reversals on survey internal consistency: If you feel the need, there is a better alternative to using those negatively worded stems. Educational and Psychological Measurement 2000; 60: 361–70.
[61]
Weng L-J, Cheng C-P. Effects of Response Order on Likert-Type Scales. Educational and Psychological Measurement 2000; 60: 908–24.
[62]
Nicholls MER, Orr CA, Okubo M, Loftus A. Satisfaction Guaranteed The Effect of Spatial Biases on Responses to Likert Scales. Psychological Science 2006; 17: 1027–8.
[63]
Hofmans J, Theuns P, Baekelandt S, Mairesse O SN, and Cools W. Bias and Changes in Perceived Intensity of Verbal Qualifiers Effected by Scale Orientation. Survey Research Methods 2007; 1: 97–108.
[64]
Yan T, Keusch F. The Effects of the Direction of Rating Scales on Survey Responses in a Telephone Survey. Public Opinion Quarterly 2015; 79: 145–65.
[65]
Maeda H. Response option configuration of online administered Likert scales. International Journal of Social Research Methodology 2013; 18: 15–26.
[66]
Christian LM, Dillman DA. The influence of graphical and symbolic language manipulations on responses to self-administered questions. Public Opinion Quarterly 2004; 68: 57–80.
[67]
Dillman DA. Survey Mode as a Source of Instability in Responses across Surveys. Field Methods 2005; 17: 30–52.
[68]
Topoel V, Das J. W. M., van Soest A. Design of Web Questionnaires: The Effect of Layout in Rating Scales. Journal of Official Statistics 2009; 25: 509–28.
[69]
Morse JM, Barrett M, Mayan M, Olson K, Spiers J. Verification strategies for establishing reliability and validity in qualitative research. International journal of qualitative methods 2002; 1: 13–22.
[70]
Day J, Bobeva M. A generic toolkit for the successful management of Delphi studies. EJBRM 2005; 3: 103–16.
[71]
Rowe G, Wright G. The Delphi technique: Past, present, and future prospects — Introduction to the special issue. Technological Forecasting and Social Change 2011; 78: 1487–90.
[72]
Diamond IR, Grant RC, Feldman BM, et al. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. J Clin Epidemiol 2014; 67: 401–9.
[73]
Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PLoS ONE 2011; 6: e20476.
[74]
Sinha IP, Smyth RL, Williamson PR. Using the Delphi technique to determine which outcomes to measure in clinical trials: recommendations for the future based on a systematic review of existing studies. PLoS Med. 2011; 8: e1000393.
[75]
Hasson F, Keeney S. Enhancing rigour in the Delphi technique research. Technological Forecasting and Social Change 2011; 78: 1695–704.
