Rigor and Flexibility in Computer-based Qualitative Research - International Journal of Multiple Research Approaches

home > archives > 2/1 > Rigor and Flexibility in Computer-based

Rigor and Flexibility in Computer-based Qualitative Research: Introducing the Coding Analysis Toolkit

Chi-Jung Lu
Department of Library Information Sciences, School of Information Sciences, University of Pittsburgh, Pittsburgh PA, United States of America

Stuart W Shulman
Director, Qualitative Data Analysis Program, University Center for Social and Urban Research, University of Pittsburgh, Pittsburgh PA, United States of America

PP: 105 - 117

Abstract

Software to support qualitative research is both revered and reviled. Over the last few decades, users and skeptics have made competing claims about the utility, usability and ultimate impact of commercial-off-the-shelf (COTS) software packages.

This paper provides an overview of the debate and introduces a new web-based Coding Analysis Toolkit (CAT). It argues that knowledgeable, well-designed research using qualitative software is a pathway to increasing rigor and flexibility in research.

| More

Keywords

qualitative research, data analysis software, content analysis, multiple coders, annotation, adjudication, inter-rater reliability, tools

Article Text

Effective qualitative data analysis plays a critical role in research across a wide range of disciplines. From biomedical, public health, and educational research, to the behavioral, computational, and social sciences, text data analysis requires a set of roughly similar mechanical operations. Many of the mundane tasks, traditionally annotations performed by hand, are tedious and time-consuming, especially with large datasets. Researchers across disciplines with qualitative data analysis problems increasingly turn to commercial-off-the-shelf (COTS) software for solutions. As a result, there is a growing literature on computer-assisted qualitative data analysis software (CAQDAS), with Fielding and Lee's study (1998) serving as an enduring landmark in the canon.

These various COTS packages (eg NVivo, ATLAS.ti, or MAXQDA, to name just a few) evolved from pioneering software development during the 1980s, with the first programs available for widespread public use early 1990s (Bassett 2004). Software tools provide a measure of convenience and efficiency, increasing the overall level of organization of a qualitative project. Researchers enhance their ability to sort, sift, search, and think through the identifiable patterns as well as idiosyncrasies in large datasets (Richards & Richards 1987; Tallerico 1992; Tesch 1989). Similarities, differences and relationships between passages can be identified, coded and effectively retrieved (Kelle 1997a; Wolfe, Gephart & Johnson 1993). The research routine is substantially altered. In the best cases, more intellectual energy is directed towards the analysis rather than the mechanical tasks of the research process (Conrad & Reinharz 1984; St John & Johnson 2000; Tesch 1989).

The first part of this article summarizes the claims associated with using software for qualitative data analysis. Next, we acknowledge the oft-mentioned concerns about the role of software in the qualitative research process. We then introduce a new web-based suite of tools, the Coding Analysis Toolkit (CAT), designed to make mastery of core mechanical functions (eg managing and coding data) as well as advanced projects task (eg measuring inter-rater reliability and adjudication) easier. Finally, we assert that CAQDAS tools are useful for increasing rigor and flexibility in qualitative research. Assuming a user (or team) running a project has a basic level of technical fluency, the requisite research design skills, and domain expertise, coding software opens up important possibilities for implementing and transparently reporting large-scale, systematic studies of qualitative data.

The second author is the sole inventor of the Coding Analysis Toolkit and has a financial interest in it should it ever be commercialized.

View references

References

Agar M (1991) The right brain strikes back. In NG Fielding and RM Lee (eds) Using computers in qualitative research, pp.181-193. London: Sage.

Barry CA (1998) Choosing qualitative data analysis software: Atlas.ti and NUD*IST compared. Sociological Research Online 3. Retrieved January 30 2008 from http://www.socresonline.org.uk/socresonline/3/3/4.html.

Bassett R (2004) Qualitative data analysis software: Addressing the debates. Journal of Management Systems XVI(4): 33-39.

Berends L and Johnston J (2005) Using multiple coders to enhance qualitative analysis: The case of interviews with consumers of drug treatment. Addiction Research & Theory 13(4): 373-382.

Bong SA (2002) Debunking myths in qualitative data analysis. Forum of Qualitative Social Research 3(2). Retrieved January 30 2008 from http://www.qualitative-research.net/fqs-texte/2-02/2-02bong-e.htm.

Bourdon S (2002) The integration of qualitative data analysis software in research strategies: Resistances and possibilities. Forum Qualitative Sozialforschung 3(2). Retrieved from www.qualitative-research.net/fqs-texte/2-02/2-02bourdon-e.htm.

Brown D (2002) Going digital and staying qualitative: Some alternative strategies for digitizing the qualitative research process. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research [On-line Journal] 3(2). Retrieved January 30 2008 from http://www.qualitative-research.net/fqs-texte/2-02/2-02brown-e.htm.

Coffey A, Holbrook B and Atkinson P (1996) Qualitative data analysis: technologies and representations. Sociological Research Online 1(1). Retrieved January 30 2008 from http://www.soc.surrey.ac.uk/socresonline/1/1/4.html .

Conrad P and Reinharz S (1984) Computers and qualitative data: Editor's introductory essay. Qualitative Sociology 7(1/2): 3-13.

Dolan A and Ayland C (2001) Analysis on trial. International Journal of Market Research 43(4): 377-389.

Drass KA (1980) The Analysis of Qualitative Data: A Computer Program. Urban Life 9(3): 332-353.

Fielding NG and Lee RM (1998) Computer analysis and qualitative research: New technology for social research. London: Sage.

Ford K, Oberski I and Higgins S (2000) Computer-aided qualitative analysis of interview data: Some recommendations for collaborative working. The Qualitative Report 4(3/4).

Gibbs GR, Friese S and Mangabeira WC (2002) The use of new technology in qualitative research. Introduction to Issue 3(2) of FQS. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research 3(2). Retrieved January 30 2008 from http://www.qualitative-research.net/fqs-texte/2-02/2-02hrfsg-e.htm.

Hinchliffe SJ, Crang MA, Reimer SM and Hudson AC (1997) Software for qualitative research: 2. Some thoughts on ‘aiding' analysis. Environment and Planning A, 29: 1109-1124.

Kelle U and Laurie H (1995) Computer use in qualitative research and issues of validity. In U Kelle (ed) Computer-aided qualitative data analysis: Theory, methods and practice, pp.19-28. London/Thousand Oaks/New Delhi: Sage.

Kelle U (1997a) Theory building in qualitative research and computer programs for the management of textual data. Sociological Research Online 2(2).

Kelle U (2004) Computer-assisted Qualitative Data Analysis. In C Seale, G Gobo, JF Gubrium and D Silverman (eds) Qualitative Research Practice, pp.473-489. London: Sage.

King G, Keohane R and Verba S (1994) Designing Social Inquiry: Scientific Inference in Qualitative Research, Princeton, NJ: Princeton University Press.

Lee RM and Fielding NG (1991) Computing for qualitative research: Options, problems and potential. In NG Fielding and RM Lee (eds) Using computers in qualitative research, pp.1-13. London: Sage.

Lewins A and Silver C (2007) Using software in qualitative research: A step-by-step guide. London: Sage.

Lombard M, Snyder-Duch J and Bracken CC (2005) Practical resources for assessing and reporting intercoder reliability in content analysis research projects. Retrieved March 17 2008 from http://www.temple.edu/sct/mmc/reliability/.

Lonkila M (1995) Grounded theory as an emerging paradigm for computer-assisted qualitative data analysis. In U Kelle (ed) Computer-Aided Qualitative Data Analysis. Theory, Methods and Practice, pp.41-51. London: Sage.

MacMillan K and Koenig T (2004) The wow factor: Preconceptions and expectations for data analysis software in qualitative research. Social Science Computer Review 22(2): 179-186.

MacQueen KM, McLellan E, Kay K and Milstein B (1998). Codebook development for team-based qualitative analysis. Cultural Anthropology Methods 10(2): 31-36.

Manderson L, Kelaher M and Woelz-Stirling N (2001) Developing qualitative databases for multiple users. Qualitative Health Research 11(2): 149-160.

Mangabeira W (1995) Computer assistance, qualitative analysis and model building. In RM Lee (ed) Information technology for the social scientist, pp.129-146. London: UCL Press.

Northey Jr WF (1997) Using QSR NUD*IST to demonstrate confirmability in qualitative research. Family Science Review 10(2): 170-179.

Pfaffenberger B (1988) Microcomputer Applications in Qualitative Research. Newbury Park, CA: Sage.

Ragin CC and Becker HS (1989) How the microcomputer is changing our analytical habits. In G Blank, JL McCartney and E Brent (eds) New technology in sociology, pp.47-55. New Brunswick NJ: Transaction Publishers.

Richards L and Richards T (1991a) The transformation of qualitative method: Computational paradigms and research processes. In NG Fielding and RM. Lee (eds) Using Computers in Qualitative Research, pp.38-53. London: Sage.

Richards L and Richards TJ (1987) Qualitative data analysis: Can computers do it? Australia and New Zealand Journal of Sociology 23: 23-35.

Roberts KA and Wilson RW (2002) ICT and the research process: Issues around the compatibility of technology with qualitative data analysis. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research 3(2). Retrieved January 30 2008 from http://www.qualitative-research.net/fqs-texte/2-02/2-02robertswilson-e.htm.

Ryan G (1999) Measuring the typicality of text: Using multiple coders for more than just reliability and validity checks. Human Organization 58(3): 313-322.

Sandelowski M (1995) On the aesthetics of qualitative research. Image: Journal of Nursing Scholarship 27: 205-209.

Seidel J (1991) Method and madness in the application of computer technology to qualitative data analysis. In NG Fielding and RM Lee (eds) Using Computers in Qualitative Research, pp.107-116. London: Sage.

Sin Chih Hoong (2007a) Teamwork involving qualitative data analysis software: Striking a balance between research ideals and pragmatics. Social Science Computer Review.

St John W and Johnson P (2000) The pros and cons of data analysis software for qualitative research. Journal of Nursing Scholarship 32(4): 393-397.

Tallerico M (1992) Computer Technology for Qualitative Research: Hope and Humbug. Journal of Educational Administration 30(2): 32-40.

Tesch R (1989) Computer software and qualitative analysis: A reassessment. In G Blank, JL McCartney and E Brent (eds) New technology in sociology, pp.141-154. New Brunswick NJ: Transaction Publishers.

Tesch R (1991a) Introduction. Qualitative Sociology 14(3): 225-243.

Thompson R (2002) Reporting the results of computer-assisted analysis of qualitative research data. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research 3(2). Retrieved January 30 2008 from http://www.qualitative-research.net/fqs-texte/2-02/2-02thompson-e.htm.

Webb C (1999) Analyzing qualitative data: Computerized and other approaches. Journal of Advanced Nursing 29(2): 323-330.

Weitzman EA (1999) Analyzing qualitative data with computer software. Health Services Research: Part 2 34(5): 1241-1263.

Welsh E (2002) Dealing with data: Using NVivo in the qualitative data analysis process. Forum: Qualitative Social Research 3(2). Retrieved January 30 2008 from http://www.qualitative-research.net/fqs-texte/2-02/2-02welsh-e.htm.

Wolfe RA, Gephart RP and Johnson TE (1993) Computer-facilitated qualitative data analysis: Potential contributions to management research. Journal of Management 19(3): 637-660.

Navigate: Archives | Contents | Top of page