Weighing Up Triangulating and Contradictory Evidence in Mixed Methods Organisational Research

P Lynne Johnstone

PP: 27

Abstract

This paper explores the role of the context-familiar researcher in the interpretation of research data, specifically in terms of applying a transparent process to weighing up triangulating evidence in mixed methods research.

It is erroneous to assume that all research data will converge on an undisputable ‘truth' or ‘fact', but few writers on the concept of data triangulation offer advice on how researchers might handle conflicting evidence in their research projects. Furthermore, little appears to be written about whether or not some evidence in a study employing multiple sources can assume greater importance as evidence than other data. In other words, are all data equal?

In this paper, I critically reflect on how I applied trustworthiness principles that are implicitly reflexive to resolve these issues in a research project undertaken in the professional health services context in which I have extensive prior experience.

Keywords

mixed methods research, triangulation, contradictory evidence, reflexivity, transparency, operating department

Article Text

Statement of problem/issue and discussion of its significance

The qualitative researcher as a participant, to a greater or lesser extent, in the study phenomena, is faced with numerous challenges. One of these challenges derives from the sheer volume and diversity of data that are collected – a situation that is compounded when mixed and/or multiple methods are employed. A related challenge concerns the methods the researcher employs to extract ‘the truth’ (ie a verifiable explanation of the phenomena) from these data, free from the preconceptions that he/she had or has about the research phenomena, and how he/she reflects such objectivity when weighing up potentially contradictory evidence in the quest for ‘the truth’.

The problem is that few writers on the concept of data triangulation offer advice on how researchers might handle conflicting evidence in their research projects. So, my first question is: ‘how might the researcher decide which evidence to accept and which evidence to regard as insignificant?’ Furthermore, little appears to be written about whether or not some evidence in a study employing multiple sources and types of data can assume greater importance as evidence than other data. So, my second question is: ‘are all data equal?’ If not, how might the researcher adjudge the relative importance of diverse evidence and/or contradictory evidence?

These were some of the questions that I was confronted with when I was engaged in research in which I was exploring the impact of technological change in surgery on operating department work. The basic tenet of this paper is that the processes I employed in resolving these issues in my research are congruent with reflexivity principles, particularly in terms of trustworthiness evidenced by transparency in both my position and perspectives on the research problem and the processes of analysis.

Mays and Pope (2000) advance a number of procedures and principles that contribute to the trustworthiness of qualitative research. Among these are: triangulation, respondent validation, clear detailing of methods of data collection and analysis, attention to negative cases, and fair dealing. In this paper, I focus on four: triangulation, clear detailing of methods of data collection and analysis, reflexivity, and attention to negative cases because I regard them as the most relevant of the factors to my discussion here about weighing up contradictory evidence in mixed methods research.

Theoretical principles

This section discusses the theoretical principles associated with four of the procedures and principles that Mays and Pope (2000) and others, such as Morrow (2005), Tobin and Begley (2004), and Lietz, Langer and Furman (2007), contend contribute to the trustworthiness of qualitative research: triangulation, clear detailing of methods of data collection and analysis, reflexivity, and attention to negative cases.

Triangulation

Most researchers experienced in employing qualitative methods will be familiar with the data analysis technique called triangulation. Denzin, in the late 1970s, apparently borrowed the term from navigation where it referred to the point of convergence of various navigation points. He proposed that by using a combination of methods in a study of a phenomenon, any bias in a particular data source, investigator, or method, would be neutralised when multiple sources of evidence are employed (Creswell 1994, p.174; Yin 1994, p.93). Consequently, researchers can have greater confidence in the conclusions they draw.

According to Tobin and Begley (2004), triangulation was originally seen as a means of combining rationalistic and naturalistic paradigms, but it subsequently gained acceptance in naturalistic only studies. A critical issue for the researcher is determining why triangulation is the preferred data analysis technique. Sandelowski (1995), for example, posited that triangulation should only be used when data from one source is used to corroborate data from another, and when such convergent and consensual validity is valued.

Yin (1994) has described triangulation in relation to case studies involving multiple sources and/or types of evidence converging on a ‘fact’. When one reads about the concept it sounds simple enough – but it is not. I propose that many writers, such as Yin (1994; 1999), Erlandson, Harris, Skipper and Allen (1993), Creswell (1994; 2003) and Begley (1996), whose work I have studied over the past decade, omit to convey that all the evidence rarely, if ever, converges on ‘the fact’. Regarding non-convergent data in terms of completeness or expansion, whereby explanations about the phenomena of interest are enriched, has been one response to this conundrum (Creswell 1994; Farmer, Robinson, Elliot & Eyles 2006; Curtin & Fossey 2007). Beyond this, however, no useful discussion or explanation has been forthcoming about how researchers might handle conflicting evidence in their research projects, or whether one type or source of data might assume greater importance in the conclusions drawn than other types or sources of data.

It could be argued that the logic of triangulation in mixed methods research is essentially the same as that employed in legal proceedings in which a defendant is judged guilty or not guilty on the basis of multiple types of evidence from multiple sources converging on a finding that is, on the balance of all probabilities, ‘beyond reasonable doubt’. This is the notion that a judgement is made, or a conclusion drawn, after all relevant and available evidence from all sources is considered by the researcher(s). I believe that this concept extends our thinking about triangulation, particularly in terms of its practical application, and helps us resolve the questions about contradictory evidence and the equality of data in studies involving multiple types and/or sources of data.

Clear detailing of methods of data collection and analysis

The principle of clear detailing of methods of data collection and analysis is embraced in two of the four trustworthiness characteristics advanced by Erlandson et al (1993): dependability and confirmability.

The dependability of an inquiry relates to its capacity to be ‘replicated with the same or similar subjects in the same (or similar) context’ and produce the same findings (Lincoln & Guba 1985, p.290). In naturalistic research, this criterion is generally regarded to equate with both the reliability and trackability of the research process, meaning that the methods of investigation are appropriate to the task and the researcher makes it possible for an ‘external check to be conducted on the processes by which the study was conducted’ (Erlandson et al, 1993, p.34). Crawford, Jenkins and Murray-Prior (1999) argued that this is all the more important in a single researcher study.

In positivist research, confirmability is inextricably tied to the ontological assumption that reality is objective, and to the axiological assumption that the research outcomes are free from investigator bias (Lincoln & Guba 1985; Creswell 1994; Stake 1995). However, in naturalistic research, the researcher needs to be able to demonstrate the confirmability of the data themselves (Lincoln & Guba 1985). This is achieved using the techniques that contribute to dependability – in particular, the trackability of data – whereby the researcher is transparent about all aspects of the research process so that others not involved in the study could audit the data from its sources to analyses and the conclusions drawn.

Moreover, it is generally recommended that in case study research (Yin 1994; Stake 1995), as is the present research, the research report should be accompanied by ‘a companion volume that includes items … such as original interview notes, actual survey responses, member-checking forms, peer debriefing notes, reflexive journals, photographs, audio tapes …’ (Erlandson et al 1993, p.166; Creswell 1994; Rolfe 2006) – what Yin (1994) refers to as a case study database. Its principal purpose is that it serves as a journal or ‘chain of evidence’ for the external observer to adjudge the reliability of the information and the conclusions contained in the case study report. For example, it would provide details of the circumstances under which evidence was collected, and make it possible to follow the derivation of the evidence from the initial research questions via explicit links to the data collected, and the conclusions drawn (Yin 1994). Erlandson et al (1993) cite Lincoln and Guba (1985) who regard the reflexive journal component of this ‘companion volume’ as ‘a kind of a diary in which the investigator on a regular basis records information about him- or herself. The journal provides information about the researcher’s ‘schedule and logistics, insights [and] the reasons for methodological decisions’ (Erlandson et al 1993, p.143). Each argues that it supports the credibility, transferability, dependability and confirmability of the study, and is an important part of the study’s audit trail. This journaling technique was employed in the present research, resulting in a 300-page word processed companion volume.

Reflexivity

Objectivity, as deified in rational positivism, is not the Holy Grail in knowing; and subjectivity is not an alternative, second-rate way of knowing. Rather, according to Hufford (1995), ‘all knowing [and knowledge] is subjective, and the ‘objective world’ is what knowers claim to know about’ (pp.57-58; Cunliffe 2003, p.984). This, Hufford says, is the central point of reflexivity, which he explains as:

‘a metaphor from grammar indicating a relationship of identity between subject and object, thus meaning the inclusion of the actor (scholar, author, observer) in the account of the act and/or its outcomes. . . [B]ecause scholars are human beings, the study of human life is always and inescapably reflexive. . . the scholar is always the subject of scholarship in the grammatical sense, that is, the doer of the action (DeVinne 1991), and those we study are the objects of our scholarship. . . [Importantly] reflexivity in knowledge-making involves bringing the subject, the ‘doer’ of the knowledge-making activity, back into the account of the knowledge’ (Hufford 1995, pp.57-58).

There are many ways whereby reflexivity, thus defined, can be demonstrated by the ‘doer’ of the knowledge-making. One such technique is transparency in all the processes of the research (Curtin & Fossey 2007), as outlined in the previous section. A second is a commitment to being open to competing conclusions – an overtly acknowledged pluralistic perspective that leads to self questioning (Winter 2002) – a principle that is discussed in the following section in relation to attention to negative cases. A third involves the researcher acknowledging, from the outset, his or her preconceptions about the research phenomena arising from personal background and theoretical paradigms (Malterud 2001).

Everett (2002, p.71) reports the views of the renown French anthropologist and sociologist, Pierre Bourdieu, who similarly argued that the call for reflexivity ‘is a call to acknowledge the way in which the researcher’s knowledge about the world influences research claims’. This knowledge comes from previous personal and professional experiences, education and interests, which influence, for example, the researcher’s motivation to conduct the research, his/her pre-study beliefs about the research phenomena, and his/her views about how it should be investigated (Malterud 2001, pp.483-484). Throughout the process, the researcher needs to be cognisant of three sources of bias: social, field, and intellectualist (Bourdieu & Wacquant 1992, p.39), biases that derive from the researcher’s sociological characteristics (eg ethnicity, gender, education, or social status). According to Malterud (2001), however, biases are not the same as preconceptions unless the researcher fails to mention them; and identifying those preconceptions is the start of reflexive practice in research.

Attention to negative cases

Another dimension of reflexivity involves the researcher looking at the research data, or its interpretation, for competing conclusions (Malterud 2001). I propose that the notion of competing conclusions, possibly as a result of the extent to which the researcher gives attention to ‘negative cases’, is implicit in the two questions I raised at the start of this paper: ‘how do we deal with contradictory evidence?’ and ‘are all data equal?’

However, as previously stated, little appears to be written, beyond broad principles, to guide researchers in their responses to these questions. These broad principles are those of transparency and trackability (Erlandson et al 1993) discussed earlier in connection with Malterud’s trustworthiness principle of clear detailing of methods of data collection and analysis.

In this connection, therefore, the researcher needs to be transparent in the way he/she intuitively weighs up all the evidence and how he/she determines strategies to deal with some evidence being more influential than other evidence in a decision or ‘proof of fact’. I propose that these activities are at the heart of reflexive research practice.

Background to my research

In 1996 I started to think about how I might investigate the impact of technological change in surgery on work in operating departments. My hunch was that the technological changes since the late 1980s had made surgical production more labour intensive. I limited my definition of technological change to changes in the hardware of surgery, that is, intraoperative artefacts that I categorised in terms of surgical instruments, biomedical machines, and surgical materials. The term, surgical production, referred to the bundle of activities undertaken before, during and after procedures within an operating department that relate to the use, management, and maintenance of those intraoperative artefacts and the associated perioperative activities (Johnstone 2001). My interest in this phenomenon was dominantly managerial – specifically in terms of how the technological changes had changed the nature and volume of work within the operating department and what had been the consequences of those changes for individual operating department staff.

My main research proposition was as follows:

The characteristics of new intra-operative artefacts, the reasons for their adoption in surgical production in hospitals, the decision processes associated with their adoption, and their consequences for receiver stakeholders, cannot be explained using the set of theories and managerial perspectives, which I refer to collectively as ‘the techno-economic theories of production’, that are typically operationalised in new technology adoption scenarios in organisations by strategies that emphasise return-on-investment.

This proposition was derived from five research questions:

1. What are the dominant technical characteristics and functional goals of new intra-operative artefacts adopted between 1988 and 1998?
2. What are the benefits expected by key internal stakeholders of adopting new intra-operative artefacts?
3. What are the actual consequences for surgical production within operating theatre services of new intra-operative artefact adoption?
4. Are the consequences for surgical production within operating theatre services of new intra-operative artefact adoption congruent with expectations and, if not, why not?
5. By what processes are decisions made to adopt new intra-operative artefacts, and how are the benefits expected by key internal stakeholders of adopting them influential in these decision processes?
   
   

Research methods

I decided upon a collective case study design involving five diverse hospitals and various types and sources of quantitative and qualitative data:

• Interviews with four categories of informants: operating theatre nurses, sterilizing department technical aides, procedural specialists, and senior health services managers (n=67)
• Questionnaire (one informant group)
• Observation of Operating Department work
• Clinical literature
• Time study of selected work processes associated with four common surgical procedures of different specializations that had undergone technological change early in the ten year period of interest
• Analysis of diverse Operating Department and hospital records
• National and NSW surgical activity data

The qualitative data were analysed inductively and deductively during different iterations of the research process within an overarching triangulation framework (Johnstone 2001, p.27; see Figure 1).

In keeping with the principle of transparency of research methods, Figure 1 is explained. It distinguishes data sourced from within operating theatre services (OTSs) (shown in Boxes 4 to 16 contained within the ‘circle’) from data sourced from outside OTSs (see Boxes 1 to 3), and data sourced from either (see Box 17). It reveals how various qualitative and quantitative data constitute the ‘multiple sources of evidence’ in the study. All six of Yin’s (1994, p.78; 1999, p.1218) sources of evidence are represented here.

Documents and archived records from which quantitative data were collected included OTS staffing rosters (see Box 10) and the OTS Surgical Registers (see Boxes 9 and 11) for the three month periods at the beginning, middle, and end of the ten-year study timeframe. Staffing rosters were used to derive ‘full time equivalent’ (FTE) staffing levels (see Box 10). The data collected from the surgical registers containing records of all surgical procedures were used to initially derive, for each three-month period at each of the five hospitals:

(i) the total number of all procedures undertaken

(ii) the total time taken to perform all procedures (ie total operating minutes)

(iii) the frequency of each of the six selected procedures; and

(iv) the total operating minutes for each of the six selected procedures (see Boxes 9 and 11).

The data in (i) and (ii) were subsequently analysed in conjunction with FTE staffing data to trend changes in staff workloads over time (see Box 12). External to OTSs, diverse data pertaining to surgical activity in acute public and private hospitals, and other related data, were accessed, in the main, from the NSW and Commonwealth Departments of Health (see Boxes 3 and 17).

Interviews were conducted with three categories of OTS personnel. The interviews with operating theatre nurses (see Boxes 13 and 16) and sterilising department technical aides (see Box 14) were unstructured, whilst the interviews with procedural specialists (predominantly surgeons) were semi-structured (see Box 15). Furthermore, anyone working within OTSs had the potential to be an informant via informal conversations (see Box 5). External to the OTS, semi-structured interviews were conducted with senior health service managers (see Box 1), who also completed a small survey questionnaire containing eight questions requiring Likert-scaled responses during the course of their interviews. The average interview duration was one hour, and all formal interviews were audio tape-recorded for subsequent text transcription.

Direct observations were made of all OTS personnel in the course of their work (see Box 6) with particular attention to the three OTS informant groups. Also, for each of the six procedures, I conducted a time study of all of the pre- and post-procedure human labour input to their production (see Boxes 4 and 8).

Physical artefacts in the form of surgical instruments, machines/equipment, and related surgical materials were central aspects of the research. Consequently, direct observations were made of the technologies employed in both the six procedures selected for detailed study and in surgical production generally.

Figure 1 also shows how the empirical and descriptive literature constituted data (see Box 2), as did the tacit knowledge that I, as someone with an OTS work background, brought to the study (see Box 7). The latter endogenous nature of my research, combined with the many continuous periods of five days in which I lived amongst the OTS staff, represent the participant-observation elements of the research.

Finally, evidence is provided in the model, by way of ‘other leads?’ boxes, of the flexibility that existed in the research process (Yin 1999, p.1216; Commonwealth of Australia 1995, p.21). Central to this flexibility was the overarching iterative nature of data collection and data analysis that occurred in various ways until the final report was written.

In this way, triangulation was not a single event, but it involved a sequence of iterations of data collection and analysis, during which time some data were regarded as insubstantial, often because data saturation had occurred in another direction. This is an example of the reflexive dimensions of the research, whereby ‘the ‘human instrument’ allows data to be collected and analysed in an interactive process [and] as soon as data are obtained, tentative meaning is applied to them, [and] when new data are obtained, meaning is revised’ (Erlandson et al 2003, p.39). Figure 2 (following) is a conceptual representation of this process as it was applied to my research (Johnstone 2001, p.20). It is the focus of a paper published in 2004 in Qualitative Health Research (Johnstone 2004, p.266).

Data analysis

This section does not present a comprehensive treatment of data analysis as would be expected if my intention in this paper was to present my research findings. Rather, it is limited to a description of my approach to analysing some of the interview texts, and pays particular attention to how I resolved my previously articulated questions about contradictory data and the (in)equality of data.

I had elected to use the RESEARCHWARETM software, HyperRESEARCH to undertake my thematic analysis of the text. I developed a sequenced set of rules to test each of the following four propositions:

1. that most changes in the nature of work are a consequence of the adoption of new surgical instruments and machines
2. that technological change in surgery had not increased employee productivity
3. that new surgical technologies did not have an automating function; and
4. that new surgical technologies did not have a deskilling function

Extracts from one proposition, proposition 3, are presented as Figure 3. It represents a sequence of rules whereby ‘goal reached’ – that is, support for the proposition – is achieved only if a specific informant’s interview data satisfy each of the rules, each of which has its own conclusion stipulated by the phrase prefaced by the term, ‘ADD’.

My problem lay in my thorough thematic analysis of the interviews whereby I coded both evidence of phenomena as well as contradictory evidence by the inclusion of the phrases ‘AND NOT’ and ‘OR NOT’ in the rules. For example, in Proposition 3 Rule 3, an informant could have made many statements indicating that he/she had on numerous occasions attributed the increased volume of work to an increase in the manual component, the proposition would be confirmed. However, consistent with my belief that all evidence, including contradictory evidence, must be ‘weighed up’, I incorporated ‘not’ statements in my proposition testing rules (eg AND NOT VOW_MANU_DEC), with the result that it took only a single occurrence of contradictory evidence to bring about an unsupported proposition.

After much soul searching, my solution to this problem was to ditch the proposition testing function, and to use the HyperRESEARCH software only to count the occurrences of all coded text – both evidence of the phenomena of interest as well as contradictory evidence – for various report purposes. I then tabulated the frequency of all evidence to depict my process of weighing up the evidence. For example, ten statements by an informant providing evidence of the phenomenon of interest (coded NOW_TECH CHANGE _INC), along with one statement, such as ‘oh, no, that new machine didn’t change how I did my work’ (coded NOW_TECH CHANGE _STAT), would not change the overall message of that informant, or overall support for the proposition. On the other hand, if the contradictory statement had been, for example, ‘overall, the technological change in surgery over the ten years has made little difference to how I do my work’ – something that did not happen – the effect on the process of weighing up the evidence would likely have been the reverse.

The following table (Table 1) provides an example of the process I have just described. It concerns the theme ‘changes in volume of manual work’. The informants are sterilising department technicians, representing seven of the thirty-nine nursing or technical staff interviewed in operating departments. Twenty-six chunks of text were coded on this theme: one indicated a decrease in manual work; two indicated no change; whereas twenty-three indicated an increase in the volume of manual work. The conclusion, on balance, is that the volume of manual work has increased, a conclusion that I believe is strengthened by the fact that all seven provided a minimum of two statements in support of this phenomenon. Other themes, such as the impact of technological change on the complexity of work, multi-skilling, and specialisation, were analysed along similar lines.

On the other hand, if an informant had made several statements such as the one above indicating little or no change in nature of work, but also made a statement such as, ‘Overall, my work has changed enormously [over the past ten years] because of the technological changes in surgery’ (coded NOW_TECH CHANGE _HI), the question arises as to whether the latter statement assumes greater weight as evidence over other statements relating to specific technologies.

As portrayed in Figure 1, the conclusions drawn from textual data were triangulated with evidence derived from my operating department work and time study of selected work processes, and my analysis of diverse operating department and hospital records to draw my final conclusions.

Discussion

The catalyst for this paper was my desire to share with other researchers how I resolved two questions that arose during the course of data analysis in my PhD research. One question was in relation to triangulation of multiple sources and/or types of data and what I should do with non-convergent data. In other words, ‘how might I decide which evidence to accept and which evidence to regard as insignificant?’ The second question, ‘are all data equal?’ is concerned with the matter of whether or not some evidence in a study employing multiple sources and types of data can assume greater importance as evidence than other data. I further questioned: ‘if all data are not equal, how might I adjudge the relative importance of diverse evidence and/or contradictory evidence?’

I have argued that central to resolving these issues is transparency of methods which, along with other techniques, demonstrates a reflexive approach to the entire research process. I have advanced the legal notion of ‘on the balance of probabilities’ as providing a useful approach to weighing up the relative importance of various data and, also, being able to adjudge whether some, possibly contradictory, data can be regarded as insignificant, on balance, in the process of drawing research conclusions.

A simple example from law is a useful analogy. A defendant is accused of committing a crime. Evidence, such as eye witness accounts and witness identification of the defendant, will be tendered in court by the prosecutor. The defendant’s legal counsel will tender evidence to the contrary. Some evidence will be more convincing than other evidence. For example, if the evidence on which the defendant’s case was based was limited to the word of a friend or the transaction time on an ATM cash receipt that was at about the same time as the crime was committed, the case would be tenuous. However, evidence could be tendered that the defendant was undergoing surgery in hospital at the time of the crime. I suggest that this latter evidence would carry much more weight in the individual’s defence than, for example, the testimony of a friend; but, if necessary, the two sources of evidence could triangulate to produce a decision in which the court would have greater confidence, despite contradictory evidence tendered by the prosecutor.

I advance the view that qualitative and mixed methods researchers need to go through similar intuitive processes when analysing the voluminous data they have collected during the course of their research projects. They need to do this in order to weigh up the diverse evidence and to adjudge which evidence is of little or no consequence or, otherwise, inconsistent with more convincing evidence.

I have also advanced the view that researchers must be open and honest – transparent – about when and how they have applied such logic to their data analysis processes, and that this transparency is at the heart of reflexivity in research practice. Figure 1 and Figure 2, and the explanations I have provided in the body of this paper, exemplify how I endeavoured to produce trustworthy research into the impact of technological change in surgery on operating department work by triangulating different types of data from multiple sources, clearly detailing my methods of data collection and analysis, and giving attention to negative cases.

Conclusion and implications

From this discussion, I have drawn several conclusions. First, I believe that there is no substitute for the application of human intuition to the analysis of qualitative data. Secondly, I propose that computer software can help enormously in the collation and interrogation of those data, but it cannot substitute for human logic in the interpretation of those data. And, thirdly, I have presented the view that we need to acknowledge contradictory evidence emanating from our diverse data and use appropriate techniques to ‘weigh it up’ against evidence of specific phenomena when we test our research propositions. In the latter connection, I have advanced the notion of ‘on the balance of probabilities’ as a useful tool to help researchers determine the relative importance and relevance of the various data in their research. Finally, the entire research process must reflect trustworthiness principles, particularly reflexivity, transparency and trackability, whereby the role and interests of the researcher in the research, along with the details of all the research processes, are clearly articulated.

View references

Navigate: Archives | Contents | Top of page

Search

Translate

Related Publications from eContent Management