Empirical Elephants—Why Multiple Methods are Essential to Quality Research in Operations and Supply Chain Management

Much like the blind men and the elephant, researchers in the field of operations and supply chain management (OSCM) often study natural phenomena (business processes are our “elephants”) using widely varied perspectives. Though we seek objectivity, each of us is prone to form an attachment to a given perspective, influenced by our training and by the perspectives of our mentors and peers. As editors, numerous conversations with researchers have given us cause to be concerned that there is a tendency in the academic community to discount the findings of others who are using their different “senses” (research methods) to examine various parts of the elephant. In this editorial, we take the opportunity to briefly review a variety of research methodologies. It is our strong belief that multiple approaches are required in order to develop a holistic understanding of operations and supply chain management phenomena. Only by being open to reading, digesting and synthesizing multiple perspectives can we get a true picture of the elephant. For a couple of reasons, we limit our comments to address only empirical research methods. First, while recognizing that modeling and purely analytical techniques have produced large advances in OSCM, we also take the position that the study of OSCM is a social science. Research in OSCM is more than problem solving; it is the study of business processes with the objective of developing theories that explain them. Business processes contain both natural and social phenomena. As such, while systems and decisions affecting business processes can be modeled, examinations of empirical data are crucial for the development and validation of models. It is especially important that we uncover the often complex social and behavioral elements involved in OSCM. Biological sciences can explain the universal characteristics of an elephant, but in practice different elephants have personalities and individual characteristics that must be observed in order to be understood. A second reason for limiting this discussion to empirical methods is the fact that the Journal of Operations Management is entirely dedicated to the publication of theory-driven, empirical research. The JOM is the only OM journal stating such a focused mission, and we are committed to it. At the same time, we would not want the journal to become too focused in terms of the empirical methodologies employed. We want to encourage research employing a wide range of perspectives and methods. In the following paragraphs we very briefly examine the benefits and limitations of several broad methodological approaches, including primary data collection (survey, case, experiments) and secondary data collection. This is clearly not an exhaustive guide or comparison of empirical techniques. There are numerous resources for this, including the recent article by Roth (2007). Our goal is to remind readers that every approach has good and bad points. There is no universally superior methodology. In the discussion we also present several recent or forthcoming JOM articles that exemplify each approach. Finally, we discuss potential synergies between the various approaches which can provide a holistic view of the elephant. Following the seminal work of Flynn et al. (1990), there has been a notable growth in empirical research generally, and in survey-based research in particular. While Flynn et al. (1990) provided an overview of numerous empirical techniques, their work focused on survey-based research. Survey research is a common methodological approach in the fields of management and marketing, with well-established protocols that are summarized by Flynn et al. (1990) and many others. The question is not whether survey-based research is appropriate, but when is it appropriate? A survey provides a low cost, non-invasive means for measuring aspects of an operational or supply chain issue. Measures included in a survey can be designed to target specific factors or attributes which may not be directly observable. For example, many behavioral variables affecting an operational process are “latent,” and can be assessed only via perceptual measures. Moreover, even directly observable attributes are often not measured by organizations, either because they are not of interest or because they cannot be measured in a cost effective manner. Frequently, measures that are published by organizations do not report data at the needed unit of analysis. For example, measures of costs or sales at the plant or lower levels are not typically available from published sources. Via a survey, data can be collected directly from the individual or unit directly responsible for managing a certain operation of interest. Much of the controversy surrounding surveys has to do with the use of perceptual measures. The use of perceptual measures is not necessarily wrong. In fact, they may be necessitated by the nature of the variable of interest. This is clearly the case when variables are a function of behaviors or organizational norms. Even when variables are materially observable, a perceptual measure may be more efficient due to problems with comparability. Take, for instance, the case of measuring operating costs. In many cases an OSCM researcher is not interested in the absolute cost performance of an operating process; he/she is interested in the unit's costs relative to those of comparable operating units. The problem, of course, is that rarely are any two operating units completely comparable. Moreover, the adjustments required to achieve comparability are not feasible given the researcher's limited resources and access to information. Consequently, it becomes efficient to rely on a respondent's rating of relative costs, given his/her expert knowledge and ability to mentally adjust cost performance estimates for relevant differences between operating units. Under these circumstances, a respondent's “perceptual” estimate of relative cost may be more accurate than an estimate provided by reports created by the organizations’ respective accounting systems. While a more objective measure of cost would be ideal, it might be infeasible to attain. Survey-based research also has important limitations. A well known difficulty with perceptual measures is the potential for measurement errors stemming from subjectivity and bias. These issues have been thoroughly discussed in the literature, and numerous controls and remedies have been forwarded as means for dealing with the limitations of perceptual measures. In the case that the survey is the only source of data used in the study, common method bias is of course also a serious concern. Again, the literature offers preventive measures and controls. It is clearly important that researchers take the utmost care to lessen and control for sources of bias, thereby improving confidence in the validity of the research findings. This need is important for any research methodology, but bias is perhaps a most salient threat in survey research using perceptual measures. Other concerns related to survey research address difficulties with respondents’ interpretations of measures, potential lack of knowledge, and representations of the unit of analysis. In addition, we have heard a common lament that business managers, particularly in manufacturing, are suffering from survey fatigue, and increasingly unlikely to respond to surveys. While each of these limitations is important, none necessarily represents a fatal flaw in the methodology. Numerous publications in the operations and broader business literatures offer suggestions on how to mitigate inherent problems. Bad research using any methodology usually starts with poor, or even lazy, research design. This is analogous to a blind man examining an elephant's foot. If he isn’t careful, he is likely to get stepped on. Rabinovich et al. (2007) use transaction cost theory and data from a survey of 196 internet commerce firms (ICFs) to show that low levels of asset specificity and uncertainty drive ICFs to establish relationships with third party logistics firms. Notable in this research is the evidence explaining why firms choose to establish multiple party relationships across a supply chain. The findings refine the higher level observation from publicly available data that these relationships exist. Schoenherr and Mabert (2008) use 252 responses from purchasing professionals to show that a more homogenous bundle of goods/services in the request for quotation process leads to better outcomes. This study illustrates how a potential shortcoming, a low response rate of 5.4%, can be mitigated and must be evaluated within the specific study context. The authors use a carefully targeted contact sample and employ several measures to assess non-respondent bias. The end result is a study that offers critical insight to a modern tool for purchasing professionals (online reverse auctions), captures important behavioral factors, and mitigates the limitations through the use of appropriate study design and application of statistical tools. Survey research has clearly been the most common methodology represented in JOM articles in the past. We expect that this methodology will continue to be well represented in the journal, and we will continue to insist that the highest standards and the best practices for ensuring reliability and validity are maintained. Case or field-based studies provide a qualitative approach to studying a phenomena in-depth, particularly poorly understood or emerging phenomena. Primarily used as a theory-building approach, cases studies have been effectively employed in a large variety of situations, and excellent guides for conducting such research exist in both the broader business (Eisenhardt, 1989; Yin, 1989) and in the operations management literature (Meredith, 1998). Benefits of case studies include the ability to examine a topic in great depth. Researchers can focus on a specific topic and/or company(s), allowing a thorough examination of numerous factors and nuances. Case studies provide a richness of description and first-hand observation of phenomena in a natural setting. Often ***case studies yield unintended insights, which can lead to new avenues of inquiry. The best case studies provide a foundation for further examination. Limitations of case studies include cost and time, inability to generalize and prescribe, and potential for bias in the perceptions of the researchers. This research approach, when well done, can be expensive in terms of both financial cost, as well as the time spent by both researchers and participating managers. While case studies can provide great depth, it is difficult to generalize results from a few companies to a broader population. Finally, since there is a high degree of involvement of researchers with the subjects, there is a good chance of bias in terms of interpretation. As with other research techniques, each of these and other limitations can be substantially mitigated with the use of proper techniques. Wu and Choi (2005) examine eight cases, gathering data from pairs of suppliers to identify five archetypes of supplier–supplier relationships. This study aptly demonstrates how careful and thorough application of appropriate case study techniques can yield substantial insights and provide a strong platform for future research. For this reason, the article by Wu and Choi (2005) was recognized as the best JOM paper of 2005. Closs et al. (in press) study business units in six Fortune 500 companies to better understand drivers of product portfolio complexity and important competencies for managing them. Complexity management competencies are explicated using a socio-technical systems theoretical perspective. Case based research can be seen as risky, as it is time consuming and may yield fewer publication opportunities per research hour of effort. For this reason, advisors sometimes caution their doctoral students against conducting case research early in their careers. However, this type of research is excellent for establishing a foundation for follow-on research streams. Many theory-building opportunities remain in the field of OSCM, and we encourage the publication of more case study research in the JOM. The primary advantage of experiments is that they provide a high degree of control over threats to internal validity by limiting influences that might otherwise explain observed relationships between independent and dependent variables. Through controls and random assignments to treatments, specific effects can be isolated, and strong inferences of causality can be developed. The obvious limitation to experiments pertains to external validity, as the laboratory environment creates an artificial setting that may not adequately represent true decision making conditions. Carter and Stevens (2007) conducted experiments involving 97 MBA and business Ph.D. students in order to study the interplay of electronic reverse auction parameters. Their findings indicate the relative effects of price information, number of competing bidders, and supplier need for contract have on final bid price and perceptions of buyer opportunism. Bendoly and Swink (2007) designed experiments studying the effects of various levels of information availability upon the actions and perceptions of 362 MBAs in an environment where multiple projects are pursued simultaneously. The findings suggest that greater visibility of situational information impacts project outcomes mainly by affecting a project manager's perceptions regarding both the behavior of project managers and the priority given to his/her task. The study extends existing information processing theory by developing and examining notions of informed rationality and reciprocity, as well as examining the effect of information on post-task sense making. As interest grows in studying the effects of behavioral and cognitive processes on decision making in OSCM, we expect to see more experiments in our field. Seawright and Sampson (2007) provide an interesting empirical examination of wait perception bias. They use a video method of data collection to study customer perceptions, finding that the presence of an explanation for long waits is helpful. Interestingly, it makes little difference whether the explanation focuses on customers or servers as the cause of waiting. This type of work provides important theoretical grounding and refinement of the intersection between psychological and operations management principles. The growing use of communications and information systems to collect data throughout the supply chain has created numerous opportunities for research that employs secondary data. The availability of data collected by corporations, government agencies, news agencies, industry groups, and other parties is growing very rapidly. A pair of recent articles (Roth, 2007; Fisher, 2007) have called for more research using secondary data, and examined some of the benefits and limitations involved with this type of data. We wholeheartedly endorse and support the use of such data and believe it is of great importance for the future of OSCM research. Benefits of secondary data sources include wide availability and lower data collection expense. For example, collaboration of companies using IT systems to exchange data offers greater opportunities to collect data across multiple supply chain entities. Secondary data are generally more “objective” than self-reported survey data, yet the quality of the data is dependent on the standards and protocols that govern their collection, and these factors are usually outside the control of the researcher. Since the researcher is dependent on another party for data collection, she/he must assume that the data were collected correctly. Other limitations of secondary data research also exist. First, available data might not include measures that tap the construct of interest exactly. Researchers are frequently left with proxy measures that may only partially represent or reflect the theoretically relevant variables. A second limitation is that data from computers or reports are not always accurate, or devoid of bias. Many data (e.g., financial reports) are quite noisy, making statistical significance of hypothesized relationships hard to establish. In addition, errors and misreports in data can be systemic, as Enron and accounting scandals have shown us. After all, the observations of people are often the source of most business data, even if it is ultimately delivered via a computer. Third, while there are vast amounts of data locked away in corporate databases—it takes careful work and high levels of trust to gain access to such data. Many of us have experienced the disappointment of being denied promised data despite having signed numerous non-disclosure agreements and having promised confidentiality to the point of exhaustion. A popular study by Hendricks and Singhal (2003) estimates the shareholder wealth affects of supply chain “glitches” that resulted in production or shipment delays using a sample of 519 public announcements made during 1989–2000. The insight that “Supply chain glitch announcements are associated with an abnormal decrease in shareholder value of 10.28%” (Hendricks & Singhal, p. 501) is tremendously motivating to managers and academics alike. Accordingly, this paper was awarded the annual JOM best paper prize. Shah and Shin (2007) use data collected from Bureau of Economic Analysis at the U.S. Department of Commerce to examine relationships between inventory, IT investment and financial performance. The study demonstrates that inventory levels have changed across and In addition, the results provide further support for the of a between IT investment and financial performance. These results some important for future research. and (2007) offer an examination of over via data collected from an customer of Their study to some of the drivers of customer et al. use results of a survey to the importance of processes in and improving supplier While IT and exchange of information, their work that relationships between and the ability of and important information to and through a supply and use a case study approach to examine some of the psychological factors the application of and The case study of a from the of The findings indicate that time leads to an in variety of As by and such research the behavioral factors and a understanding of is needed to develop better training for in order to limited training of a this approach. et al. a in which managers were to a of responses to various purchasing Their study evidence that the approach to making a decision is not necessarily but can be based on several In and both have substantial on the end This work offers with the work of and in that do not decisions with While than a the work of et al. was recognized with the JOM best paper at the of Management and (2005) customer survey data from two companies to customer from corporate that survey and secondary data. Bendoly and (2008) a pair of case studies to examine of following of systems. The case studies are carefully to pair of with protocols using the same The data from the case studies were by a involving and to managers. The study findings indicate that of IT systems may strong to systems in the presence of and p. The of case study and provide important validation for the findings. A of research made of a variety of research methodologies is not a portfolio of financial The collection is more likely to yield with (in this case, of Our review of a of recent articles published in JOM the variety of approaches employed by our We and encourage such of empirical approaches in the Journal of Operations While we cannot to in every potential methodology, we can to do our level best to appropriate and to work to our portfolio of tools. In we want to our belief that methodologies have to Moreover, the of methodology do not necessarily It is nature to the for with which we are most and We believe that OSCM and the OSCM community as a will if we can our in an to understand and the benefits and limitations of empirical methodologies. After all, blind men use their to for the lack of we as researchers any of that will the