Replication
Philosophers have long identified replication as an important facilitator of scientific progress. Several terms have been used to denote the ability to assess past work through replication, including "intersubjective testability," "reliability," and "verifiability by repetition." Authors of scientific papers typically describe the methods and materials they used in their research so that, at least hypothetically, others can repeat the work and reproduce the reported results. Successful replication of their own and others' work gives researchers confidence in its validity and reassures them about the fruitfulness of the general line of inquiry they are following. In contrast, inability to replicate one's own or others' results casts doubt upon the validity of the previous work. Critics argue that because sociologists infrequently attempt to replicate findings, they are both less able to identify valid lines of inquiry and more likely to follow spurious ones.
One can identify a continuum ranging from exact to weakly approximate replication. The former, also called repetition, consists of attempts to use the same materials and procedures as previous research to determine whether the same results can be obtained. Approximate replication, on the other hand, consists of using some but not all of the conditions of a previous study. By systematically varying research conditions in a series of approximate replications, it may be possible to determine the precise nature of a previous study's results and the extent to which they also hold for different populations and situations (Aronson et al. 1998). Researchers usually value successful approximate replication more than successful exact replication because the latter contributes less to existing knowledge.
In the natural sciences, experimentalists are usually expected to carry out successful exact replications of their own work before submitting it for publication. This reduces the likelihood of reporting spurious results and of misleading one's colleagues. Exact replications of others' research are often difficult and costly to execute, however, and natural scientists rarely attempt them except in cases where the original work is theoretically important or has high potential practical value, or where there is suspicion of fraud. Another disincentive for carrying out exact replications of already published work is that such work is usually difficult to publish. This is true not only because little new knowledge results from an exact replication, but also because the meaning of a failure to replicate exactly is often ambiguous. Failures can indicate that the original work was flawed, but they may also be due to inadequate specification of research procedures, the existence of a stochastic element in the production of results, or errors in the replication itself (Harry M. Collins 1985). By contrast, approximate replications, especially those involving the modification of research instruments and their application to new areas of inquiry, are common in the natural sciences, and this has led some to identify them as constituting a central element of "rapid-discovery, high-consensus science" (Randall Collins 1994).
Many hold that social scientists' opportunities to carry out replications, especially exact replications, are severely limited. This is partly because social scientists often use nonexperimental research techniques that are difficult to repeat exactly. In addition, changing social and historical contexts can influence studies' results. As a result, failures to obtain the same results as reported by previous studies are even more ambiguous in the social sciences than in the natural sciences (Schuman and Presser 1981). This ambiguity may account for social scientists' continued interest in concepts and theories stemming from studies whose results have repeatedly failed to be replicated (e.g., sex differences in fear of success and patterns of moral development).
Nevertheless, critics have long argued that behavioral scientists need to attempt more replications of previous research because their dependence on statistical inference produces many spurious reports of "statistically significant" results. Statistical inference allows researchers only to reject or fail to reject a null hypothesis. Each of these two outcomes is subject to error due to the probabilistic nature of statistical hypothesis testing; sometimes researchers reject null hypotheses that are actually true (type one error), and sometimes they fail to reject null hypotheses that are actually false (type two error). However, failure to reject a null hypothesis does not justify accepting it, and studies that do not yield rejections therefore are often judged as contributing little. As a result, scholarly journals tend to publish only papers that report the rejection of null hypotheses, some of which are the result of type one errors (Sterling 1959). Furthermore, to ensure that they will be able to reject null hypotheses, researchers sometimes use inappropriate analytic procedures that maximize their chances of obtaining statistically significant results (Selvin and Stuart 1966), increasing the likelihood that published findings are due to type one errors. To counteract these patterns, some have argued that behavioral science editors should set aside space in their journals for the publication of replication attempts, and to publish studies that fail to replicate earlier results even when the replications themselves fail to reject null hypotheses.
Despite the calls for increased replication, behavioral science journals publish few papers reporting replication attempts. In an early examination of this issue, Sterling (1959) reported that among 362 articles in psychology journals, 97 percent of those reporting a test of significance rejected the null hypothesis, but that none was an explicit replication. Ironically, many have replicated Sterling's results (cf. Dickersin 1990; Gaston 1979; Reid et al. 1981). These studies probably underestimate the prevalence of replication, because they do not count papers reporting a set of experiments that comprise both an original result and one or more approximate replications of it. By not encouraging more replication, however, behavioral science journals may foster elaborate and vacuous theorizing at the expense of identifying factual puzzles that deserve theoretical analysis (Cook and Campbell 1979, p. 25).
Although the traditional view of replication entails the collection of new data—including data on additional cases or additional measures—statisticians and social scientists have suggested alternative replication strategies. One is to build replication into a study from the start. For example, a researcher can draw a sample large enough to allow its random partition into two subsamples. Data from one subsample can then be used to check conclusions drawn on the basis of analyses of data from the other. Another approach, requiring the intensive use of computing resources, is to draw multiple random subsamples from already collected data and then use these subsamples to crossvalidate results (Finifter 1972). This general strategy, which includes such techniques as "jackknifing" and "bootstrapping," is also used to assess sampling variances for complex sampling designs (see Sampling Procedures). Still another elaboration of the basic idea of replication is the general approach called meta-analysis. Here the analyst treats previous studies on a topic or relationship as a sample of approximate replications. By statistically analyzing whether and how studies' results vary, one can determine how generalizable a finding is and the extent to which differences in study design account for variation in results (Hunter and Schmidt 1990). Finally, replication may also be fostered by the increased availability of already-collected data sets stemming from the establishment of data depositories, and funding agency requirements that data from supported projects be made accessible to other researchers. Access to previously collected data makes it possible to carry out both exact replications of previous analyses and approximate replications that alter the analytic procedures used by the original researcher.
Sampling Procedures)
References
Aronson, Elliot, Timothy D. Wilson, and Marilynn B. Brewer 1998 "Experimentation in Social Psychology." In Daniel T. Gilbert, Susan T. Fiske, and Gardner Lindzey, eds., Handbook of Social Psychology, 4th ed. Boston: McGraw-Hill.
Collins, Harry M. 1985 Changing Order: Replication and Induction in Scientific Practice. London: Sage.
Collins, Randall 1994 "Why the Social Sciences Won't Become High-Consensus, Rapid-Discovery Science." Sociological Forum 9:155–177.
Cook, Thomas D., and Donald T. Campbell 1979 Quasi-Experimentation: Design and Analysis Issues for Field Studies. Chicago: Rand-McNally.
Dickersin, Kay 1990 "The Existence of Publication Bias and Risk Factors for Its Occurence." Journal of the American Medical Association 263:1385–1389.
Finifter, Bernard M. 1972 "The Generation of Confidence: Evaluating Research Findings by Random Subsample Replication." In Herbert L. Costner, ed., Sociological Methodology 1972. San Francisco: Jossey-Bass.
Gaston, Jerry 1979 "The Big Three and the Status of Sociology." Contemporary Sociology 8:789–793.
Hunter, John E., and Frank L. Schmidt 1990 Methods of Meta-Analysis. Newbury Park, Calif.: Sage.
Reid, L. H., L. C. Soley, and R. D. Rimmer 1981 "Replications in Advertising Research: 1977, 1978, 1979." Journal of Advertising 10:3–13.
Schuman, Howard, and Stanley Presser 1981 "Mysteries of Replication and Non-Replication." In Howard Schuman and Stanley Presser, eds. Questions and Answers in Attitude Surveys. New York: Academic Press.
Selvin, Hannan C., and Alan Stuart 1966 "Data-Dredging Procedures in Survey Analysis." American Statistician 20:20–23.
Sterling, Theodore D. 1959 "Publication Decisions and Their Possible Effects on Inferences Drawn from Tests of Significance—Or Visa Versa." Journal of the American Statistical Association 54:30–34.
This is the complete article, containing 1,445 words
(approx. 5 pages at 300 words per page).
