Much of this blog post will describe some basic meta-analysis procedures and how they are applied. First, quantitative procedures are only applicable to research syntheses, not to literature reviews with other foci or goals (see Chapter 1). For instance, if a literature reviewer is interested in tracing the historical development of the concept “intrinsic motivation,” it would not be necessary to do a quantitative synthesis. However, if the synthesist also intended to make inferences about whether differ ent definitions of intrinsic motivation lead to different research results, then a quantitative summary of the relevant research would be appropriate. Also, meta analysis is not called for if the goal of the literature review is to critically or historically appraise the research study by study or to identify particular studies that are central to a field. In such instances, a proper integration likely would treat the result of studies as an emerging series of events, that is, it would use a historical approach to organizing the literature review rather than a statistical aggregation of the cumulative findings. However, if the synthesists are interested in whether the result of studies change over time, then meta-analysis would be appropriate.
Second, the basic premise behind the use of statistics in research syntheses is that a series of studies address an identical conceptual hypothesis.If the premises of a literature review do not include this assertion, then there is no need for cumulative statistics.Related to this point, a synthesist should not quantitatively combine studies at a broader conceptual level than reader would find useful.At an extreme, most social science research could be categorized as examining a single conceptual hypothesis-social stimuli affect human behav ior.Indeed, for some purposes, such hypothesis test might be very enlighten ing.However, the fact that “it can be done” should not be used as an excuse to quantitatively lump together concepts and hypotheses simply because methods are available to do so (see Kazdin, Dorac, &Agteros, 1979, for a humorous treatment of this issue).Synthesists must pay attention to those distinctions in literature that will be meaningful to the user of the synthesis.For example, in the meta-analysis of the effects of choice on intrinsic motivation, we did not combine study result across the nine different outcome measures.Doing so would have obscured important distinctions among the outcomes and might have been misleading.Instead, the highest level of data aggregation was within outcome types. Another instance of too much aggregation occurs when a hypothesis has been tested using different types of controls. For example, one study examining the the effect of meeting weekly with a physical therapist on order adults’ level of activity might compare this treatment to a no-treatment control while another study compares it to a treatment in which participants receive written information.It might not be informative to statistically combine the results of these two studies To what comparison does the combined effect relate? Synthesists might find that a distinction in the type of control group is important enough not to be obscured in a quantitative analysis (but separate meta-analyses might be done for each type of control group).
Third, under certain conditions, meta-analysis might not lead to the kinds of generalizations the synthesists wish to make.For example, cognitive psychologists or cognitive neuroscientists might argue that their methodologies typically afford good controls and reasonably secure findings because the things they study are not strongly affected by the context in which the study is conducted.Thus, the debate about effects in this areas of research usually occurs with reference t the choice of variables and their theoretical, or interpretive, significance.Under these circumstances, a synthesist might convince ingly establish generalization using conceptual and theoretical bridges rather than statistical ones.
Finally, even if synthesists wish to summate statistical results across studies on the same topic, they may discover that only a few studies have been con ducted and these use decidedly different methodologies, participants, and outcomes measures.In circumstances where multiple methodologies, distinctions are confounded with one other and difference in study results, the statistical combination of studies might mask important differences in research findings.In these instances, it may make the most sense not to use meta-analysis, or to conduct several discrete meta-analyses within the same synthesis, by combining only studies that similar clusters of features, to make summary statements about relationships.