Further advice for navigating the waters of mediation analysis
Subscribe to Decision Science News by Email (one email per week, easy unsubscribe)
MEANINGLESS MEDIATION
Decision Science News has posted before on Zhao, Lynch, and Chen’s practical article on mediation analysis. John Lynch has written the following, re-emphasizing the article’s main points:
Meaningless Mediation
John G. Lynch, Jr., University of Colorado
January, 2011
In August of 2010, JCR [Journal of Consumer Research – Ed] published an invited paper by Zhao, Lynch and Chen on common abuses of mediation analysis.
Zhao, Xinshu, John G. Lynch, Jr., and Qimei Chen (2010), “Reconsidering Baron and Kenny: Myths and Truths about Mediation Analysis,” Journal of Consumer Research, 37 (August), 197-206.
In a note accompanying the paper, the editor suggested that authors either follow its recommendations or take them into account if they chose to use an alternative approach. The paper made four points. As I observe how the paper is being used and adopted by JCR authors and authors at other journals, the least original of our recommendations is the most widely adopted, so in this note I want to restate the recommendations in order of importance.
1. Consider the discriminant validity of the mediator. Our single most important point is stated on the last page of Zhao et al. Many, many reports of mediation tests in consumer research and psychology are utterly meaningless because the authors have not demonstrated that the mediator is distinct from the independent variable or the dependent variable. When it is not distinct, the data will appear to support “full mediation” in Baron and Kenny’s terms and “indirect only” mediation in the parlance of Zhao et al.
A great many meaningless mediations are published in leading journals in which the mediator M is essentially a manipulation check (and hence, no discriminant validity from X) or an alternative measure of the conceptual dependent variable (and hence, no discriminant validity from Y). Some reviewers looking for any evidence of process may give “partial credit” for even meaningless mediations; this would encourage defensive insertion of meaningless mediation analyses by authors. We could save a lot of page space by deleting reports of these mediation results from the pages of JCR, JMR, and JCP. Until very recently, I have not seen much evidence that the Zhao et al. paper has had any deterrent effect on this error.
2. Embrace partial mediation and use unexpected “direct” effects to stimulate theorizing about omitted mediators. Our second most important point was that X-Y relationships are likely to have multiple mediators, and we researchers are usually not smart enough to test for more than one. In that case, it is likely that the data will sometimes indicate “indirect only” mediation (or “full mediation in Baron and Kenny’s terms), but more often will support either the “competitive mediation” or “complementary mediation” outlined by Zhao et al. Here, an unexplained direct effect of X on Y accompanies a significant indirect effect X – M – Y as posited by the researchers. Followers of Baron and Kenny viewed those direct effects with mild embarrassment. We pointed out that the sign of the direct effect can often be a hint to the sign of some omitted mediator. I should note that model misspecification and omitted variable bias can lurk as easily in data that seem to be consistent with “indirect only” (“full”) mediation as in data where there is an unexplained direct effect. The great advantage of the latter case is that the sign of the direct effect gives the authors some tip that there is more to learn, and a hint of what direction to look – for omitted indirect paths matching the sign of the “direct” effect. Write to me for an easy-to-understand example of “indirect only” results hiding omitted variable bias due to an omitted second mediator.
3. Test only for the indirect effect X – M – Y and not for an “effect to be mediated.” The Baron and Kenny procedure required that authors show a significant zero order effect of X on Y to establish “an effect to be mediated.” We showed that this effect is algebraically equivalent to the “total effect” of X on Y: the sum of the indirect effect of X on Y through M and the direct effect of X on Y. We noted that this total effect test is meaningless or superfluous. If the signs of the direct and indirect effects are opposite, it is easy to fail to observe an “effect to be mediated” or to observe an “effect to be mediated” of the wrong sign despite strong evidence for the posited indirect pathway. If the signs of the direct and indirect effects are the same, the test of the zero order effect of X on Y will always be significant when the indirect effect is significant – hence the test is superfluous here. We pointed out how nonsensical it was to treat a result as publishable when a posited indirect effect matched the sign of an unexplained direct effect, but not in the equally likely case in which the unexplained direct effect was opposite in sign. Ironically, about the time our paper was coming out, I received a rejection from a top journal with an AE report citing, among other failings, the marginal significance of the “effect to be mediated” in one of two replications.
4. Use Preacher and Hayes boostrap instead of Sobel test. The least important and least original point in Zhao et al. is, ironically, the one that seems to have caught on: use bootstrap tests rather than Sobel tests for the indirect effect X-M-Y. This one is a “no brainer.” Bootstrap tests using the very simple-to-use Preacher and Hayes (2008) macro are almost always more powerful than Sobel tests for reasons explained in our paper. There are no published bootstrap tests of mediation of within-subjects effects, where Sobel tests can be used. But in the usual between-subjects case, authors should head to Andrew Hayes website http://www.afhayes.com/spss-sas-and-mplus-macros-and-code.html
Though many consumer researchers have started using bootstrap tests, I have had colleagues tell me that reviewers told them to remove bootstraps tests and replace with Sobel. AEs should be vigilant to contradict such clearly incorrect advice if it appears in JCR reviews.
Though not emphasized in Zhao et al., the other major advantage of the Preacher and Hayes (2008) macro is that it makes it easy to test multiple mediator models.[1] Most published mediation tests consider a single mediator, though we assert in Zhao et al. that most X-Y relations likely have multiple mediators. Authors who are insightful enough to posit dual mediators almost always test each one piecewise using the Baron and Kenny tests we criticized. That’s wrong. With the Preacher and Hayes macro, it takes the same single line of code in SPSS or SAS to specify a multiple mediator model as to specify a single mediator model.
[1] Use MPLUS to analyze latent variable versions of the same multiple mediator models.
[…] rest is here: Further advice for navigating the humour of … – Decision Science News Share and […]
October 24, 2011 @ 1:08 am
Have to disagree with 3. If X is experimental, and M is not, the analysis is even more sensitive to problems like omitted variables. (The “mediation” in this case is more likely to be completely driven by an unobserved variable that affects M and Y.)
October 24, 2011 @ 10:22 am
I agree with Anonymous. I was struck that there is no mention of the much bigger problems of omitted variables here. This is the primary reason many statisticians think psychologists are nuts when it comes to mediation.
October 24, 2011 @ 3:11 pm
By now, there is an emerging scientific consensus that it is not necessary to have a zero-order effect of an IV on a DV to make a claim of mediation: X → M → Y.
Hayes, Andrew F. (2009), “Beyond Baron and Kenny: Statistical Mediation Analysis in the New Millennium,” Communication Monographs, Vol. 76 Issue 4, 408-420
Judd, Charles M. and David A. Kenny (2010), “Data Analysis in Social Psychology: Recent and Recurring Issues,” in The Handbook of Social Psychology, 5th ed., ed. Daniel Gilbert, Susan T. Fiske, and Gardiner Lindzey, New York: Wiley, 115–139.
Zhao, Xinshu, John G. Lynch Jr. & Qimei Chen (2010), “Reconsidering Baron and Kenny: Myths and Truths about Mediation Analysis,” Journal of Consumer Research, 37 (2), 197-206.
Rucker, Derek D, Kristopher Preacher, Zakary Tormala and Richard Petty. 2011. Mediation Analysis in Social Psychology: Current Practices and New Recommendations. Social and Personality Psychology Compass . 5(6): 359-371.
All four of these papers give reasons to toss out the old Baron and Kenny requirement that one must establish “an effect to be mediated” to make a claim of mediation. By the say, the authors who now agree with us in this conclusion are the people who have highly cited earlier papers making the point that they now disavow: Hayes, Judd, Kenny, Preacher.
Zhao et al. is most directly relevant to the points of Dean and Anonymous, because it is in fact about the role of omitted variables – specifically, omitted mediators. Any time one has what Zhao et al. called “competitive mediation” (direct effect opposite in sign to indirect effect), it is quite easy to find the “total effect” (zero order effect of IV on DV) to be nonsignificant or even significant and opposite in sign to the indirect effect.
This is true whether there is a “real” direct effect (as in the example in ZLC of condoms affecting STDs) or a “direct” effect due to omitted mediators (as in the bastarized / fictional example from Mitra and Lynch on p. 199 of ZLC). )
ZLC noted that authors routinely hand-wave their “direct” effects – “partial” mediation in Baron and Kenny terms is more common than “full.” We argue that in many cases, it is quite plausible that “direct” effects reflect omitted mediators. We show that if the direct and indirect effect are the same in sign, the “total” effect of X on Y — so called “effect to be mediated” — will necessarily be significant any time the indirect effect is significant. But this is not true if they are opposite in sign.
Imagine two authors who submit their papers, both reporting a predicted indirect effect and an unpredicted (and as yet unexplained) direct effect. However, the direct and indirect effects have matching signs for the first paper and have opposite signs for the second. If Anonymous were Editor, the first paper would be accepted and the second rejected – even though they both make partial progress in understanding the mechanism underlying the X→ Y link and leave for future research the task of explaining the unexpected direct effect.
One of the most fundamental problems in methodology is how to make scientific process despite only partial understanding of the phenomenon one studies. Any author who would claim to have established all of the mediators of the X->Y relations s/he studies would be painfully naïve. Every paper we publish will later be shown to reflect an incomplete understanding of the phenomena we studied. Despite this, our publications reporting empirical findings can contribute to scientific progress.
Also, it should be noted that the problem of omitted variable bias is just as much of a concern when the data appear to support “full” mediation. (Write to me for an easy-to-understand example of a case where it is clearly true that an X→ Y effect has at least two mediators, but entering only one creates the illusion of what Baron and Kenny label “full” mediation and what Zhao et al. call “indirect only” mediation.)
Thus, Dean and Anonymous are right that we should be concerned about model misspecification due to omitted variable bias. But they are incorrect in thinking that there is any protection against such bias afforded by adding a requirement of a significant total effect in making claims about mediation.
October 24, 2011 @ 5:37 pm