Decision Science News

YOU CAN NEVER THINK UP ALL THE NON-SINISTER EXPLANATIONS FOR THAT FACT

This week, DSN suggests watching a couple videos.

First, this captivating Errol Morris short, The Umbrella Man, over at the New York Times.

Next, this clip of Sofitel Hotel security guards dancing (or is it hugging?) in a backroom shortly after their colleague reported a sexual assault by Dominique-Strauss Kahn. More videos here.

Lastly, the famous case of People vs. Collins.

A robbery was reported as having been carried out by an African-American man with a beard and mustache in the company of a blond Caucasian woman with a ponytail who escaped in a yellow convertible. A couple fitting that description, the Collinses, was picked up. Faulty mathematics were use to infer “that there could be but one chance in 12 million that defendants were innocent and that another equally distinctive couple actually committed the robbery.” (See the Wikipedia page for how they arrived at this). A computation later in the Supreme Court of California Decision (People v. Collins, 68, Cal.2d 334, 335 (Cal. 1968)) figures that even with the prosecutor’s assumptions:

we would derive probability of over 40 percent that the couple observed by the witnesses could be “duplicated” by at least one other equally distinctive inter-racial couple in the area, including a Negro with a beard and mustache, driving a partly yellow car in the company of a blonde with a ponytail. Thus the prosecution’s computations, far from establishing beyond a reasonable doubt that the Collinses were the couple described by the prosecution’s witnesses, imply a very substantial likelihood that the area contained more than one such couple, and that a couple other than the Collinses was the one observed at the scene of the robbery.

We close with the closing words of the umbrella man:

If you have any fact, which you think is really sinister … is really obviously a fact that can only point to some sinister underpinnings … hey forget it, man, because you can never on your own think up all the non-sinister perfectly valid explanations for that fact. A cautionary tale.

We’re not stating an opinion on guilt or innocence in the above cases. We’re just encouraging a bit of reflection on what “beyond a reasonable doubt” should mean.

ADDENDUM: A lighter take on correlation and causation.

Hat Tip to Mark Fasciano for the Umbrella man.

2012 BEHAVIORAL DECISION RESEARCH IN MANAGEMENT CONFERENCE

The Behavioral Decision Research in Management Conference is held biennially and brings together the best of behavioral research within, but not limited to, the areas of consumer behavior, organizational behavior, negotiation, managerial decision making, behavioral finance, experimental and behavioral economics, decision analysis, behavioral strategy, behavioral operations research, behavioral accounting, and medical and legal decision making.

The conference will hosted by the Leeds School of Business at the University of Colorado, Boulder June 27-29, 2012 at the St. Julien Hotel in Boulder, CO.

Inquiries about BDRM 2012 should be directed to bdrm2012inquiries@colorado.edu

Conference Website

Call for papers

Submit to the conference (Deadline Feb 10, 2012)

THE SAME EVENT DESCRIBED IN DIFFERENT-SOUNDING WAYS

We’ve written before about avoiding relative risk formats in communicating risks to the general public.

One issue is that subtle changes in terminology lead to vastly different figures. For example, imagine there are two coaches, A and B. When Coach A  is working, the team catches the ball 16 times per 100.  When Coach B is working, the team catches the ball 8 times per 100. Those are the facts. Now lets look at the range of numbers we can generate with relative risks from those facts.

• Going from Coach B to Coach A, the relative chance of catching the ball increases 100%. Let’s publish it in Science!
• Going from Coach A to Coach B, the relative chance of catching the ball drops 50%. Somebody call the newspaper!
• Going from Coach A to Coach B, the relative chance of dropping the ball increases 9.5%. Yawn
• Going from Coach B to Coach A, the relative chance of dropping the ball decreases 8.7%. Zzz….

After reading, many people will forget details such as whether the statistic was going from A to B or B to A,  whether it concerned catching or dropping, or what the sign was, leaving different people with different vague ideas that something in the set of -100%, -50%, -9.5%, -8.7%, 8.7%, 9.5%, 50%, 100% had something to do with coach quality.

We see here that relative risk formats can make a catch rate change from 16% to 8% look like a very different animal than a drop rate change from 84% to 92%. But they are not different animals, they are two ways of referring to the exact same event.

Don’t believe me? Without looking up, try to remember the relative change in drop rate going from Coach A to Coach B.

So. Is the best practice just to give two numbers, that is, tell the readers the catch rate went from 16% to 8%? Or to tell them that the drop rate went from 84% to 92%?

The problem is, even when avoiding relative risk formats, changes between small numbers just look bigger than changes between big numbers, even when the small numbers and the big numbers express the exact same event. That is, 8 vs 16 looks like a bigger deal than 92 vs 84.

Here’s an experimental demonstration. Stanislas Dehaene and colleagues surveyed Mundurucu participants (in the Amazon) and Western participants (in the USA) on where they felt numbers lie on a scale from from 10-100. Specifically, participants had an interface like that pictured above, with 10 dots on the left and 100 dots on the right. They were then shown between 10 and 100 dots and asked to click on the line where they would fall.

Data on the left are from the Mundurucu participants, those on the right from American participants.

Both the Amazonians and the US participants had negatively accelerated response curves. For instance, when shown 30 dots, the Mundurucu clicked the middle of the scale, and the Americans placed 40 dots in the middle of the scale. With such curves, differences between small numbers take up more space on the scale than equal differences between large numbers, which is why a rate change of 16 to 8 sounds like a bigger deal than a  rate change of 84 to 92. This is another instance in which people seem attuned to relative differences over and above absolute differences: the Weber-Fechner idea.

When expressing changes in probabilities, it makes us nervous that people can play games. They can choose between small numbers (here catch rates) and large numbers (drop rates) to play up or play down a point.  This post discusses how this can happen whether one uses relative risk calculations or simply provides before and after numbers. This is one reason we are fans of providing absolute changes in probability. Going from Coach A to Coach B, catches decreased 8 percentage points, which is to say drops increased by 8 percentage points. With this formulation, no matter how the thing is expressed, the only thing the forgetful can get wrong is the sign.

REFERENCES

Dehaene, Izard, Spelke, Pica. (2008). Log or Linear? Distinct Intuitions of the Number Scale in Western and Amazonian Indigene Cultures. Science, 320, pp 1217-1220.

Addition Is Useless, Multiplication Is King: Channeling Our Inner Logarithm

XKCD: Percentage Points

R CODE

options(digits=2)
CoachAcatch=16 # Coach 1 catch rate
CoachAdrop=100-CoachAcatch # Coach 1 drop rate
CoachBcatch=8 # Coach 2 catch rate
CoachBdrop=100-CoachBcatch # Coach 2 drop rate
#Going from Coach A to Coach B
sprintf("Relative risk of catching the ball: %.1f%%",(CoachBcatch-CoachAcatch)/CoachAcatch*100)
sprintf("Relative risk of dropping the ball: %.1f%%",(CoachBdrop-CoachAdrop)/CoachAdrop*100)
#Going from Coach B to Coach A
sprintf("Relative risk of catching the ball: %.1f%%",(CoachAcatch-CoachBcatch)/CoachBcatch*100)
sprintf("Relative risk of dropping the ball: %.1f%%",(CoachAdrop-CoachBdrop)/CoachBdrop*100)
#Going from Coach A to Coach B
sprintf("Absolute risk of catching the ball: %.1f%%",(CoachBcatch-CoachAcatch))
sprintf("Absolute risk of dropping the ball: %.1f%%",(CoachBdrop-CoachAdrop))
#Going from Coach B to Coach A
sprintf("Absolute risk of catching the ball: %.1f%%",(CoachAcatch-CoachBcatch))
sprintf("Absolute risk of dropping the ball: %.1f%%",(CoachAdrop-CoachBdrop))


SOCIETY FOR CONSUMER PSYCHOLOGY FIRST INTERNATIONAL CONFERENCE, FLORENCE ITALY

SOCIETY FOR CONSUMER PSYCHOLOGY
First International Conference
La Pietra Conference Center, Florence, Italy
June 29-July 1, 2012

SUBMISSION DEADLINE: Thursday, December 15, 2011

Conference Co-chairs:
Simona Botti, London Business School
Vicki Morwitz, New York University
Stefano Puntoni, Erasmus University

The aim of this conference is to foster the exchange of innovative theories and findings among psychologists and consumer researchers coming from different parts of the world. The small size of the conference (max120 attendees), the intimate Villa La Pietra location, the stunning Florence background, and the inspiring keynote speeches from Anne Maass (University of Padova) and Peter Gollwitzer (New York University), should facilitate idea generation and new collaborations.

Conference Web Site

Call For Papers

ABSTRACT SUBMISSION DEADLINE: DECEMBER 15, 2011

Hi Folks – This year BDRM will back back to back with with Boulder Summer Conference on Consumer Financial Decision Making. What an excellent reason to attend both!

What: 2012 Boulder Summer Conference on Consumer Financial Decision Making
When: June 24-26, 2012
Where St. Julien Hotel, Boulder, Colorado
Deadline for Submitting Abstracts: Dec. 15, 2011

To submit an extended abstract (1 page single spaced), please visit the conference website http://leeds.colorado.edu/event/bouldersummerconference#overview and click on the Submit Paper Abstract link.

Conference Overview
Consumer welfare is strongly affected by household financial decisions large and small: choosing mortgages; saving to fund college education or retirement; using credit cards to fund current consumption; choosing how to “decumulate” savings in retirement; deciding how to pay for health care and insurance; and investing in the stock market, managing debt in the face of financial distress. This conference brings together outstanding scholars from around the world in a unique interdisciplinary conversation with regulators, business people in financial services, and consumer advocates working on problems of consumer financial decision-making.

Our goal is to stimulate cross-disciplinary conversation and improve basic research in the emerging area of consumer financial decision-making. This research can inform our understanding of how consumers actually make such decisions and how consumers can be helped to make better decisions by innovations in public policy, business, and consumer education. Please see the 2010 and 2011 programs on the conference website to see abstracts of research by scholars in economics, psychology, sociology, behavioral finance, consumer research, decision sciences, behavioral economics, and law. Our format allows a very high level of opportunity for conversation and interaction around the ideas presented.

Conference Format
We begin with a keynote session late Sunday afternoon: ten 75-minute sessions over the next two days. We begin with financial decision making of consumers in distress because of poor financial decision making or situational stress. We then turn our focus to more basic processes that guide everyday consumer financial decision making, both good and bad. Throughout the conference we schedule significant time for informal interaction outside of the sessions.

The conference co-chairs will select papers for presentation at the conference based on extended abstracts. Selected papers must not be published prior to the conference, but those researchers presenting their work at the conference must commit to have a paper that is complete and available for review by discussants one month prior to the conference. Selections will be based on quality, relevance to consumers’ financial decision-making, and contribution to breadth of and complementarity of topics and disciplinary approaches across the conference as a whole.

Registering for the Conference and Booking a Room
There are links on the conference website for booking at the St. Julien Hotel and for registering for the conference.
The conference will be held in the St. Julien Hotel & Spa. We have negotiated very attractive room rates for conference attendees (and families). Please note that the Conference has not guaranteed any rooms, rather they are on a “first come” basis. We encourage you to book your rooms as soon as you can. Boulder is a popular summer destination and rooms go quickly at the St. Julien Hotel!

JDMers please note that this year the Boulder Summer Conference immediately precedes the Behavioral Decision Research in Management conference, held later the same week at the St. Julien Hotel. Our hope is that many of you will attend both conferences.

COSTLESS FILE SYNCHRONIZATION TECHNIQUES IN INCREASING ORDER OF COMPLEXITY

It is not uncommon to have two computers at work, four at home, and a server out on the wide open internet. How to keep all these files in sync? Here are some file synchronization tools that we use, listed in increasing order of complexity.

Dropbox dropbox.com
Setup: Easy.
OS: Windows, Linux, or Mac
We use this at Decision Science News, but only for some of our files. New users get 2GB of storage for free (or 2.25 GB if they use this link). If used sparingly it can last a long time. We find this especially useful for open-updated files that one doesn’t want out of sync for even a minute. For files that only need to be synced every day or so, we use Unison, covered next.

Unison www.cis.upenn.edu/~bcpierce/unison
Setup: Moderate for USB drive use, hard for network use (requires installing server software)
OS: Windows, Linux, or Mac

We sync about 10GB of files with unison. Unison works across the network or with a portable USB drive. Like the other solutions listed here, it magically only needs to sync the differences between files, which is much faster than moving whole files around. We have unison run as a scheduled task to make sure files get synched at least daily.

The best Unison tip is to set up a “star” configuration. That is, you designate one server (or one USB drive) as the hub and all your other machines as spokes off of it. You sync each spoke with the hub, and never sync one spoke directly to another spoke.

On a Windows7 system, unison will create a .unison folder in the C:\Users\YourUserName directory. You can put configuration files (with .prf extensions) there to tell unison what to do. Here’s a sample config file to sync the directory C:\DG on your machine to a folder E:\DG, on a USB drive.

==myconfig.prf == (assumes Unison 2.27.57 is installed on the server)
root = C:\DG
root = E:\DG
batch=true
fastcheck=true
log=true


We wrote a little batch file to start the sync process:
==sync.bat contents== (assumes Unison is installed under C:)

"C:\Unison-2.27.57 Text.exe" myconfig

When getting started, there’s a GUI version of Unison that helps you get the knack of it. For everyday use, the text version (called from our batch file, above) is the way to go.

Want to sync to a server instead of a USB drive? Here is an example config file we use to sync a local directory (C:\DG) to a directory on a linux server (/home/dsn/DG). We sync all our computers (the spokes) with this same linux server directory (the hub), which keeps all our computers in sync.

==myconfig.prf == (assumes Unison 2.27.57 is installed on the server and that ssh is installed on the Windows machine)
root = C:\DG
root = ssh://dsn@ourserver.com//home/dsn/DG
batch=true
fastcheck=true
log=true

Subversion subversion.tigris.org
Setup: Hard (Need to know how to install and configure client and server software)
OS: Windows, Linux, Mac
Built as a version control system for programmers, some people use Subversion to keep all their files in sync. It is a programmer’s tool and not easy to learn, though if you read the free subversion book and are handy with computers, you can learn it. You’ll want to have a server running on the network somewhere to make this a viable option.

We use subversion to keep our research projects (R source code, documentation, LaTeX writeups, images, PDFs of articles, small data sets) synched across many machines.

Lsyncd code.google.com/p/lsyncd
Setup: Hard (Need to know how to install and compile server software)
OS: Linux only

Also not built for the purpose, Lsyncd can be used in conjunction with Unison to keep files in sync. Lsyncd (or “live synching daemon”) is a program that watches a bunch of files waiting for any of the to be changed. Once a change occurs, it can trigger arbitrary actions, such as synching them. J D Long uses lsyncd to keep his R files (specifically, R Studio output) in sync with his local machine. Post 1. Post 2. At DSN, we use lsyncd to create a magic folder on our server that pushes R plots generated on the server back to our PC automatically.

ADDENDUM

Some other ideas have been coming in through the comments. I will list them here for posterity.

• Box.net
• DVCS-Autosync
• Rsync
• Sparkleshare
• Sugarsync
• Ubuntu one
• Wuala

QUANTITATIVE / COMPUTATIONAL POSTDOC AT YAHOO RESEARCH IN NEW YORK CITY

The Human Social Dynamics Group in Yahoo! Research is seeking highly qualified candidates with strong quantitative and computational skills for a post-doctoral research scientist position. The successful candidate will contribute to the HSD group’s research agenda, and also work on applied problems of relevance to Yahoo! consumer and advertising products. Candidates must have completed (or be able to complete before starting) a PhD, preferably but not necessarily in CS/IS, statistics, or in quantitative social science and must be skilled programmers.

Desired skills include but are not limited to:
* Mining web data
* Statistical analysis of large data sets
* Building 3rd party apps (Open Mail, Facebook)
* Designing, building, and running web-based “virtual lab” experiments
* Designing, building, and running web-based field experiments

Application Instructions

Applications, including a CV and names/addresses of three referees, should be emailed to:

Duncan Watts, djw at yahoo-inc.com

For more information on HSD personnel and publications, see

http://research.yahoo.com/Duncan_Watts
http://research.yahoo.com/Sharad_Goel
http://research.yahoo.com/Dan_Goldstein
http://research.yahoo.com/Jake_Hofman
http://research.yahoo.com/Siddharth_Suri

SOCIETY FOR JUDGMENT AND DECISION MAKING NEWSLETTER & 2011 CONFERENCE PROGRAM

This weekend (Nov 4-7, 2011) is the annual Society for Judgment and Decision Making Conference. Find the preliminary program for the conference in the current SJDM newsletter available for download here.

It’s not too late to make a spontaneous decision to attend the conference. Here are the facts.

WHERE: Sheraton Seattle Hotel & Washington State Conference Center, 1400 Sixth Avenue, Seattle, WA
WHEN: Nov 4-7, 2011
REGISTRATION: It’s is too late to register early, do it once there
MAP: http://bit.ly/tnuhCz

View Larger Map

MEANINGLESS MEDIATION

Decision Science News has posted before on Zhao, Lynch, and Chen’s practical article on mediation analysis. John Lynch has written the following, re-emphasizing the article’s main points:

Meaningless Mediation
John G. Lynch, Jr., University of Colorado
January, 2011

In August of 2010, JCR [Journal of Consumer Research – Ed] published an invited paper by Zhao, Lynch and Chen on common abuses of mediation analysis.

Zhao, Xinshu, John G. Lynch, Jr., and Qimei Chen (2010), “Reconsidering Baron and Kenny: Myths and Truths about Mediation Analysis,” Journal of Consumer Research, 37 (August), 197-206.

In a note accompanying the paper, the editor suggested that authors either follow its recommendations or take them into account if they chose to use an alternative approach. The paper made four points. As I observe how the paper is being used and adopted by JCR authors and authors at other journals, the least original of our recommendations is the most widely adopted, so in this note I want to restate the recommendations in order of importance.

1. Consider the discriminant validity of the mediator. Our single most important point is stated on the last page of Zhao et al. Many, many reports of mediation tests in consumer research and psychology are utterly meaningless because the authors have not demonstrated that the mediator is distinct from the independent variable or the dependent variable. When it is not distinct, the data will appear to support “full mediation” in Baron and Kenny’s terms and “indirect only” mediation in the parlance of Zhao et al.

A great many meaningless mediations are published in leading journals in which the mediator M is essentially a manipulation check (and hence, no discriminant validity from X) or an alternative measure of the conceptual dependent variable (and hence, no discriminant validity from Y). Some reviewers looking for any evidence of process may give “partial credit” for even meaningless mediations; this would encourage defensive insertion of meaningless mediation analyses by authors. We could save a lot of page space by deleting reports of these mediation results from the pages of JCR, JMR, and JCP. Until very recently, I have not seen much evidence that the Zhao et al. paper has had any deterrent effect on this error.

2. Embrace partial mediation and use unexpected “direct” effects to stimulate theorizing about omitted mediators. Our second most important point was that X-Y relationships are likely to have multiple mediators, and we researchers are usually not smart enough to test for more than one. In that case, it is likely that the data will sometimes indicate “indirect only” mediation (or “full mediation in Baron and Kenny’s terms), but more often will support either the “competitive mediation” or “complementary mediation” outlined by Zhao et al. Here, an unexplained direct effect of X on Y accompanies a significant indirect effect X – M – Y as posited by the researchers. Followers of Baron and Kenny viewed those direct effects with mild embarrassment. We pointed out that the sign of the direct effect can often be a hint to the sign of some omitted mediator. I should note that model misspecification and omitted variable bias can lurk as easily in data that seem to be consistent with “indirect only” (“full”) mediation as in data where there is an unexplained direct effect. The great advantage of the latter case is that the sign of the direct effect gives the authors some tip that there is more to learn, and a hint of what direction to look – for omitted indirect paths matching the sign of the “direct” effect. Write to me for an easy-to-understand example of “indirect only” results hiding omitted variable bias due to an omitted second mediator.

3. Test only for the indirect effect X – M – Y and not for an “effect to be mediated.” The Baron and Kenny procedure required that authors show a significant zero order effect of X on Y to establish “an effect to be mediated.” We showed that this effect is algebraically equivalent to the “total effect” of X on Y: the sum of the indirect effect of X on Y through M and the direct effect of X on Y. We noted that this total effect test is meaningless or superfluous. If the signs of the direct and indirect effects are opposite, it is easy to fail to observe an “effect to be mediated” or to observe an “effect to be mediated” of the wrong sign despite strong evidence for the posited indirect pathway. If the signs of the direct and indirect effects are the same, the test of the zero order effect of X on Y will always be significant when the indirect effect is significant – hence the test is superfluous here. We pointed out how nonsensical it was to treat a result as publishable when a posited indirect effect matched the sign of an unexplained direct effect, but not in the equally likely case in which the unexplained direct effect was opposite in sign. Ironically, about the time our paper was coming out, I received a rejection from a top journal with an AE report citing, among other failings, the marginal significance of the “effect to be mediated” in one of two replications.

4. Use Preacher and Hayes boostrap instead of Sobel test. The least important and least original point in Zhao et al. is, ironically, the one that seems to have caught on: use bootstrap tests rather than Sobel tests for the indirect effect X-M-Y. This one is a “no brainer.” Bootstrap tests using the very simple-to-use Preacher and Hayes (2008) macro are almost always more powerful than Sobel tests for reasons explained in our paper. There are no published bootstrap tests of mediation of within-subjects effects, where Sobel tests can be used. But in the usual between-subjects case, authors should head to Andrew Hayes website http://www.afhayes.com/spss-sas-and-mplus-macros-and-code.html

Though many consumer researchers have started using bootstrap tests, I have had colleagues tell me that reviewers told them to remove bootstraps tests and replace with Sobel. AEs should be vigilant to contradict such clearly incorrect advice if it appears in JCR reviews.

Though not emphasized in Zhao et al., the other major advantage of the Preacher and Hayes (2008) macro is that it makes it easy to test multiple mediator models.[1] Most published mediation tests consider a single mediator, though we assert in Zhao et al. that most X-Y relations likely have multiple mediators. Authors who are insightful enough to posit dual mediators almost always test each one piecewise using the Baron and Kenny tests we criticized. That’s wrong. With the Preacher and Hayes macro, it takes the same single line of code in SPSS or SAS to specify a multiple mediator model as to specify a single mediator model.

[1] Use MPLUS to analyze latent variable versions of the same multiple mediator models.

LUCK VS SKILL IN MUTUAL FUND MANAGEMENT

This week, two fun Econ-Finance papers, both with a Chicago link.

First is Steven Levitt and Thomas Miles‘ analysis of whether poker is a game of skill:

Using newly available data, we analyze that question by examining the performance in the 2010 World Series of Poker of a group of poker players identified as being highly skilled prior to the start of the events. Those players identified a priori as being highly skilled achieved an average return on investment of over 30 percent, compared to a -15 percent for all other players. This large gap in returns is strong evidence in support of the idea that poker is a game of skill

It is a game of skill, they conclude. Read more here.

Next the famous Fama–French duo ask the same question of mutual fund management. They find [drumroll] there is a bit of evidence for skill (and its opposite) in the extreme tails of the distribution:

The aggregate portfolio of actively managed U.S. equity mutual funds is close to the market portfolio, but the high costs of active management show up intact as lower returns to investors. Bootstrap simulations suggest that few funds produce benchmark-adjusted expected returns sufficient to cover their costs. If we add back the costs in fund expense ratios, there is evidence of inferior and superior performance (nonzero true α) in the extreme tails of the cross-section of mutual fund α estimates.

However, they wouldn’t advise trying to find a top manager over passive indexing:

In other words, going forward we expect that a portfolio of low cost index funds will perform about as well as a portfolio of the top three percentiles of past active winners, and better than the rest of the active fund universe.

REFERENCES

Levitt Steven D. and Thomas J. Miles. (2011). The Role of Skill Versus Luck in Poker: Evidence from the World Series of Poker, NBER Working Paper No. 17023. [link]

Fama, Eugene, F. and Kenneth R. French (2010) Luck versus Skill in the Cross-Section of Mutual Fund Returns. The Journal of Finance, LXV(5). [link]

Photo credit: http://www.flickr.com/photos/vizzzual-dot-com/2655969483/