Decision Science News

THE REPRODUCIBILITY IS NOT GOOD, BUT THE FIELD CAN FIX ITSELF

Science Magazine has just published the article Estimating the reproducibility of psychological science/ (PDF).  We’ll let the abstract speak for itself.

ABSTRACT

Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. Replication effects were half the magnitude of original effects, representing a substantial decline. Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.

The work was done by the Open Science Collaboration, a group of 270 authors, led by Brian Nosek.

Now, the question on everyone’s mind. What do you think about this, Decision Science News?

• When we review papers, we see a lot of quibbling about theoretical contributions, which editors readily embrace and push back on authors. These debates are almost entirely subjective. Rarely do reviewers say “given the sample sizes here, and researcher degrees of freedom (i.e., the countless ways that authors can deceive themselves by massaging the data until desired results emerge), I don’t believe there’s anything here. I want to see this replicate on a larger sample.” And when they do, editors rarely push the authors to replicate as if it’s impolite, like it’s accusing the authors of cheating. Let’s change that. The first order concern is establishing the effect or its absence. Worry about the theory next. To do otherwise is a waste of time.
• People think this is bad news for psychology.  For psychology past, sure. But for psychology future, it’s a good thing. Decision Science News has met many psych researchers in recent years who are embracing replication, favoring larger samples, and being skeptical of what they read in Psychological Science. They’ve also started researching topics like: false positive psychology, p-hacking, and researcher degrees of freedom. People seem eager to break with the past.
• Non-replication doesn’t imply shenanigans in the original study. Non-replications can happen for a lot of reasons. However, when a lot of findings don’t replicate, it’s a sign that something’s wrong. We suspect that a lot of what doesn’t replicate is the result of file-drawer effects and p-hacking.
• We probably would have seen better replication results if the original studies were larger. Large samples make it much more difficult for researchers to deceive themselves during data analysis because small adjustments (i.e., exercising “researcher degrees of freedom”) don’t change things much when samples are large.
• Fields that haven’t run their own massive replication projects shouldn’t throw stones.

MEMBERS OF THE OPEN SCIENCE COLLABORATION (APPLAUSE TO ALL)
Alexander A. Aarts, Joanna E. Anderson, Christopher J. Anderson, Peter R. Attridge,, Angela Attwood, Jordan Axt, Molly Babel, Štepán Bahník, Erica Baranski, Michael Barnett-Cowan, Elizabeth Bartmess, Jennifer Beer, Raoul Bell, Heather Bentley, Leah Beyan, Grace Binion, Denny Borsboom, Annick Bosch, Frank A. Bosco, Sara D. Bowman, Mark J. Brandt, Erin Braswell, Hilmar Brohmer, Benjamin T. Brown, Kristina Brown, Jovita Brüning,, Ann Calhoun-Sauls, Shannon P. Callahan, Elizabeth Chagnon, Jesse Chandler,, Christopher R. Chartier, Felix Cheung,, Cody D. Christopherson, Linda Cillessen, Russ Clay, Hayley Cleary, Mark D. Cloud, Michael Cohn, Johanna Cohoon, Simon Columbus, Andreas Cordes, Giulio Costantini, Leslie D. Cramblet Alvarez, Ed Cremata, Jan Crusius, Jamie DeCoster, Michelle A. DeGaetano, Nicolás Della Penna, Bobby den Bezemer, Marie K. Deserno, Olivia Devitt, Laura Dewitte, David G. Dobolyi, Geneva T. Dodson, M. Brent Donnellan, Ryan Donohue, Rebecca A. Dore, Angela Dorrough,, Anna Dreber, Michelle Dugas, Elizabeth W. Dunn, Kayleigh Easey, Sylvia Eboigbe, Casey Eggleston, Jo Embley, Sacha Epskamp, Timothy M. Errington, Vivien Estel, Frank J. Farach,, Jenelle Feather, Anna Fedor, Belén Fernández-Castilla, Susann Fiedler, James G. Field, Stanka A. Fitneva, Taru Flagan, Amanda L. Forest, Eskil Forsell, Joshua D. Foster, Michael C. Frank, Rebecca S. Frazier, Heather Fuchs, Philip Gable, Jeff Galak, Elisa Maria Galliani, Anup Gampa, Sara Garcia, Douglas Gazarian, Elizabeth Gilbert, Roger Giner-Sorolla, Andreas Glöckner,, Lars Goellner, Jin X. Goh, Rebecca Goldberg, Patrick T. Goodbourn, Shauna Gordon-McKeon, Bryan Gorges, Jessie Gorges, Justin Goss, Jesse Graham, James A. Grange, Jeremy Gray, Chris Hartgerink, Joshua Hartshorne, Fred Hasselman,, Timothy Hayes, Emma Heikensten, Felix Henninger,, John Hodsoll,, Taylor Holubar, Gea Hoogendoorn, Denise J. Humphries, Cathy O.-Y. Hung, Nathali Immelman, Vanessa C. Irsik, Georg Jahn, Frank Jäkel, Marc Jekel, Magnus Johannesson, Larissa G. Johnson, David J. Johnson, Kate M. Johnson, William J. Johnston, Kai Jonas, Jennifer A. Joy-Gaba, Heather Barry Kappes, Kim Kelso, Mallory C. Kidwell, Seung Kyung Kim, Matthew Kirkhart, Bennett Kleinberg,, Goran Kneževic, Franziska Maria Kolorz, Jolanda J. Kossakowski, Robert Wilhelm Krause, Job Krijnen, Tim Kuhlmann, Yoram K. Kunkels, Megan M. Kyc, Calvin K. Lai, Aamir Laique, Daniël Lakens, Kristin A. Lane, Bethany Lassetter, Ljiljana B. Lazarevic, Etienne P. LeBel, Key Jung Lee, Minha Lee, Kristi Lemm, Carmel A. Levitan, Melissa Lewis, Lin Lin, Stephanie Lin, Matthias Lippold, Darren Loureiro, Ilse Luteijn, Sean Mackinnon, Heather N. Mainard, Denise C. Marigold, Daniel P. Martin, Tylar Martinez, E.J. Masicampo, Josh Matacotta, Maya Mathur, Michael May,, Nicole Mechin, Pranjal Mehta, Johannes Meixner,, Alissa Melinger, Jeremy K. Miller, Mallorie Miller, Katherine Moore,, Marcus Möschl, Matt Motyl, Stephanie M. Müller, Marcus Munafo, Koen I. Neijenhuijs, Taylor Nervi, Gandalf Nicolas, Gustav Nilsonne,, Brian A. Nosek,, Michèle B. Nuijten, Catherine Olsson,, Colleen Osborne, Lutz Ostkamp, Misha Pavel, Ian S. Penton-Voak, Olivia Perna, Cyril Pernet, Marco Perugini, R. Nathan Pipitone, Michael Pitts, Franziska Plessow,, Jason M. Prenoveau, Rima-Maria Rahal,, Kate A. Ratliff, David Reinhard, Frank Renkewitz, Ashley A. Ricker, Anastasia Rigney, Andrew M. Rivers, Mark Roebke, Abraham M. Rutchick, Robert S. Ryan, Onur Sahin, Anondah Saide, Gillian M. Sandstrom, David Santos,, Rebecca Saxe, René Schlegelmilch,, Kathleen Schmidt, Sabine Scholz, Larissa Seibel, Dylan Faulkner Selterman, Samuel Shaki, William B. Simpson, H. Colleen Sinclair, Jeanine L. M. Skorinko, Agnieszka Slowik, Joel S. Snyder, Courtney Soderberg, Carina Sonnleitner, Nick Spencer, Jeffrey R. Spies, Sara Steegen, Stefan Stieger, Nina Strohminger, Gavin B. Sullivan, Thomas Talhelm, Megan Tapia, Anniek te Dorsthorst, Manuela Thomae,, Sarah L. Thomas, Pia Tio, Frits Traets, Steve Tsang, Francis Tuerlinckx, Paul Turchan, Milan Valášek, Anna E. van ‘t Veer,, Robbie Van Aert, Marcel van Assen, Riet van Bork, Mathijs van de Ven, Don van den Bergh, Marije van der Hulst, Roel van Dooren, Johnny van Doorn, Daan R. van Renswoude, Hedderik van Rijn, Wolf Vanpaemel, Alejandro Vásquez Echeverría, Melissa Vazquez, Natalia Velez, Marieke Vermue, Mark Verschoor, Michelangelo Vianello, Martin Voracek, Gina Vuu, Eric-Jan Wagenmakers, Joanneke Weerdmeester, Ashlee Welsh, Erin C. Westgate, Joeri Wissink, Michael Wood, Andy Woods,, Emily Wright, Sining Wu, Marcel Zeelenberg, Kellylynn Zuni

SJDM CONFERENCE NOVEMBER 20-23, HILTON CHICAGO HOTEL

The preliminary program for the 2015 conference of the Society for Judgment and Decision Making, which takes place November 20-23, 2015 at the Chicago Hilton, is now available online. Get it.

Some key times / dates:

• Friday November 20th, 5PM – Opening reception & registration
• Saturday, November 23rd, 1:30PM – Interview with Danny Kahneman
• Saturday, November 23rd, 6PM – Einhorn award presentation
• Sunday, November 22nd, 1:30PM – Keynote by Max Bazerman
• Sunday, November 22nd, 4:45PM – Presidential address by Ellen Peters
• Sunday, November 22nd, 9PM – Social event
• Monday, November 23rd, 12:45PM – Conference ends

SAVE THE DATE

WHAT: Behavioral Decision Research in Management (BDRM) 2016
WHEN: June 9-11, 2016
WHERE: Rotman School of Management, University of Toronto, 105 St. George Street, Toronto, ON, Canada M5S 3E6, Desautels Hall, 2nd floor in south (new) building.
HOTEL: Park Hyatt Yorkville. Conference rate booking info. Maps (One map, Two map)
WHO: bdrm2016 at rotman.utoronto.ca

The 15th biennial conference on Behavioral Decision Research in Management (BDRM) will be held at the Rotman School of Management, University of Toronto, on June 9-11, 2016. Please save the date!

Information and the call for papers will be available at the BDRM 2016 website

BDRM is the leading conference for behavioural research conducted in business schools. It brings together the best of behavioral research within, but not limited to, the areas of consumer behavior, organizational behavior, negotiation, managerial decision making, behavioral finance, experimental and behavioral economics, decision analysis, behavioral strategy, behavioral operations research, behavioral accounting, and medical and legal decision making. Previous meetings have been held at Cornell (1986), Texas (1988), Wharton (1990), Berkeley (1992), MIT (1994), Miami (1998), Arizona (2000), Chicago (2002), Duke (2004), UCLA (2006), UC San Diego (2008), Carnegie Mellon (2010), Boulder (2012), and LBS (2014).

We look forward to seeing you in Toronto in June 2016!

MANHATTAN IS MORE EXTREME THAN YOU THINK IN POPULATION DENSITY

Slinging numbers around all day, one adage we believe is that most surprising statistics are wrong.

But here’s one that’s not: When you look at the 100 most populous counties in the USA, Manhattan (aka New York County) has about twice the population density of the next densest county (Brooklyn, aka Kings County), four times the density of the 5th densest county (San Francisco), and 13 times the density of the 10th densest county (Cook County, IL, home of Chicago).

Population density drops off sharply as you look at highly populated US counties, and New York City has 4 of the top 5 densest. We color code by state in this graph:

The graph at the top of this post represents a square kilometer and draws a dot for every person in various counties. This representation is deceptive at high densities. It would look like a black square long before it got to 1,000,000 people (1,000 people by 1,000 people, each taking up a square meter). We just can’t show 1,000 by 1,000 dots on a graph that size.

We can be more faithful, and make things easier to imagine, if we talk about people per hectare.

But what’s a hectare? Glad you asked. It’s 100 meters by 100 meters. As we see below, it’s roughly one (US) football field by one football field:

Now the differences in densities are still dramatic, but it doesn’t look like people in Manhattan are packed in like sardines.

CODE FOR R, GGPLOT, AND ANIMATION FANS

UNCROWDEDR

For a few years now, there has been a sticky note at our desk that says “UNCROWDEDR” to remind us (DSN and Jake Hofman) to build an app that tells us when other people are doing things (like going to the movies or coffee shops or grocery stores) so that we can do these things when nobody else is doing them. We liked to call this idea:

• Uncrowdedr
• The anti-social app
• The app that gets less useful as more people use it
• The Andy Warhol

The last one comes from something Jeff Nickerson told us about Mr. Warhol, who apparently liked to do certain things at the least popular times, such as going to movies on Monday. (If anyone knows the reference, let us know).

Good news. Now we don’t have to build it, as our friends at Google have built it into location search results on Android phones.

See the TIME magazine article Google Fixed the Worst Part About Grocery Shopping.

Here’s a screenshot. To see for yourself, search for “blue bottle williamsburg” from the Google search bar in Android (it doesn’t seem to be in map result data yet).

What’s cooler is that data quality/production work on this product was done by one Quang Duong, who was an intern when we were all at Yahoo Research.

AN ESTIMATE

We travel a lot, rent a lot of cars, drive around in far flung corners of the USA, and correspondingly, hear a lot of country music.

One song we’ve heard quite a few times goes:

I dug my key into the side
Of his pretty little souped-up 4 wheel drive,
Carved my name into his leather seats.
I took a Louisville slugger to both headlights,
Slashed a hole in all four tires.
Maybe next time he’ll think before he cheats.

It’s a Carrie Underwood song called “Before He Cheats” and is also featured in the movie Pitch Perfect 2.

Anyway, the last time we heard this song, we couldn’t help but think, “Darn, that’s a lot of damage she does to his truck. But how much?”

Then we thought, “Estimation is a vital skill in decision science. We should really estimate this.”

Then we thought, “blog post.”

Let’s break it down

Lyric: “I dug my key into the side of his pretty little souped-up 4 wheel drive”
Corresponding repair: Fixed a “keyed” truck door
Estimate: $650

Lyric: “Carved my name into his leather seats”
Corresponding repair: Two front seats professionally reupholstered
Estimate: $750

Lyric: “I took a Louisville slugger to both headlights”
Corresponding repair: Replacing two headlight assemblies
Cost: $950

Lyric: “Slashed a hole in all four tires”
Corresponding repair: Replacing four truck tires
Cost: $800

What’s the total damage? $3,150.

Is that a big crime? Is is! It’s a felony offense with a potentially lengthy prison sentence in Carrie Underwood’s native Oklahoma.

So, when you hear that song, maybe imagine it’s being sung from prison.

(Yes, we’re assuming she got caught, but come on, she carved her name into the leather seats).

A PACKING LIST AND MORE

Esquire recently posted some of Anthony Bourdain’s travel tips. Some good stuff there. Seeing the list reminded the editorial team at Decision Science News that we’ve always wanted to do a post on this. We travel a lot.

A PACKING LIST SAVES US TIME AND PREVENTS ERRORS

This packing list is based on one that David Allen (of Getting Things Done fame) shared. We’ve modified it to suit our needs. We pack from a list every trip. It gets packing time down to 20 minutes and zero errors. Here is the PDF (source for editing), ready to print four-per-page.

=ESSENTIALS
Socks
Underwear
T-shirts short + long sleeve
Shirts
Pants
Shoes
Belt
Glasses + contacts
Razor + cord
=FITNESS
Sneakers
Exercise clothes
Bathing suit
=BUSINESS ONLY
Dress shirts
Dress pants
Suit or jacket
Tie
Cufflinks #so easy to forget
=WEATHER DEPENDING
Sunglasses
Sandals
Shorts
Sunscreen
Umbrella
Gloves
Coat
Sweater
=IN WALLET
Business cards
Airport lounge ID card
=IN WALLET-INTERNATIONAL
Foreign credit cards
Foreign transit cards (UK: Oyster; NL: OV Chip)
=IN CARRY ON
Toiletries ziploc
Phone + headphones w/ microphone
Hoodie #avoid freezing on the plane
Inflatable Neck pillow #see below
Laptop + charger
USB wall plug adapter
Retractable micro USB cord #see below
Plane/train/car/hotel info
Maps (cached to phone) + directions
USB memory sticks
Conference programme
Prescription medicine
=IN CARRY ON-INTERNATIONAL
Passport
Foreign currency
Foreign plug adapters #see below
=DO BEFORE LEAVING
Weather checked?
Computer synced?
Presentation to USB and emailed to self?
Withdrew cash?
Checked in + chosen seats?

GETTING A PHONE WITH FREE GLOBAL DATA WAS GAME CHANGING

Since we moved from Verizon to t-mobile, our phone now works everywhere in the world and gets free data everywhere in the world. Much better than schlepping around a second unlocked phone and switching sim cards.

INFLATABLE NECK PILLOWS MAKE US FEEL GOOD

For a long time, we looked at those silly looking people with their neck pillows and felt superior. Then we tried one. They really make you feel better. We’ve tried four kinds, including some expensive ones, but our favorite is the Samsonite Inflatable Neck Pillow.  It folds up to an impossibly small size and disappears so you won’t look a doofus marching around the airport with one tied to your bag.

IT’S NOT THE JET LAG, IT’S THE SLEEP DEPRIVATION

More information.

BOARDING TIME CAN BE UP TO 10 MINUTES BEFORE THE PUBLISHED BOARDING TIME

This matters if you have status with the airline and get to board first. If you show up at the time on the ticket, you may be boarding with group 4.

WE PREVENT LIQUIDS FROM LEAKING THIS WAY

Two things. 1) Squeeze a little air out 2) Put a little square of plastic wrap over the mouth of the bottle. Screw the cap on. Voila.

WE LIKE RETRACTABLE USB CORDS

Like these. Keeps things tidy and untangled. Apologize to the squirrels who were nesting in your bag.

WE LIKE INTERNATIONAL PLUG ADAPTERS WITH BUILT-IN USB PORTS

Like these. Use your computer and charge your phone at the same time. Amaze your friends with what you can charge in one outlet, anywhere in the world.

WE LIKE COMPRESSION BAGS

Like these. They make your stuff, especially winter stuff, take up less space. And they act as laundry bags for the return trip. And they will keep a wet bathing suit from getting the rest of your stuff wet. And you don’t need a vacuum cleaner to use them. Just seal and roll. The air escapes through magical one-way valves. Not recommended for things that wrinkle easily. BTW, dry cleaning bags do a pretty good job of keeping things wrinkle free.

MEMORIZING OUR PASSPORT NUMBER WAS WORTH IT

If you travel internationally a lot, this is more useful than it sounds. You get asked for it surprisingly often, and you never feel like digging up your passport when filling out forms. We even have advice on how to remember long numbers. Lock it in once and you’re good for life.

SWIM DON’T RUN

And avoid packing an extra pair of shoes for exercise. We read about this in an interview once.

WE TRY NOT TO BOOK FLIGHTS THAT DEPART BETWEEN 11AM and 10PM

This helps us avoid flight delays.

 WE ALWAYS RETURN HOME WITH ZERO FOREIGN COINS

And this is how we do it.

If you have tips, feel free to share them in the comments.

Photo credit: http://www.flickr.com/photos/araswami/1948449158/

CHICAGO BASED POSITION

The Group: On the Behavioral Insights Team, we use behavioral science to help people engage with and improve their finances. We then run rapid, digital experimental studies to determine what works, and what doesn’t, and publicize the results.

The Role: Help millions of people retire with dignity and in comfort. Morningstar is looking for an applied behavioral scientist, to help understand and overcome the behavioral obstacles that individuals face to financing their retirement. You’ll pursue original research alongside a team of like-minded behavioralists and leading academics from around the country. This position is based in our Chicago office.

Responsibilities

• Lead our retirement-focused initiatives, developing our strategy to better understand and to apply lessons about financial decision-making behavior.

• Design, execute, and analyze large-scale randomized control trials (Morningstar interacts with millions of individual investors, their advisors, and their employers).

• Work with the marketing, product, and research teams to identify their pressing questions around retirement behavior and assist them in designing and running experimental studies.

• Coordinate with our Behavioral Science Advisory Board – amazing and thoughtful researchers from around the country – to develop new interventions, and measure their real-life impact.

• Publicize your findings in the media, papers, and at conferences.

Requirements

• Postgraduate training in behavioral social science, especially on financial decision making, or equivalent professional experience required.

• Professional experience in applying behavioral interventions specifically to improving retirement outcomes is ideal.

• Demonstrated ability writing about behavioral research for a general audience.

• Experience with digital communications (marketing, software products), and A/B testing tools for them, is a big plus.

• At least two years of experience using R or STATA to analyze complex datasets and panel data (academics: yes, GRAs count).

• Experience doing data processing and analysis in SQL is also a plus.

In your cover letter, mention how you’d approach the issue of helping people prepare for retirement.

Morningstar is an equal opportunity employer.

See: https://www.linkedin.com/jobs2/view/56477808?trk=job_view_browse_map&trk=job_view_browse_map

A DOZEN YEARS OF ESCAPE DATA FROM NEW YORK STATE

Click to enlarge

The recent prison break in upstate New York got us wondering how long prison escapes tend to last. We found some data on prison escapes in New York State. See the dot plot above.

We then found a table in the Atlantic, which cites this paper, on perhaps more skillful inmates who managed to stay on the run a bit longer, though 80% of them were back in prison inside a week.

The upshot from both analyses is the same: if you escape from prison, you’re probably going to be re-captured within a day.

R CODE

CORRELATION WITH REPUBLICAN VOTING IN THE 2012 PRESIDENTIAL ELECTION

Click to see a larger, stripped down version

In light of the good news today that the Supreme Court ruled that same-sex couples have the right to marry in the United States, we thought we’d look at when same-sex marriage laws became effective by state (if they did at all) before today.

Noting that same sex marriage is a dividing issue between the left and right in the US, we thought we’d compare the date of when same-sex marriage laws become effective in each state with each state’s Republican (two-party) vote share in 2012. Results are below.

Click to enlarge

Note that California’s same-sex marriage law was effective in 2008, then it was overturned, and then made effective again in 2013. We went with the earlier date because you’ve got to chose something.

The next question is “what about the states that were added by the Supreme Court decision today”? We made a second plot, below, in which we put today’s date for those states in the “effective date” column. The result is pretty much the same. See at the top of the post the ribbom plot contributed by David Lallemant in the comments.

It is interesting to note that none of the plurality Romney-voting states had same sex marriage laws in effect before 2014.

A couple last things. You probably want to see the data, because the graph can be hard to read.

THE DATA

Or you can get the data and the R code both

R CODE