Chart criticism
Subscribe to Decision Science News by Email (one email per week, easy unsubscribe)
DE GUSTIBUS NON EST DISPUTANDUM?
This post got us thinking about chart critique. Charts are things we like to judge, as graph rating systems, and the name of the blog Junk Charts suggest.
What we wonder is will science ever be able to separate chart opinions from chart knowledge? Chart doxa from chart episteme? Consider that the Napolon’s March (above), so celebrated by chart guru Tufte
The Decision Science News editor cannot help but be reminded of teaching acting, which he did professionally for a dozen years. Actors make decisions while acting teachers take notes, and the notes usually take the form of “good move there”, “bad move there”, or “questionable move there”.
While acting is an art, one can learn to discriminate between good, bad, and questionable choices. This is because teachers make predictions and get feedback as to whether these predictions are correct.
A bad choice has been identified correctly if the scene falls flat two minutes later (as judged by everybody in the room, including the people acting in it). A questionable choice has been identified correctly if, two minutes later, some of the room liked it and the rest didn’t. A good choice has been identified correctly if, two minutes later, everybody says “excellent move there!”.
The reader might be thinking, yes, but isn’t good and bad acting just a matter of taste? Two responses. First, not as much as you might think when dealing with people learning to act. Mistakes just pop out, like a beginning clarinetist squeaking their reed. Second, there are questions of taste, and those are the ones that divide the audience reacation. The clearly good and bad moves are plain to see.
(By the way, actors who make bad choices almost never get work. They have trouble getting agents, and even if they do, with hundreds people trying for each paying job, they never get past the audition phase. A lot of prize-winning actors split the public opinion. A lot of actors who always make good choices never get a nod, though they do get steady work).
So, in acting, we can use the reaction of the room as a measurement. But in statistics, the science of measurement, there doesn’t seem to be as clear a criterion. Fields like Interaction Design may suggest good candidates, such as the number of milliseconds it takes to answer a question based on a chart, the correctness of answers based on people reading the chart, and so on. Can there ever be such a thing as expertise in judging what constitutes good, bad, and questionable choices in chart design?
ADDENDUM
David Weiss and Jim Shanteau have proposed a solution to the problem we pose above. See the comment below and this paper in Human Factors, and this one in the Journal of Behavioral Decision Making.
DSN poses the question of whether there can “ever be such a thing as expertise in judging what constitutes good, bad, and questionable choices in chart design?” One might similarly ask whether there can be expertise in judging wines, or figure skating routines, or prioritization ratings made by occupational therapists. These domains are characterized by the lack of an accepted, objective standard of performance.
James Shanteau and I have proposed a solution to this question, a performance index we refer to as CWS. The gist is that skilled judgment must simultaneously exhibit both discrimination (different ratings of different stimuli) and consistency (similar ratings of similar stimuli). These observable characteristics of expert judgment are not sufficient, because one can discriminate and be consistent on the wrong attributes. Nevertheless, because they are necessary, we have been able to make considerable progress in domains in which subjective evaluation has been the norm. Some of our work has been presented in papers in Human Factors and the Journal of Behavioral Decision Making.
September 25, 2008 @ 6:37 pm
Спасибочки:) Классная тема, пишите чаше – у вас отлично получается 🙂
May 28, 2009 @ 11:58 am