There’s a problem with cross-disciplinary work where no one person can check all the facts because there’s nobody on earth with the relevant background knowledge. I’ve heard this most often as a criticism of Jared Diamond, he of the popular anthropology / history / biology books like “Guns, Germs, and Steel”. When that first came out they historians said “His biology and anthropology are good, but his history is lacking” and the biologists said “His anthropology and history are good, but his biology is lacking” and I’ll let you guess what the anthropologists said.
This happens when someone is very good at constructing a reasonable narrative but they overstep their own abilities at fact-checking. You see it in journalism as well… you know full well that when there is a news article talking about your own employer or industry, it’s riddled with errors and biases. Yet if you don’t apply that principle more broadly, you get the impression that the news does a better job covering other topics.
Over the past couple of days I’ve been reading “The Power of Habit: Why We Do What We Do In Life And Business”, by Charles Duhigg. It came out a few years ago and was apparently quite popular, though I never got around to it.
It’s... well, it’s a business book, something I’ve seen far too many of. It’s long on narrative and short on facts. It’s very well written, it tends to agree with all my preconceived biases, and it makes its points through a series of compelling narratives rather than statistics.
But there’s a problem. The two subjects I knew about already, where I’m familiar with the statistics behind the narratives… he got them wrong. In one case it’s only a little wrong, in the other case it’s all the way wrong.
Now, I’m not finished yet, so maybe all this is fixed in the end. But I can review the first two thirds of it. The book is all about how you can change your life by developing good habits and routines so that you don’t have to make the conscious decision every day to do useful things. It’s a simple premise that I agree with.
He starts out talking about how new routines are developed, and the example is of a marketer from the early 20th century who convinced everyone to brush their teeth. You make a cue, a simple response, and a reward… the filmy feeling, the brushing, the minty aftertaste.
Next he talks about a football coach who turned the game around by breaking down each player’s job into very simple parts… watch that player’s shoulders until they move, if they move right watch that next player’s feet until they move, if they go forward you move towards that third player there, etc. Turning football from a game football players play with a ball to a game of chess coaches play with their players as pieces. This apparently worked very well and revolutionized the game, just by breaking down the complexity into a discrete series of simple steps.
Then he talks about Alcoholics Anonymous… and that’s where it starts to break down. It’s easy to see where it went wrong and he even admits it up front. The chapter leads off by saying that there’s no data to show that AA actually works at all. In fact, though this goes unmentioned, though the studies are limited and contradictory the studies seem to show a success rate that’s indistinguishable between AA and Not AA.
If you want to learn lessons from how AA teaches its participants, acknowledging that there’s no reason to think it works is a good start. But if you go on to treat those lessons as fact, then clearly any conclusions you draw from it can’t be trusted.
After that unfortunate chapter he continues with a section on building willpower. The narrative here is built around how Starbucks trains their employees, all well and good. But it also draws heavily on a psychology experiment called the Marshmallow Test. Have you heard of this one? It’s where researchers leave a toddler in the same room as a treat, telling them that if they wait to eat is until the researcher get’s back, they get a second treat as well. If they’re too impatient they only get the one they could see. In the original study they followed up years later and found that the people who could resist the temptation as toddlers had more success as teens and adults.
There have been a lot of follow-ups, confirmations, and elaborations to this study over the decades. The basic principle seems sound, but there’s a correlating factor that might better explain the results. You see, the original experiment doesn’t just measure willpower, it measures trust. The toddlers have no proof that the researchers will keep their word. There was a fascinating elaboration (which I couldn’t find today, sorry) where before the main test a researcher would make some small promise, leave for a bit, return, and either keep or break their original promise. Unsurprisingly, toddlers who knew the researcher to be unreliable didn’t trust their promise of a second marshmallow later, and those who knew the researcher to be reliable had more patience than the original subjects.
This leads to another possible conclusion… if those children were only impatient because they were untrusting, perhaps it’s because they have had bad experiences with adults before. Perhaps their parents and teachers are similarly unreliable. You would almost expect such children to be less successful later in life, wouldn’t you?]
This is a relatively minor quibble… there’s no question that the data is real and it’s telling us something important, it’s just that we aren’t quite sure what that something is.
It remains to be seen how good the rest of the book turns out to be, but I’m not sure if I’ll be able to make that judgement. Because you see, the whole thing so far has sounded really good. It’s only because I have that previous familiarity with the material that I could spot the flaw in these two chapters. What if the other stories have similar problems? How would I know?
I’ll definitely read the rest of the book, but I don’t know if I’ll trust it. As the marshmallow showed us, when the author’s been unreliable in the past, it can be hard to take things on faith.
No comments:
Post a Comment