8 Ways We Deceive Ourselves With Metrics (And How To Avoid Them), Part I
“Oh, What A Tangled Web We Weave, When First We Practice To (Self) Deceive” With apologies to Sir Walter Scott for my mangling of his most popular turn of phrase, I’d like to take some time now to discuss how people −and particularly people involved in metrics and evaluation of “how did we do this week/month/quarter?” […]
“Oh, What A Tangled Web We Weave, When First We Practice To (Self) Deceive”
With apologies to Sir Walter Scott for my mangling of his most popular turn of phrase, I’d like to take some time now to discuss how people −and particularly people involved in metrics and evaluation of “how did we do this week/month/quarter?” − are prone to self-delusion.
Along the way, I run the risk of insulting you, but I see that as a necessary risk when confronting a topic like this. It takes a healthy sense of self-worth to say “holy crap! I’m a dufus!”
Now, let’s start off with some assumptions. First, that (most) people are honest, and don’t really intend to deceive themselves or others. (If you’re doing it intentionally, then bad karma on you, but I leave your fate to another writer.) This leaves with us two groups: the honest self-deceivers who do so consciously and those who do it unconsciously.
Conscious self-deception? Surely not! Yet, how else do we explain common phrases for just this very thing:
- “I don’t want to know”
- “It’s got nothing to do with me”
- “Don’t make waves”
- “Look the other way”
- “Nothing I can do about it!”
- “Let sleeping dogs lie” (personally I love the double entendre here)
- “Ignorance is bliss”
- “Brush it under the carpet”
- “Why didn’t I listen to my intuition?”
All of these phrases are used in situations where the facts are uncomfortable and we consciously choose to ignore them. We make believe the situation is different, all the while knowing it isn’t so. I like to call this “quantum shifting” into an alternate universe where the unpleasant is no longer so.
Many of the situations where you’d use a phrase like the above are ones in which you feel you may have little or no control over things. So the natural human reaction is to ignore and hope that things change.
Is there a way to minimize self-delusion? Actually, yes, there is somewhat of a process you can use, though with varying levels of success.
- Admit that it occurs; realize, as a human, you will do it again
- Recognize when it’s happening
- Look for the opposite
- Friends don’t let friends self-delude: have a buddy who is willing to level with you
Again, I’d stress that the more compelling the bad news is, the more likely one is to steadfastly hold onto one’s convenient alternate reality. Let’s all group-self-deceive right now and chalk that up as “part of the charm of being human.”
Eight Common Self-Deceptions
Now, I can’t help you with that last buddy part. Nevertheless, here are eight common ways we unconsciously self-deceive using stats, ignorance, bad logic, or other wily human tricks. I’ll present them as encouragements − makes them funnier that way − so you can recognize when they might occur and be prepared to challenge yourself.
To encourage your own self-deception, it helps if you let yourself:
#1: Be Innumerate
Example: “what’s the big deal going from 2% conversion to 3%?? That’s only a 1% lift! We want more!”
Uh, no. Going from 2% to 3% is a 50% lift.
As the saying goes, “Innumeracy is a serious problem which affects 8 out of every 5 people.” I’m listing this as #1 on the self-delusion hit parade, because I see it the most often, and the people who practice innumeracy really do believe they “get math.” In some ways, you can argue this is easy to fix. In other ways, you can argue “you can’t fix stupid.” You decide which group you see yourself in.
#2: Start With your Wished-For Outcome… Then Look For Supporting Proof
Example: “Our Customers love us! We had 317 positive reviews on Yelp last month” (conveniently ignoring the 2,182 negative reviews and a poor Net-Promoter score)
I call this one the “Creationist Ostrich” technique. (Oh boy, I can see the comment box filling up now!) You start off in a comfy zone, and you really, really, really want to stay there, so you ignore all the evidence that contrasts with your view. Usually, this is because the alternative suggested by the evidence is bothersome in some way, and the more deeply bothersome, the easier it is to just dismiss, hold out with your head in the sand till 5 p.m., then go grab dinner. Problem solved!
Instead, under the part of the process mentioned above as “look for the opposite,” consider when you have a wished-for outcome that the best thing you can do for yourself is to try to find evidence against what you’d like the outcome to be. At least, if your goal is to uncover truth.
Virtually everyone involved in metrics, analytics, and testing-driven improvement online does, in fact, have a wished-for outcome: “get higher conversion,” “increase revenue,” “decrease cart abandonment.” Every metric you have for success implies you’d like more or less of that metric, so you always have a wished-for outcome at hand.
So, you’re subjecting yourself regularly to this sort of self-deception. Cut it off from the start by always trying to prove yourself wrong. Give yourself permission to be wrong, by constructing your tests in such a way that before you even run the test you can say, “heck if I see such and such from the test results, we know we’ve completely missed the mark” − and then look for such-and-such.