Can’t get it right? Start measuring. Maybe.
“For a variety of reasons, measuring doctor skill is a tricky affair” so argues Steven Levitt and Stephen Dubner about rating doctors, in their “Superfreakonomics“. That’s because – first there is selection bias, sicker patients seek out better doctors and no matter how good the doctor is, their high risk makes them more vulnerable. Then, the doctor, knowing that he is measured against patient outcomes, is more likely to “cream-skim” the patients and turning down high-risk patients. That’s not what was intended.
Levitt and Dubner say “…the law of unintended consequences is among the most potent laws in existence“. Disabilities Act, meant to prevent discrimination of disabled, in fact results in fewer people with disabilities getting hired as employers worry they cannot discipline or fire them. Endangered Species Act, supposed to protect endangered species and their habitats, actually makes landowners cut down trees to make their land less attractive to endangered species, lest their land is claimed by the government.
Public policies, economics, law abound with such examples. Organizations too.
It’s stale but still worth repeating the Dilbert strip with Wally’s “I gonna write me a new minivan this afternoon” in response to PHB’s decision that engineers will be paid on how many bugs they fix. People behave the way they are going to get measured by.
Someone, the other day, was trying to measure agility of teams practising Scrum within an organization. I cannot blame if that makes teams trying to game the system. Then there was that ‘programmers productivity’. A muddy subject, if not useless. In any case, why bother? And don’t get me started on Lines of Code. I have seen managers trying to rewards developers with pens for writing unit tests for their own code. I have seen people shying away from mentoring others if individual performance is all they are measured against. And it gets worse if it’s all a zero-sum game.
Do we really need to measure everything? Robert Glass, in Facts and Fallacies of Software Engineering, cites their Fallacy 1 as: You can’t manage what you can’t measure. However, he argues, we do manage things we cannot measure – think of software design, think of cancer research. Edward Deming includes in his 14 points: “Eliminate management by objective. Eliminate management by numbers, numerical goals. Substitute leadership“.
This is not to undermine the value of data and its analysis. It’s to recognize that there will be nooks and corners where data, at least the obvious ones, will mislead more than show. Then how do we get around managing those dark corners? Gut factor. Or Gladwellian blink factor. Or flip a coin.