Monday, April 11, 2016

The Tyranny of Metrics: you get what you ask for

Image used under under Creative Commons CC0
I am a Silicon Valley engineer and a product manager.  As such, I have been bathed in the holy waters of data and baptized into the religion of metrics.  Numbers are hard, knowable, and unassailable.  They give you real ammunition to prove your success (or failure) and charts to show progress and the velocity of your team.  The scripture of metrics demands that you set measurable goals for success and then track those metrics.  Most large organizations will then incentivize and judge employees by the results of these metrics.  Where humans can be biased, (Jane is awesome because she likes rugby, just like me!) numbers are not.  Trust in the numbers.

This works great in most cases where success is about operational efficiency.  In a call center, the things you care about are number of calls handled, duration of each call, and the percentage of calls that are resolved in a positive (as defined by each company) manner.  Great.  Track this data, reward based on it, and each operator will be motivated to improve her numbers.  You have achieved operational efficiency.

But innovation is not efficient.  An organization or an individual can become better at innovation and guarantee long term success, but the path required is not a straight one.  Sometimes it will seem like you are moving backwards instead of forwards.  You’ll be constantly opening and closing the funnel.  It’s messy and somewhat unpredictable in the short term.  You should absolutely still use metrics and data to make your progress transparent, but resist the urge to move down the slippery slope of tying those metrics to incentives like compensation, promotion, and survival of the team.

Metrics should remain a tool to inform decisions, not to automate decisions.  In the innovation space, individuals and teams need to make judgements on what tasks, technologies, user research, relationships, experiments, infrastructure, etc. to invest their time and money in.  The problem is that you are now comparing not just apples and oranges, but apples and oranges, the taste of victory, the memory of a loved one, and the sound of silence.  These are so different that they do not condense into a few metrics.  But once you tie incentives to metrics, you are locking down the algorithm for making complex decisions.  In other words, the incentive system will do exactly what it was designed to, and nothing more.  The smart people you hired because of their ability to make good judgments in complex, ambiguous environments will instead make decisions based on the much simpler task of fulfilling the objectives set out by your incentive system.  If you set a metric on number of experiments, then you will get more experiments.  But all incentive systems are also disincentive systems.  People will stop doing other important tasks.  The experiments may not even be useful, but more will occur because that’s the metric.  The complex balancing that humans can do will go out the window.  What a waste.  Some people may still “do the right thing”, but now it is against their own financial self interest.  This will frustrate them to no end until they leave.

So what’s the alternative?  Hire smart people.  Align your goals and motivation.  And then TRUST your people.  In Daniel Pink’s book, “Drive”, he explores the science of human motivation and boils this down to autonomy, mastery and purpose.  Note, that not one of these key elements is monetary compensation.  Yes, money is important, but more as a feedback mechanism and a judgement of fairness.  When the very clear guidance of financial motivations conflict with the powerful but ambiguous motivations of autonomy, mastery, and purpose, you create non-beneficial tension in your innovation workers (i.e. unhappiness).  So please, avoid this.  Trust your people, and avoid letting metrics drive your process.  Don’t let the certainty of numbers cloud your team into making bad decisions and you’ll have a happier, more creative, and more successful team.