Why You Should Avoid Optimizing One Product Metric?

Don't optimize one product metric at the expense of another.

To illustrate this, think of an extension wire.

You know how there are certain chargers with ungainly-sized plug heads that don't play nice with other slots on the wire?

As soon as you force one into an extension slot, it unhinges the plug next to it.

And you're simply not able to find an angle which will accommodate both plugs together at one time.

A similar problem arises in product metrics a lot.

You try to optimize for a certain metric and become so obsessed about it that you lose sight of another key metric that's taking a nosedive.

Example? Sure.

I was working on an applicant tracking system once and we were trying to maximize the number of jobs posted.

The team came up with the theory of a lighter & quicker job posting flow as our existing one was quite intricate.

We rolled that out and it was an instant hit. We increased our job posting counts by 2x in a week's time.

However, we also realized that job quality - the score we assign to determine how well the job description is written and something that would affect application quality - also tanked.

There was a whole flurry of thin jobs being posted and we had to scramble to address that challenge.

Other examples:

• I was trying to open up the top-of-the-funnel by landing more traffic on the vFairs marketing site by "renting" paid and social traffic. However, while pageviews started booming, the lead quality dropped sharply.
• In early 2010s, Airbnb was rolling out features in an attempt to optimize for the number of nights booked. However, this aggressive pursuit left defects in the platform that translated to an undesirable 30% increase in customer support tickets.
• Facebook front-loaded the initial versions of their mobile apps with tons of features, however, this affected performance as the app took a number of seconds to load.

Something to keep in mind is that some related metrics will always tend to vary when you optimize for another. The decision to be made is how much loss/deficit you're willing to bear.

Thus, it's important to look at your product metrics matrix in totality when considering a feature.

1. For each feature you're building, outline primary and secondary KPIs that you're setting out to improve.
2. Along with this, identify "contra-indicators" which you do not want to sacrifice as metrics in (1) are improving. Set minimum thresholds for these to help you identify when an experiment has failed. This keeps you honest in your optimization efforts.
3. Once you release the feature, look out for strong negative correlations between primary & related metrics over time. If you find one, you might need to iterate in another direction.

See more: Metrics used for your Marketing evaluation