The Future of Product Management Metrics

Product managers answer for a product that customers judge later, and most of the numbers they get arrive after those customers have made up their minds. If you run user tests, you already hold the raw material for measures that arrive while you can still act. This guide is for product managers. It covers where the usual metrics fall short, what your user testing can tell you, and the three scores built on Delta Testing, the approach to user testing that Centercode created: Delta Health, Delta Success, and Delta Impact. It closes with why automating those scores matters. New to running tests? Start with The Ultimate Guide to Beta Testing.

Why do most product metrics arrive too late?

Many of the best-known product metrics arrive too late because they record what customers have already done or felt. Product managers have no shortage of metrics. You manage the vision, and you turn what the numbers say into a strategy that several teams rely on. These are the numbers you know well:

• Monthly active users (MAU) and daily active users (DAU)
• Customer retention rate
• Net Promoter Score (NPS)
• Feature and product adoption
• Customer satisfaction score (CSAT)
• Customer lifetime value (LTV)
• Customer acquisition cost (CAC)
• Customer conversion rate

These are powerful tools, and they are essential to understanding and shaping a product. So what is the problem? Several of them are lagging indicators or reactive, such as NPS, CSAT, retention, and lifetime value.

What are leading and lagging indicators?

Amplitude’s explainer on leading and lagging indicators draws the line this way: leading indicators tell you what is likely, lagging indicators tell you what already happened, and you check the first against the second. Even the definition of an active user varies by business, according to Amplitude’s Guide to Product Metrics (PDF), so write down how you count each one.

Why does relationship NPS arrive months after launch?

Relationship NPS shows the lag clearly. Bain & Company’s Net Promoter System site describes a relationship survey as one sent with no triggering event, often once or twice a year. A first full reading can therefore arrive months after launch. By then the product is already in customers’ hands, and the gap between the score you aimed for and the one you got is hard to close. Many things besides the product shape satisfaction, such as the purchase and support experience, and you are held accountable for things you could not have accounted for.

Recommended reading: How to Use Net Promoter Score in Your Beta Program.

What can your user testing tell you?

So how do you know whether people will like your product, and how do you close the gap between what you think you know and how it all turns out after launch? What role can metrics play before star ratings, reviews, or support tickets arrive? There is a practical answer, and it calls attention to the issues you can fix before they reach your satisfaction scores. If you already run user tests, you are halfway there. The future of product management metrics is deepening the insights you get from them.

Product managers take the blame when a product fails and share the credit when it wins. User testing eases some of that pressure, because it shows you how real customers react before launch.

If you treat your tests only as a gut check for glaring issues, you miss the chance to learn how well the product is likely to do. Exploring your user testing metrics gives you four things:

• A read on how the product is likely to be received
• Confidence that you are catching the most critical issues
• A way to see how what customers tell you changes what you build
• Proof of how customer feedback is shaping your product

Why do so many teams struggle to track them?

Many people who run user tests say they cannot show what the tests are worth. Centercode’s Delta Testing Industry Report 2021, built on a December 2020 survey of 124 people who run user tests, found that 49 percent say they spend too little time demonstrating the value or ROI of their tests, and that about 1 in 3 has issues proving the ROI of their testing efforts. It also found that 59 percent now use three or more tools to collect feedback. Without one home for your test data, you pull numbers from spreadsheets, tester emails, and several tools, and that takes time you do not have. With no clear sense of what to measure, counts fill the gap: bugs collected, bugs fixed, surveys completed, and active testers.

What is a better way to measure?

Without earlier measures, product managers fall back on one of two habits, and there is a better third path:

1. Nothing. You measure little and feel uneasy about it.
2. Vanity metrics. You report numbers that look good in a presentation but say little about the product or its customers on their own, such as bugs found or surveys completed.
3. A few meaningful measures. You track a few measures that deepen your understanding of your product and your customers, so you can make your case, bring everyone who signs off along, and solve problems on new and existing products.

The third path takes two things. The first is a small set of core user testing metrics. The second is automation and reporting that does the math for you, so you can run tests alongside your other responsibilities. Because the numbers come from your own testers, they are finer-grained than whole-experience measures like NPS or CSAT, which makes it easier to build focused fixes and rank what to fix first.

Recommended reading: What Most Teams Don’t Understand About User Testing Metrics.

What are the three Delta scores?

The three scores are Delta Health, Delta Success, and Delta Impact. Each answers one question, and each builds on the one before. Delta Health tells you how far to trust your results. Delta Success tells you how the product is doing in your testers’ eyes, and it shows as a range until Delta Health reaches A+. Delta Impact compares Delta Success with where it would stand had you fixed nothing. All three can be read for a whole project, a single phase, or a single feature, and in the Centercode Platform they sit at the top of every project’s dashboard.

An earlier edition of this guide called Delta Health the Delta Health Score, Delta Success the Product Success Score, and Delta Impact the Test Impact Score. They are the same measures under the names the Centercode Platform uses today.

What is Delta Health?

Delta Health measures how much your testers are taking part, graded from N/A up to A+. It is set against the tester target you choose when you create the project, and the Centercode Platform recommends one from what you tell it about the project. Half of the grade comes from activities completed and half from feedback submitted, so a few testers filing a lot of feedback cannot carry it alone. Delta Health grades the test, not the product. A low grade means you do not have enough data to be confident yet, and an A+ does not promise five stars at release.

Taking the temperature of a user test is hard. Without a score, you do it by gut feeling or by comparing against past tests, and neither helps when you have little time to deal with a problem like low engagement. Delta Health makes it plain when a project needs your attention: a feature that needs more feedback, a phase that needs more time, or testers who are blocked. If a feature is still graded D+ three weeks in, find out whether testers are stuck or have nothing to tell you. If they have nothing to tell you, move it to a later week of testing. Delta Health pays off in three ways:

• A pulse on your test. When Delta Health is strong, you can turn to other work with confidence. When it is not, you know where to steer tester attention.
• Proof for decisions. Areas with strong Delta Health give you solid ground for decisions you can defend to the people who sign off.
• A yardstick across projects. The same grade reads the same way from one project to the next, so it is easier to improve how you run tests.

Recommended reading: How to Measure User Testing Engagement.

What is Delta Success?

Delta Success is how the product is doing in your testers’ eyes if you released it today, from 0 to 100. Issues and ideas pull the score down and praise lifts it. Fixing an issue or building an idea removes its hit to the score. Each piece of feedback counts by its Impact Score, not by a simple tally, and you control the weighting for each feature, so the features that matter most move the score most.

Star ratings and NPS cover the whole customer experience, not the product on its own. That is why they cannot stand in for a read on the product before launch: they arrive after customers have the product, and they mix in price, the purchase, and the support experience. Delta Success does not replace them. It sits beside them and tells you about the product itself.

It is an estimate, and it tightens as Delta Health rises. It begins as N/A, shows a wide range once Delta Health has built up some data, narrows as Delta Health climbs, and becomes a single number at A+. Set a Success target for each product when you start, and the Centercode Platform shows what you need to do to reach it. Delta Success pays off in four ways:

• An early answer for the people who sign off. You can tell them whether the product is on track without waiting for reviews or support calls.
• Faster fixes. Data from your own testers makes it quicker to find and fix what is getting in people’s way, one feature at a time.
• A read on the product itself. The score reflects how testers feel about the product, apart from the purchase and support experience.
• Priorities by impact. Look at each feature to see how much it affects overall acceptance, then fix in that order.

Recommended reading: Measuring Project Success With Delta Health.

What is Delta Impact?

Delta Impact is the gain in Delta Success from the feedback you have addressed, shown as a percentage. It compares Delta Success as it stands, with the feedback you have addressed, against where it would stand if you had shipped without addressing any of it. When you mark an issue fixed or an idea implemented, its hit leaves the score, and the higher a piece of feedback’s Impact Score, the more Delta Success moves. Counting issues in against issues fixed is useful, but not every change helps the customer equally, and Delta Impact shows which ones did.

It is notoriously hard for product managers to show their value in numbers, especially when so much that shapes satisfaction is out of their hands and the product’s success is still on their shoulders. Delta Impact puts a number on what your work changed. It is a return measurement in the sense of what your fixes gained, not a figure in dollars.

Delta Impact pays off in three ways:

• Direction you can steer by. Movement in Delta Impact shows how well your effort is aimed, so you can change course while the test is still running and keep your team on the areas that pay off most.
• Proof whatever the market does. When sales swing for reasons you cannot control, you still hold numbers that show the difference your work made.
• A feature-by-feature view. Read Delta Impact for each feature to see exactly where you and your team made the biggest improvements.

Recommended resources:

• Centercode Key Performance Indicators: Impact Score
• The 3 Elements of Feedback Scoring

What is the difference between Impact Score and Delta Impact?

Every piece of feedback gets its own Impact Score, which ranks each issue, idea, and praise by how serious it is and how many testers share it. Delta Impact is the change in Delta Success that your fixes bought, readable for a project, a phase, or a feature. The two share a word and answer different questions: Impact Score ranks one piece of feedback, and Delta Impact reads what all your fixes added up to.

Why automate your metrics?

You could calculate all three scores by hand, but nobody has the time. Automating them changes what you can do with them:

• You decide in real time. Your tests are only as useful as the insights you take from them. Automated metrics show at a glance how many improvements you have made and how the test is running, so you know when to act and when to relax.
• Triage and math happen without you. Delivering these metrics by hand is tedious and slow. The Centercode Platform centralizes your data and does the math as feedback arrives, which frees your time for the rest of your job.
• The measuring stays consistent. Automation does not forget how it reached a number two months ago. It measures the same thing the same way every time, so you can set dependable benchmarks for tracking your progress and performance.

You can run whole user testing processes in the Centercode Platform, which keeps your data in one place and shows you where every test stands. The same thinking carries into UserVolley, which gives you three similar reads: how the product is doing, how engaged your participants are, and what your fixes changed. Our post on why automation is essential during Delta Testing goes deeper.

Key takeaways

• Several common product metrics, such as NPS, CSAT, retention, and lifetime value, trail behind or react to events, and relationship NPS can arrive months after launch.
• User testing already gives you the raw material for earlier measures, but only if you track a few meaningful ones and not vanity counts like bugs found.
• Delta Health shows how far to trust your results, Delta Success shows how the product is doing in your testers’ eyes, and Delta Impact shows what your fixes gained.
• Delta Success is a range until Delta Health reaches A+, and a high Delta Health grade is a grade for the test, not a promise about the product.
• Impact Score ranks one piece of feedback. Delta Impact is the change in Delta Success that your fixes bought. Do not mix them up.
• Automating the scores gives you real-time decisions, math off your plate, and consistent benchmarks.

Conclusion

Product managers have always had to judge a product through measures that arrive after the verdict. The three Delta scores move that judgment earlier. Delta Health tells you how far to trust the data, Delta Success tells you how the product is doing, and Delta Impact tells you what your work changed.

Track a few meaningful measures, let automation do the math, and you can show your team and everyone who signs off where the product stands while there is still time to improve it.

Want to see these scores on your own tests?

Centercode has run more beta tests, and more kinds of them, than any company, because we build the platform and run tests on it ourselves, so we know what these scores look like on real tests. If you want to see Delta Health, Delta Success, and Delta Impact on a test of your own, talk to Centercode about the three scores or take a look at the Centercode Platform.

Prefer to read it as an ebook? Get it from the resource library.