Metrics PM interview questions

An overview of metrics questions in PM interviews, includes example PM interview questions, context on why they matter and sample answers.

Vlad Ilchenko, Senior Product Manager at Walmart Published: Jan 6, 2022

Key definitions | Interview context | Full case example

Overview of term definitions (Top)

Metrics questions are a super common part of PM interviews. And it can be intimidating to get hit with a "X metric is down Y%, how would you figure it out?" right off the bat. But don't worry, in this post, we'll lay out common types of metrics, sample questions and how to approach these with confidence.

First, let's clear up some confusion. There are lots of different types of metrics so let's quickly touch on a few key types and their purpose:

• North star metric
• Feature level metrics
• KPIs

Let's start with KPIs - quantifiable measures of performance to track key business activities. They are often keeping the lights on quarter after quarter and show the health of an organization going forward.

For example, product usage as a KPI may aim to keep a given average number of monthly users and track how close the actual number is to the targeted one (e.g., we want to hold 30-day retention rate at 50% or higher). Another potential KPI which is more financial in nature can be monthly sales or revenue numbers (e.g., we want to hit $100K in daily revenue). In either example, a KPI puts a stake in the ground that the team can measure their progress against.

Feature level metrics - this category is the most popular among interviewers and is often asked in success metric questions or product execution cases.

For example, you might be asked how you'd measure the success of a new feature launch (e.g., imagine you just launched the ability to post videos on LinkedIn, how would you measure the success of this feature?). Another example might be: your see that the number of direct messages sent in Instagram has spiked up 10% day-over-day (DoD), why do you think it's up?

North star metric - is a single ideal goal set by product leadership and understood by everyone who has an impact on it. The canonical example here is Facebook which early on set the % of users who had 10 or more friends in the first 7 days. They set this as their "north star" because they realized if a user connected with 10 friends in the first week they were extremely likely to get hooked (e.g., be a highly retained user). Thus, getting people to that point drove the business in the right direction.

Another example from a logistics team in the e-commerce industry may be OTIF: On-Time-In-Full delivery to customers, which means 100% of orders are delivered on time without defects or missing items. Here, this makes sense as well - if customers get the right stuff at the right time, it seems indicative that the logistics team is doing its job well!

Essentially, North star metrics help galvanize the team around a hyper-specific metric that's been determined to drive the business in the right direction.

Metrics in the interview context (Top)

Why do interviewers care about the candidate's comfort with metrics? The simple answer is that answers to metrics questions are a good proxy for the hiring manager to assess the candidate's understanding of key product management activities (e.g., measuring what's working in the product or not or investigating a decline in core metrics and usage).

In the interview context and generally in the earlier rounds of the recruiting process, metrics questions are usually asked by hiring managers ranging from the Senior PM level to the PM Director herself.

Some examples of common metrics questions include:

How would you measure performance of new feature X and why?

Metric Y dropped by X% - can you figure out why?

You have noticed that engagement for your website has gone up recently - which steps will you take to investigate the increase?

Imagine you're the lead PM for product X. What do you think a good North star metric to track would be and why?

Mini case study: "Clear Mind" mobile app (Top)

Let's discuss the metrics case study using the fictional late stage tech startup which developed a mental health tracking mobile application called "Clear Mind". The mobile application is targeting users who want to track their emotional well being on a daily basis and share weekly progress reports.

Currently, the mobile application has been downloaded and used by hundreds of customers. As an incoming Product Manager you see that recently user retention has dropped by 5%. Your task is to find the root cause for this dramatic change (NOTE: This is a KPI metric).

This is a super common type of metrics interview question which can be described as: "Metric Y dropped by X, % - can you explain why?" NOTE: Facebook, in particular, loves to ask interview questions like this.

For our startup to measure mobile app's performance, focusing on retention is important, because this metric can be considered a proof of concept which customers accept by returning to "Clear Mind" and spending their time there.

The PM's goal here is to run a structured, clear investigation to identify what's going on. The best way to do this is start with a framework.

Looking into retention or any other metric, the following framework would be a good start to search for the root cause.

1. Geography: is the recent drop in retention specific to any place or region?
2. Platform: in our case - do "Clear Mind" users show different behaviors on iOS vs Android platforms?
3. Channel: if the startup was not mobile only, a good practice would be to compare Retention on the Web vs Desktop vs Mobile apps. Or even if mobile only, are users acquired from a specific channel retaining worse (e.g., a poorly performing ad network).
4. Market: were there any new competitive products launched that appeal to our user base and cover similar customer needs?

If the interviewer answers "yes" to any of the above questions, that's a good hint to dig into that specific area further.

If not, you can dig in further using a more product centric framework to identify root cause(s):

1. New features: Were there any UX/UI improvements or major changes made to the application during the period in question?
2. Experiments: are there any active experiments running on the "Clear Mind" right now?
3. User base: do we see drop/ increase in the recently joined users (<30 days) or long-term users (more than 1 month) etc?
4. Data analysis: are we 100% sure the data integrity is trustworthy, data sources are validated and methodology for analysis is confirmed?

Let's imagine that the interviewer's answer to the last question sounds something like this: "There was a recent update to the methodology of calculating Retention: instead of looking into returning users by calendar days we switched to Retention on a 24hr increments basis."

This is great, because it can explain the recent drop in Retention by 5%!

For example, previously the user who logged in to the app at 23:50 on December 29 and then, 20 minutes later, again at 00:10 on December 30 was considered a returning user from calendar day perspective because she returned the next day.

If we change the measurement to the 24hr methodology, that same user has to login again between 23:00 December 30 and 23:00 on December 31. This condition would be harder to execute because now it is similar to the returning user on Day 2 from the calendar day standpoint.

As we see, asking the above questions of the interviewer can help the aspiring product managers narrow the scope of the case and come up with ideas on how to prioritize success metrics and which workstreams will be the most valuable to the young company.

💡 Got a PM interview? Our PM interview drills help get you in top form

Conclusion

In this article we looked into one of the popular types of PM interview questions: metrics questions.

First, it's important to understand the different types of metrics that you might get asked about - from north star metrics to high level KPIs.

Second, it's helpful to prepare and think about how you'd respond to the common types of metrics questions. Over time, you can begin to develop your own custom framework for you'd respond but to get started, feel free to use the above as a baseline.