What Product Managers Forget About User Feedback

Written by: Ty Magnin Ty Magnin 

Keep tabs on our content. One or two emails a week:

In every meeting Jeff Bezos attends, there’s an empty chair among the C-suite employees and board members to represent the customer. The idea is to remind decision-makers that customers can’t speak at the meeting, but the company still has to prioritize them.

For PMs at the drawing board, it’s easy to get lost in the numbers of usage behavior and statistics. They don’t look at the empty chair, but they still need to consider what the customer would think.

Too many PMs ignore this. With so much behavioral data out there, what’s the use in pestering the customer to ask for insight? PMs are afraid to contact their customers through email, NPS surveys, or in-app messaging, too worried they’ll annoy users, or get skewed data since only their happiest or unhappiest customers will respond.

Asking your users for feedback sounds like a great way to give the customer a seat at the table. In fact, Sean Ellis, founder of Qualaroo and GrowthHackers, found that having the customer’s voice “heard” through qualitative data boosted conversion by 300%.

For PMs, your user is the best possible resource for gaining insight into how people are (or aren’t) using your product—you just need to know how to approach the process and analyze the results. 

Get the Timing Right

The first step to getting quality feedback from users is to know when to ask. The timing of your request colors the data you get, so it has to be carefully calculated in order to get honest responses and avoid bugging users.

Asking for information before a customer even starts can feel like a risk, but it can provide valuable data about who is signing up. There will be a segment of users that sign up, but never really use your product. They were interested, so it's worth trying to understand their motives.

The language learning app Babble approaches this well by showing new users this screen before they can use the product.

babble-welcome-screen.png

It’s a barrier to entry, but a really low one. If a user wants, they can skip this survey. Most don't, however, since the expectation is that it will be short. Plus, the design makes it feel like onboarding, not a data grab. For additional guidance on how to phrase requests for feedback, take a look at Zapier’s guide on how to ask users questions.

Another common time to gather insight is when customers leave. If they’ve churned, it means there’s room for improvement and getting qualitative feedback is a great way to figure out exactly what that is.

The concern for PMs here is that there’s no real incentive for churned users to give feedback. While some people get around this by offering actual incentives, Harvest does this by simplifying the process. Rather than a series of open-ended questions on the landing page, they show users this list:

harvest-exit-email.png

What’s smart about this survey is it literally only takes one click to fill out—users don’t have to fill out bubbles and submit or anything. It’s low-friction, making it incredibly easy for users to provide valuable feedback.

Other apps, like Shopify, will ask for feedback from active users. By framing the questions around helping the user — and offering an $800 prize to one of the respondents — they get users to want to fill out the survey.

shopify-survey.png

How to Analyze Results

If you’re asking the audience for feedback in order to tap into one voice of the customer, it’s crucial for PMs to remember that not all customers are created equal. PMs need to know how to properly analyze the results they get from asking users for feedback. This starts with knowing how to interpret the data via segmentation.

Odds are, you have a bunch of different kinds of users. There are people who use your product every single day, and those who tune in every once in a while. There might be users who pay big bucks for your service, and others who are on a freemium plan. If they’re all answering one survey, or one question, their answers shouldn’t be given equal weight.

Each of these customers will have different needs and jobs-to-be-done.

Let’s say you’re using the average of all your Net Promoter Scores (NPS) to gauge how happy everyone is. PMs need to remember that an NPS is just an average. You might have several cohorts within that number—some that are really happy, and others that are unhappy. The average is a really nuanced figure, and you really need to know what you’re looking for.

A much more powerful way to analyze the data is to introduce segments to get granular with your analysis.

Break your data down into segments

Segments are useful because there’s not just one, holistic voice of the customer—there might be 2, 3, or more. By introducing segments when asking users for feedback, they can think about product changes that help specific user personas and address granular problems. 

As the experts at Wootric have discovered, NPS scores are much more accurate and meaningful when they’re segmented, not just an average of all customers. As they write:

“It’s unlikely, after all, that promoters and detractors will be evenly dispersed among all the segments, or that they will have the same desires or delights.”

That’s why it’s important to segment and filter, and then look at the breakdown within there. Wootric can automatically recalculate your NPS score based on segments and filters that you check off. That means you can tell that your NPS score for individuals on the Lite plan is 41, but much lower for enterprise companies.

wootric-nps-survey.png

Looking at segmented responses tells you that there’s a gap between these different personas. You can then try to understand and fix that gap by looking at their qualitative responses. Finding the subdivisions that make the most sense isn’t that difficult, and doesn’t even require calling up a developer. You can break users down into really specific segments, like:

  • New users on the premium plan
  • SaaS companies with >$5 ARR
  • SMBs that use the app 3-5 times per week

Before you change an entire product based on what one persona said, you need to take into account a holistic view of the voices of all your customers (or at least, a more comprehensive one). 

How to Implement What You’ve Learned

Successfully implementing feedback is one of the trickiest parts of product development. Utilized properly, talking to your customers allows you to validate your ideas and cost-effectively plan product improvements. Done poorly, it bloats your product with feature creep, makes it unwieldy, and detracts from its ability to get the job done. 

Without aligning the way you use customer feedback to your larger product vision, it’s easy to tumble down the rabbit hole of bad data. Airbnb co-founder Joe Gebbia points out:

“I'm not sure how useful data is if you don't have meaningful scale to test it against. It may be misleading. The way that we do things is that if we have an idea for something, we now kind of build it into the culture of this idea that it is okay to do something that doesn't scale. You go be a pirate, venture into the world and get a little test nugget, and come back and tell us the story that you found.”

As Janet Choi at Customer.io says, the real value in asking for feedback comes from “how you digest and address survey responses and resulting conversations — whether it’s a simple thanks for feedback, solving customer problems, or delving further for more information.”

intercom-product-roadmap.png

Is the Customer Always Right?

PMs often wonder how much they should trust users. Just because a ton of your users want something doesn’t mean you should build it, especially if you’re waiting on user testing data to back up feature roadmap prioritization.

Of course, user testing and data is hugely helpful, but customer responses still provide insight. As the folks at Atlassian say, “You can make a strong case to prioritize these features in your backlog with all of the data, but keep in mind that NPS data will not tell you what to build next— it is one input of many at your disposal.”

They sort NPS surveys by issues the customers are commenting on. Each mention merits a point. They then assemble a chart that looks like this:

Atlassian-nps-survey.png

By looking at how pressing the categories are (and how happy the customers are who comment on them), PMs can get a gut sense of what customers are thinking and feeling. It might not be the compass metric for guiding your product roadmap, but it’s certainly worth considering.

Intercom doesn’t allow customer comments to dictate the product roadmap, but this feedback definitely shapes product development. The company has its PMs read up to hundreds of customer conversations a week, and create a “hit list” of common customer problems that they need to address.

Intercom’s whole business is a platform for communicating with customers. If you aren’t Intercom, you’ll need a way to ask for feedback in your app.

If you don't have a way of engaging with customers—either by teaming up with the customer success team or directly asking for feedback in-app or via an email—you’ll have no way of knowing what the most important common customer problems are. 

Avoid the “Squeaky Wheel” Syndrome

One problem PMs are all too aware of is what Cindy Alvarez, author of Lean Customer Development, deems the “squeaky wheel syndrome,” where the loudest voices are the customers who are the most upset with your product. One unhappy customer might have a lot to say, while 15 relatively happy customers might keep their thoughts to themselves.

As Steli Efti of Close.io says, the way to mitigate the “squeaky wheel syndrome” is to balance user feedback out with quantity. As he writes, PMs need to “listen to 100 people instead of 1. Extract trends and patterns from the breadth of feedback and insight you receive throughout customer development—shared pains and problems that your product can solve.” 

Feedback Gives Product Managers Perspective

As the startup Homebase found, they were quickly losing the customer’s perspective as they scaled. They used to know all their customers, and be small enough that they could call them up if they wanted to. But as they grew, this wasn’t possible. Now, even though customer opinions and feedback don’t guide their product roadmap, they play an important role in the product team’s day-to-day. 

As one employee writes, “We send the NPS feedback out to everyone on the Homebase team every week. It’s a great rallying cry for everyone on the team to remember why we’re here.” When it comes down to it, that’s what customer feedback can really be: a rallying cry, and a reminder of why everyone is really there.

PMs sometimes get overly tied up in product—it’s their job. That’s why talking to customers is super important. Not only does it provide PMs with the feedback they need to improve product and scale, but it shows them how people are actually putting what you've built to use.

user onboarding academy

The User Onboarding Academy

Everything you need to know about user onboarding in 7 weekly emails.

Recommended reading