Product Level 1 - Running effective customer interviews to validate your product hypothesis, no matter who you're talking to
Why it’s so easy to run bad customer interviews, and how to do it right
Customer interviews are an essential tool in allowing a product manager to understand the thoughts, goals, emotions, problems, and priorities of your customer personas and target audience.
There’s a fine line between running a good customer interview and running a terrible one, but the good news is that it’s not hard to learn how to get much better at them. By absorbing and understanding those differences, you’ll level up a critical product skill.
Customer interviews are an excellent tool to:
Find the right customer pain points to solve
Gain insight into how your customers experience your product
Learn what your customer’s goals and priorities are
So why are they hard?
Many readers are probably already familiar with The Mom Test: How to talk to customers & learn if your business is a good idea when everyone is lying to you by Rob Fitzgerald. If you haven’t already, definitely check it out - it’s a fast read, packed with actionable information, presented in easy-to-relate-to examples and concepts.
The core idea of the book is that if you were to interview your mother about your product or idea, she'd tell you how great it is, how excited she is for it, and how she can totally see it working. However, she's not the only one who will spoon feed you what you want to hear; most people, even strangers, are going to take the easy way out of an awkward conversation, and end up telling you similar things. If you let them.
Today’s level is about learning how to run customer interviews so that no matter who you interview, even if it’s your mother, you can come out with objective truth, and use that truth to validate or invalidate your product hypotheses.
Asking the Right Questions
The right questions stay away from opinions and hypotheticals, and are presented in open-ended ways. Good questions focus on past actions, are centered around goals, and probe into the problems encountered trying to reach those goals.
Bad questions are hypothetical, leading, and narrow ones that only allow you to get back very misleading data. If the person you’re interviewing knows what you want to hear, you’ll often end up hearing it. It’s awkward to say that your idea sucks, and so most people won’t, and will feign interest or agreement. Bad questions detach the conversation from reality, and render all of the feedback useless.
Here are some bad questions:
Do you think it’s a good idea?
Your customers aren’t experts at designing solutions, only at understanding their problems
Would you buy a product that did X?
Anything involving one’s future self is an over-optimistic lie
People will lie to you if they think it’s what you want to hear
How much would you pay for X?
Avoid hypothetical questions - they are not grounded in reality, or, in this case, the actual sale process
Would you use the product if it had X feature?
Another one that’s both hypothetical, and solution-oriented. Get back to the problem
Good questions, on the other hand, are open ended, ask for specific examples from the past, and let you dig into the what, how, and the why of the problems your user has. Good questions often don’t even need to involve your own product at all.
Here are some good questions:
How are you dealing with it now?
It’s rare for someone to tell you precisely what they’ll pay you, but they’ll often show you what it’s worth to them
What else have you tried?
If they haven’t looked for ways of solving it already, they’re not going to look for (or buy) your solutions
Talk me through the last time that happened
Learn from what they did, not what they “would” do
What are the implications of that?
Some problems don’t actually matter - always try to understand the impact of the problem, and your customers’ tolerance for it
Why do you do it that way?
Get past the first layer of the problem, see if there are deeper issues at the root.
Can you give me an example?
An excellent go-to whenever there’s a face-value type statement (eg. “I don’t like when this happens”), or you aren’t sure what they’re getting at - ask the person to put it into an experience for you
If you are testing out a specific solution your team has built or designed, bring interactive mockups, or something the interviewee can play with right in front of you. In this instance, try to do the following to ensure you’re staying away from opinions, and able to get good data:
Ask them to do a task, or a series of tasks
Note where they get confused, how they navigate things, and what problems they have
Ask them what they expect a certain action or button to do, then follow up with questions like “Why do you expect that?”, “What would that do for you?”, and “Why is that important to you?”
When seeing something for the first time, we all sort of subconsciously have our expectations, and often those expectations are grounded from where our wants and needs are. Just be sure to ask the follow-up questions
Create a Hypothesis, and then Try to Disprove It
Ideally, you should try to always have one or two product hypotheses that you want to test in your interview. This will help make sure you’re using your interviews to continue progressing towards or into a problem.
Then, in each interview, set your mindset so that your goal is to try to disprove your hypothesis as fast as possible, so that you can move on to the next one. While this feels backwards, framing it like this will keep you more honest and objective in your questions, as well as your perception of their answers. It will also help you stay less eager to accept the fluff, compliments, and politeness you might receive if you accidentally ask some bad questions. Nobody’s perfect.
Let’s look at a real world example of how this might work. Early in ClassPass’s history, after some initial funding, they had started hitting it off with a product they called the Passport, which allowed a user to sample a handful of different gyms in the area, once each, in order to see which one they liked best.
At this stage, ClassPass was operating under the hypothesis that “As a user, I want to try out several different gyms before joining one, so that I know I’m making the right choice.”
What they ended up finding was that some users were working around this limitation by signing back up under different email addresses, and paying a second time, or even a third time.
By running customer interviews against those users, ClassPass was able to disprove their hypothesis and learn something very important about their customers: these users didn’t want to join a single gym at all. They loved the increased variance and freedom that the Passport provided them so much so that they were manipulating the system in order to pay for such a service.
Further reading: FirstRound Review | ClassPass’ Founder on How Marketplace Startups Can Achieve Product/Market Fit
Running the Customer Interview
Heading into the interview, make sure to line up some good questions, and have them ready on the backburner. Your goal is not to simply get through this list of questions, but rather to have them ready to steer the conversation if it needs it.
Hopefully, your conversation ends up being a good dialogue. Ask questions, ask follow-up questions, just remember to keep them open-ended. When in doubt, start a sentence with the word “Why”, and you’ll probably end up putting a good question out there. If you ever start a sentence with the word “Would”, stop and rephrase.
Some general tips to running effective interviews:
Keep them short and focused
Start with 30-45 minute time slots. As you get better, and more effective at extracting what you need, you can even drop them down to 20 minutes.
Use helpful tools to reduce the friction around the logistics
Teams and Zoom are great tools for the video conference
Calendly makes scheduling much easier, and saves you both from the back-and-forth. I have found the free version very satisfactory
Record the meeting if possible
Especially mid/post-COVID, everybody is familiar with videoconferencing now, so use one of those conferencing tools to set up the call. It makes recording a breeze
Bring a notetaker (especially if you can’t record it)
Conversations go fast, and sometimes you might not catch all the details, or realize all of the importance of something in the moment
Ask a fellow PM or maybe your PMM to do you a solid and take notes or use it as a learning opportunity for an APM or someone new to the team
Turn your video on
Being able to see the person you’re talking to will allow you to get that much more data from your conversation, as you can see their facial expressions and reactions throughout
Have your video on to start the interview, and you’ll find that a lot of people will feel subconsciously obligated to turn theirs on as well
If after ten minutes or so they don’t, and you feel awkward about it, you can always turn yours off by pulling a casual move like, “My audio is getting a little choppy, I’m going to turn my video off to save some bandwidth”
Don’t pitch
If you ever feel yourself talking a lot about your own product or its features, stop, take a breath, and switch gears. You can even excuse it with a light hearted handwave such as, “Sorry about that, I got a little too excited about what we’re doing, and slipped into pitch-mode. Can you tell me more about how you…”
Focus on their problems, their workflows, their thoughts, their emotions
Talk less and listen more
Similar to the “don’t pitch” advice, if you ever find yourself talking more than the customer, you are most likely botching your interview, or at least you should be finding a way to get things back on track
Ignore fluff, and deflect / reflect compliments
Don’t let fluff bog you down, and be prepared to deflect or reflect compliments, distracting questions, and unrelated topics to either move them out of the current discussion, or use them to bounce back into it.
Some examples of how they could be used:
Deflect distracting questions that don’t have a clear path to disproving your hypothesis.
A questions about an unrelated part of the roadmap, can be deflected by promising to circle back to it at the end
A compliment can be reflected by asking them what interests them so much about that, if it is one of their top priorities, and why
Keep your hypothesis at top of mind
Make sure that one way or another, you’ll have furthered your understanding of your hypothesis
Try to save the last 5 minutes for a recap
Reiterate the main ideas that you’ve taken away from the call, and invite the interviewee to jump in if they think you missed something. This is a nice way to make sure you correctly identified the level of importance of the different things they talked about
How Many Customer Interviews is Enough?
As the Intercom guys put it, customer interviews are like flossing. Ask most product managers and they’d say they do it, but ask them to be honest, and they’d admit they don’t do them enough.
“A solution can only be as good as your understanding of the problem you’re addressing.”
- Paul Adams, VP of Product @ Intercom
There is no end to customer interviews, because every time you’re successful in disproving a hypothesis, it’s time to form a new one to test. It’s a continual evolution of your understanding of the problem space, the customer persona, or the opportunities in the domain.
It’s also how you, as the product manager, can make sure you’re constantly growing your product judgment, and remaining on top of shifting tides in your market.
Product judgment is essentially your ability to emulate your customer effectively, playing out multiple situations and futures of a user, workflow, or impact related to your product. This helps you to bring the voice, thoughts, and concerns of your customers into your product development process, and is the power of the product management role.
Ultimately, when we build things, we’re always making guesses and assumptions about what the right way to solve the problem is. Growing your product judgment helps you improve the probability with which you make good decisions for your product during the solution design, planning, and development phases.
Your product judgment is also always fixed in time - consider it a snapshot taken the last time you interviewed a customer. Every day thereafter, it becomes increasingly more stale. If you let this sit for long enough, you’ll miss major shifts in technologies, and their impact on industries as a whole, which brings about changes in the wants, needs, goals, and expectations of your customers and the wider market.
Even if your competitors miss these tidal forces as well, newcomers to the market will ride in on them, and they’ll catch the incumbents off guard and flat footed. Don’t get left behind.
Action Items
There’s no substitute for doing, and as a product manager, this is a tangible path to continue leveling up on multiple fronts. Make sure to take time to prepare beforehand, and review afterwards - note down what went well and what didn’t, and you’ll see yourself improving rapidly within just 2 or 3 interviews.
Try to set some goals for yourself around customer interviews, and if it helps you stay diligent about meeting those goals, share them with your manager or a colleague.
Some suggestions:
Difficulty: Novice - Try to schedule 2 customer interviews within the next 2 weeks
Difficulty: Beginner - Try to average 1 customer interview per month
Difficulty: Expert - Try to average 1 customer interviews per week
Difficulty: Guru - Try to average 3 customer interviews per week
If you enjoyed today’s level, and want to be notified of new ones, sign up here:
Sources and Further Reading
Note - I am not affiliated in any way with any of these organizations. I simply found these to be valuable reads during my own time spent learning about this topic
FirstRound Review | ClassPass’ Founder on How Marketplace Startups Can Achieve Product/Market Fit
ProductPlan | 10 Great Questions Product Managers Should Ask Customers
Product Tribe | Customer Interviews: When You Need Them and How to Make Them Efficient
Intercom on Product | Understanding your customer is key to good product judgment
Roadmunk | How to find the best features to build using the right customer interview questions
Congratulations, you have leveled up!
I hope you enjoyed the post, and I’d love to hear any feedback on the content, length, style, or ideas for future posts if you have any!