When users have questions, timing matters. The closer we can get answers to the moment they need them, the better their experience.

Knowing that, we pulled over 2,500 support tickets to examine when and where customers need help. Every time, our support team has done an incredible job helping the people who reached out—but they’re not stopping there. They’ve been eager to figure out how we can answer more of those questions before they ever need to hit the inbox.

So we dug into the data, combing through every ticket to find patterns, uncover the questions users ask most, and start translating what we learned into in-app guidance—so users get answers right when they need them, without having to ask.

Our goal is simple: reduce support tickets by answering common questions inside the product. But first, we needed to figure out exactly where people were getting stuck—and what questions were coming up the most.

Here’s how we're attempting to go from those 2,500+ tickets to a punchlist of key questions we can solve inside the product:

Step 1: Export and filter tickets to product-related questions

We exported every support ticket from the past six months and filtered out anything that wasn’t about using the product. That gave us a focused set of questions we could actually address with in-app guidance.

Step 2: Group tickets by product area or feature

We organized the filtered tickets into groups based on which part of the product they were about. This helped us see patterns tied to specific features or workflows.

Step 3: Summarize ticket trends with AI

We ran each group of tickets through AI to get a clear write-up of:

• What users were trying to do
• Where they were getting stuck
• How many tickets fell into that category

This turned hundreds of conversations into a digestible summary for each area.

Here’s the AI prompt we used if you want to try the same approach—it might need small tweaks depending on your dataset, goals, or where you want the AI to send or format the outputs:

AI Prompt Template

Review the attached case history file (a collection of customer support tickets) and summarize the top 2–5 product areas or features where users most commonly experienced issues.

For each of those key areas, provide:

• A clear summary of the main issue users encountered
• 3–5 bullet points describing the specific user needs, pain points, or questions represented in the tickets
• An estimate of how many tickets fall into this category (if possible)

The goal is to identify which product issues have the highest volume and impact so we can prioritize them for in-app help solutions. Write the summaries in plain, easy-to-understand language for use by a product and support team.

Text copied

Step 4: Identify the most common, high-impact questions

We used those summaries to spot the seven question variations that came up the most—and accounted for nearly 70% of all “How do I do X?” tickets. Each sticking point became a to-do on our in-app roadmap: Pins, tooltips, messages—whatever works best to meet users in the moment and solve those top questions.

Next up: building the in-app help. We’re mapping each question to the right Appcues pattern, getting them live, and tracking what changes. We’ll be watching ticket volume closely to see if we’re moving the needle—and making adjustments as we go.

We’ll share what we build (and why) in the next post. Subscribe to stay in the loop.