Drowning in data, yet starving for insights?
If you’re a researcher, this probably sounds all too familiar. You’re juggling user interviews, surveys, analytics, and competitor analysis. There’s never enough time to analyze everything. Synthesizing it all into actionable insights? Even harder.
So how can AI, the buzzword of our era, help? How can it help you reclaim your time and amplify your findings? Well, you’re in for a treat! In our latest expert webinar, we hosted Mike Glezos, a seasoned design leader and fractional design executive, and Jenna Harmon, Senior UX Researcher at Pantheon.
In their conversation with Marvin’s co-founder and CEO, Prayag Narula, they shared insights into leveraging AI as a “thought partner.” They also discussed strategies for stakeholder management, measuring experience quality, and streamlining research with AI UX tools.
Watch the entire conversation on accelerating your research impact with AI, or read ahead for key takeaways.
The Core Trio: Design Leadership, Research, and the Customer
The conversation started with the interplay between design leadership and research. “It’s a very tight relationship… because… the most important person in everything is that customer,” Mike said. Both design and research teams focus on the customer. They act as their advocates within the organization.
This highlights a crucial point: integrating research into product development isn’t just about adding steps. It’s about shifting the focus to the customer at every stage.
Mike laid out the foundational questions that every project should begin with:
- Who’s the customer?
- What problem are we solving?
- What’s the goal?
Seems obvious, right? But as Mike admitted, it’s common for projects to launch without this clarity. Jenna added that having a researcher in the room is valuable. A researcher can keep the team focused on the core questions, especially at the start. This underscores the importance of embedding research early in the product development lifecycle.
The Experience Quality Score: Beyond NPS
One of the most insightful parts of the discussion was about Pantheon’s “Experience Quality” (EQ) score. This was a project Mike and Jenna collaborated on. It addressed a common SaaS problem: inaccurate metrics for measuring the true quality of the user experience.
As Jenna explained, NPS and CSAT scores tell you something. But “they’re not good measures for… what customers and users are actually experiencing [and] feeling within… the product.”
The EQ score combines behavioral metrics (what users are doing) with sentiment metrics (how they feel about it). This approach acknowledges a few possibilities. Users might successfully complete tasks (high task success rate), but they may find it untrustworthy or aesthetically unappealing.
“It can be one thing [for customers to] come in, and the task success rates are like 90%. Like they’re very successful at doing the thing, but they hate it,” Jenna said.
The EQ score breaks down sentiment into four key areas:
- Usability
- Trust
- Appearance
- Loyalty
“Combining both these behavioral measures with these sentiment measures is what creates the EQ score for us,” Jenna explained.
It provides a more nuanced view and helps to bridge the gap between product experience and business goals. The EQ score serves not as a diagnostic tool but as a starting point for deeper investigation.
For instance, you might see a decline in trust scores soon after a new release or launch. However, you may also find that usability remains high. This triggers further investigation. Perhaps the new release disrupted workflows or removed crucial information. Thus, the EQ score helps in achieving consensus across the company on user problems and desired outcomes. These insights directly inform product decisions.
Design Crits: Research Joins the Critique
Mike revealed that Jenna actively participates in weekly critique sessions.
Jenna shared her enthusiasm for this practice, drawing a parallel to her academic background:
“Learning about weekly crits was probably the number one thing that made me most excited to start at Pantheon. It’s a credit to the kind of culture that Mike and the other UX leaders have at Pantheon that it’s also a very safe space… we feel very comfortable and confident being honest, but… supportive with one another.”
Researchers present works-in-progress. They get feedback on clarity and raise questions. Designers learn about upcoming research. They provide visual feedback. It fosters a strong, collaborative environment focused on craft. This regular exchange of information is a powerful way to integrate research. It makes research a continuous feedback loop rather than a one-off activity.
Stakeholder Personas: Know Your Audience (Inside and Out)
The conversation took an interesting turn. The focus became gaining executive buy-in for research initiatives. Mike dropped a truth bomb: We spend countless hours on customer personas. But how much time do we invest in stakeholder personas?
“If you were a lawyer, you would be trying to understand the jury, right?”
As lawyers study juries, researchers and design leaders should understand their stakeholders. Learn their priorities, pain points, biases, fears, and preferred involvement. This helps you anticipate objections, prepare responses, and make it easier for them to say “yes.” Creating a research-centric culture requires strong relationships. It means understanding the internal audience as well as the external one.
Jenna added another powerful tactic: “Asking them to have hypotheses before you start the research… Get them to have hypotheses at the beginning, [and] they are now invested in what it is that you find.” This shifts the dynamic from “we already knew that” to genuine engagement. Actively involving stakeholders in the research process creates buy-in. It fosters shared ownership of the findings, even before research begins.
AI as a Research Thought Partner
Mike acknowledged the current AI landscape. He called it “entertaining… creative… odd… alarming… intimidating.” Instead of focusing on the hype, though, Mike and Jenna grounded the discussion in practical applications.
Mike addressed the concerns of younger designers who worry about AI replacing their jobs. He emphasized that “somebody who knows AI can possibly take [their jobs] because they’re going to provide more value.” The key lesson? “Keep learning, investing yourself in education, gaining awareness, understanding what’s going on, and learning and digging more.”
Both Mike and Jenna stressed that they use AI as a tool, not a solution. Tactically, they use Marvin’s AI for:
- Interview Summary and Transcript Digestion: The AI helps them to summarize interview transcripts, make them searchable, and save them time.
- Research Repositories: AI-powered repositories, like Marvin, make research findings easily accessible. This happens across the organization, breaking down silos and democratizing research insights.
Jenna highlighted the transformative impact of AI on user interview analysis. She stated it “has dropped the threshold for how intensive, how labor intensive user interviews used to be.” She mentioned that for every hour of interviewing, she used to spend 4-5 hours on follow-up work. AI has reduced this drastically, potentially down to just 2 hours.
Jenna explained, “It is now probably down to two. And that is because we record the interviews directly in Marvin. And I know that there’s going to be an automatically generated transfer transcript, which means that I don’t need note-takers.”
Mike illustrated the practical benefit. He used Pantheon’s Weekly Customer Call program as an example. AI enables them to process and analyze a vast amount of information efficiently. This information comes from these calls, making customer insights readily available. These insights are offered to stakeholders across the company. He demonstrated how he could quickly search the repository. He searched for “mobile design” and could access summaries and relevant quotes for analysis.
AI in UX research can significantly enhance design and research workflows. Focus on tactical applications like interview analysis and research repositories. You’ll realize tangible benefits in efficiency and accessibility of insights. Continuous learning and adaptation are crucial. This is true in this rapidly evolving AI landscape.
Tackling Challenges with AI: Fewer Resources, More Impact
The conversation then shifted to the challenges facing design and research teams today. There are fewer resources, tighter budgets, and increased pressure. Teams must do more with less. How do you conduct research with limited resources?
Mike acknowledged the reality of layoffs. He pointed out the shrinking ratios of researchers to PMs. He invoked a quote from Alex Hormozi: “Rich people buy time, poor people buy stuff, ambitious people buy skills, and lazy people buy distractions.”
Mike connected this to AI. He emphasized its potential to “save you time and improve productivity.” He reiterated that AI should be viewed as a time-saving tool. This is especially true in specific tactical use cases.
Jenna echoed this sentiment. She emphasized that “AI tooling can help you save time if you know what it is that needs to be done, and you have the appropriate tool to do it.” However, she stressed something else. The most significant time-saving strategy, even more potent than AI, is “defining the problem.” Effective stakeholder management and ruthless prioritization are paramount. This ensures research efforts are focused on the most impactful projects. These projects should be aligned with business goals.
Jenna explained, “The biggest way to know that I am using my time appropriately and wisely and not getting spun up into some kind of swirl that is going to eat my time and not be impactful is exactly how Mike grounded this conversation at the very beginning. It’s defining what the problem is, who we are doing it for, who we are solving it for, and how might we go about doing that?”
In an environment of limited resources, AI offers valuable tools. They help with efficiency and productivity. However, strategic problem definition, ruthless prioritization, and effective stakeholder management remain the cornerstones. These are the foundations of impactful research and design, even more so than the latest AI tool.
Key Takeaways About Accelerating Research with AI
The webinar concluded with a dynamic Q&A session. It addressed pressing questions from the audience:
How would you gain approval for using AI in research, especially considering sensitive data and company policies?
Mike tackled this head-on. He emphasized the importance of understanding company policy and AI governance. He outlined three company stances: complete prohibition, permitted use with no sensitive data, and progressive approaches. These approaches use walled-garden APIs like Google Vertex. He stressed the need to align with company policy. He also highlighted the need to adhere to ethical best practices. These include customer confidentiality, disclosures, and NDAs.
Jenna added that at Pantheon, they always obtain explicit verbal consent. This is to record interviews for research purposes. They also anonymize data when necessary. She also advocated for internal AI governance councils. Ultimately, government regulation is needed to ensure data safety and ethical AI use.
Can you share any resource articles on how to measure EQ?
Jenna pointed to the SuperQ article by Jeff Sauro. This provides the survey instrument for measuring sentiment. She reiterated that the behavioral component relies on understanding your product’s user journeys. It also requires defining task success metrics. Excitingly, Jenna hinted at a potential blog post. It will be on the EQ Score methodology in the future!
What are some of the ways you can train AI to understand your stakeholders?
Mike suggested training AI models (like ChatGPT or Gemini) in separate threads. These threads should be dedicated to specific stakeholders to continuously feed the model information. This should be about stakeholder decisions, rationale, and communication styles. He emphasized that the AI model’s effectiveness hinges on the quality of training data.
Jenna added the tip of treating stakeholder meetings as mini-user interviews. Observe their questions, information preferences, and communication styles. This helps to refine your understanding and anticipate their needs.
If you were to define one more metric of work experience that AI features like Marvin have impacted, what would they be?
Jenna circled back to the user interview analysis time. She quantified the dramatic reduction in time spent on qualitative data analysis, specifically for user interviews. This is due to AI-powered features like automatic transcription and summarization within Marvin. This time-saving allows researchers to focus on deeper insights. It extends research visibility across the organization.
Words to Live By
Mike and Jenna left the audience with powerful closing thoughts as the webinar wrapped up.
Jenna emphasized problem definition and stakeholder alignment as the cornerstones of impactful research.
“It doesn’t matter if you’re running late in class analyses; it doesn’t matter if you’re doing user interviews. Know what questions you are answering. Be clear and aligned on the problems that need to be solved and who you’re solving them for. That’s where good research happens.”
Mike highlighted the imperative of AI literacy and the importance of viewing AI as a collaborative tool for design and research professionals.
“Invest in education. Invest in your AI education. Learn or get left behind. Gain awareness. Understand the space, at least at a rudimentary level. And get proficient at, you know, your craft. And treat AI like your thought partner.”
The Bottom Line
From the EQ Score to stakeholder personas to AI-powered research tools, the key message was loud and clear: Focus on the fundamentals. Understand your users, define the problem, and embrace AI as a partner. Do this in your journey to create exceptional user experiences.
This event blended the essential principles of design leadership and research with the potential of AI. Mike and Jenna offered theories, actionable strategies, and real-world examples.
Ready to accelerate your research with AI? Request a free demo to get started!
We look forward to seeing you at our next expert webinar!
