Researchers, rejoice! Your days of manually carrying out research tasks are over. User research software promises to rid us of the drudgery of research.
Now a permanent fixture in the UX toolkit, user research software can streamline data collection, enhance analysis and foster collaboration (well, the good ones do). Furthermore, it allows researchers to focus wholly on insights related to the user experience.
We know what you’re thinking… No, this ISN’T our guide to choosing the right research tools (we already covered that, remember?).
This IS a guide on how best to integrate research software into your process. We share some DOs and DON’Ts for mastering user research software. Use these helpful tips to supercharge your productivity and ensure your research is ironclad.
Here’s a quick summary of what we’ll go over:
- Where Software Fits into the Research Process
- DO’s — Conduct Rigorous Research
- How Microsoft Organizes their Qualitative Data
- DON’Ts — Avoid these Pitfalls
- The User Research Software Landscape
Where Software Fits into the Research Process
Our team breaks down the research process into three main steps:
At each stage, researchers must deal with:
- Project & study management
- Participant management & recruiting
- Data preparation
It’s important to note: All UX research stage incur administrative overhead. User research software automates these clerical tasks, allowing researchers to focus on insights.
With a fictional example, we’ll illustrate how researchers can integrate user research software at each stage of the research process:
‘Mov3’ is a company that manufactures and sells fitness trackers. It sensed an opportunity to make improvements to its application to attract non-fitness enthusiasts.
Mov3 decides to conduct research to understand more about:
1. Potential Users. What are their behaviors, motivations and needs?
2. Market. What competition do they face? How do they position themselves?
3. Value. What value does the product create for users? What are they willing to pay?
How can Mov3 leverage user research software to conduct effective market research and reach deep insights faster?
The DOs of Conducting Rigorous UX Research
Define Strategic Objectives
We begin at stage 1 – Preparation.
Behind every winning product are a company’s research and product teams, completely in sync.
To achieve this, we suggest creating a research roadmap. A living document, research roadmaps are an organizational tool for managing and executing projects in line with a company’s strategy. It connects user research with business goals.
Mov3 uses a research repository as its research planning hub. A central location where employees can understand more about various research initiatives and studies.
For each study, Mov3 researchers detail the following:
- Background
- Rationale
- Objectives
- Design
- Methodology
- Statistical considerations
Equipped with this information, Mov3 can decide on the product features and functionality worth implementing in their new app. They choose processes that are quick and cost-effective, including a health assessment quiz aimed at sedentary folks. With access to real insights, Mov3 undoubtedly saves time, money and resources.
Accessible to everyone, a research roadmap creates organization-wide awareness. New research insights (in one place) build on foundational knowledge about customers. Moreover, it allows new team members to become aligned with the focus of a company’s research practice.
Use a research repository to document your work. Write it down and create a paper trail!
We can’t emphasize this enough.
Want to know more about your options when choosing a research repository? Head over to our comprehensive guide.
Recruit Representative Samples
Gone are the days of recruiting participants of a study one-by-one. Panel management software allows you to streamline participant recruitment and tracking en masse. Mov3 uses it to automate:
- Research participant recruitment and panel building. Mov3 had a simple screening criteria — they recruited participants who answered “NO” to the question “are you meeting the recommended 120-150 minutes per week of exercise?” Since their new app features are aimed at inactive individuals, they want to understand all they can about these potential customers. Mov3 assembled the right panel in no time.
- Scheduling. What a nightmare it
is(used to be). Coordinating everyone’s schedule and blocking their calendars made researchers feel like they were someone’s PA. Mov3’s panel management software uses triggers and actions to send out invitations and reminders to participants. Just provide an instruction, and let the application take care of all communications and data entry. Email and calendar integrations mean less typing, less back and forth and less admin work.
- Incentive Delivery. Companies often use incentives (such as gift cards) to get people to participate in studies. Using their panel management tool, Mov3 distributes a virtual $10 gift card to all participants. This saves themselves from a logistical nightmare (not to mention the time savings).
Establish specific criteria for the demographics you’re after. Panel management software allows you to quickly assemble, segment and track the right users, which means less noise and bias in your data.
Collect Data Consistently
Jump right into Stage 2 – Field Work.
Researchers can use two different approaches (and therefore types of user research tools) when collecting data:
Passive Data Collection
| Type | Description | Examples |
| Passive | Insights are collected on an ongoing basis, usually as users interact with your app or website. Passive insights are independent of any research project. | Product and behavioral analytics & in-app feedback. |
Mov3 continuously collects data about the number of users, free or paid subscribers, and user navigation within their application.
Automated passive insights can be invaluable for time-sensitive research. They also provide great coverage for further analysis. Mov3 can use a tool like Google Analytics to observe many user behavior scenarios — such as mobile phone interactions vs. desktop, resolution, connection speed, geographical region… You get the drift.
Active Research
| Type | Description | Examples |
| Active | Research tools used to conduct studies related to a specific research question | Interviews, surveys & usability studies |
Mov3 also sent out a survey to the fitness averse demographic that they zeroed in on. Turns out, their sample size needed to be large. When running a large-scale survey, a tool like Surveymonkey can streamline Mov3’s data collection and handle an extensive volume of data for analysis. Survey routing is when a survey adapts and changes course based on the answers provided. This ensures that many types of participants are covered.
These two approaches are NOT mutually exclusive — companies must rely on both. Researchers must remain consistent in:
- Choosing the right measurement method – Check out this list of Practical UX Research Principles. Then head over to our guide to determine whether you need quantitative or qualitative research.
- Standardizing research conditions for all participants – make sure the research environment is the same, limiting the influence of external factors.
- Applying methods consistently – if multiple researchers are involved, document your steps with discussion guides and templates to ensure they’re easily replicated. Define how specific behaviors are counted and use a carefully worded guide to ensure questions are delivered as intended).
Using research tools consistently for both active and passive research can reduce costs, save resources, and most importantly, gives you an ROI on the design and research work that’s gone into improving a product.
Discover & Share Findings Across the Organization
And finally, over to stage 3 – Post Game Analysis.
Once you’ve collected all the data, use design and visual thinking tools to organize and visualize your findings. Popular data visualization tools such as Tableau can help turn boring spreadsheets into informative graphs and charts.
Create visuals to enable a deeper understanding of customer pain points and feedback. Mov3 wants to map their feedback channels. The research team visualizes the feedback survey (from above), and finds that many people complete around 50% before they drop off. Surely this will create gaps in the collected data. It might be time for a survey redesign. Sigh.
To increase everyone’s awareness of the research process and findings, share your insights with peers and the rest of the organization.
Where are all these insights stored? How will they access the files?
As we’ve outlined, a research repository or ‘‘insights management tool’ (in fancy speak), contains research plans, interview or discussion guides. A repository also houses the recordings and insights that you’ve gained along the way. It’s the one stop shop for anyone in the organization who wants to
Communication is key — a good research repository makes it easy to share insights across an organization. It optimizes internal flow of information (some more fancy speak for you).
Continuously Train Staff to Update Knowledge
At Marvin, we believe user research is a team sport. Part of a senior researcher’s remit is to train junior staff in the practice. We recommend conducting training sessions to educate juniors, to reduce friction when they begin using any software.
Create documentation that establishes standard operating procedures (SOPs) for various studies. Templates and guides will help them as they learn the types of research methodologies and how to best leverage the software at their disposal. This enables them to conduct research the right way.
New researchers will have no trouble following the steps if you’ve laid them out clearly. Make these documents available to everyone on your repository software platform.
It’s imperative for researchers to stay up to date with the latest advancements in the field. It’s imperative to. Document your processes. Iterate, reflect, improve and constantly refine them over time.
What worked and what didn’t?
How Microsoft Organizes their Qualitative Data
Now a quick story about Microsoft. Mihaela Vorvoreanu and her research team ‘Aether’ sought to create a framework for companies to pursue responsible AI (RAI). They interviewed over 90 AI practitioners and amassed over 80 hours from interviews and focus groups.
See how Microsoft used Marvin to analyze hours of interviews to create the world’s first empirically-based RAI Maturity Model.
Using Marvin, transcripts were generated within minutes of completing video calls. Microsoft saved thousands of dollars when compared to traditional human transcription.
Qualitative analysis requires several iterations of refining tags, codes, labels or themes (all interchangeable). Traditionally at Aether, they used native Microsoft applications such as Word or OneNote to maintain the codebook.
With this particular study, there was inherent complexity in unwrapping the various themes that came out of discussion with AI experts. Themes were so intertwined that they had several overlapping codes. Marvin’s Analyze feature allowed them to easily consolidate tags and codes to maintain and update their codebook.
Check out our guide for the best ways to use qualitative tags in user research, expedite your analysis and reach insights faster.
Avoid These Research Software Pitfalls
Don’t Over-Rely on Automation
Automation is here to take the tedium out of our lives. Case in point — Google finished that first sentence for us. As AI and automation continue to permeate into our work, it’s essential to understand its limitations, and the dangers of relying too heavily on them without any human intervention.
Mihaela and her team at Microsoft Aether recently published a paper on the overreliance on AI. In addition to widely differing demographic and psychographic traits, each individual has different levels of:
- AI (or automation) Literacy
- Domain Expertise
- Task Familiarity
It’s important to stay objective and unbiased. Being aware of a software’s limitations keeps researchers alert to any errors it may make. Here are four types of bias to watch out for:
- Automation bias – tendency to favor recommendations from automated systems, disregarding information from non-automated sources.
- Confirmation bias – tendency to favor information that aligns with one’s prior assumptions, beliefs and values. Users believe AI uses logic and reasoning similar to their own.
- Ordering effects – changing the order of presented information alters user perceptions and decisions. If AI does well in your early interactions, you’re more likely to over-rely on it. If it fails to convince you, you’ll establish distrust in the technology.
- Overestimating explanations – detailed and convincing explanations can lead users to develop blind trust in AI, especially when they’re not experts on the subject.
Automation requires more, not less attention to training and onboarding, interface and interaction design.
You must know the point where the machine’s involvement stops, and humans take over.
Use Marvin’s guide to learn how to effectively integrate AI into your research workflow.
Don’t Neglect Data Security & User Privacy
Oil has long been ousted as the world’s most valuable resource. Its usurper? Data.
Today, data behind a users’ every click, keystroke, swipe or tap is collected and used in company decision making. Amassing data in droves is one thing. The 2018 Cambridge Analytica-Facebook debacle taught us that protecting data is a whole different ball game. The world’s largest companies aren’t immune to accidental leaks or data breaches or misuse.
Data security refers to the steps taken to safeguard the privacy, availability and integrity of sensitive data. Here’s a list of data security must-haves:
- Encryption – converting readable text into unreadable text using an algorithm.
- Data masking – obscuring data so it can’t be read. When data is de-identified, any indicators linking the data to someone’s identity are removed.
- Access control – authentication verifies the user is the actual user. Limits data access to authorized individuals only.
- Data backup – creating copies of files and databases to a secondary or tertiary location.
In qualitative studies, (unless responses are ‘anonymous’) the onus is on researchers to maintain confidentiality of a participant’s Personal Identifiable Information (PII).
At Mov3, they store people’s highly confidential demographic and health data. Overlooking data security and user privacy protocol has several repercussions.
Non-compliant companies face legal implications such as fines, litigation or worse, dissolution. Data leaks of sensitive information cause mistrust between customers and the company. This can cause irreparable reputational damage and eventually a decline in sales.
Ready for an acronym attack?
When choosing user research software, make sure it meets at least one (or more) of the following types of data compliance:
- GDPR – General Data Protection Regulation
- HIPAA – Health Insurance Portability & Accountability Act
- SOC2 – System and Organization Controls (AICPA)
Let your user software worry about blurring faces and de-identifying data. You worry about the research. Better safe than sorry.
Read more about different types of compliances here. And don’t worry. Marvin is HIPAA, GDPR and SOC2 compliant. Your user data is always protected.
Don’t Ignore Data Validation
In any study, data preparation is key. Ask anyone who works with data, and they’ll tell you they spend the bulk of their time vigorously cleaning and sanitizing datasets for use.
Due to its subjective nature, qualitative research is prone to use of incomplete or poor quality data. Incomplete or inaccurate data restricts its use. Companies form incorrect hypotheses (and therefore decisions) due to a lack of data validation.
A small but simple example – Mov3 collected participant’s dates of birth in their survey. Without appropriate data validation, 01/12/2013 can be read as Jan 12th 2013 or Dec 1st 2013.
Which one is it?
Cleaning up this mess requires more data manipulation and processing before the dataset can be used. Incorrectly classifying participants, in turn, causes incorrect conclusions and can render the research useless.
Data validation aims to remedy this by providing accuracy, cleanliness and completeness to a dataset.
By creating controls and data validation (a strict format: MM-DD-YYYY), Mov3 ensures no erroneous responses in their dataset and don’t waste any time in cleaning the data further before analysis.
A little data preparation goes a long way.
User Research Software Landscape: An Overview
Confronting the universe of user research software can be daunting.
If you’re building out a firm’s research practice, it’s key to establish the functionality that’s most important for your team.
What type of active research tools will you choose?
User Research Tools
| Tool Type | Description | Software |
| Participant Recruitment | Find participants for interviews and studies. | User Interviews, Ethnio, Ribbon, Respondent, Qualtrics |
| Surveys & Feedback | Create and roll out surveys on a large scale to collate user feedback. | SurveyMonkey, Typeform, Jotform, SurveyGizmo, Google Forms, Survicate, YesInsights |
| Interviews | Audio / video conferencing tools give you live feedback from users. | Marvin, Lookback, UserZoom, dscout |
| Transcription / Note-Taking | Generates transcripts from interviews, freeing up a researcher’s focus. Look out for these features. | Marvin, Rev, Reduct, Otter.ai, Trint, ConfirmKit, Evernote, Pear Note |
| User & Usability Testing | Users put products and their information architecture to the test. Companies can see how easy / difficult it is for users to navigate through a product’s features or website. | Lookback, PingPong, Maze, Loop11, UserTesting, Userlytics, UsabilityHub, Optimal Workshop, Validately, kardSort, UXarmy, Optimizely, VWO |
| Research Repository | Where all your insights live. Marvin’s comprehensive guide dives into the pros and cons of a generic versus a purpose built research repository. Use ‘SAAS ASPIRE’, our memorable acronym, to help you decide features crucial to your business. | A 2023 comparison of research repositories lives here. You can see the most current research repository reviews on G2.com. Rather obviously, we recommend Marvin (and so do users — we were the only Leader in G2’s Winter 2023 Research Repository category). |
| Product Analytics | Provide details of how users interact with your product. | Hotjar, Kissmetrics, Mixpanel, CrazyEgg, FullStory, Heap, Google Analytics |
| Data Visualization | Transform your data into engaging visuals to share insight nuggets with management and colleagues. | Tableau, Power BI, Looker, Plotly |
More resources for User Research Software
The table isn’t exhaustive. Take advantage of free trials — a try-before-you-buy period without any financial commitments.
Calling all fantasy fans and adventure aficionados — our friends at User Interviews created an interactive map of the user research software landscape. We suspect they’ve drawn inspiration from the Lord of the Rings / Game of Thrones. See if you can spot Marvin!
Navigating User Research Software Mastery
User research used to be heavily time and resource intensive. Senior researchers of yesteryear conducted every aspect of research manually for decades.
A plethora of new user research software has brought significant changes to the way they work. Researchers now have the luxury of evaluating and selecting tools that enhance and expedite their workflow. They’re spoiled for choice.
While machines perform some heavy lifting, human intervention becomes more crucial than ever. You’ll always need someone to steer the ship and ensure the research is of a high quality.
The bottom line?
Adopters of user research software reap the benefits. Studies cost less in terms of time, money and resources. Crucially, it frees up researchers and designers from admin work, allowing them to focus on gaining a deeper understanding of their users. Isn’t that the whole point anyway?
Photo by Aaron Burden on Unsplash
