AI moderation has turned qualitative research from a project that takes weeks into one that takes days.

But “AI moderation” has come to mean a wide, wide range of different things. Not all “AI moderators” are built the same, and the approaches and quality vary. Here are seven questions worth asking.

1. What kind of AI moderator is it, and does it get the depth that you need?

Some tools labelled ‘AI moderator’ are really surveys with a handful of open-ended probes. They run through a script set in advance, largely accept whatever comes back and move on. Others, like Motives, respond to what participants actually say, follow up on unexpected answers, probe for the ‘why’ and adapt the conversation as it unfolds.

The format of the interview also makes a big difference to quality:

  • Text-based vs audio-only vs video+audio - Moderators that speak out loud with participants create a more natural feeling conversation, whereas text-based questions tend to get results more like a survey. We’ve found that participants give much more thoughtful answers when on video, speaking out loud back and forth with the moderator. At Motives, all of our interviews are conducted over video to achieve this higher quality.
  • Desktop vs mobile - Research participants responding on their cell phone tend to give shorter answers and are often distracted by being on the go or doing something else simultaneously.

Make sure to read some of the transcripts and watch some of the interviews for yourself to assess the quality. Compare multiple transcripts for differences to see how much the moderator is really diving deep on the right topics in each interview, versus following a strict guide. And look at how engaged participants seem, including how thoughtful their answers are.

2. Does it follow research best practices?

An AI moderator should behave like a seasoned qualitative researcher. Make sure that the platform supports research best practices - some good ones to check are:

  • If you’re showing stimulus to participants, does the platform support rotating the order in which stimulus is shown to participants to avoid bias?
  • Does the moderator run a high-quality interview: Are questions neutral? Does it ladder and probe for ‘why’? Does it avoid leading questions?

3. Are the results transparent and traceable?

The most important question here is whether you can trust the research, and how you would know. Being able to trace every finding back to the evidence from the fieldwork is how that trust gets proven.

At this point, everyone knows that AI can sound authoritative while being wrong, inventing details or citing sources that don’t support the claim. In research, the equivalent is a finding backed by a quote that doesn’t quite say what the summary claims.

When you evaluate a provider, ask:

  • Can I click from any insight to the quotes and clips that support it?
  • Do the quotes say what the summary says they say?
  • How many participants does a theme actually rest on?
  • Can I see the full discussion guide, the probing logic and the raw data?
  • How is the analysis checked, and what stops the model from inventing a pattern?

At Motives, every finding is backed by cited quotes from participants, so you can see exactly what was said and by whom. Reports also come with the video clips behind each insight.

4. Is there a research team behind it?

An AI moderator is only as good as the researchers who shaped it. Behind every strong platform should be a team of experienced qualitative researchers supporting you before, during and after fieldwork. At Motives, every study is supported by an experienced qualitative researcher, from guide design to the final report.

When you evaluate a provider, ask:

  • Who designed the moderation approach? Look for researchers with real qualitative experience, not only product and engineering teams.
  • Can I speak to a researcher about my study?
  • Can an experienced researcher help me plan when and how to use an AI moderated study vs other types of study?
  • How is the AI's output quality-checked? Ask what human review happens, and when.
  • How do you improve the moderator over time? The best teams test it against what an expert human moderator would do, and keep refining.

5. How good is the participant quality control?

The scale that makes AI interviews so powerful also makes participant quality more important to get right. Fraud is a growing problem in research panels. A 2026 NORC review cites industry estimates that 15-30% of market research responses are fraudulent.

Start by asking how the provider chooses their panels. Most panels were built for quant, where a few bad responses get averaged out. In qual, every participant can become a headline quote. Ask whether a provider sources and screens specifically for qualitative depth.

Then ask how participant identity and quality are verified:

  • Video verification. Is the participant on camera, and is that the same person who passed the screener?
  • ID checks. Is there document-level verification?
  • Human and AI vetting. Does a person review responses for signs of fraud, or is it automated end to end?

Video interviews raise the bar, but they don’t make fraud impossible. At Motives, we’ve spent an enormous amount of time vetting each panel we work with, using a mixture of automated and manual checks. We once caught a participant who had completed the same interview three times under different identities on the same panel - so we continue to carefully vet our partners and the individual studies we run.

6. Does it work for your stakeholders as well as your team?

Insights teams and the people they serve need different things from the same platform, which means setting different access permissions for each project and each group of people.

Your insights team needs full control over guides, stimulus, transcripts and analysis. Your stakeholders need fast, trustworthy answers without having to dig through the raw data.

Look for role-based access that serves both.

7. Will it get through procurement?

As an insights lead, you're judging a provider on research quality. Your procurement, IT and security teams will judge it on something else, and they can stall a decision for months. Check these early: SSO, custom roles and permissions, data residency and GDPR, security certifications and documentation, and clear terms on whether your data trains AI models.

Final thoughts

We built Motives around these seven questions because they're the ones we'd want answered before trusting any research tool. Whichever AI moderator provider you choose, ask them to show you, not just tell you.

If you'd like to talk any of this through for your next study, we're always happy to help. Get in touch with us: hello@motives.ai