We Let an LLM Interview Our Customers — The Results Were Wildly Accurate13
We Let an LLM Interview Our Customers — The Results Were Wildly Accurate
By Dr. Elena Vasquez
AI Inspired
We built a system where a large language model conducts structured interviews with our customers, transcribes the conversations, and generates insight reports. No human interviewer. No recording equipment. Just a prompt, a voice-to-text pipeline, and a model with a PhD in pattern recognition. The results surprised us.
Customer Insight Accuracy
┌─────────────────────────────────────────┐
│ Human Analyst ████████████████████ 82% │
│ LLM Pipeline █████████████████████ 85% │
│ Random Guess ████████ 28% │
└─────────────────────────────────────────┘The Setup
We run a B2B SaaS product. Every quarter, our research team interviews 200–300 customers. The process is expensive: coordinators schedule calls, analysts take notes, and a senior researcher synthesizes findings. A single quarterly cycle costs us roughly $140,000 in labor and tooling.
The bottleneck was never the interviews themselves. It was the synthesis. Analysts heard the same 12 recurring pain points across 250 calls but still spent three weeks writing the report. And they missed patterns. Human memory is lossy. A nuance in call 47 gets forgotten by call 180.
So we asked: what if the model doesn't just transcribe, but understands?
Prompt Architecture
The interview prompt is not a single string. It is a layered structure:
Layer 1 — Persona. The model is told to be a calm, curious product researcher. Not a salesperson. Not a therapist. A researcher who asks follow-ups based on what the customer actually said, not a script.
Layer 2 — Question Tree. We give the model a decision tree, not a fixed list. If the customer mentions onboarding friction, the model drills into which step felt unclear. If they praise the dashboard, the model asks what specifically works. The branching is conditional, which means no two interviews follow the same path.
Layer 3 — Synthesis Contract. After the conversation, the model produces a structured output:
Verbatim quotes (3–5, verbatim, with context)
Latent needs (needs the customer implied but didn't state)
Emotional valence per topic (positive / neutral / negative)
Feature requests (explicit vs. inferred)
Churn risk signals (any language suggesting evaluation of competitors)
The model is told to distinguish between what a customer said and what they meant. This distinction is where most human analysts add value, and it is also where they most often introduce bias.
The Pilot
We ran 40 interviews in week one. The model conducted the calls, generated the reports, and we compared them against the ground truth we'd collected from 40 human-conducted interviews on the same customer segment.
Task Human LLM Δ
─────────────────────────────────────────
Quote accuracy 96% 94% -2
Latent need detection 71% 78% +7
Emotional valence 68% 74% +6
Feature request extraction 84% 88% +4
Churn risk flags 55% 63% +8
─────────────────────────────────────────
Weighted overall 75% 80% +5The LLM wasn't just matching human performance. It was beating it on the tasks that require consistent pattern recognition across many data points. Humans are better at a single conversation. Models are better at across conversations.
What the Model Caught That Humans Missed
Three findings stand out.
1. The "quiet dissatisfaction" cluster. Across 12 interviews, customers used the phrase "it's fine" when describing a core workflow. A human analyst reads this as neutral. The model, trained on thousands of support tickets and review texts, flagged it as a low-arousal negative. These 12 accounts all ended up churning within 60 days. The human report had categorized them as "satisfied."
2. Feature requests that were really complaints. A customer said "it would be nice if the export format was CSV." The model noted the underlying need was interoperability with their data warehouse, not a format preference. The human analyst logged it as "CSV export request." Different granularity. Different product decisions.
3. Competitor mentions buried in tangents. Customers rarely lead with "we're evaluating Competitor X." They mention it in passing. "Oh, we looked at their tool last year." The model caught 9 of these. The human notes caught 3.
The Failure Modes
This wasn't a clean win.
Over-interpretation. The model would infer a latent need from a single ambiguous sentence. We added a confidence threshold: the report now includes a probability estimate per inferred insight, and only insights above 0.7 make the final report.
Tone calibration. The model occasionally came across as too formal for a customer who was being casual. We adjusted the persona prompt and added a style-matching instruction: "Match the customer's register. If they're informal, be informal."
Hallucination in quotes. In 2 of 40 interviews, the model slightly paraphrased a quote and presented it as verbatim. We added a verification step: the pipeline checks that each quoted string appears in the transcript.
Language. The model handles English fluently. For our Spanish-speaking customers, accuracy dropped from 85% to 68%. We're building a multilingual pipeline for Q3.
Cost and Time
Metric Human Team LLM Pipeline
──────────────────────────────────────────────
Time per 250 interviews 21 days 4 days
Cost per cycle $140,000 $12,000
Analyst hours 640 48
Report turnaround 3 weeks 36 hoursWe still need humans. A senior researcher reviews the model's reports, validates the insights, and writes the narrative that goes to the product team. But the data collection and first-pass synthesis is automated. The human role shifts from note-taker to editor.
What This Means for Research at Scale
The fundamental constraint on customer research is bandwidth. A team of five analysts can do 300 interviews per quarter. A model can do 3,000. And because the model doesn't fatigue, doesn't confirm-bias, and doesn't forget call 47, the quality of the synthesis doesn't degrade as the dataset grows.
This doesn't replace human judgment. It replaces human labor. The questions we ask, the segments we choose, the hypotheses we test — those are still human decisions. But the tedious, lossy, expensive middle layer — transcribing, categorizing, aggregating — that's what we handed to the model.
The result: we run interviews continuously, not quarterly. We track sentiment drift in near-real-time. We catch a feature regression in the voice of customers within a day of the release, not six weeks later in the next research cycle.
The model isn't a better analyst. It's a consistent analyst. And at scale, consistency is the feature that matters.
Continuous vs. Quarterly Research
┌──────────────────────────────────────────────┐
│ Data points / quarter Human: 300 LLM: 3000 │
│ Insight latency Human: 21d LLM: 1.5d │
│ Cost per insight Human: $467 LLM: $4 │
│ Pattern consistency Human: 0.71 LLM: 0.89 │
└──────────────────────────────────────────────┘We let an LLM interview our customers. The results were accurate, cheap, and fast. The wild part wasn't the accuracy. The wild part was how good the model was at the things humans do inconsistently.
Dr. Elena Vasquez
