🧠 How One Clever Prompt Helped Us Rescue Churning Customers

By Dr. David Jones, PhD in Artificial Intelligence

Have you ever noticed how customers rarely give a long, detailed farewell speech before they cancel? Instead of writing a thoughtful essay explaining why they're leaving, most people just send two words: "Want to cancel." It's efficient, it's polite (barely), and it gives your retention team almost zero information to work with.

We ran into exactly this problem at our company last year. Our churn rate had crept up to 14% over six months. The customer success team was drowning in cancellation requests, and every single one looked the same — a short message saying they wanted to cancel, sometimes with no reason given. We were reacting after the decision was made, not helping prevent it in the first place.

Then our data science lead, Marcus, did something that sounds almost too simple: he wrote a prompt for our internal LLM that analyzed the full conversation history of each customer interaction and identified which accounts showed early behavioral signals of churn. Not after they'd already asked to cancel — before. The results were striking. Let's walk through how it worked, why it matters, and what you can steal from this approach.

The Problem With "Want to Cancel"

Before we get into the prompt itself, it's worth understanding why traditional churn prediction is so hard in practice. In academic literature, customer churn modeling typically relies on structured data: usage frequency, payment history, ticket volume, feature adoption metrics. These are clean, quantifiable signals that a logistic regression or gradient-boosted tree can chew on without breaking a sweat.

But the qualitative side of the customer relationship — tone shifts in support tickets, the way someone's questions get more specific over time, whether they start asking about "alternatives" or "competitor pricing," the subtle change from enthusiastic exclamation marks to dry one-word replies — this rich signal is locked inside unstructured conversation logs. And most companies have terabytes of these logs sitting in their CRM or ticketing system, underutilized.

That's where a well-crafted LLM prompt becomes a force multiplier. Not because the model is doing some magical statistical inference, but because you can ask it to read like a human account manager would and then structure its observations into something your team can act on.

The Prompt Itself

Here's essentially what Marcus wrote (I've simplified for clarity):

You are an experienced customer success analyst reviewing the full 
conversation history of [Customer Name]'s account over the past 90 days.

Analyze all interactions and identify:

1. TONE TRAJECTORY: Is there a shift from engaged/enthusiastic to 
   neutral or disengaged? Cite specific messages that mark the turning point.

2. USAGE PATTERN CHANGES: Are they using fewer features, asking about 
   basic functionality they should already know, or reducing session frequency?

3. COMPETITIVE SIGNALS: Any mention of alternative tools, pricing comparisons,
   or "just evaluating options"?

4. STAKEHOLDER SHIFTS: Has a different person taken over the account? 
   Is there evidence of an internal handoff at their company?

5. UNRESOLVED ISSUES: Any open tickets, unanswered questions, or 
   complaints that never got a satisfying resolution?

6. COMMITMENT LANGUAGE: Have they made future plans with us (roadmap 
   alignment, team onboarding) or has that language disappeared?

Rate overall churn risk as LOW / MEDIUM / HIGH with a one-paragraph 
justification. Then list 3 specific account manager actions to take this week.

A few things make this prompt effective:

• It's role-specific. Telling the model "you are an experienced customer success analyst" anchors it in domain knowledge rather than generic NLP.

• It asks for evidence, not just labels. Requiring cited messages means the output is auditable. Your account manager can actually go read those lines and verify the analysis.

• It's structured for action. The final line isn't "what's your opinion?" — it's "list 3 specific actions." This turns a reading-comprehension exercise into an operational to-do list.

The Results

We ran this prompt against our top 200 accounts (by revenue) every Monday morning, and the output fed directly into the account manager's weekly pipeline review. Over eight weeks:

• 37% of high-risk flags were confirmed — the customer either downgraded or canceled within two weeks

• 61% of medium-risk flags showed measurable engagement drops in the following week

• Account managers reported feeling "less reactive" — they had a structured reason to call rather than guessing

The churn rate for those 200 accounts dropped from an annualized 18.4% to 11.2%. Not a revolution, but for a B2B SaaS company where your top 5% of customers drive roughly 60-70% of revenue (a pattern that holds across most subscription businesses), a ~40% relative reduction in churn among the key accounts is worth a small house in a decent neighborhood.

Why This Works When Classical ML Doesn't (As Well)

Now, I want to be precise here, because we're all too eager to crow about "AI magic" and not precise enough about which part of the pipeline is doing the work.

A gradient-boosted tree trained on usage metrics will outperform an LLM at predicting churn when you have clean structured features. That's well-established in the literature (think: Churn Prediction with XGBoost papers, there are hundreds). What the LLM adds is interpretability of unstructured text and actionable specificity. A GBDT tells you "account 4821 has a 0.73 churn probability." An LLM tells you "Sarah's tone shifted from collaborative to transactional after ticket #3402 was closed without resolution, and she started asking about [Competitor] pricing in the March 12th call."

The second output is what an account manager can actually do something with. And that distinction — between a probability score and a narrative with cited evidence — is where the practical value lives.

A Few Practical Caveats

Let's be honest about the limitations:

• Cost scales with context window. If your customers have 200+ messages of history, you're looking at meaningful token costs per account per week. We batch-processed on a schedule rather than in real-time to keep this manageable.

• LLMs can hallucinate evidence. We always include the cited message IDs and let account managers verify before acting. Treat the output as a strong hypothesis, not ground truth.

• This is a supplement, not a replacement for structured churn models. Use both. The GBDT catches the statistical patterns; the LLM reads between the lines of the conversations that got left out of your feature set.

What You Can Do This Week

You don't need an ML team or a GPU cluster. If you have access to any decent LLM API and a way to pull conversation history from your CRM or ticketing system, you can build a simplified version of this in an afternoon:

1. Pick your top 50 customers by revenue

2. Pull their last 90 days of interactions (tickets, emails, call transcripts)

3. Write a prompt structured around the six dimensions above

4. Run it weekly, feed outputs into your pipeline review

5. Track which predictions came true and refine the prompt accordingly

The prompt will get better with iteration. That's the beauty of this approach — it's not a one-shot deployment, it's a collaborative tool where your domain experts (your account managers) keep teaching the model what to look for.

The Bigger Picture

What I find genuinely exciting about this pattern is that it represents a shift in how we think about LLMs in business operations. We spent years treating them as either "chatbots" or "code generators." But their real superpower might be structured reading of unstructured information at scale — doing the work of 20 account managers each skimming 4,000 conversations a week to catch subtle tone shifts that no dashboard would surface.

It's not replacing your team. It's giving your team eyes everywhere, all at once. And in a business where a single key account walking away can dent your quarterly numbers, that kind of early warning isn't a luxury — it's infrastructure.

So next time you see "want to cancel" in your ticketing system, maybe the real story is in the 47 messages before it. You just need someone — or something — willing to actually read all of them. 📊🔍