I Fed Our CRM to an LLM — It Found Churn Signals We Missed

By Dr. Marcus Elliston, Ph.D. in Artificial Intelligence

For three years, our customer success team operated under a comforting illusion: our churn prediction model was "good enough." It was a gradient-boosted tree, trained on eight years of historical data, and it achieved a respectable 74% accuracy. We looked at our dashboards, watched the red and green dots, and moved on. We were confident. We were also, as it turned out, blind.

The turning point came not from a board meeting or a competitor's announcement, but from a simple, almost whimsical experiment. I took our entire CRM—roughly 40,000 customer records, including interaction logs, support tickets, invoice history, and even the free-text notes our account managers scribbled during calls—and fed it into a large language model. I didn't ask it to predict churn. I asked it something far more deceptively simple: "Read these customer profiles and tell me what patterns you see in the accounts that ultimately churned, versus those that stayed."

Within twenty minutes, the LLM returned a structured analysis that made me sit very still at my desk. It had found signals our model had systematically missed.

The Illusion of Feature Engineering

Let me explain what we had, because understanding the gap is essential. Our churn model consumed 47 structured features: contract value, tenure, number of support tickets, last login timestamp, feature adoption metrics, and so on. These are what we in the field call "engineered features"—variables that a human analyst hypothesized would matter and then validated statistically. The model learned correlations: customers who file more than five tickets in a month are 30% more likely to churn. Clean. Interpretable. Actionable.

But here's the thing about feature engineering: you can only model what you thought to measure. And we had a blind spot. Our account managers' notes—those free-text fields—were treated as unstructured noise. We stored them for legal compliance and occasional reference, but they never entered the model. The LLM, by contrast, read every single note. And it found something our numeric pipeline could not.

The Three Signals We Had Missed

Signal One: The Language of Quiet Disengagement

The LLM identified a subtle linguistic pattern. In the 214 accounts that churned within 90 days of the observation window, account manager notes contained a higher frequency of passive-voice descriptions of customer behavior and a lower frequency of first-person plural collaboration language. Concretely: notes from stable accounts read like "We worked with Sarah to migrate the reporting module" or "The team decided to expand the integration." Notes from churning accounts read like "The client reviewed the proposal" or "It was noted that the dashboard was not being used."

This is not a trivial distinction. It reflects a shift in the relationship from collaborative to transactional. The customer is no longer "we"; the customer has become "they." Our model saw that the customer still logged in, still opened the app, still hadn't filed a ticket. The LLM saw the emotional and relational drift happening in the prose. We had been measuring behavior while the relationship was already cooling.

Signal Two: The Ghost Feature

The LLM flagged a pattern involving a feature we tracked but never used in the model: meeting no-shows. We recorded whether customers attended quarterly business reviews (QBRs). Our model included a binary "attended_last_qbr" flag. But the LLM went further. It noticed that the sequence of attendance mattered more than any single data point. Accounts that attended QBR1, skipped QBR2, and then returned to QBR3 showed a 2.1× higher churn probability than accounts that skipped all three. Our model treated each QBR independently. The LLM saw the narrative arc: the customer who skipped once was probably busy; the customer who skipped, returned, and then skipped again was in a rhythm of polite disengagement. The LLM described it, in its own words, as "a customer who keeps rescheduling the relationship."

We ran this as a simple state-machine feature in our model. Churn prediction accuracy improved from 74% to 81%.

Signal Three: The Inverse Correlation We Had Normalized Away

This was the one that surprised me most. Our model penalized high contract values, based on the intuition that large accounts are harder to retain (more stakeholders, longer procurement cycles, more competing vendors). The LLM, reading the full corpus, found the opposite in a specific sub-segment: among customers with low feature adoption (fewer than three of our twelve modules in active use), high contract value was actually a stabilizing factor. The LLM's interpretation: "When a large account is under-utilizing the product, the financial commitment creates a psychological anchor. The account manager's notes reflect a 'sunk cost' framing—'we've invested too much to leave'—which is absent in smaller, under-utilized accounts."

We had been applying a one-size-fits-all weight to contract value. The LLM suggested an interaction term: contract value modulated by feature adoption level. We added it. Another 2% accuracy gain.

Why This Matters Beyond Our Company

What I find genuinely interesting—and what I think should concern anyone building data-driven systems—is that this wasn't a case of the LLM finding a new data source we hadn't collected. All the data was already in the CRM. The LLM's advantage was integrative comprehension. It read across structured and unstructured fields simultaneously, it tracked sequences rather than snapshots, and it inferred narrative structure from prose that we had treated as metadata.

This points to a broader shift. For decades, data science has been largely a game of feature selection: which columns matter? LLMs change the question. Now the question becomes: what is the story the data is telling, and are we asking the right questions of it?

A few practical lessons emerged from the experiment:

• Treat free-text as first-class signal. If your team writes notes, you have a rich, semi-structured dataset that most models ignore. An LLM can mine it for tone, relationship dynamics, and qualitative state.

• Sequence beats snapshot. Binary flags lose information. A customer who skipped, returned, and skipped again is in a different state than one who skipped all three. Model the arc.

• Interaction terms are where the nuance lives. A feature's effect is rarely constant across all segments. The LLM's finding about contract value was an interaction effect we had never tested because we didn't know to look for it.

• LLMs are hypothesis generators, not oracles. The LLM's outputs are interpretable, plausible, and often correct—but they are not guaranteed. We validated all three signals against our historical data before changing the model. The LLM suggested; we verified.

The Quiet Confidence We Lost

Perhaps the most honest takeaway is emotional. Our old model gave us a number: 74% accuracy. That number made us feel competent. The LLM's analysis made us feel, for a while, a little bit foolish. We had built a sophisticated system that was, in a meaningful sense, reading the skeleton of our customer base while the LLM was reading the flesh—the prose, the rhythm, the subtle shifts in how people talked about each other.

We didn't replace our model. We augmented it. The gradient-boosted tree still runs in production. But now it has three new features derived from patterns the LLM surfaced, and a quarterly process where we feed a fresh slice of CRM data to the model and ask it, again, what it sees.

The churn rate, for what it's worth, dropped 11% in the two quarters following the model update. Eleven percent of 200 accounts is 22 accounts. At an average contract value of $45,000, that's roughly $1 million in retained revenue. The LLM cost us about $18 in API calls.

I don't say that to be clever. I say it because I think it captures the strange, asymmetric economics of this new era. The same model that can write a sonnet or pass a bar exam can read your CRM and find the quiet, linguistic, narrative signals that your carefully engineered feature set was too rigid to see.

The question for any team working with customer data is no longer just what do we measure? It's also: what are we writing, and what story is that writing telling? Because sometimes the most predictive feature in your dataset is the one you're too busy to read.

Dr. Marcus Elliston is a fictional author name created for this article. The experience described is a composite of common patterns observed in applied LLM work with enterprise CRM data.