📊 We Cancelled 12,000 Projects. Then Taught a Machine to See It Coming

By Dr. Elias Thornwood

Every company that has ever promised more than it could deliver knows the smell of churn. It smells like stale dashboards, half-finished onboarding flows, and Slack threads titled "quick sync" that were anything but quick. We are a mid-sized SaaS company selling to mid-market teams, which means we live in the most interesting failure mode there is: customers who genuinely want our product, mostly use it, and quietly decide they no longer need us — usually three weeks before their account manager finds out.

For years we treated cancellations as weather. You watched the sky darken, you hoped for rain or drought, and you wrote a postmortem after the fact. Last year I got tired of that. We had about 14,000 customer accounts with a full history: usage telemetry, support tickets, billing events, CRM notes, and an unfortunate amount of "just checking in" emails from our sales team. I asked for six months and one engineer's time to see if any of that noise actually contained a signal. What came back surprised me enough to write this down properly.

The Data We Had Was Ugly, So the Model Had To Be Honest

Before anyone mistakes this for a clean research paper: our data was not beautiful. Usage telemetry arrived in 17 slightly different schemas depending on which product line the account sat under. Support tickets were free text written by tired agents at 9pm. CRM notes were even less structured — one of our AEs writes in bullet points, another in full Shakespearean paragraphs, and a third appears to use the notes field as an emotional journal.

We didn't try to normalize all of it. We normalized only what mattered: per-account daily active users, sessions, feature adoption depth, ticket volume and sentiment, payment success rate, and a rolling 90-day trend on each. Sentiment was computed with a small transformer fine-tuned on our own ticket corpus — off-the-shelf sentiment models misread "this is great but the export is broken" as positive, which is exactly the kind of sentence that predicts churn.

The final feature set came out to roughly 210 dimensions per account, sampled weekly for two years of history, giving us about 84 time points per customer. We deliberately kept the temporal structure instead of flattening everything into a single row per account — more on why in a moment.

Why Not Just Use a Classic Survival Model?

The obvious first answer to "predict when will this customer leave" is survival analysis: Cox models, Kaplan-Meier curves, a nice confidence interval, done. I built one. It worked fine and it taught me something important about our own business: cancellations in our data are not Poisson-like. They cluster. Accounts don't drift away steadily; they commit to leaving somewhere between 2 and 4 weeks out, often right after a renewal discussion or a support escalation that didn't land.

That clustering is exactly what you want a sequence model to see. So we went with a temporal setup: an encoder-decoder transformer over weekly feature vectors, trained to output a per-week cancellation probability for the next 12 weeks. The loss was weighted binary cross-entropy with heavier weights on true cancellations in the first four forecast weeks — because that is the window where a retention team can actually do something.

Training details worth sharing: batch size 64 accounts, AdamW at lr = 3e-4 with cosine decay, early stopping on validation log-loss, and a 70/15/15 temporal split so no account appeared in two splits — leaking your own history into your test set is a quiet way to flatter yourself.

The 94% Number, And What It Actually Means

Here is where I owe you an honest footnote, because "predicted 94% of cancellations at 3 weeks out" sounds like a press release and deserves unpacking.

We ran the model over a holdout window covering 2,140 accounts across one quarter. In the true 21-day pre-cancellation window, 687 accounts cancelled. The model flagged 590 of them with a probability ≥ 0.70. That is:

$$\ text{Recall} = \frac{590}{687} \approx 0.858$$

So the recall was about 86%, not 94%. Where did 94% come from? We also measured early warning coverage: of all accounts that cancelled in that quarter, 94% had at some point between T-21 and T-7 days shown a model probability ≥ 0.50. In other words, the model saw almost every churning account coming — but it didn't always flag them with high confidence until close to the wire.

Both numbers matter, and they mean different things:

The precision number is the one that keeps me up at night and it's why I'd push back on anyone quoting only recall. You're not just predicting; you're spending human attention budget, and every false positive is a sales rep bothering a happy customer with "just checking in."

A Simple Visualization of What the Model Sees

Here's what the probability trajectories look like for three representative accounts:

P(cancel) over weeks before cancellation (T-12 to T0):

Account A (smooth decay — classic churn)
  Week:   -12  -9   -6   -3    0
  Prob:   .08  .14  .31  .72   1.00     ← clean ramp-up, easy catch

Account B (support-escalation trigger)
  Week:   -12  -9   -6   -3    0
  Prob:   .05  .07  .09  .84   1.00     ← flat then spike; model tracks the ticket

Account C (quiet fade — hardest case)
  Week:   -12  -9   -6   -3    0
  Prob:   .04  .05  .07  .18   1.00     ← slow drift; model only confident at T-3

Account B is the money case — the escalation ticket lands, and within one weekly sampling the probability jumps from under 0.1 to above 0.8. Account C is the reminder that not every churn is loud. The model handles both, which is most of why I trust it more than any single-feature rule we tried before.

What Actually Drove Predictions (The Boring Truth)

Feature attribution via integrated gradients, averaged over the test set:

• Rolling 21-day active user trend: contributed roughly 34% of total signal

• Ticket volume + sentiment shift: about 22%

• Payment success rate dips: about 11%

• CRM note sentiment (AE-written): about 9%

• Feature adoption depth / new-feature trial: about 8%

• Everything else (plan tier, seat count, industry, tenure): the remainder

The first one is almost tautological — you can predict a leak by watching water leaving. The interesting finding was how much weight CRM notes carried. Our AEs write things like "client seems distracted this quarter" and "feels like they're comparing us to X." Humans are surprisingly good narrators of customer drift, even when the rest of our instrumentation is noisy. The model essentially learned to read our own body language.

What We Do With the Predictions (This Is Where It Gets Real)

A prediction that lives in a notebook is just math. We wired it into a simple weekly workflow:

1. Every Monday, accounts with P ≥ 0.50 go into a shared "watchlist" grouped by predicted time-to-cancel.

2. Accounts at T-4 to T-6 weeks (probable churn within ~1 month) get a value outreach — not a discount offer, but a use-case review with our solutions team. The goal is to make the product visibly useful again, because that's what "using it less" usually means.

3. Accounts at T-2 to T-3 weeks get a relationship touch: their AE reaches out personally, no script, asking about their quarter and what we can adjust.

4. Accounts at T-1 week or earlier trigger our formal save playbook — this is where discounts live, because by now the cost of losing them has crossed most thresholds.

After two quarters in production, the watchlist team saved an estimated 19% of flagged accounts from cancelling — which sounds modest until you multiply it by ARR. The more subtle win: we stopped guessing at who to call. Retention attention is a scarce resource and we now spend ~70% of it on accounts that were genuinely at risk, up from maybe 40% when we worked off gut feel.

Three Lessons I Wish Someone Had Told Me Sooner

Lesson one: predict the window, not just the event. "This account will churn" is a boring answer; "this account has a 72% chance of churning between the 3rd and 5th week of next month" is an actionable one. The whole design decision to forecast per-week probabilities instead of binary classification paid for itself in how naturally it mapped onto our retention cadence.

Lesson two: instrument for narration, not just metrics. Our best single predictor after usage trends was the free-text CRM notes — data we had been collecting for years without knowing what to do with it. If you're building this kind of system, make sure your field team is writing honest, specific notes. "Client unhappy" is worth less than "client's VP mentioned a competitor demo on Tuesday." The model can't recover information that was never written down.

Lesson three: calibrate or die. We track reliability diagrams monthly and recalibrate the output probabilities with Platt scaling when drift shows up — which it does, roughly every 4 to 6 weeks as product changes shift behavior baselines. A well-calibrated probability is what lets a non-technical retention lead read "0.72" and know it means approximately seven out of ten similar accounts left within the window. Without calibration you're just guessing with extra steps.

The Honest Limits Of This Whole Effort

It's worth closing with what the model does not do well, because anyone selling churn prediction will happily skip this paragraph:

• New customers under 6 months are noisy — not enough history for the temporal encoder to work well. We run them on a simpler rule-based flag instead and only graduate them into the model after six monthly usage cycles.

• Accounts in the middle of a large implementation can trigger false positives because usage spikes look like churn-risk in our training data (we had that correlation inverted early on, which was embarrassing).

• Competitor launches, industry shocks, or macro events are only captured indirectly through usage and notes — the model learns them with a lag, so we keep a small manual override channel where an AE can add context.

And there's a philosophical one: predicting churn well is not the same as preventing it. The model tells you when to act; it does not tell you what will make this specific customer stay. That part remains stubbornly human, and I'd argue that's how it should be — we're using the machine to buy us time to do the relationship work better, not to replace it.

What I'd Do Differently If We Started Over

Three things, in rough order of importance: first, I would have instrumented feature-adoption events at a finer granularity from day one; our usage telemetry captured sessions but not which features were actually used, and that cost us an early version of the model. Second, I would have built the watchlist UI before the model was perfect — we spent two months on architecture for something the retention team never touched because it lived in Jupyter. Third, and most importantly, I would have involved the AEs during training, not after. Their intuition about which accounts "feel" at risk is a dataset of its own, and validating the model against their gut was far more informative than any split metric we computed.

None of this is a replacement for good product work — because ultimately the best churn prediction is a product people don't need to leave. But as one small layer in that system: six months of effort, two engineers part-time, and a decent GPU or two, in exchange for a model that sees 94% of our cancellations coming from up to three weeks out. For a business where churn quietly erodes everything else you're building, I'd pay for that number twice.

Dr. Elias Thornwood is a research engineer focused on applied machine learning for customer analytics.