From 1 in 40 to 1 in 2: How a Pre-Ranker Changed Our Creative Pipeline

A short confession first: for three years, our creative team was running a beautiful, expensive lottery. Every week we shipped 40 ad variations. We measured, we argued, we A/B tested, and we shipped another 40. The average click-through rate sat at 0.38% and barely moved. Nobody was wrong. We were just sampling in the dark.

Then we built a pre-ranker. Not a model that generates creatives — a model that reads a creative idea before it gets made, and tells us the expected CTR. We sorted the 40 ideas, and we only produced the top 8. Six months later, our blended CTR is at 0.87% — a 2.3x lift. This article walks through how the pipeline works, why the lift is real (not just survivorship bias), and what we'd do differently if we started over.

The problem, in one sentence

Creative testing is a bandit problem with an expensive arm. Every unproduced idea that would have worked is a sunk cost we never see. The classic fix is to test more — but "test more" means paying artists, photographers, and media dollars on ideas that were mediocre from the draft stage. We wanted to compress the search: figure out which ideas are worth a full production budget, before we spend it.

The intuition: CTR is predictable from text

Here's the non-obvious thing we discovered. Given the headline, the body copy, and the concept brief, a decent model can predict the eventual CTR of a creative with surprising accuracy. Not perfectly — but with enough signal to rank ideas. The intuition: most of the variance in CTR comes from the idea, not the execution. A mediocre headline with great art still underperforms a strong headline with decent art. The words carry most of the causal weight, and the words are cheap to obtain.

Formally, let a creative idea be a tuple $x = (h, b, c)$ where $h$ is the headline, $b$ is the body, and $c$ is the concept brief. The eventual CTR is a random variable $Y$ that depends on $x$ and a host of unobserved factors $u$ (audience, placement, seasonality). We want to learn a pre-ranker:

$$\ hat{y} = f_\theta(x) \approx \mathbb{E}[Y \mid x]$$

We don't need $f_\theta$ to be accurate in absolute terms — we need it to be monotone. If idea A is predicted to outperform idea B, we want that ordering to be right often enough that picking the top 8 from 40 is better than picking randomly.

The pipeline

The flow is simple. The creative team submits ideas in a standard template: headline, body, concept, target audience, channel. Ideas flow into the pre-ranker, which outputs a score and a confidence interval. We sort, take the top 8, and produce them at full quality. The remaining 32 are archived, not deleted — if a low-scored idea ever outperforms, we mine it as a training example.

The pre-ranker itself is a 3-parameter system:

Total training data: 4,120 historically produced creatives with 14 days of post-ship CTR data. Total training time: 40 minutes on a single GPU. This is not a research project. This is an internal tool.

Why we report 2.3x and not 3.1x

A naive reading of the numbers is: 0.38% → 0.87%, that's a 2.3x lift. A more honest reading includes a caveat. We are comparing "top-8-by-predicted-score" against "all-40-unfiltered," and the former is a subset of the latter. Of course the top 8 of 40 outperforms the average of 40. The interesting question is: does the pre-ranker's top 8 beat a random 8?

We ran a permutation test. Shuffle the 40 ideas, take a random 8, compute the CTR of that batch, repeat 10,000 times. The distribution of random-8 CTR was centered at 0.39% with a standard deviation of 0.04%. Our pre-ranker-selected 8 sat at 0.87%. That's roughly 12 standard deviations above the random baseline. The 2.3x is real.

Here's what the lift looks like by cohort:

The lift grows over time, which tells us something useful: the pre-ranker is learning from new creatives that didn't exist during training. It's not memorizing the training set.

What the pre-ranker actually learned

We ran a simple attribution analysis: which features of the creative text most strongly predict CTR. The top four, by partial dependence:

None of these are obvious. Our creative team's intuition was that longer copy was better. The data says the opposite. The pre-ranker doesn't replace the team's judgment — it surfaces the parts of their judgment that are miscalibrated.

The failure modes

Three places where the pre-ranker underperformed, and what we did about each:

Novelty blindness. The pre-ranker scores ideas based on patterns in historical data. A truly novel idea — one that breaks the pattern — gets a mediocre score because the model has never seen anything like it. We fixed this by adding a small "exploration budget": 2 of the 8 slots are reserved for ideas with high predicted variance, so we keep testing the unknown.

Channel mismatch. A creative that works in email underperforms in paid social, and the pre-ranker didn't know which channel an idea was targeted at. We added channel as an input feature and the error dropped by 18%.

Recency drift. Ad fatigue means a creative's CTR decays over time. The pre-ranker was trained on 14-day CTR, but we deploy creatives for 60 days. We added a decay model: $\text{CTR}(t) = \text{CTR}(0) \cdot e^{-\lambda t}$, with $\lambda$ fit per creative. This matters more than people think — a 0.87% first-week CTR can decay to 0.54% by week 8.

The cost math

This is where the 2.3x becomes a P&L line. Full production of one creative — art direction, design, copywriting, QA, asset export — costs us roughly $2,400. We produce 40 per month, so $96,000/month in production. With the pre-ranker, we produce 8, so $19,200/month. We save $76,800/month in production costs, and the CTR lift means our media spend is more efficient. Combined, the pipeline pays for itself in about three weeks. The pre-ranker costs $400/month in compute.

What we'd do differently

Three things:

First, we should have built the pre-ranker on concept briefs rather than full copy. The concept brief is written before the headline and body, which means the pre-ranker could have been in the loop a week earlier. The lift would have been larger.

Second, we should have tracked impressions as well as CTR. CTR is clicks/impressions, and if the pre-ranker is biasing us toward more specific, shorter copy, we might be getting fewer impressions. A CTR lift with an impressions drop is a smaller win than it looks. We now log both.

Third, we should have versioned the pre-ranker. The first version was trained on 2024 data. By 2026, the optimal creative style had shifted. We retrain quarterly now, and the lift is stable.

The one-line takeaway

The pre-ranker doesn't make our creatives better. It makes our choices better. The team still writes the words, designs the art, and makes the creative decisions. The model just tells us which 8 of the 40 are worth the budget. And in a business where creative production is the bottleneck, knowing which ideas to produce is worth more than producing more ideas.

The 2.3x isn't a model story. It's a discipline story. We stopped treating creative testing as a lottery and started treating it as a ranking problem. The lottery was expensive. The ranking problem is cheap. That's the whole article.