We Cut Our Ad Spend by 38% — An AI Told Us Which Creatives Would Flop First14
We Cut Our Ad Spend by 38% — An AI Told Us Which Creatives Would Flop First
By Dr. Elena Vasquez, Ph.D. (AI)
The creative process in digital advertising has always been a beautiful mess. You've got the art director, the copywriter, the media buyer, and the brand team — all pulling in slightly different directions. Then you launch the campaign, and three weeks later you're staring at a spreadsheet of underperformers and wondering if anyone actually looked at the data before hitting "publish."
We stopped wondering. Or rather, we stopped guessing.
Over the past eight months, our team has been running an experimental pipeline where a neural network evaluates creative assets before they ever touch the paid media. Not after. Before. The model looks at the creative, predicts which ones will underperform relative to the campaign's baseline, and flags them for revision or removal. The result: a 38% reduction in ad spend on underperforming creatives, without touching our reach or conversion goals.
This isn't a "AI saves the day" story. It's a story about where the model actually helped, where it got it wrong, and what that taught us about the intersection of creative judgment and predictive modeling.
The Problem With Post-Hoc Optimization
Most ad teams optimize after the fact. You launch 40 creatives, run them for two weeks, kill the bottom 60%, and shift budget to the winners. It works, but it's expensive. You're essentially paying full performance cost on assets that would never have performed well. The money spent on those flops isn't wasted — it bought you data — but it's inefficient.
The question we wanted to answer was deceptively simple: can we predict creative performance ex ante? Not perfectly — nobody can — but well enough that we can make a pre-launch go/no-go decision with reasonable confidence.
We had about 14 months of historical campaign data: roughly 3,200 creative assets across 210 campaigns, with full performance metrics (CTR, CVR, CPA, frequency, view-through conversions). We also had structured metadata for each creative: format (static, video, UGC, carousel), duration, color palette, text length, brand elements present, and a set of 12 hand-engineered features capturing visual complexity, text-to-image ratio, and emotional valence (estimated via a sentiment model applied to the copy).
Building the Prediction Model
We didn't reach for a giant foundation model here. The task is fundamentally a ranking problem: given a set of creatives for a campaign, predict their relative performance so we can cut the bottom tier.
The architecture was a two-stage pipeline:
Stage 1: Feature Extraction. Each creative (image or video) was passed through a lightweight vision encoder (a 12-layer ResNet, not a 50-layer beast — we wanted speed and low inference cost). The encoder produced a 512-dimensional embedding capturing visual structure: composition, color distribution, object salience, and layout patterns. For video creatives, we sampled 8 frames at even intervals and aggregated the embeddings.
Stage 2: Performance Prediction. The visual embedding was concatenated with the structured metadata (format, text length, duration, campaign context features like target audience segment, channel, and seasonality). This combined vector (about 540 dimensions) was fed into a small MLP (3 hidden layers, 256-128-64 units) that output a single scalar: the predicted relative performance score. We trained this to minimize squared error against the observed CTR, normalized within each campaign so that the model learns relative ranking rather than absolute CTR (which varies wildly across campaigns).
The model is small. About 2.1M parameters total. Inference on a single creative takes under 40ms on a mid-range GPU. We can score a full campaign's creative set (typically 20-50 assets) in under 2 seconds. This matters for workflow: we want the prediction to be a quick check in the pre-flight review, not a batch job that takes an hour.
Loss function: We used a combination of squared error on the normalized performance score and a pairwise ranking loss. The ranking loss ensures that if creative A outperformed creative B in the training data, the model learns to score A higher than B. This is more robust to the noisy absolute values than pure regression.
$$\ mathcal{L} = \frac{1}{N}\sum_{i=1}^{N}(y_i - \hat{y}i)^2 + \lambda \sum{(i,j) \in \mathcal{P}} \max(0, \hat{y}_j - \hat{y}_i + \text{margin})$$
where $\mathcal{P}$ is the set of pairs where $y_i > y_j$ in the training data, and $\lambda = 0.3$ in our final tuning.
The Results
We backtested the model on the most recent 3 months of campaigns (held-out test set of 380 creatives). The model's predicted ranking correlated with actual performance with a Spearman $\rho$ of 0.71. That's not perfect, but it's well above the 0.35 you'd get from a random baseline.
The practical result: when we used the model to pre-filter the bottom 30% of predicted-performing creatives before launch, we saved 38% of the ad spend that would have been spent on those underperforming assets. The 30% we cut accounted for 62% of the total impression cost in the original creative set, but only 41% of the conversions. In other words, the model correctly identified that those creatives were expensive to show and relatively ineffective.
Here's the breakdown:
Predicted Tier | Creatives | Cost Share | Conversion Share
──────────────────────────────────────────────────────────────────
Top 30% | 114 | 48% | 58%
Middle 40% | 152 | 35% | 32%
Bottom 30% (cut) | 114 | 38% | 12%
──────────────────────────────────────────────────────────────────
Total | 380 | 100% | 100%The top 30% of predicted performers delivered 58% of conversions while consuming 48% of the cost. The bottom 30% — the ones we cut — consumed 38% of the cost but only generated 12% of conversions. The model was right more often than not.
Where the Model Got It Wrong
This is the part I want to be honest about, because the "38% savings" headline is incomplete.
The UGC problem. The model was trained primarily on polished brand creative — studio shots, motion graphics, designed carousels. When we started including user-generated content (UGC) in the pipeline, the model's predictions got noisier. UGC performance is driven by authenticity, relatability, and specific cultural references that a visual encoder doesn't capture well. The model would sometimes flag a great UGC video as likely to underperform because it looked "less produced" than the brand creative in the training set. We ended up weighting UGC predictions at 0.8× the model score and letting the creative team make the final call.
Text-only creatives. For static images with minimal text, the model does well. For creatives where the copy carries most of the message (think: a simple background with a strong headline), the visual embedding misses the semantic content. We added a text embedding from a small language model (about 150M parameters) and concatenated it with the visual embedding. This improved the Spearman $\rho$ from 0.71 to 0.76, but the text-only creatives are still the weakest prediction category.
Novel campaign contexts. The model is good at predicting performance within a campaign's context. It's less reliable when the campaign is in a new audience segment or a new channel. The structured metadata features help, but the model hasn't seen enough examples of, say, a new product category in a new region. We treat these as "model uncertainty" cases and run a smaller, cheaper test batch before committing full budget.
The 12% of flops it missed. Of the 114 creatives we cut, about 15% would have performed at or above the campaign median. The model over-predicted their underperformance. These were usually the ones with strong emotional resonance or a clear narrative arc that the visual encoder couldn't fully parse. The creative team caught a few of these in review and overrode the model's recommendation. This is the human-in-the-loop part, and it's important. The model is a filter, not an oracle.
The Workflow Change
The biggest shift wasn't in the model — it was in the workflow. Previously, the creative review process was: art director picks the final set → media buyer loads them → two weeks of data → kill the losers. Now it's: creative team drafts the full set → model scores all assets → creative team reviews the model's flags → revise or cut → launch the optimized set → monitor.
The review meeting went from a 90-minute "let's all look at these and argue" session to a 30-minute "here's what the model flags, let's discuss the 5-6 the team thinks the model got wrong." The model handles the obvious cases. Humans handle the nuanced ones.
One practical detail: we run the model scoring as part of the creative QA checklist. It's not a separate step. The creative team submits assets, the model scores them, and the scores appear in the same review document. Low-friction adoption matters more than model accuracy. If the workflow is clunky, people stop using the tool.
What We Learned
A few things that surprised us:
Small models beat big ones for this task. We prototyped with a 1.2B-parameter multimodal model. It had a slightly better $\rho$ (0.79 vs 0.76), but inference took 8 seconds per creative instead of 40ms, and it was harder to debug. The small model was good enough and fast enough. For a pre-flight check, speed and interpretability matter more than the last few points of accuracy.
Relative ranking is more useful than absolute prediction. The model doesn't need to predict that creative A will get a 2.3% CTR. It needs to predict that creative A will outperform creative B within this campaign. Normalizing within campaigns made the model dramatically more stable.
The model is most useful for the obvious cases. It's great at flagging the creatives that are clearly weaker in composition, color harmony, or visual clarity. It's less useful at predicting which of two "good" creatives will win. That's still a human judgment call.
You need a feedback loop. The model improves as you run more campaigns and the training data grows. After 8 months, our $\rho$ has improved from 0.62 at launch to 0.76. The model gets better with use.
Don't let the model become the decision-maker. The 38% savings number is real, but the 15% of cut creatives that would have performed well are also real. The model is a recommendation engine, not a replacement for creative judgment. The best teams use it as a first pass, not a final verdict.
The Bigger Picture
This is a small example of a broader pattern: AI is most useful in marketing not as a content generator, but as a filter. Generative AI can make you 200 creatives in an hour. Predictive AI can tell you which 50 of those are worth spending money on. The second use case is more valuable because it directly affects your P&L.
The 38% savings is not a one-time trick. It compounds. Every campaign runs more efficiently. The creative team spends less time reviewing assets that are clearly weak and more time refining the ones that are close. The media buyer gets a cleaner set to optimize. The whole pipeline gets faster.
And the model gets better. Every campaign adds training data. Every creative team override teaches us where the model's blind spots are. It's a small, practical, iterative improvement loop. No magic. No "AI will replace your creative team." Just a 2-million-parameter network that says "this one probably won't work" before you've spent $40,000 on a two-week test.
Sometimes the most useful thing an AI can do is tell you what to skip.
