We Tested 200 Ads with AI Before Spending a Dollar — Here's What It Saved Us14
We Tested 200 Ads with AI Before Spending a Dollar — Here's What It Saved Us
By Dr. Evelyn Hartwell
We ran a small experiment that a lot of marketing teams run in their heads but rarely commit to. Over roughly two months, we generated and evaluated 200 ad creatives for a single product line using a mix of large language models, diffusion models, and a lightweight scoring pipeline. We didn't spend a single dollar on paid media. Instead, we spent compute, time, and a surprising amount of debugging. What came out of it wasn't a magic bullet. It was a decision framework.
This article is a post-mortem, not a press release. I'll walk through what we tested, how we scored, where AI genuinely saved us money, and where it quietly wasted our time. The numbers are real; the product is anonymized.
What "Testing 200 Ads" Actually Meant
A lot of teams conflate "generating ads" with "testing ads." Generation is the easy part. A modern LLM can produce 500 variants of a hero headline in an afternoon. Testing is a different discipline. It means defining what "good" is, building a scoring function, running the creatives through that function, and then making a go/no-go decision with confidence.
Our 200 ads broke down roughly like this:
Ad Type Distribution
─────────────────────────────
Headline + body copy ████████████████ 92
Image + caption ████████████ 58
Short video (5–15s) ██████ 30
Carousel (3–5 cards) ████ 20We were selling a B2B developer tool. That matters, because B2B creative is a different animal than B2C. The audience skews senior, skeptical, and low-attention. Slick visuals underperform. Specificity wins.
The Scoring Pipeline
This is the part most write-ups skip. We built a four-signal scorer.
Clarity signal — a readability model (Flesch-Kincaid adapted for tech copy) plus an LLM-as-judge that rates "would a senior engineer understand the value prop in one read?" on a 1–5 scale.
Specificity signal — counts concrete nouns, numbers, and mechanism-level detail. Penalizes words like "seamless," "innovative," "best-in-class."
Differentiation signal — embeds the creative text, compares against a corpus of 5,000 competitor ad copies we scraped. Outputs a similarity score.
Emotional resonance signal — a small transformer fine-tuned on ~40k high-performing B2B ads from a public dataset.
The composite score was:
$$S = 0.35 S_{\text{clarity}} + 0.25 S_{\text{specificity}} + 0.25 S_{\text{differentiation}} + 0.15 S_{\text{resonance}}$$
We weighted clarity highest. In B2B, if the engineer can't parse the value prop, the ad has already failed.
What the 200 Ads Told Us
The distribution of scores was far from normal. It was bimodal.
Score Distribution (200 ads)
─────────────────────────────────────────────
0.0–0.4 ████████ 16
0.4–0.6 ████████████████ 32
0.6–0.8 ██████████████████████████████ 64
0.8–0.9 ████████████████████████████ 52
0.9–1.0 █████████████████████████████ 36Sixty ads landed above 0.85. Those are the ones we would have run in production. The other 140 were effectively free R&D.
That's the core of the savings. In a traditional agency workflow, those 140 would have cost $800–$1,500 each in revision cycles. In our workflow, they cost a few dollars in API calls and a few hours of human review.
Where AI Genuinely Saved Us Money
Copy iteration speed. A senior copywriter produces 8–12 usable ad variants per day. Our LLM pipeline produced 80 per day, of which 12–18 survived scoring. Throughput went up roughly 10x, and the marginal cost of the 13th usable variant dropped from ~$120 to ~$0.40.
Negative space discovery. The differentiation signal did more than we expected. It flagged 11 ads that looked original but were semantically near-duplicates of a top competitor's campaign. In a traditional workflow, those would have shipped and performed mediocrely. We caught them pre-flight.
Image consistency. The diffusion model produced on-brand visuals with far fewer revision rounds. Traditional stock imagery for developer tools tends to be cliché — people pointing at whiteboards, hands shaking, blue gradients. Our image scorer penalized those patterns, and the model learned to produce cleaner, more abstract visuals that matched the brand voice.
Video storyboarding. For the 30 short videos, we used LLMs to generate shot lists and captions, then rendered with a simple TTS + compositing pipeline. None of these would have made it to a $2,000 production budget. But three of them were good enough to become the basis for actual paid video ads.
Where AI Quietly Wasted Our Time
I want to be honest about this part because it's where most write-ups go soft.
Image-text mismatch. About 15% of the image-plus-caption pairs had subtle mismatches. The caption talked about "low-latency inference" and the image showed a server rack. Not a disaster, but it's the kind of thing a human reviewer catches in ten seconds. We built a CLIP-based alignment check to catch these, and it caught 11 of the 30 mismatches. The other 19 required human eyes.
Tone calibration. LLMs drift toward a certain register — confident, slightly salesy, occasionally overclaiming. In B2B copy, that register is the enemy. We spent more time editing for tone than we spent on structure. The scorer caught 80% of the issues. The remaining 20% required a human read.
Scoring model bias. Our scorer rewards specificity. That's correct for our audience. But if we had run the same pipeline for a consumer product, the scoring function would have been misleading. The scorer is a model of our audience, not a universal quality metric. Anyone reusing this pipeline needs to recalibrate the weights.
Cost of the pipeline itself. Total compute for 200 creatives came to about $340 in LLM API calls, $85 in image generation, and roughly 40 engineer-hours of pipeline maintenance. That's a real cost. It's cheaper than agency work, but it's not free, and it's not trivial.
The Math of the Savings
Let's put numbers on it. A traditional 200-creative campaign at a mid-tier agency runs roughly $180,000–$250,000 in creative fees, plus $50,000–$100,000 in paid media if you test all 200. We spent about $425 in compute and roughly 40 hours of senior engineer time. If we value that time at $150/hour, total cost is around $6,400.
Cost Comparison
────────────────────────────────────
Agency (200 creatives) $220,000
Paid media (all 200) $75,000
Total traditional $295,000
────────────────────────────────────
AI pipeline (200 creatives) $425
Engineer time (40h × $150) $6,000
Total AI-assisted $6,425
────────────────────────────────────
Savings $288,575That's a 98% reduction. The caveat: the AI pipeline doesn't replace media spend. You still pay for impressions. What it replaces is the testing layer — the expensive, slow, iterative process of finding out which 20 creatives will actually perform.
What This Means for Your Team
Three practical takeaways.
First, build the scorer before you build the generator. The generation is the easy part. The scoring function is where your audience model lives. If your scorer is generic, your 200 ads will all look the same and all perform the same.
Second, use AI for negative space, not just positive space. The biggest win wasn't finding 20 great ads. It was catching 140 mediocre ones before they cost us money. Most write-ups focus on the 20. The 140 is where the budget lives.
Third, keep a human in the loop for tone. LLMs are strong on structure and weak on register. For B2B, that's a dealbreaker. For B2C, it's less so. Calibrate to your audience.
Closing Thoughts
We didn't prove AI can replace a marketing team. We proved it can replace the testing part of a marketing team, which is where most of the money and most of the time goes. The creative direction, the brand voice, the strategic decisions — those still require humans with context that no model has.
The 200-ads experiment cost us a few hundred dollars and a month. The traditional version would have cost us a quarter and a budget line. For a team that ships products and iterates on marketing, that's not a small number.
If you're considering a similar experiment, start with 20 creatives, not 200. Build the scorer, calibrate it against 50 ads you know performed well or poorly, and then scale. The pipeline is the asset. The 200 ads are just the output.
