The 15-Minute AI Check That Replaced a Two-Week Test Cycle for Us

By Dr. Elena Vasquez

Published under AI Inspired

Two weeks. That's how long our regression suite used to run. Two weeks of engineers babysitting CI queues, triaging flaky tests, re-running jobs, and debating whether a green build actually meant the build was good. Two weeks where the person who wrote the change was long gone from the branch, the context had evaporated, and the next person had to re-derive intent from a wall of stack traces.

We're a 40-person team shipping a large-scale data platform. The old cycle looked like this:

write code  →  open PR  →  wait for queue  →  run 40k tests  →  triage failures
   →  fix  →  re-run  →  ship (if lucky, in 10-14 days)

When a PR sat in that pipeline for two weeks, context decayed. Reviewers had to re-read the diff from scratch. The author had moved on to a different ticket. Bugs that needed a one-line fix from the original writer turned into a full postmortem. We started calling it "the long weekend of CI."

Then we did something that sounded almost too simple to be true: we stopped running the full suite on every PR. We built a 15-minute AI-assisted check that runs on every commit, and we let the two-week cycle handle the cases the 15-minute check couldn't resolve. The result, after six months of production use, is what I want to walk through below.

The Problem Wasn't More Tests

A common reflex when test cycles are slow is to write more tests. We had already done that. We had 41,200 tests across 6 repos. We had property-based tests, golden-file tests, integration tests against a staging cluster, and a 200-page document describing which tests were "authoritative."

The problem was never coverage. The problem was signal-to-noise.

Of our 41,200 tests, only about 1,800 were "truth tests" — tests that would reliably fail if a real behavior changed. The other 39,400 were guard-rail tests, snapshot tests, mock-happy-path tests, and tests that verified the test harness itself. They ran, they passed, and they told us almost nothing.

Here's the distribution:

Truth tests            ████████████████        1,800  (4.4%)
Snapshot / golden      ███████████████████████ 12,400 (30.2%)
Mock happy-path        ███████████████████████ 10,100 (24.5%)
Integration / staging  ██████████████████████   8,700 (21.1%)
Harness / meta         ██████████████████████   8,200 (20.0%)

The two-week cycle was dominated by the bottom four bars. The top bar — the part that actually predicted regressions — finished in roughly 90 minutes.

What the 15-Minute Check Actually Does

The check has four stages. I'll walk through each, because the design choices matter.

Stage 1 — Static impact analysis (2 min).

A small LLM reads the diff and the call graph. It doesn't write code. It answers one question: which public behaviors could plausibly change from this diff? The output is a list of 5 to 30 candidate behaviors, each with a short justification and the code paths that touch it.

Stage 2 — Test-selection (3 min).

A second LLM pass takes the candidate behaviors and walks our test index. It picks the smallest set of tests whose union of covered code paths intersects the candidate behaviors. On our platform, this typically lands around 350 tests, not 41,200.

Stage 3 — Execution (7 min).

Those 350 tests run in parallel on a warm container pool. Wall-clock is usually 6 to 8 minutes. This is the only stage that actually executes code.

Stage 4 — Failure interpretation (3 min).

If any test fails, a third LLM pass reads the failure output, the relevant diff hunk, and the test source. It writes a one-paragraph interpretation: is this a real regression, a flake, a test that needs updating, or a test that was testing the wrong thing?

Total wall-clock, including queue time: 14 to 15 minutes.

A Note on the Math

The speedup isn't a magic number. It's arithmetic.

Let $T_{\text{full}} \approx 14$ days be the old cycle. Let $T_{\text{check}} = 0.25$ hours be the new check. The raw ratio is:

$$

\frac{T_{\text{full}}}{T_{\text{check}}} = \frac{14 \times 24}{0.25} \approx 1{,}344

$$

But that's not the real speedup, because the old cycle was serial and the new one is largely parallel. The effective cycle-time reduction — the time a PR spends waiting for a green light before review — dropped from ~14 days to ~15 minutes plus ~2 hours of human review. That's a ~55× reduction in time-to-first-signal, and a ~3× reduction in end-to-end time-to-merge, because reviews now start while the long-running suite is still warming up.

The 80/20 We Kept

We didn't throw out the two-week cycle. We demoted it.

The full 41,200-test suite still runs on every PR — but it runs in the background, and its results are treated as advisory unless the 15-minute check has already passed. If the full suite finds a failure the 15-minute check missed, we get a signal that our test-selection model is under-covering. That signal goes into a monthly review of the call graph and the behavior index.

If the 15-minute check finds a failure the full suite didn't, we get the opposite signal: we're running too many tests for the risk, and we can prune.

Both directions of mismatch are useful. The two cycles are now a pair of cross-validations instead of a single serial bottleneck.

What the AI Actually Decides — and What It Doesn't

This is where I push back on the "AI replaced our test cycle" framing. It didn't replace it. It pre-filtered it.

The 15-minute check never decides a PR is good. It decides which tests are worth running first. The human review still happens. The full suite still runs. The only thing that changed is order and parallelism.

In production terms:

The last row is important. We still run all 41,200 tests. We just don't gate the PR on them.

The Failure Modes We Hit

I'd be doing you a disservice if I skipped these.

Over-optimistic impact analysis. Early on, the LLM would sometimes miss a behavior change in a deep call graph. A PR with a subtle change to a serializer was greened by the 15-minute check, and a downstream consumer broke in staging. We caught it within 24 hours. Fix: we added a "depth budget" — if the call graph depth exceeds 4 hops from the diff, the check automatically promotes to the full suite.

Snapshot tests that mask real changes. Snapshot tests pass when a behavior changes in a way the snapshot already captured. The LLM learned to discount them in the selection pass.

Flaky tests masquerading as regressions. The failure-interpretation stage had to learn to distinguish "this test is flaky" from "this test caught a real bug." We fed it 3 months of historical flake data to calibrate.

Prompt drift. We version our prompts the same way we version code. A prompt change that quietly shifts selection behavior shows up as a change in the test-set size histogram, and we alert on that.

What This Looked Like in Practice

One example from our logs. A PR that changed a cache-eviction heuristic. Old world: 12 days to green, then a reviewer asks "why did you change the eviction order?" and the author has to re-derive intent from a diff they wrote two weeks ago. New world: 15 minutes to a 350-test green, 2 hours of review, 6 hours of background-suite confirmation, merged in under a day. The reviewer had the diff fresh in mind. The author was still in the PR thread.

The qualitative shift was as big as the quantitative one. PRs got smaller. Authors wrote more focused diffs because they could see the cost of scope creep in minutes instead of weeks. Code review conversations got shorter because everyone was looking at the same 15-minute signal, not a 14-day black box.

A Few Things I'd Tell a Team Starting This

1. Start with your call graph, not your LLM. If your codebase has no stable call graph, the LLM is just guessing. Build the index first.

2. Treat the AI check as a router, not a judge. It picks tests. It doesn't pass PRs.

3. Keep the long cycle running. You need the slow, complete signal to calibrate the fast, partial one.

4. Version your prompts like code. Prompt drift is real and it's measurable.

5. Instrument the selection pass. Log which behaviors the LLM identified and which tests it picked. That log is your debugging tool.

6. Don't delete the old tests. You're not replacing the suite. You're adding a fast lane in front of it.

The Part I Don't Say in the Marketing Deck

The 15-minute check doesn't work for every PR. PRs that touch the build system, the test harness, or the LLM pipeline itself have to go through the full cycle, because the fast lane is made of the same code the PR is changing. We estimate ~8% of our PRs still take the long path. The other 92% get the fast path.

And the LLM in the loop is not the same LLM that would write your production code. We use a smaller, cheaper model for impact analysis and test selection, and reserve the larger model for failure interpretation, where nuance matters more than speed.

It's an unglamorous stack. A call graph index, a prompt, a parallel test runner, and a failure-interpretation model. No one is going to write a keynote about it. But it turned a two-week bottleneck into a 15-minute signal, and that's about as close to a free lunch as software engineering gets.

Dr. Elena Vasquez