We Replaced Our Entire Attribution Team With One AI Model

By Dr. Elena Vasquez, Ph.D. in Artificial Intelligence

Published March 2026

The Morning We Fired the Pipeline

For six years, our analytics stack ran on a chain of eight ETL jobs, three SQL views, a Python notebook nobody fully understood, and a team of four data engineers who collectively owned the logic that told the CEO why revenue moved. It was a beautiful, fragile machine. When the CEO asked "what drove the Q3 dip in EMEA?" the answer took two days. Two days of logins, joins, and a 400-line notebook that one engineer had stopped updating in 2023.

In January, we did something that still feels a bit like a press release we wrote for ourselves: we retired the entire attribution pipeline and replaced it with a single, well-prompted, well-evaluated LLM-powered service. Four engineers. One model. One API key.

This is not a hagiography. It's a post-mortem of the transition, the numbers, and what surprised us.

What "Attribution" Actually Meant For Us

Before the model, "attribution" at our company was a polite word for a set of heuristics:

• Last-touch: credit the final click.

• First-touch: credit the initial impression.

• Linear: split the credit evenly.

• Data-driven: a Markov-chain model that we refreshed quarterly.

Each of these was a choice disguised as a fact. The CEO wanted one number. The CFO wanted a different one. The growth team wanted a third. The data team kept a spreadsheet of "which attribution we're using today." It was theater of precision.

What we actually needed was simpler and harder: given a revenue event, produce a defensible explanation in natural language, with a confidence interval, in under 30 seconds, on demand.

That's the spec we wrote down. That's the spec the model now meets.

The Architecture (It's Smaller Than You'd Think)

The full system is roughly 1,200 lines of Python, one vector store, one LLM, and a small evaluation harness.

┌─────────────┐     ┌─────────────────────┐     ┌──────────────────┐
│  Event Log   │────▶│  Feature Builder    │────▶│  LLM Service     │
│  (Parquet)   │     │  (embeddings +     │     │  (reasoning +    │
└─────────────┘     │  retrieval)         │     │  explanation)    │
                    └─────────────────────┘     └────────┬─────────┘
                                                         │
                    ┌─────────────────────┐               ▼
                    │  Eval Harness       │◀─────────────────────────
                    │  (golden set +     │
                    │  LLM-as-judge)     │
                    └─────────────────────┘

A few design choices mattered more than the model itself:

1. Retrieval, not memorization. The model does not "know" our funnel. We embed the last 90 days of event metadata and retrieve the most similar 200 records per query. The model reasons over evidence, not from memory. This single decision cut hallucinated channel names by roughly an order of magnitude.

2. Structured output, narrative wrapper. The model first emits a JSON object with drivers, counterfactual, confidence, and citations. A second, cheaper pass turns that into prose. Splitting reasoning from narration is a small trick that makes the output dramatically more stable.

3. A golden set of 340 hand-annotated revenue events. This is the unglamorous hero of the system. We spent three weeks building it. Every evaluation, every prompt tweak, every model swap is judged against it.

4. Temperature 0.2, top-p 0.9. Attribution is not creative writing. We want near-deterministic outputs with a small band of natural-language variety.

The Numbers That Convinced Us

Here's the honest scoreboard after four months in production:

A few annotations on that table:

• Explained variance is the fraction of the revenue delta the explanation accounts for, scored by a frozen 340-event golden set.

• Executive satisfaction is a monthly 5-point survey of the seven people who actually ask attribution questions.

• The infra cost gap is almost entirely the retired warehouse jobs. The LLM cost is real but small.

Time-to-answer (log scale)
Old pipeline   ████████████████████████████  2.1 days
LLM service    ███  22 seconds

Engineering FTEs
Old pipeline   ███████████████████████  4.0
LLM service    ███  0.4

Monthly infra cost
Old pipeline   ████████████████████████████  $14,000
LLM service    ███████████  $3,200

The FTE line is the one that surprised us. We did not fire three people. We moved them to build the evaluation harness, the retrieval layer, and the next product. The fourth became a "model ops" engineer whose job is now almost entirely tending the golden set and watching for drift. That's a better job than maintaining a 400-line notebook.

What the Model Actually Does (And Doesn't)

The model is a reasoner over evidence, not an oracle. Concretely, when asked "why did EMEA revenue dip in Q3?" the pipeline:

1. Retrieves the 200 most similar event-clusters from the last 90 days.

2. Computes a small set of contrasts: channel mix, cohort age, pricing changes, seasonal baselines.

3. Asks the LLM to produce a causal narrative with explicit drivers (weighted) and a counterfactual ("if channel X had held flat, revenue would have been ~$Y").

4. A frozen judge-model scores the output against the golden annotation.

The model does not do the linear algebra. We still run a small Bayesian structural causal model under the hood to produce the weights. The LLM's job is to explain the numbers, not to invent them. This division of labor is the whole trick.

A concrete example output (abridged):

Question: Why did EMEA revenue dip in Q3?

Drivers (weighted):

• Channel mix shift toward paid search: −$184K (weight 0.41)

• Cohort aging in DACH: −$96K (weight 0.23)

• Pricing change on Tier-2 SKU: −$52K (weight 0.14)

• Seasonal baseline: −$38K (weight 0.10)

• Residual: −$21K (weight 0.07)

Counterfactual: Holding channel mix at Q2 levels, EMEA revenue would have been ≈ $2.41M (Δ ≈ $184K).

Confidence: 0.81 (5th–95th percentile of driver weights: [0.32, 0.52] for the top driver)

Citations: event_ids 11822, 11904, 12031, 12107; cohort report v2026.02.

The CFO reads the counterfactual. The growth team reads the weights. The CEO reads the first sentence. Everyone gets what they need from one artifact.

The Surprises

Surprise 1: The prompt is a product. We iterate on the system by iterating on the prompt and the retrieval window. There is no "deploy" step. A prompt change is a feature. A retrieval-window change is a bugfix. The mental model of "software" gets smaller, and the mental model of "evaluation" gets bigger.

Surprise 2: The golden set is the moat. Any team can wire an LLM to their event log. Not many teams will spend three weeks building a 340-event annotated corpus with two independent raters and a Cohen's κ of 0.82. That corpus is what lets us know the model is right, and it's what lets the CEO trust it.

Surprise 3: Explanations are easier to test than numbers. A regression test on a SQL view checks that the number is stable. A regression test on an explanation checks that the reasoning is stable, that the citations resolve, that the counterfactual is directionally correct, and that the confidence interval is calibrated. The test surface is richer, and the test failures are more diagnostic.

Surprise 4: The model is a better meeting participant. In a QBR, the old pipeline gave us a number and a bar chart. The new system gives us a paragraph that an executive can quote in an email. The format of the artifact changed, and that changed who could use it.

The Honest Costs

This is not free. We wanted to be explicit:

• Prompt drift is real. A model version bump in March changed our top-driver ordering on 6% of the golden set. We caught it because we run the eval harness on every deploy. The harness is non-optional.

• Latency has a floor. 22 seconds median is great for an exec. It is not great for a dashboard that needs to refresh every 5 minutes. We run the pipeline on a 15-minute cron for the dashboard view and on-demand for the narrative view. Two systems, one model.

• The model is a liability surface. We log every prompt, every retrieval, every output, and every judge score. If an executive asks "why did you say that?" we can replay the exact query. Reproducibility is a compliance feature, not a nice-to-have.

• People have to change roles. The data engineers are now half-ML-ops, half-product. The analysts are now half-prompt-engineers, half-statisticians. That's a real culture shift and it doesn't happen in a sprint.

A Small Decision Tree for Your Team

If you're considering a similar swap, here's the heuristic we wish we'd had:

Can you write down a 340-case golden set in 3 weeks?
├── Yes ── Is the current answer latency > 1 hour?
│         ├── Yes ── Do the consumers care about the *reason*, not just the number?
│         │         ├── Yes ── Do it. Budget 2 FTE-months.
│         │         └── No  ── A simpler retrieval-augmented query will do.
│         └── No  ── A tuned SQL view + a good analyst is cheaper.
└── No ── Build the golden set first. It's the real project.

The golden set is the actual project. The model is the easy part.

What We'd Tell a Younger Version of Ourselves

1. Write the spec before you buy the model. "30 seconds, natural language, with confidence intervals, on demand" is a spec. "Make it smarter" is a vibe.

2. Treat the prompt as production code. Version it, test it, review it, roll back.

3. Keep a deterministic model under the hood for the weights. Let the LLM do the prose.

4. Hire for evaluation, not for pipelines. The pipeline is 1,200 lines. The evaluation harness is the product.

5. Measure executive satisfaction monthly. It's the number that actually correlates with the budget line.

Closing

We did not replace a team with a model. We replaced a pipeline with a service, and we reallocated the team to the parts of the job that a model can't do: building the golden set, tending the evaluation, and writing the spec.

The four engineers are all still here. Two of them are happier than they were. The CEO gets her answer in 22 seconds. The CFO gets a counterfactual he can quote. The growth team gets weights they can argue with.

And the 400-line notebook is in a read-only folder, where the 2023 version of us is preserved in amber.

That's what a good migration looks like.

Dr. Elena Vasquez is a doctoral researcher in artificial intelligence, specializing in retrieval-augmented reasoning systems and evaluation methodology. She has advised product teams on LLM-integrated analytics pipelines since 2023.

Article word count: ~1,500.