Measurement you can defend to a skeptical CFO.
Five studies, one system. Experiments anchor it, the model extends it, held-out evidence validates it, and reconciliation is honest about what agrees, what disagrees, and what was never independent in the first place.
Evidence tiers
Not all evidence is equal, so we don't treat it equally.
Most measurement failure starts by averaging strong and weak evidence into one blended number. We hold the tiers apart: ground truth anchors, models extend, and biased instruments get reconciled but never trusted.
Ground truth
Incrementality tests · Geo experiments
Randomized and quasi-experimental designs: the strongest causal evidence for what they cover. These anchor the system, and everything else is checked against them.
Broad, modeled
Marketing Mix Modeling
Covers every channel, every week, including where no experiment can reach. On its own, MMM is weakly identified; in our system it is anchored to experiments rather than left to fit a story.
Biased instruments
Attribution · Brand-lift surveys
Useful but systematically biased: attribution over-credits branded search and retargeting, and surveys measure stated intent rather than spent dollars. We reconcile against them and never calibrate to them.
Common currency
Every study speaks the same language.
Studies can't be compared if they answer in different units. Everything revenue-affecting is normalized to incremental return on ad spend (iROAS) per channel per period, always with a 90% uncertainty interval. In plain terms: incremental revenue per dollar spent.
- Attribution is labeled, not laundered. Attribution ROAS is carried alongside, tagged as biased, and its divergence from the model is treated as a finding worth reading.
- Brand lift stays off the numeric track. Surveys measure brand outcomes, not revenue; forcing them into iROAS would be false precision. They're checked directionally against market behavior instead.
- Low-spend guard. Return per dollar is meaningless as spend approaches zero, so channels below a minimum-spend threshold report incremental revenue only and sit out reconciliation that period.
The pipeline
Four operations, one direction: evidence flows from experiments to the model.
01
Calibrate
Experiment results enter the mix model as evidence about the channel's response curve at the spend level the experiment actually measured, never as hand-picked priors. Trust is mechanical: a tight geo test constrains the model hard, a noisy platform readout constrains it loosely. Old experiments decay, and evidence more than two years old drops out entirely, so the system stays able to detect that effectiveness changed.
02
Extend
The calibrated model covers what experiments can't: every channel, every period, always-on. That is MMM's job in the system: breadth, anchored by ground truth.
03
Validate
Every new experiment is first scored against the current model, which has never seen it, before it is allowed to calibrate the next one. Each experiment is an honest out-of-sample test exactly once. You cannot grade your own homework, so we let time do the split.
04
Reconcile
Where two independent reads exist, we compare them with their uncertainty and classify agreement on the interval of the difference rather than gut feel or a fixed tolerance. Genuine disagreement is surfaced as a finding, because it tells you exactly where to run the next experiment.
How agreement is classified
When two methods are compared, the verdict comes from the uncertainty of their difference: aligned when zero sits comfortably inside it, watch when it's drifting to the edge, and conflict when the methods disagree by more than their combined uncertainty. The thresholds are deterministic and documented once. There is no tolerance knob to quietly turn until the answer looks right.
Rigor per study
How each study earns its place.
Geo experiments: designed before they're believed
Matched-market tests are analyzed with Bayesian synthetic control: treated markets against a modeled counterfactual from untreated ones. Designs are powered by simulating placebo tests on the client's own history, and an underpowered test doesn't launch: noise that would then calibrate the model is worse than no test. Every design is registered before the outcome is known, which makes “the model called it before the test read out” provable.
Incrementality: honest about where causal evidence comes from
Platform conversion-lift studies (Meta, Google, TikTok) are real RCTs, but the seller is grading its own homework, so a platform readout is never the sole anchor where a self-run geo test is feasible. Owned channels (email, SMS, direct mail) get true self-run holdouts. Provenance travels with every number.
MMM: gated before it can speak
A Bayesian mix model (weekly grain; adstock and saturation per channel; seasonality, weather, events, and price controlled) must pass deterministic QA gates before a result exists:
- Convergence. The sampler must actually converge; divergent or unstable fits fail loud and never publish.
- Configuration sanity. Simulated response must land within plausible bounds, so a config error fails the run before it can become a published number.
- Predictive checks. The fitted model must reproduce the observed series without systematic residual structure.
- Held-out accuracy. A validation refit holds out the final eight weeks; the model's 90% interval must cover what actually happened, within a documented tolerance.
A failed gate fails loud: the run is recorded, marked unpublishable, and a human investigates before anything reaches you.
Decisions on the margin, not the average
A channel with a healthy 3.0× average return may already be saturated, and the next dollar might return 0.8×. Budget guidance comes from the response curve and marginal return at the current spend level, with its uncertainty band. Average ROAS presented as allocation advice is the industry's favorite way to mislead; we don't do it.
The honesty rules
The rules that keep the system from flattering itself.
Circular agreement is never reported as agreement.
Where the model was calibrated by an experiment, model-matches-experiment is true by construction. That isn't evidence, and we never present it as a win. Reported reconciliation covers only genuinely independent reads: model vs. attribution, and model vs. experiments it was never calibrated on. Calibration provenance is tracked so the two can't blur.
Expected bias isn't news; unexpected bias is.
Attribution predictably over-credits branded search and retargeting. When it diverges in the expected direction, that's annotated as expected; headlines are reserved for divergences in the wrong direction or of surprising size.
Every published number can be reproduced, months later.
If a Q2 number is challenged in Q4, we re-run and get the identical result: input data is snapshotted at publish time, external covariates are owned copies that a provider restatement can't rewrite, the compute environment and random seeds are pinned and recorded, and the published posterior itself is stored as the first line of audit.
We don't report numbers the platforms are still changing.
Ad platforms restate their own recent history: conversions revise for weeks and spend adjusts retroactively. Published results read only settled windows, data old enough that the source has stopped rewriting it. That costs a little recency, and it buys the number a shelf life: what you saw at publish is what a re-run reproduces.
AI never decides a number.
The model fits, the causal inference, and every agreement verdict are deterministic, auditable statistics; no generative model sits in the measurement path. Where AI does appear (drafting the explanation of a conflict, answering an analyst's question about your results) it works from figures the pipeline already computed and cites what it read. It can explain the measurement; it cannot change it.
Deliberately not built.
No grand unified meta-model pooling all five studies: academically satisfying, un-debuggable, unexplainable to the person paying for it. No automated conflict resolution: deterministic logic decides whether a conflict exists, a human decides what it means. Restraint is part of the methodology.
Want this level of rigor on your measurement?
Whether you're standing up a system from scratch or want a second opinion on the numbers you have, let's talk.