How to determine incrementality on Walmart Connect
We held search spend at $350/day for a full week, down 94.6% from the ~$7k/day we'd been running. In-store sales barely moved while digital penetration tanked. This test gives us a real number on how much incremental revenue Walmart search is actually driving for Brand A.
$2.15of incremental total GMV per $1 of search spend
Six different baselines all land between $2.00 and $2.36. Block bootstrap puts the 90%
interval at $1.74–$2.63 — every single one of 5,000 resamples came back above
$1.00. This effect is real, and it's not small.
That's how much this channel is really pulling in: turn it off for a week and you lose
$49,136 in net GMV — $91,726 in sales, to save $42,590 in media.
THE WEEK ITSELF
Digital fell 35%. In-store didn't change.
All three comparison weeks run Thursday to Wednesday, so this isn't a day-of-week thing. Everything below is day-of-week adjusted and indexed to the Aug 13–19 baseline.
In-store GMV stays inside a 93–104 band the whole month, blackout included. Digital drops to 64. If this were a demand shock, an out-of-stock, a price change or a seasonal dip, both lines would've moved together. Only the advertised channel did.
Rolling week
Spend / day
Digital GMV / day
In-store / day
Total GMV / day
Units / day
Digital share
Aug 6–12Thu–Wed
$6,214
$33,738
$34,539
$68,277
10,765
49.4%
Aug 13–19Thu–Wed · baseline
$6,434
$35,883
$36,253
$72,136
11,647
49.7%
Aug 20–26Thu–Wed · spend at floor
$350
$23,293
$36,553
$59,846
9,805
38.9%
Aug 27–30Thu–Sun · ramp back
$4,953
$37,878
$39,567
$77,445
12,284
48.9%
−94.6%
Search spend, Aug 20–26 vs prior week
−35.1%
Digital GMV, same comparison
+0.8%
In-store GMV — statistically indistinguishable from no change
−10.8pp
Digital share of basket, 49.7% → 38.9%
CHANNEL MIX
Only 5% of the lost digital volume showed up in the store
Digital GMV fell ~$96k against its counterfactual; total GMV fell $91,726. The $4,967 gap is the in-store clawback — people who would've ordered online instead grabbed it off the shelf. That's real and it's statistically detectable, but it only recovers a twentieth of the loss.
The other 95%? Just didn't happen.
Digital share of basket tracks spend almost one-for-one, ramp back included. Recovery lag is about a day: on 8/27 spend was already back to a third of baseline, but share was still stuck at 37.7%. By 8/28 it was back to 48.1%.
HOW LARGE, AND HOW SURE
Every reasonable way of drawing the baseline gives the same answer
The easiest way to break a number like this is to move the comparison window around. So we ran six variations — one drops 8/19 (spend was already being throttled that day), one drops 8/20 in case of same-day carryover. Spread across all six: 36 cents.
Filled dots = total GMV per $1. Open rings = digital GMV per $1. The shaded band
is the block-bootstrap 90% interval on our headline spec.
Outcome (log), day-of-week controlled
Elasticity
Std. error
95% interval
p
R²
Read
Digital GMV
+0.1394
0.0071
+0.126 to +0.153
<0.001
0.953
clear effect
Total GMV
+0.0550
0.0050
+0.045 to +0.065
<0.001
0.973
clear effect
Units
+0.0439
0.0078
+0.029 to +0.059
<0.001
0.954
clear effect
Digital penetration
+0.1517
0.0059
+0.140 to +0.163
<0.001
0.899
clear effect
In-store GMV (placebo)
−0.0122
0.0041
−0.020 to −0.004
0.003
0.979
economically null
n = 30 days. Newey–West standard errors, 3 lags, because daily GMV is autocorrelated
(Durbin–Watson 1.12) and naive OLS errors would be too narrow. Aug 31 is excluded — no spend
was reported against it.
The falsification test holds up
In-store GMV is our placebo — search ads shouldn't touch it much. Its elasticity is −0.012, statistically detectable but economically trivial, and pointing the opposite way. That's the small substitution effect from above, showing up again. If the model were just picking up general demand swings, in-store would've moved just as hard as digital. It didn't.
We also re-ran the same model on every possible seven-day window inside the pre-blackout period, where nothing was cut. The biggest fake effect any of those windows produced was 9.3%. The real window produced 31.2% — 4.2x larger than the worst false positive the data can generate.
THE LIMIT OF THIS DATA
This proves search should be on. It says nothing about the right budget.
This is the part the platform deck always skips — and it's the part that should shape how the next test gets designed.
Outside the blackout, spend isn't independent of demand, it follows it. Daily spend correlates at r = 0.816 with the same day's sales last year — a pure demand proxy that couldn't possibly have been caused by this year's ads. Budgets pace up into high-traffic days, auction volume rises with shopper volume. So the natural swing between $4,000 and $10,000 a day is contaminated: high-spend days look productive partly because they were always going to be good days.
The on/off comparison is tightly estimated because the cut was forced on us, not
chosen. The marginal return at current spend levels isn't: point estimate is $0.72, but the
interval runs from −$1.12 to $2.55 (p = 0.45). A quadratic curve fit to the same range puts
saturation near $10,600/day, but that curvature term isn't significant either.
Bottom line: this isn't "we're overspending" and it isn't "we should spend more" either. One forced blackout in one month can tell you the channel works. It can't tell you where the curve bends. Any diminishing-returns claim off this dataset alone is just noise.
The one distinction that matters
$2.15 tells you the channel is real and worth keeping funded. It doesn't tell you whether dollar #10,001 on a given day is as productive as dollar #1. The blackout only ever tested the extremes — on vs. off. It never tested "off vs. slightly less on," which is the actual budget call you're making day to day.
THE TAKEAWAY
This is what actually moves digital penetration
Digital penetration — the share of Brand A's Walmart sales happening online instead of in-store — sat at 49.7% for weeks, then fell to 38.9% the moment search spend hit the floor. That's not noise: the regression shows a +0.1517 elasticity on digital penetration, one of the clearest effects in this whole analysis.
Flip the framing and this is the real headline: Walmart search is what captures the demand before it goes somewhere else. Cut spend and you don't just lose the sale — the shopper still searches, still buys, just not from Brand A. A competitor's product fills that spot instead, and that demand is gone for good. Fund the channel, and it's Brand A's SKU sitting there when the shopper looks.
