Detection: Top ad got more impressions than it earned

Key: false_winner_meta_bias Severity: High Confidence: 75–90%

What this rule detects

We flag an ad set when its apparent "winning" ad earned a disproportionate share of delivery without correspondingly better conversion economics. The signal is that the delivery system has rewarded an early lead rather than the genuinely best-performing creative. Concretely, the rule fires inside an active ad set with at least three active ads, where every ad has run for at least seven days, the top-impression ad has at least twice the impressions of the second-place ad, and the top ad's conversion rate (purchases ÷ clicks) is at least 15% lower than the combined conversion rate of the other ads — and that gap survives a Wilson confidence-interval test for significance. This is the detection we polish hardest. If a user only ever runs Honest Growth once, this is the finding we want them to leave with.

Why it matters in 2026

Meta's delivery model decides which ad in an ad set to serve based on a combination of predicted action rate, bid, and ad quality. The system has a well-documented tendency to lock in on the early leader: the first ad to record a meaningful conversion is rewarded with more impressions, which produces more conversions, which produces more impressions — a feedback loop that can persist long after a better sibling has entered the auction. Two 2024–2025 platform shifts make this worse. First, Advantage+ campaign structures and Advantage+ creative consolidate more delivery decisions into the system and reduce the per-ad levers a practitioner has to correct it. Second, the "Andromeda" retrieval and ad-ranking update — Meta's rebuilt machine-learning ad-retrieval stack — widened the candidate pool the system ranks from, which makes the early-leader feedback loop converge faster and harder. Layered on top, iOS 14.5+ App Tracking Transparency and the conversion modeling Meta uses to fill the resulting signal gaps mean the per-ad conversion counts the system optimizes against are themselves estimates. The practical result: the ad with the highest impression volume is presented to the practitioner as the obvious top performer, the practitioner pauses the others to "concentrate spend on what's working," and account performance degrades because the actually-best ad was starved out, not the one that won the lottery on day three.

The math

The core comparison is between the top-impression ad's conversion rate and the pooled conversion rate of every other ad in the ad set. Conversion rate is purchases divided by clicks:

p_hat = purchases / clicks

top:  p_top  = top_purchases  / top_clicks
rest: p_rest = sum(other_purchases) / sum(other_clicks)

A raw percentage gap between two rates is not evidence on its own — a small ad with few clicks can show a large gap by chance. So before we trust the gap we put a confidence interval around each rate. We use the Wilson score interval for a binomial proportion, which behaves correctly even when the rate is small or the sample is modest (unlike the normal-approximation "Wald" interval):

            p_hat + z^2/(2n)      z          /  p_hat(1 - p_hat)     z^2     \
center  =  ------------------    +/- -------- * sqrt( ----------------  +  -------- )
             1 + z^2/n           1 + z^2/n   \        n              4n^2    /

  p_hat = observed proportion (purchases / clicks)
  n     = sample size (clicks)
  z     = 1.96  for a 95% two-sided interval

The interval is [center - margin, center + margin]. We compute one interval for p_top and one for p_rest. If the two intervals overlap, the difference is within statistical noise and we refuse to fire — no matter how large the headline gap looks. Only when the intervals are disjoint do we proceed to the effect-size check.

For ranking which of several competing comparisons is most likely real, the two rates can also be compared with a two-proportion z-test, which yields a p-value. When an account produces many candidate comparisons at once, we guard against false discoveries with the Benjamini-Hochberg procedure. Sort the m p-values ascending, p(1) <= p(2) <= ... <= p(m), then find the largest rank k for which the p-value still sits under its rank-scaled threshold:

find largest k such that:   p(k) <= (k / m) * Q

  m = number of comparisons
  Q = target false-discovery rate (we use Q = 0.10)

reject (keep) comparisons ranked 1..k; drop the rest

A useful way to think about the choice between ads is the Beta-Binomial conjugate model. Treat each ad's true conversion rate as unknown and start from a Beta(alpha, beta) prior. Observing s purchases in n clicks updates it in closed form:

prior:      rate ~ Beta(alpha, beta)
data:       s purchases out of n clicks
posterior:  rate ~ Beta(alpha + s, beta + n - s)

posterior mean = (alpha + s) / (alpha + beta + n)

With a weak uniform prior Beta(1, 1), the top ad's posterior is Beta(1 + top_purchases, 1 + top_clicks - top_purchases) and the rest bucket's is Beta(1 + rest_purchases, 1 + rest_clicks - rest_purchases). The probability that the rest bucket's true rate beats the top ad's is the overlap integral of the two posteriors — when that probability is high, the "winner" is winning on delivery share, not on merit. The shipped rule uses the Wilson non-overlap test as its kill-switch because it is exact and cheap; the Beta-Binomial framing is the same conclusion expressed as a posterior.

The thresholds we use and why

Confidence is 90% when impression dominance is ≥ 3× and the CR deficit is ≥ 25%; otherwise 75%. These values are taken directly from the detection source and are the shipped defaults. The 15% deficit floor and the 2× dominance ratio are tuned defaults rather than statistically derived constants — they are the line we draw between "winning fairly" and "winning by attrition," and Nachiket should review them against real audit data.

Known false-positive cases and how we mitigate them

• The top ad is genuinely the best. Sometimes the early leader is also the best long-term performer. If the top ad's conversion rate is similar to or better than the rest, no finding is surfaced. The 15% deficit floor plus the Wilson non-overlap test together require the gap to be both large and statistically real.
• The other ads only recently launched. A new ad live for five days has low impressions because the system is still exploring it. We require every ad in the comparison set to have at least seven days of impressions.
• Small samples. A dramatic-looking CR gap on an ad with 12 clicks is noise. The 30-purchases-per-bucket floor and the Wilson interval test both exist to suppress this.
• The CR difference is by design. Some teams intentionally include a retargeting-style or brand creative in a prospecting ad set; it converts lower on purpose. If a specific ad is meant to underperform, the finding does not apply — though in that case the ad probably should not share an ad set with the prospecting creative.

A worked example

A $30K/month account runs an active prospecting ad set with four active ads, all live for the full 30-day audit window. Insights for the window:

Impression dominance: 480,000 ÷ 150,000 = 3.2× — clears the 3× strong threshold. Rest bucket combined: 191 purchases on 7,500 clicks, so p_rest = 2.547%. Top ad: p_top = 1.80%. Both buckets exceed 30 purchases.

Wilson 95% intervals (z = 1.96):

top:   p_hat = 173/9600 = 0.01802   ->  CI approx [0.0155, 0.0209]
rest:  p_hat = 191/7500 = 0.02547   ->  CI approx [0.0221, 0.0294]

The intervals do not overlap (0.0209 < 0.0221), so the gap is significant. CR deficit: (0.02547 − 0.01802) / 0.02547 = 0.2925, i.e. 29.3% — above the 25% strong threshold. Both strong conditions are met, so confidence is 90%.

Recoverable monthly value uses the deficit applied to the top ad's spend. If Ad A spent $9,000 in the 30-day window:

recoverable_in_range = top_ad_spend * deficit = 9000 * 0.2925 = $2,632
monthly_cost         = recoverable_in_range * (30 / days_in_range)
                     = 2632 * (30 / 30) = $2,632 / month

The finding tells the practitioner that roughly $2,600 a month of Ad A's spend is buying conversions at a rate a better sibling would beat — the fix is to reset the auction, not to pause Ad A.

Limitations

• The rule sees per-ad counts, not the auctions behind them. It infers early-leader lock-in from the impression-dominance plus CR-deficit pattern; it cannot observe Meta's internal predicted-action-rate scores directly.
• It compares conversion rate, not revenue or ROAS — deliberately, because revenue is noisier per ad and would bias the rule toward high-AOV creatives. An ad that converts less often but at much higher order value can still be a real winner the rule will flag.
• It operates within a single ad set. Cross-ad-set creative concentration is covered by a separate rule (creative_concentration_risk).
• Modeled conversions from ATT signal loss mean per-ad purchase counts are partly estimates. The 30-purchase floor and Wilson test reduce, but do not eliminate, the chance that modeling noise drives the gap.
• It cannot tell you why the better ad lost the auction — only that the delivery split and the conversion economics disagree.

Source

This methodology page is generated from apps/api/app/services/detections/false_winner_meta_bias.py. The detection code is open for inspection on GitHub. Disagree with how the rule fires? Open the file. Read the code. Tell us where we're wrong.