When an Overall Trend Reverses by Segment: How to Check for Simpson’s Paradox

A conversion rate falls from 50% to 35.5%. That looks like a clear deterioration.

Now split the result by campaign. One campaign rose from 80% to 90%. The other rose from 20% to 30%. Both improved, yet the combined rate fell.

This is the business form of Simpson’s paradox: a trend in the total disappears or reverses when the data are examined within meaningful groups. It is less a statistical magic trick than a question about changing mix.

In this example, the lower-converting campaign brought a far larger share of visits in the second period. Its improvement was real, but its much greater weight pulled down the overall result.

Neither number is automatically wrong. The observed 35.5% describes what the business experienced across its actual May traffic. The campaign-level rates describe performance among comparable campaign types. The useful question is which one answers the decision at hand.

Start with counts and changing shares

Percentages alone can hide the mechanism. Keep the numerator and denominator for every segment, then calculate each segment’s share of the eligible population.

Here is the example in full:

April’s total was 100 conversions from 200 visits, or 50%. May’s was 390 from 1,100, or 35.5%.

The total is a weighted average. Each campaign’s rate matters, but so does its share of all visits. In May, nearly all traffic came through the campaign that still converted at the lower rate. The mix shift outweighed improvement within both campaigns.

A compact table like this is often enough to identify the issue. Add one more optional column—rate multiplied by segment share—to show each segment’s contribution to the total rate.

Ask two questions, not one

A Simpson’s-paradox check separates questions that reports often blend together:

1. Did performance change within comparable segments? 2. What outcome did the business experience under its actual mix of activity?

For campaign optimization, the first question may matter most. If both campaigns improved, it would be misleading to claim the campaigns became less effective based only on the total conversion rate.

For revenue planning or staffing, the second may matter more. A 35.5% observed conversion rate still affects sales volume, capacity, and budget, regardless of why it occurred.

The practical interpretation could be: campaign effectiveness improved, but acquisition mix moved sharply toward lower-intent traffic. That points toward decisions about targeting, routing, channel economics, or capacity planning—not a conclusion that either campaign failed.

Use a fixed mix to isolate rate change

A standardized comparison makes the distinction visible. Choose one set of segment weights, then apply both periods’ segment rates to those same weights.

Using April’s 50/50 campaign mix, May’s standardized conversion rate is:

(50% × 90%) + (50% × 30%) = 60%

Under the April mix, May’s within-campaign performance would produce a 60% rate, compared with April’s observed 50%. That is a useful comparison of segment-level improvement.

It is not May’s actual conversion rate. The actual rate remains 35.5%, because May did not have April’s mix.

Choose the weights deliberately and label them. Prior-period mix is common for a before-and-after comparison. A target customer mix may be more useful for planning. A pooled mix can be reasonable for a neutral descriptive comparison. Different defensible weights can yield different standardized summaries.

Choose segments before hunting for a reversal

Potential segments should come from how the work happens, not from an open-ended search for a striking chart. Depending on the metric, plausible candidates might include acquisition channel, customer size, product family, geography, tenure, case complexity, shipping service, or lead-intent tier.

Then run a short audit:

• Preserve the outcome count and eligible denominator for each period.
• Calculate total and segment-level rates.
• Compare each segment’s share of the denominator across periods.
• Check whether the total trend reverses inside the relevant segments.
• Calculate a fixed-weight comparison when separating mix from within-segment change would help.
• Flag segments with sparse denominators before drawing a strong conclusion.

The same approach applies to defect rates, renewal rates, on-time delivery, support resolution, and many other business measures.

Segmentation is not an automatic correction

Splitting data does not always produce a fairer comparison. A segment can be a pre-existing driver of outcomes, but it can also be part of the action being evaluated or the result of a selection process.

For example, a downstream funnel stage might change because of a marketing treatment. Adjusting for that stage could remove part of the treatment’s effect rather than create a like-for-like comparison. Likewise, dividing customers into arbitrary bins can invent or conceal patterns.

Use process knowledge to decide whether a variable belongs in the analysis. Be modest about causal claims: a segmented observational comparison can describe associations and mix-adjusted comparisons, but it does not establish what caused the change.

The Berkeley graduate-admissions analysis from the 1970s remains a well-known historical example. Aggregate admissions figures suggested one pattern, while department-level patterns and differences in application destinations materially changed the interpretation. Its lesson is not that totals should be dismissed. It is that the structure of the population can matter as much as the headline result.

When an overall KPI and its segments tell different stories, report both. State the observed total, show the segment rates and mix, and explain which measure fits the decision. A reversal is a prompt to inspect the composition of the data—not permission to pick the most convenient number.