How to Use Difference-in-Differences to Evaluate a Business Change

A business change rarely happens in a vacuum. Sales may rise because demand is seasonal. Conversion may fall because a competitor cut prices. A new workflow may look successful because the teams selected for a pilot were already improving.

Difference-in-differences, often called DiD, is a practical way to ask a better question: what changed because of the intervention, rather than because the business environment changed around it?

It is useful when a clean randomized test is unavailable, but a credible unchanged comparison group remains. That condition matters more than the statistical method used afterward.

Start with four observations

DiD compares an outcome across two groups and two periods:

The calculation is:

> Estimated effect = (treated after − treated before) − (comparison after − comparison before)

Suppose a retailer pilots a checkout change in some stores.

• Pilot-store conversion rises from 4.0% to 5.2%, an increase of 1.2 percentage points.
• Comparison-store conversion rises from 4.5% to 5.0%, an increase of 0.5 points.

The DiD estimate is +0.7 percentage points: 1.2 minus 0.5.

The pilot stores improved by 1.2 points, but not all of that improvement should automatically be credited to the checkout change. The comparison stores suggest that 0.5 points may reflect a broader shift, such as seasonality, demand, or a companywide campaign. DiD removes common changes when they affect both groups similarly.

It can also accommodate stable baseline differences. The comparison stores started with higher conversion, and that is not automatically disqualifying. The central question is whether the groups would have moved along similar paths without the change.

The assumption that carries the analysis

That claim is called the parallel-trends assumption. It does not mean treated and comparison units need identical outcomes before launch. It means their average outcome trends would plausibly have been similar if the intervention had not occurred.

This is a counterfactual claim, so it cannot be proven from the data alone. But it can be made more or less credible.

Plot several pre-change periods for both groups. Look for persistent divergence, not just whether one pre-launch statistical test happens to be non-significant. A quiet pre-period supports the design, but a short or noisy history may fail to reveal meaningful differences.

Also document why the groups share relevant operating conditions: demand patterns, product availability, measurement rules, customer mix, and seasonal exposure. A comparison group that merely looks similar in a summary table may still be a poor counterfactual.

Design the evaluation before launch

DiD is more dependable when the rollout plan preserves the ability to evaluate it. Before implementation, make several choices explicit:

• Define the unit of analysis. It might be store-week, account-month, warehouse-day, or agent-week. The unit should match both how the change is delivered and how the outcome is measured.
• Choose one primary outcome. Define its numerator, denominator, and observation window before reviewing post-launch results. Secondary outcomes can help explain why a result occurred or reveal unintended effects.
• Set the exposure date carefully. Announcement, training, software enablement, and consistent operational use may be different dates.
• Keep enough pre-change history. Weekly or monthly observations before launch are needed to inspect trend patterns.
• Log concurrent changes. Promotions, territory changes, inventory disruptions, policy revisions, and reporting changes can create false signals.
• Protect the comparison group. It should remain unexposed for the evaluation period when ethical and practical.

Spillovers are especially easy to miss. A call-routing policy tested in one region may affect another if staff, inventory, advertising, or customers cross the regional boundary. Once the comparison group is affected, it no longer represents the outcome under no treatment.

Pressure-test the result

A point estimate is a starting point, not a verdict. Useful checks include trying reasonable alternative comparison groups, assigning placebo launch dates before the real rollout, excluding units with known concurrent changes, and examining results by post-launch period.

Check whether a single large store, account, or region drives the estimate. Report an uncertainty interval or range as well as the central estimate. Business data often contain related observations within stores, accounts, regions, or periods, so treating every row as independent can make an uncertain result look falsely precise.

A clear reporting statement might read:

> Relative to the comparison group, the pilot group’s conversion changed by an estimated 0.7 percentage points over the evaluation window. This estimate is consistent with an effect of the checkout change if the groups would otherwise have followed similar trends. Pre-period patterns and placebo checks provided supporting, not conclusive, evidence for that assumption.

That language is more useful than declaring that the intervention “caused” the outcome without qualification.

When DiD is not enough

Do not rely on DiD alone when there is one treated unit and no credible donor group, no pre-change observations, or no meaningful untreated comparison. A change deployed everywhere at once is another warning sign, especially if it coincides with a holiday campaign or major market shift.

DiD also struggles when the comparison group is exposed to the change, when the treated group experiences a separate shock at the same time, or when outcome definitions or group membership change during the study. In those cases, a randomized holdout, redesigned phased rollout, interrupted time series, synthetic control, or another design may be more suitable.

Staggered rollouts deserve extra care. If warehouses adopt a routing tool in January, March, and June, do not assume one conventional two-way fixed-effects coefficient gives a reliable summary. Effects may differ by adoption cohort or months since adoption. Use an approach built for differing adoption dates, compare newly treated units with not-yet-treated or never-treated units where possible, and report cohort or event-time effects.

The best time to make DiD credible is before the business change is complete. Pre-specify the adoption calendar, retain a valid comparison group temporarily, record exposure dates, and track other interventions. The formula is short. Building a comparison that deserves to be believed is the real work.