Why duplicate records are a context problem, not just an exact-match problem

A repeated name, email address, street address, or identifier is a reason to look closer. It is not, by itself, a reason to delete a row.

That distinction gets lost in many cleanup projects. Teams see two records that look alike and ask whether they match exactly. Exact matching is useful, especially when a stable identifier is available. But it answers a narrow question: do selected values agree character for character?

The business question is usually larger: are these records redundant for the work we need to do?

A practical way to answer it is to separate three questions that are often collapsed into one.

1. Do the records refer to the same entity?

This is the identity question. Two records may refer to the same company, person, location, or other real-world entity even when their values differ.

A company name may be abbreviated in one source and spelled out in another. A phone number may use different formatting. An address may have changed. One system may hold a legal name while another uses a trading name.

Exact agreement is fast, but it will miss some genuine links when data is incomplete, inaccurate, or represented differently. Conversely, an exact name match can be misleading. Common names, corporate groups, branches, and successor organizations can all produce records that look more alike than they are.

The right question is not, “Do these strings match?” It is, “Is there enough evidence that these records refer to the same entity for this purpose?”

That evidence might include an official registration number, website domain, address, source date, or other context. No single field should carry more certainty than it deserves.

2. Do they describe the same relationship, event, or object?

Even when two records identify the same entity, they may describe different facts.

Consider a transaction table with six invoices for one customer. The customer name repeats six times, but the rows are not duplicates. They are six distinct business events. Linking those invoices to one customer entity can improve customer-level reporting, while removing five rows would damage the accounting record.

The same principle applies to locations, contracts, contacts, source observations, and opportunities. A branch directory may contain three addresses for one brand. A one-row-per-brand view might consolidate them under a shared entity. A service-coverage analysis needs all three locations.

This is why row grain matters. Before deciding that rows are duplicates, state what one row is meant to represent:

• one company
• one legal entity
• one location
• one invoice
• one contract
• one observation from a source
• one version of a record over time

Two rows can be redundant in a prospect list and essential in an invoice file. The data did not change; the intended use did.

3. Are they statements about the same point in time?

Time often explains an apparent conflict.

Suppose a supplier file has an address from a 2023 filing and a different address from a 2026 verification. The records may refer to the same business, yet each can be correct in its own period. Treating them as interchangeable erases useful history.

They become a true conflict only if both records claim to describe the same attribute at the same relevant time.

Dates therefore need labels. Is a date the effective date, the observation date, the date the source published its data, or the date a system imported the record? Those are different facts. Without that context, a cleanup process can mistake change over time for bad data.

Identification is not the same as action

A likely entity match is an identification result. It does not dictate what should happen next.

Depending on the dataset’s purpose, the sensible action may be to keep both records, link them with a shared entity ID, create a consolidated reporting view, or remove a genuinely redundant import. Choosing among those actions requires business rules.

Source reliability, recency, completeness, legal requirements, and audit needs all matter. A current operational view may favor a recently verified address. A historical analysis may need every dated observation. Neither need requires destroying the original source record.

A safer pattern is often to preserve source records and their identifiers, add a justified shared entity ID, and build a purpose-specific view for reporting. This keeps analysis convenient without making the cleanup irreversible.

Measure the mistakes that matter

A high match count is not proof that matching worked well. It shows how many links were made, not whether those links were correct.

Two error types deserve attention. A false positive merges or links records that belong to separate entities. A false negative leaves records for one entity split apart. Which is worse depends on the use case.

For some work, an incorrect merge can distort a count, hide a relationship, or attach information to the wrong organization. In other settings, missed links may create fragmented reporting. Teams should evaluate both risks rather than celebrating a large number of matches.

Before removing or merging rows, retain enough context to revisit the decision: source system, source record ID, dates of observation and update, and what relationship the row represents. Route ambiguous cases for review when the consequences justify it.

The useful goal is not the fewest rows. It is a dataset whose rows mean what users think they mean. Exact matching can help get there, but context decides whether two records are truly duplicates.