Your starting point
Prerequisites: Use the ideas and vocabulary from lesson 9; revisit its worked examples if needed.
Definition: An exact duplicate repeats every field. A duplicate key repeats a field or field combination expected to identify a unique record. A repeated event may still differ in formatting or metadata, so duplication is a question about meaning as well as equality.
Define the key before searching
A customer ID is not a unique order key because a customer can place many orders. An invoice ID might be expected to be unique, but an invoice-line table legitimately repeats it for several products. State the grain first, then choose a key. Counting repeated category labels such as region is not a duplicate test for transactions.
Inspect groups rather than deleting blindly
An exact copy can be a repeated export record, while two records with the same ID and different values may be revisions or data-entry conflicts. Keep the original records and compare dates, status and source information. A keep-first rule is a convenience, not evidence that the first record is authoritative. If no version rule exists, flag the conflict for investigation.
Check the effect of removal
Record the number of groups, extra records and affected totals. Three identical rows form one group with two extra records if one is retained. This distinction matters when reporting the cleaning result. Re-run the check after trimming spaces only when trimming is justified: normalization can reveal duplication but can also merge distinct labels if applied carelessly.
Two worked examples
Exact repeated invoice
A fictional invoice table contains rows A: 100, B: 80 and a second identical A: 100. There are three rows, one duplicate group and one extra record. If the second A is confirmed as a repeated import, the total changes from 280 to 180 after removal. The confirmation is part of the evidence.
Legitimate repetition
An order-line table contains order R1 with product Pen and order R1 with product Notebook. The repeated order ID is expected. The pair of order ID and line number can identify each record. Removing one row merely because R1 repeats would undercount the order’s items.
Where this becomes useful
A nonprofit merges registration exports from two days. Some people appear in both, while others booked multiple sessions. Decide whether the analytic unit is a person, registration or session attendance before choosing a deduplication key.
Make an attempt
Open the Duplicate Detection Challenge with Small business operations. Predict the number of extra exact records, then compare the result when using invoice_id or service as a key. Explain why repeating a service category does not justify deletion.
Misconception check
Claim: Remove duplicates is always a harmless cleanup button.
Inspect the correction
It can remove real events if the selected fields do not identify the intended record. Inspect and document the key.
Independent application
Construct a table containing an exact duplicate, a repeated customer with two real orders, and a conflicting repeated order ID. Write a different decision for each case and reconcile row counts and totals before and after any confirmed removal.
Completion records practice and self-review. It is not certification of statistical competence.
