Skip to content

MODULE 2 · LESSON 9 · Working with tables

Cleaning missing values

Distinguish unknown values from observed zeros and document a defensible missing-data decision.

By Laurence Svekis · Self-paced · All worked data is synthetic unless you supply your own source.

Try the practice ↓

PredictInspectCalculateCheckExplainTransfer

Your starting point

Prerequisites: Use the ideas and vocabulary from lesson 8; revisit its worked examples if needed.

Definition: A missing value is an absent measurement or entry. It is not automatically zero, false or an average. Imputation substitutes an estimated or chosen value; exclusion omits affected observations from a particular calculation.

Investigate the reason for absence

A blank could mean not collected, not applicable, refused, lost during transfer or not yet available. These reasons have different implications. If long support cases are more likely to have missing response times, calculating an average only from recorded times may understate the service’s delays. Counting missingness is necessary but does not establish that excluding it is harmless.

Choose a policy for the question

Start by identifying required fields. A missing optional comment need not exclude a valid transaction from a revenue total. Removing every row with any blank can discard usable evidence. Conversely, filling a missing quantity with zero changes the implied event. Use a replacement only when the source or a stated model justifies it, and compare conclusions with and without that choice.

Report the denominator that remains

When a calculation ignores missing values, state the number of valid measurements alongside the original row count. Preserve the source and record the policy. In this workbench, descriptive statistics omit blank and nonnumeric entries and report both counts. Cleaning can remove rows or fill a chosen column, but the interface does not decide whether that action is scientifically appropriate.

Two worked examples

Unknown versus no time

Response times are 0, 10 and blank minutes. Zero is a recorded value, so the mean of the two observed measurements is 5. Replacing the blank with zero would produce 10 ÷ 3, about 3.33. Those results answer different assumptions; the lower mean is not automatically an improvement in accuracy.

Selective exclusion

An expense table has twenty complete amounts and three missing optional comments. A total of amounts can use all twenty records if the missing comments are irrelevant to that calculation. A thematic analysis of comments has a different usable set. The right row count depends on the fields needed for the stated question.

Where this becomes useful

A course analyst finds absent completion statuses for learners who joined late. Treating every blank as failure would encode an assumption about those learners. Investigate the reporting process and explain any unresolved classification.

Make an attempt

Load Customer support in the Cleaning Workbench. Inspect the one missing duration, compare the valid-measurement count with the row count, and write a policy before trying a fill or removal. Restore the source and compare how the policy affects the result.

Open the accompanying interactive lab →

Misconception check

Claim: Ignoring missing entries removes their influence on the conclusion.

Inspect the correction

It can change who or what is represented. Missingness may be related to the quantity you want to estimate.

Independent application

Create a small table with observed zeros, blanks and a not-applicable case. Define the meaning of each. Compare two possible policies, report their valid counts, and explain which policy fits your question without claiming that the missing values are known.

Completion records practice and self-review. It is not certification of statistical competence.