Fairness

The third factor related to the quality of test scores is fairness. If test items or a test in general are biased against (or for) a person or subgroup and this difference is not related to what is being tested, then some may consider the test to be unfair. The technical term for this is construct-irrelevant variance, meaning there is extra noise in the test scores caused by something other than the skills and knowledge you are trying to measure.

For example, a test may include items that reference topics that are taboo for a specific subgroup of test takers and may contribute to missed correct answers. In this case, the test has introduced construct-irrelevant variance.

Another example is when items include unduly complicated language beyond technical vocabulary needed for content mastery. In this case, the test is not only measuring knowledge of the content domain but also proficiency of the English language. In some cases, these kinds of items might be biased against non-native speakers.

Fairness also includes providing reasonable accommodations for students with documented disabilities. Usually, guidelines provided by your institution can help you navigate these situations, but general awareness of the topic can help you know what to expect.