Objective Measurement of Subjective Phenomena

6. Problems in Measuring Constructs

Exercise 3

The goal of this exercise is to test your knowledge of solutions to potential problems in measurement in research. 

Instructions:
A researcher wants to anticipate potential problems in measurement when planning a study.  The research has used some of the measures in prior studies and will be assessing persons from a similar population in the planned study.  Analyses of prior data has shown certain problems with some of the items, and the research would like to avoid these problems in the future. 

Match each potential problem below with the approach that would best minimize the threat of the problem.

Potential Problems:
1.  Acquiescence – “Yea-saying” or “Nay-saying”
2.  Positive skewness – Data have longer tail at the high end of the scale
3.  Negative skewness – Data have longer tail at the low end of the scale
4.  Extremity – Tendency to use only extreme, or only middle values
5.  Halo – Bias that yields a generalized positive or negative evaluation
6.  Social desirability – Tendency to respond according to positive or negative value of item content
7.  Anchoring – Bias in ratings due to prior information

Approaches to Minimize the Threat of the Problem:

  • Try a log transformation or square root transformation.
  • Try squaring values.
  • Balance items so about half are positively worded and half are negatively worded.
  • Administer a social desirability scale to measure this problem during test development, and then discard items that correlate highly with the social desirability scale score.
  • If few subjects use low values on the rating scale, make the item “harder” by re-wording the item stem to make the item more difficult to agree with.
  • Provide explicit context for raters, avoiding idiosyncratic conceptions across raters in their understanding of the construct to be rated.
  • Have multiple raters rate each subject and then model the trait and rater effects in analyses.

Answers:
1.  Acquiescence – “Yea-saying” or “Nay-saying”
[Balance items so about half are positively worded and half are negatively worded.]

2.  Positive skewness – Data have longer tail at the high end of the scale
[Try a log transformation or square root transformation.]

3.  Negative skewness – Data have longer tail at the low end of the scale
[Try squaring values.]

4.  Extremity – Tendency to use only extreme, or only middle values
[If few subjects use low values on the rating scale, make the item “harder” by re-wording the item stem to make the item more difficult to agree with.]

5.  Halo – Bias that yields a generalized positive or negative evaluation
[Have multiple raters rate each subject and then model the trait and rater effects in analyses.]

6.  Social desirability – Tendency to respond according to positive or negative value of item content
[Administer a social desirability scale to measure this problem during test development, and then discard items that correlate highly with the social desirability scale score.]

7.  Anchoring – Bias in ratings due to prior information
[Provide explicit context for raters, avoiding idiosyncratic conceptions across raters in their understanding of the construct to be rated.]