Design Decisions in Research

4. The Design and Planning Phase

Data Collection Methods

All instruments must have evidence of adequate reliability and validity. Instrument reliability refers to the extent to which an instrument consistently measures an attribute, variable or construct. Instrument validity refers to the extent to which the instrument measures what it is intended to measure. The process of instrument validation often requires several sequential studies, which often includes comparison with another similar instrument. For example, a valid questionnaire of physical activity should be highly associated with actual physical activity measured by an actigraph physical activity monitor. Researchers can find reliable and valid instruments that measure variables of interest for a particular study using computer searches and published compendiums (Bradley, 1994; Glasgow, 1997; Jones, 1996; Salek, 1999).

Table 8

Reliability and Validity

  Definition Statistical Test
Reliability The degree of consistency or dependability with which an instrument measures what it is designed to measure. Reliability coefficient (a correlation coefficient that range from 0 to 1.0) - For psychosocial measures, a reliability coefficient of .70 or higher is recommended. For physiological measures, a higher reliability coefficient is recommended.
Validity The degree to which an instrument measures what it is intended to measure. Not a single statistical test - Numerous techniques are used (ie., evaluating the correlation coefficient of the instrument with a gold standard or testing the instrument in groups anticipated to score high or low on the instrument) - Usually takes years to establish validity of an instrument.

Table previously published in: Whittemore, R. & Melkus, G. (2008). Designing a research study. The Diabetes Educator, 34, 201-216.

Bradley, C. (1994). Handbook of psychology and diabetes: A guide to psychological measurement. Switzerland: Psychology Press.
Glasgow, R. E. (1997). Behavioral and psychosocial measures for diabetes care: What is important to assess? Diabetes Spectrum, 10(1), 12-17.
Jones, R. L. (Ed.). (1996). Handbook of tests and measurements for Black populations. (Volumes 1 & 2). Hampton, VA: Cobb & Henry Publishers.
Salek, S. (1999). Compendium of Quality of Life Instruments. New York: John Wiley & Sons, Inc.