An Attempt to Develop a Scale to Measure Quality of Survey Data

Abstract

Sources of error in survey research are numerous and when described in discrete components may be said to consist of sampling and non-sampling error. One type of non-sampling error is respondent-originated error which results from respondent self reports of behavioural or psychological states and includes non-response error.
One source of respondent-originated error is response style. That is, a tendency to respond to a question or scale mostly as a function of factors other than the stimulus. Examples of response styles include random responding, yes-saying and no-saying.
Data quality is a broad issue, however, this paper focuses primarily on one aspect of data quality, response style, and specifically upon random responding. Although no systematic program of research has been undertaken to investigate their (response style) impact on data quality, an index of random responding could be constructed using the concept and approach of entropy. This paper describes such an investigation.
Broadly stated, entropy is a measure of centralized dispersion and is analagous to the concept of variance. Entropy was first introduced in the physical sciences where it is interpreted as a measure of uncertainty or randomness in a probabilistic system. Since its introduction, the concept of entropy has been applied in information theory, economic engineering, etc.
Since survey research may be considered to be operating in a communication context, the concept of entropy will be discussed here in information theory terms. As stated by Coombs, Dawes and Tversky (1970, p. 308):
The statistic "information " is a number that describes how much information is conveyed from source to receiver when given message is delivered. The assumption on which this quantification is based is that there are probabilities associated with the messages that may be delivered and that the amount ofinformation conveyed by a particular message may be determined from its probability of delivery.
The amount of information conveyed by a message is determined from its probability of delivery. (This is to say nothing about the content of the message; content is relevant only so far as it affects probability of delivery.) For example, if the author of a book conveys a highly probable message (one that had a high probability of delivery), then s/he will be considered to have said little or nothing (little or no information conveyed). However, if the message were highly improbable, (ie., a Communist author expounding upon the virtues of Capitalism), then s/he will be perceived as having said a great deal (high amount of information).