Sunday, February 24, 2008

Questionnaire redesigned


http://www.freepatentsonline.com/7121830.html

This article describes an intervention to redesign questionnaires that measure human performance, primarily of those in leadership positions. The article offers some salient limitations to the current use of evaluation and frequency scales. The intervention/invention seems feasible to me but the last part is a bit statistical or more quantitative (so those among us who use this aspect of the left brain a bit more than me can review and tell me what you think :) .

Proposed arguments

  • The author distinguishes between evaluation scales and frequency magnitude. The former judge how well the individual performs and the latter tell how often the individual displays or engages in a behavior.
  • These scales often report two extreme findings the underdone and the overdone. He asserts that it easily identifies the underdone but assumes that high scores are good and people often make invalid inferences. For example, a high rating of 4 (“Frequently, if not always”) in response to the item “Speaks up in a group” using the frequency/magnitude scale could mean one of two things: either the individual speaks up appropriately often or that the individual dominates group discussions. Therefore, a high score does not mean that the individual is an effective communicator and group leader, etc. Again, there is no provision for the overdo extreme.”
  • These scales often do not provide the reasons why the individuals were given the ratings they received.
  • They are lopsided as they do not assess complimentary attitudes and characteristics.

Solution

  • Have a questionnaire that requires participants to rate along a continuum. Where the middle indicates balance, represented by a zero (0). The deficient end will be represented by increasingly negative (-) figures and the overdone end is represented by increasingly positive (+) figures.

← Too little (-1 or more)-The right amount (0)-Too much (+1 or mote) →

  • Second the questionnaire will assess complimentary characteristics and attitudes but will do so implicitly and the questions for matching characteristics will not be placed beside each other. (This is where the statistics comes in)

1 comment:

Wayne Nelson said...

I like the suggestions about designing questions that require a continuum response instead of a specific value. The problem is that people who use the results of surveys want hard numbers, and it would be more difficult to analyze the data of a continuum.

However, I also believe you should force a choice, even if you are using whole numbers. For example, when designing responses to survey questions, always use an even number of possible choices, rather than an odd number. Then, you force the respondent to move on either side of the "middle" response. Here's a more explicit example:

Respond to the following by circling the choice that best represents your belief.

1. This post makes sense.
(a) Strongly agree
(b) Agree
(c) Disagree
(d) Strongly disagree

is better than:

1. This post makes sense.
(a) Strongly agree
(b) Agree
(c) Neutral
(d) Disagree
(e) Strongly disagree

Unless you want to know how many people are "neutral", why would you use the second question?