When you pass out questionnaires, you should seek a random distribution.
-- Ramage, Bean & Johnson p. 110

If you gather your own statistics, be sure you know how to analyze them correctly.
-- Hult & Huckin p. 135

Never mind that neither of those sets of authors discuss exactly HOW to get a random distribution or to analyze data correctly. (We give you some idea when we discuss statistics). The real problem with surveys is that it's just too easy for the survey writers to build in biases. The biases may or may not actually be their own - they may not even know they're doing it.

Consider the example published in the February 2000 issue of IEEE's Today's Engineer. It's a reader poll on the subject of "transnational workers" - internationals who come to the USA for specific short-term jobs.

There are eight questions in the survey. The questions are multiple choice, and the choices always are strongly agree/agree/disagree/ strongly disagree.

1. Current demand for high-tech workers in the US warrants the use of temporary international workers.
2. Free market forces, such as rising wages and greater investment in education and training cannot work fast enough to provide the skilled workers the US economy needs from its domestic workforce.
3. Most internationals who come to the US to study and stay for jobs these days are different from past immigrants who came to become Americans.
4. The health of our economy is more important than the health of our polity: as long as we are prospering, it doesn't matter if we develop a permanent foreight underclass of "transnational workers."
5. Whether imported workers want to become Americans is less important than our economic bottom line.
6. Transnational workers -- temporary foreign workers in the US who are then sent home -- should be an ever-increasing part of our labor market.
7. The traditional approach to immigration, that newcomers become Americans, is outmoded. Transnational workers are a better way.
8. I am prepared to accept law enforcement measures necessary to round up and deport foreign workers who don't leave when required.

Then the magazine asks for an e-mail address in the event you'd like to be interviewed on the subject.

This survey drew an angry response from Bruce Eisenstein, president of IEEE-USA. I'm not going to repeat the whole letter here. But here's an excerpt:

The survey -- as well as the tone and lexis, the choice of words, in parts of the survey - is inflammatory and improper by any definition of professionalism. In my opinion, the words of some of the survey questions were intentionally crafted to incite strong, even hateful, reactions from those responding to it. Simply stated, there is no place for this type of activity in the IEEE. As both a native-born American and an IEEE member since 1962, I will not respond to this survey. I encourage other members to do likewise. Such an incendiary and ultimately useless activity hurts everyone - including American-born engineers - and is not healthy for our organization.

Surveys are very easy to bias. Here are a few typical ways:

• Questions that are written "suggesting a right answer." The example I use is that of young men asking for a first date: "you wouldn't want to go out with me, would you?" :-)
• Data collected from a specific audience, particularly an incomplete audience. In this case, both the survey AND Eisenstein's comments are aimed at IEEE-USA, despite the facts that
  • IEEE is an international society, and
  • the entire subject is non-US citizens. Don't these folks get to answer for themselves?
• It's possible for a question to be sufficiently complex that it can't be addressed by only eight questions. There are all kinds of phenomena in engineering that can't be measured by only a few sensors; modes that can't be excited by only a few bongs or twangs. Are we oversimplifying issues to the point of strawmen when we condense them to a page? (I'm not saying that's being done here, but it might be.)
• Bias surveys by deadline. The magazine came out in March 2000 and data was to be collected by 03.31.2000. Who knows how many people didn't respond because they didn't see the survey before Eisenstein's letter? Who knows how many people responded *because of* Eisenstein's letter? In this case, only readers who could shoot off a fast response, most likely by e-mail, could be counted.
• Composing a survey on the basis of perceived emotional response. You can look at the questions and see that they reflect a feeling that transnational workers are (putting it mildly) a concern. If Eisenstein thinks that this bias is an invention of the survey writers, he may be sadly mistaken. I have seen engineers who believe the use of transnational workers has been a prime consideration in the loss of jobs for American engineers and the trend to "temp" the jobs that remain.

That may be unprofessional, as Eisenstein says, but if it is it's only the kind of unprofessional that you and I have to deal with, oh, just about every day. :-) To me, the real lesson we have to learn is that if we want a survey to actually *reveal* something, especially when that something deals with human behavior, we have to make the survey as unbiased as possible.

Here is a list of potential pitfalls to watch out for when taking surveys:

Those taking the survey

1. What people SAY about themselves may not only not be the truth, it may bear no relation to the truth!
   • information is lost when behavior is self-reported in the first place
   • information is also lost when self-reported behavior is based on what people *believe* they did
2. Sometimes people will tell you what they think you want to hear.
3. Sometimes people LIE.
4. Sometimes people are distracted by lack of interest, time of day, mood, appetite, or lack of preparation.
5. Sometimes people can't give their real answer because it's not a choice. This often happens with "I don't care."

Those making the survey

1. Sometimes questions are written so as to reflect a "right answer."

   Since data taken this way has no real meaning, sometimes you'll see types of questions repeated several times (this happens often in personnel-related surveys), using different words and in random order.

2. Sometimes questions are biased based on terminology alone.

   A question that's otherwise apparently unbiased can be perceived differently by different responders depending on how well they understand terms used. If jargon is included, the survey has a built-in bias favoring those schooled in the jargon.

3. Sometimes questions are biased based on audience.

   To get an unbiased perspective, you have to aim your survey at your entire audience.

   Political surveys will often sample just a few hundred people and project the results over the whole population. The differences between such an approach and what was done by the IEEE are

   • uncertainty involved in the approach is calculable, and those who take the surveys present that number with the results
   • the political survey is aimed (theoretically) at a cross-section of society, in terms of race, geographic region and economic status.

Our company does this every year - the survey asks "do you think management is doing a good job of implementing the customer service model?" No one at the company (especially management) understands the customer service model. And no one's addressed the issue of whether or not it's a good idea (whatever it is). I've been at meetings where people ask managers what the customer service model is and all they get back is a cloud of fog. Our written documents discussing the customer service model are likewise largely comprised of fog.

Incentives

I have seen organizations provide incentives to survey subjects, such as coupons from stores interested in (for instance) results of surveys on shopping habits. Some engineering trade publications have recently given readers each a crisp, new one-dollar bill, saying it's a way of compensating them for taking the time to complete the survey. (My time is worth much more than that, but I appreciate the gesture so much that I take the surveys anyway.)

Surveys taken online can be convenient to the survey-taker, and they can provide the advantage of instant, colorful feedback to the survey subjects. That's not necessarily an incentive, but some Web-surfers will take an online survey (especially a short one) before they can realize their time could be better spent. :-) Some Web services, such as BeSeen, offer free one-question surveys (or BeSeen used to, until August 2002 anyway... sigh...):

(answers below)

1. Aim your survey at your entire audience. That means you have to know what that audience is before you start distributing.
2. Get multiple points of view involved in the composition, to ensure multiple points of view in the response.
3. Be sure your audience understands what the survey is about.
4. Provide some incentive to your audience to complete the survey.
5. Commit to take action based on the response. If your organization will not support this, the survey will waste time and money.