From 100 feet away, Eric Burbidge saw that the #99 bus was filthy. It would smell on the inside, he was sure. Forty riders were docilely lined up to board -- one was a letter carrier, and a collection of other blue collar men and women comprised the rest. Some were idly thumbing through newspapers -- Good! Very good! -- some were chatting, and two were ruminating slowly on hero sandwiches, which he supposed they had picked up at the tavern across the street from the bus station.

Burbidge swept past the queued passengers, taking care not to make premature eye contact, and brusquely rapped his clipboard against the bus's folding door. With a whoosh the driver snapped the door open, and Burbidge heaved himself up onto the bus. No one tried to follow him up, which he would not have allowed anyway, as he needed time to interrogate the driver, to see if the man would be of any help whatsoever.

"Good evening, driver. I am Mr. Burbidge, and I am from headquarters."

"That figures -- We don't see many suits on route 99."

"Then you know why I have had to come out here." He'd been with the bus company for three months, and this was the first time he had been required to board and ride a bus. What is that smell? Machine oil . . . sandwich meats . . . what else do the route's demographics suggest would be every day tracked through this bus? Sweat, of course! "I am conducting a scientific survey to determine the newspaper readership, if any, of riders on this route."

"Yessir," murmured the driver, without enthusiasm.

"If you have been reading your employees' newsletter, you must know that the corporation is soon to announce a restructuring of the route system and schedule, pursuant to which we shall have to purchase advertising space in the leading media to reveal our new route structure, maps, and schedules."

"Won't that be a mess," muttered the driver, in the same flat tone. "And, yessir, I read your newsletter. For 15 years I have read it."

"According to our records, you have been a driver on this very route for five years, and so you may have noticed that this route runs north-to-south equidistantly between the twin cities . . . ."

"The route runs north-south, with one city on the east and the other on the west, yessir. I caught on to that long ago, sir. It's right there on the route map, sir. . . . Clean splits the line between East City and West City and never gets closer to one than the other. If you want to get to one city or the other, you transfer to another bus. My bus just goes straight north. Right up there on the route map. Sir."

"As corporate research director," said Burbidge, gracefully emphasizing his point by pointing a forefinger roofward, "I take nothing for granted, but instead provide scientific methodology, in this instance to test my hypothesis that readership of newspapers on this route would be equally divided between the East City Gazette and the West City Tribune. To that very end I have prepared this survey of your riders, which with your complete cooperation I have selected to pass out along this route, on this day, at this time."

"Newspapers, you say," said the driver, showing a flicker of involvement. "I could tell you quite a lot about those newspapers . . . ."

"Not required! This is a scientific survey of riders on route 99, and anecdotal information is not appropriate. What I require of you is to let the passengers up onto the bus so I may give them these pencils and surveys, and then to refrain from swerving or unnecessarily agitating the bus while they are filling out my surveys. Do you think you can manage that?"

"I'd better turn on the inside lights, don't you think?"

"Well, of course, turn on the lights. That goes without saying."

The passengers boarded slowly, exchanging pleasantries with the driver. They were quite a little clan, Burbidge could see.

The bus rolled steadily northward, and Burbidge was pleased to see that riders hunched over his one page survey, thought he was struck by how long it took them to answer the simplest of questions. As each completed the survey he or she shyly shuffled forward and proffered it to Burbidge without comment.

Just when he sensed everything was going as well as might be expected, two men stood up, one in the front by the driver, one in the rear, and spread their legs and straddled the aisles. A ball of newsprint was tossed into the aisle, and the passengers began whooping and batting the paper forward and back. "What is this, driver?" demanded Burbidge.

"Hockey. They are playing hockey. The idea is to knock the ball between this guy's legs or that guy's in the back."

"Aren't you going to stop them?"

"No harm done. These are a friendly bunch. Big sports fans. . . . In fact," said the driver, whose voice had gained enthusiasm when evading discussion of company business, "the East City club is playing pro hockey tonight, so when I clean out the bus, most of the newspapers will be the East City Gazette." He rattled on, contributing to Burbidge's annoyance, explaining that the riders liked to study the night's pro game in advance, the better to discuss it among themselves, so that newsstand sales were brisk in the terminal, but only for the newspaper that did the better job of covering the sport de jour. "Of course tomorrow night there is pro basketball in West City, and most of the riders will pick up the West City Tribune at the newsstands."

"That's impossible to accept," shouted Burbidge. "Such behavior would bias my scientific survey, which asks for the paper they most recently purchased!".

"Well, you had better accept it. B'cause I've been cleaning out this bus for five years. It's the Gazette before hockey, and the Trib before basketball. This is a hockey crowd tonight, which means extra Gazettes."

Burbidge was mortified and hoped he wasn't revealing his distress to the driver. Surely the driver would not understand the consequences of this predicament in biasing the survey. But the driver added, " `Course, in the mornings these folks bring the papers that the newsboys toss on their lawns, so you don't see such a situation."

"That is very difficult to accept, driver. It means that by choosing between sampling this route -- morning or evening -- I get a systematically different set of results, and by choosing a hockey night or a basketball night, I further incline toward different results." Burbidge was thinking out loud, almost in a trance.

"Naturally . . ." agreed the driver, with irritating enthusiasm.

"Be still, driver. I have to think." Burbidge was shaken.

The driver had now warmed to his exposition of trash can sociological research. ". . .Of course, by reading my newsletter, as I do, I know that by the time you announce the new routes and schedules, we will be finished with hockey and basketball and into the baseball season . . . ." Was there irony in the driver's chuckle?

"If you are furnishing me information that is at all reliable, driver, then the generalizability of my survey would be contingent on whether I ride in the morning or evening, on the professional sports schedule on a particular night, and on the season of the year."

"That's part of it, yessir."

"Part of it? Part of it? You mean, there's more? There is something else I haven't thought of?"

"Well, yessir, because, you see, most of these folks on the 5:15 bus are East City folks, and most of the people on the 5:45 bus are West City folks, so your outcomes would naturally depend on whether you took survey on the 5:15 or 5:45 bus."

Burbidge was shaken and no longer able to able muster a vigorous "impossible to accept" or even a "difficult to accept."

"How can this be? Tell me."

"Well, you see, the 5:15 bus on this route -- the one you are on now -- gets to Boght Corners at 5:55, and the East City folks transfer onto the eastbound bus that is waiting there for them. Most of the West City folks hang out in the bar across the street from the terminal, get on the 5:45 bus and rendezvous at 6:25 with the westbound bus at Boght Corners."

"I see. I see." He was reconsidering the wisdom of this survey. "AND IS THERE ANYTHING ELSE you would care to share with me?"

"Only that the 5:45 boarders don't read the newspaper much at all, as they have been watching sports on the TV in the bar, are lightly soused, some of them, and can't read the small print because I don't turn on the inside lights."

"I might as well have stayed home," Burbidge cried in despair.

The driver stopped the bus and swiveled to face Burbidge. "Wouldn't that have been a pity, sir? as I would have been deprived of this excellent lesson in scientific research."

Burbidge studied the driver carefully, struggling to believe this last remark was not the unkindest cut of all. No, he concluded, the bumpkin is too disingenuous to be capable of irony.

Source: from Donald R. Cooper and C. William Emory, Business Research Methods, 5th ed. Burr Ridge, IL: Richard D. Irwin Publishers, 1995 (text and ordering information described elsewhere on this web site).

Concerned about your firm's approach to data collection? Convinced that political agenda skew research results to benefit short term thinkers? Perturbed about data that mislead your executives on product loyalty and customer satisfaction? Then you need to read this Research Tip.

Abstract:

While it is possible to produce completely error-free measurement by perfecting the components of sampling, question design, and interviewing skills, the value for management decisions is not normally worth the cost. Therefore, it is important that the tradeoffs made in the survey are the best, most statistically sound, and most practical. Relationships among sampling, question design, and interviewing are such that the findings produced will be no better than the design component most vulnerable to error. To justify expenditures, investments made to improve one component should be proportionate to the compromises that preexist in other areas. For example, if the questions being asked are difficult to answer, increasing the sample size to reduce sampling error is a misdirected expense. The application of these suggestions will enhance the objectivity of customer satisfaction research and strengthen management's ability to improve products and services.

There are many customer activities involving measurement and data collection that people call surveys. Some are carefully planned to yield results that are as objective as possible; others are less scientific but fulfill organizational requirements. In this Research Tip, we examine surveys that have the following characteristics: (a) the research purpose is to numerically describe a study population, (b) information is collected by asking questions of customers, and (c) part of the population, a sample, is the basis for evaluating the customer population as a whole.
Improved customer satisfaction measurement though the survey method can increase the likelihood that the data collected will describe what was intended. Sampling, question design, and interviewing are three key components for improving the survey process and affirming valid and reliable results. Their contributions to enhanced knowledge for executives, research directors, and internal consultants are discussed in the sections that follow.

When we do a customer satisfaction survey, we should be able to assure the decision makers that the sample is not biased and that the data have a definable level of precision. The best means of accomplishing this is to give all customers in a target population (e.g., new buyers, warranty claimants, 800-line users, and so forth) an equal chance of being selected. Probability methods should be used for selecting them. When we are unable to provide this assurance, our sampling constraints must be made known to decision makers.
Many information gathering activities--those designed to build customer relationships, secure impressions through focus groups, or poll competitors' customers--introduce bias through convenience samples (or sampling from lists that exclude important segments of the population). This does not result in representativeness anymore than the opinion of the loudest voice at a meeting or the most convenient person on a street corner. Nonprobability sampling methods have a place in customer satisfaction research but principally for exploratory work.
Because nonprobability sampling approaches offer cost-saving compromises, companies (and consultants) use them inappropriately. For example, we infer crucial customer perceptions from people that were not sampled at all (magazine readers who return product service questionnaires) or substitute willing and available respondents for refusals and "not-at-homes." We also fail to compare the responses of respondents with nonrespondents to check for nonresponse bias. An equally common mistake is to accept low response rates in mail surveys though research consistently shows t early responders to have biased views on the research objective. The problem with these compromises is they provide no scientific basis for generalizing from the sample to the population nor statistically credible evidence for quantifying error.

Solutions. First, involve a sampling specialist in the design of the sample, calculation of the sample size, and estimation of error. This is particularly important when the study's outcome influences the direction of business. Second, chose simple random, systematic, stratified, or another probability sample whenever possible to assure a known chance of selection for each person in the study population. Calculate the sample size based on the level of precision desired and the confidence level (80%, 95%, or other) you need. Rule-of-thumb sampling rarely produces an accurate information. Third, take a comprehensive look at error sources: the sample frame, response errors, and nonresponse errors when deciding the degree of precision needed to represent the population (more on this later). Fourth, when cost or the study's objectives make probability sampling prohibitive, consider a nonprobability method (such as a quota sample) that improves the chance of population characteristics being proportionally represented.

The importance of question design for reducing error is vastly underestimated. Perhaps because asking questions seems a commonplace rather than scientific activity. But answers to survey questions are the only measures of customer response. Thus, reliability and validity are at stake. Poor questions produce unreliable answers: they cause erratic results from similar respondents when consistency should occur; and, they are biased in one direction or the other from the "true" value (as when the number of units purchased is under reported).
Question design also affects validity, producing answers different from those intended. Respondents can answer accurately. We confirm this by checking factual questions with records or other sources. However, the level of accuracy depends on the topic and how questions are constructed. The most pertinent factors affecting accuracy are: the respondent's ability or knowledge of the answer, the respondent's willingness to answer, and how understandable the question is.

Solutions. Question content, wording, response structure, and question sequence can distort or improve the accuracy of results. Remove ambiguous or misleading wording, standardize the format (including the introduction and respondent encouragements if an interview is used), and take all possible steps to make question mean the same to all respondents. Pretesting is the best insurance! A thorough pretest does several things. It determines respondent interest, discovers question meanings, examines question continuity and flow, and checks for a respondent's modification of the question's intent. It also experiments with sequencing patterns, evaluates skip instructions for the interviewers, collects early warnings on variability, and fixes the length and time of the instrument.
Question sequence can drastically affect respondent willingness to cooperate and the quality of responses given. Generally, the sequence should begin with efforts to awaken the respondent's interest in continuing. Early questions should be simple rather than complex, easy rather than difficult, nonthreatening, and obviously germane to the announced objective. (Telemarketers sour hundreds of willing respondents daily to legitimate research by deception of purpose.) Frame-of -reference or scale changes should be minimal, and questions should be sequenced so that earlier ones do not distort later replies.
Take care when devising customer studies internally so that questions borrowed from existing customer satisfaction or marketing questionnaires fit. Borrowing questions has risks. The original questions were created for a specific population at a know time with context-specific reliability and validity issues in mind. Language and idiom should also be tailored to fit the new population.
Finally, it is reasonable to ask employees and consultants to document the steps they have taken to improve reliability and validity of question design. This discussion can accompany any report that has mission-critical status for management. The reduction of error through these suggestions is an inexpensive means of improving customer survey results.

Often customer satisfaction surveys are self-administered although telephone interviews are used commonly. Interviewer quality can make or break telephone interviews. Respondents typically react more to their feelings about the interviewer than to question content. Interviewers must be trained (!) to ask questions properly, record responses accurately, and probe meaningfully. Their knowledge helps them foster a good interviewing relationship.
It is difficult to separate the interview effects from sampling effects when interviewers are assigned samples nonrandomly. However, poorly trained or unsupervised interviewers can diminish the effective sample size by up to 30 percent. This means that the investment in a sample of 1000 will be reduced to an effective precision of 700.
Biased results come from three sources: sampling error, nonresponse error, and response error. Sampling error was previously discussed. Nonresponse error occurs when interviewers are unable to locate those whom they are supposed to study. It is an especially difficult problem when using a probability sample. If forced to interview substitutes, an unknown but possibly substantial bias is introduced. In a widely cited study of nonresponse, only 31% of all first calls were completed. The most reliable solution is to make callbacks. With enough attempts, most respondents can be contacted --although unlimited callbacks are expensive.
When the data reported differ from the actual data, response error occurs. These errors occur when the respondent fails to report fully and accurately. Also, errors in tabulating and processing data can create response errors. Although neither manifestation can be controlled completely, the use of trained and supervised interviewers helps.
Interviewer error is also a source of response bias. From the introduction to the conclusion of the interview, there are many points where the interviewer's control of the process can affect the response rate and the quality of the data. Reducing the average number of calls that an interviewer is required to make (say, from 100 to 50) is an inexpensive way to reduce this problem.

Solutions. Considering these problems, it is prudent for research directors and managers to recognize the constant potential for error in the interview. Economizing on interviewer training and supervision worsens the situation. Interviewers --including employees drafted to for call center work -- should receive training ranging from brief written instructions to extensive sessions, depending on the scope of the study. Written instructions should be a part of all data collection efforts. Such instructions should cover at least the general objectives of the study, something about problems encountered in the pretests and how they were solved. In addition, questions should be discussed separately, giving interviewers insight into the purpose of each question, examples of adequate and inadequate responses, and suggestions such as how to probe for more information. Definitions also help the interviewer explain and interpret respondents' queries with consistency.
Evidence on the value of training suggests that intensive training produces significant improvements in interviewer performance. The training effect can be so great that average employees without interviewing skills can show substantial improvements with little time and resource investment. The role of the interview in customer satisfaction surveys is just beginning to receive adequate attention. Effort in this area is a very cost-effective means of improving the precision of survey estimates.

In this Research Tip, we focus on the problems of measuring attitudes and opinions using a common scale, rating scales. This is a more technical note than our previous one, but an important topic for conscientious researchers.

What we measure in business is frequently complex and abstract while the available measurement tools can be crude or imprecise. Wanting a valid measurement, we often get something between the true score and the test score. When the object is a concrete and the measurement tool is standardized, the variation between the test and true scores will be small. This is like the accuracy you would expect in measuring the length of a table with a yardstick. If the concept is abstract (attitudes toward various firms) and the measurement tool is not standardized (questions about attitudes), then you will not be confident that the test results reflect true scores. This is comparable to using your forearm as a scale to measure the table.

"Scaling," by the way, is a procedure for assigning numbers (or other symbols) to the objects being measured. A more correct definition would be that one assigns numbers to indicants of the properties of objects. For example, a number scale is used to represent the various levels of heat and cold and one calls it a thermometer. Or, paint durability can be determined with a scrub brush attached to machine that applies a predetermined amount of pressure as it scrubs. By counting the number of brush strokes that it takes to wear through the paint's thickness, the scrub count becomes the indicant of the paint's durability. Or, you may judge a person's supervisory capacity (property) by asking a peer group to rate that person on various questions (indicants) that you devise.

Rating scales allow us to judge the properties of objects without reference to other similar objects. These ratings may be in such forms as "like-dislike," "approve-indifferent-disapprove," or other classifications using even more categories. There is little conclusive support for choosing a three-point scale over scales with five or more points except on cross-cultural grounds. One argument is that more points on a scale provide an opportunity for greater sensitivity of measurement and extraction of variance. The most widely used scales range from three to seven points, and it does not seem to make much difference which number is used.

Problems in Using Rating Scales. The value of rating scales for measurement purposes depends upon the assumption that a person can and will make good judgments. Before accepting respondents' ratings, we should consider their tendencies to make errors. Three of the most common are the errors of leniency, central tendency, and halo effect.

The error of leniency occurs when respondent is either an "easy rater" or "hard rater." The latter being an error of negative leniency. Raters are inclined to score those people higher whom they know well and with whom they are ego-involved. There is also the opposite of this situation-where one rates acquaintances lower because one is aware of the leniency danger and attempts to counteract it. A way to deal with positive leniency is to design the rating scale to anticipate it. An example might be an asymmetrical graphic scale which has only one unfavorable descriptive term and four favorable terms (poor-fair-good-very good-excellent). The scale designer expects that the mean rating will be near "good" and that there will be a symmetrical distribution about that point.

Raters are reluctant to give extreme judgments, and this fact accounts for the error of central tendency. This is most often seen where the rater does not know the person or object being rated. Efforts to counteract this error are to (1) adjust the strength of descriptive adjectives, (2) space the intermediate descriptive phrases farther apart in graphic scales, (3) provide smaller differences in meaning between steps near the ends of the scale than between the steps near the center, and (4) use more points in the scale.

The halo effect is the systematic bias that the rater introduces by carrying over a generalized impression of the subject from one rating to another. You tend to expect the employee who does well on the first question of a skills test to do well on the second. You conclude that a report is good because you like its format, or you believe someone is intelligent because you agree with them. Halo is a pervasive error. It is especially difficult to avoid when the property being studied is not clearly defined, not easily observed, not frequently discussed, involves reactions with others, or is a trait of high moral importance. One way to counteract the halo effect is to rate one trait at a time for all subjects or to have one trait per page.

Rating scales are widely used in business research and generally deserve their popularity. The results obtained with careful use compare favorably with other methods. They are interesting for respondents and have a wider range of application than most other methods.

In the reference text described elsewhere on this web site, we discuss measurement strategies at length providing many examples of different scales and how to analyze the data that comes from them.

A proposal is an individual's or company's offer to produce a product or render a service. Many beginning researchers and managers view proposals as unnecessary work. In actuality, the more inexperienced a researcher is, the more important it is to have a well planned and adequately documented proposal. The same is true for managers. The research proposal is essentially a road map, showing where a journey begins, the destination, and the method of getting there. Well prepared proposals include potential problems that may be encountered along the way and ways of avoiding or working around them, much like a road maps include alternative routes for a detour.

Also known as a work plan, prospectus, outline, statement of intent, or draft plan, a proposal answers a buyer or sponsor's questions of what, why, where, how, to whom, and with what benefit is the proposed research. A good proposal does three things well. (1) It presents the problem to be researched and its importance; (2) discusses the research efforts of others who have worked on similar problems; and, (3) sets forth the data necessary for solving the problem and how the data will be gathered, treated, and interpreted. The proposal of a successful contract researcher must present its plan, services, and credentials in the best possible way to encourage its selection over competitors. In contract research, the survival of companies depends on their ability to develop winning proposals.

What's the Added Value to the Research Process?

In a corporate setting, whether the research is being done in house by a research department or under contract to an external research firm, management sponsors the research. A proposal is the first step in the evaluation process. It allows the sponsor to assess the consultant's sincerity of purpose, clarity of design, extent of background material, and fitness for undertaking the project.

For consultants. Consultants, your proposals reflect your discipline, organization, and logic. A poorly planned, poorly written, or poorly organized proposal damages your reputation more than the decision not to submit one. The proposal provides a document that the sponsor can evaluate based upon current organizational, scientific, or scholastic needs. It allows the research sponsor to assess both the researcher and the proposed design, to compare them against competing proposals, and to make the best selection for the project.

For managers. The proposal also provides a basis for the sponsor to evaluate the results of a project. By comparing the final product with the stated objectives, it is easy to decide if the research goals have been achieved. Manager skill training is another benefit proposals bring to the sponsor. Many managers, requesting an in-house, departmental research project, do not adequately define the problem they are addressing. A proposal acts as a catalyst for discussion between the person conducting the research and the manager. The researcher's job is to translate the management question into something concrete and researchable while outlining the objectives of the study. Managers may discover their interpretation of the problem does not encompass all the symptoms. The proposal, then, serves as the basis for additional refinement of the problem until all aspects of the are clearly understood. The parts of the problem which are not researchable, or at least not subject to empirical methods can be treated with an alternate design, such as a qualitative or policy study. Upon completion of the discussions, a carefully worded research question emerges that will produce results for the firm and bring credit to the managing sponsor.

For an outside contract, the process is different. Proposals are usually submitted in response to a request for bid or request for proposal (RFP). However, by responding to an RFP, the consultant has a vehicle to convince the sponsor that the real research question may be different from the problem management has envisioned in the initial RFP. In this way, a contract researcher can show superior understanding of the problem over competing proposals.

In the textbook described elsewhere on this web site, we discuss the different types of proposals and their purposes. We also show various ways to structure a proposal and describe evaluation schemes that are easily implemented.