Your Stats Are Meaningless

Election season always bring various stats from polls, interviews and anecdotal evidence. However, you don’t need to look to politics to see meaningless stats.

Let me just give one example.

Xerox has a press release about receiving the “Best” in Annual Survey of Office Workers. The survey indicates that Xerox color printers and multifunction systems are the best. Period.

You can read the full release here if you are so inclined.

There is one key problem about the designation that is lost among all the rhetoric.

From the press release, you would think that this is the most amazing award ever bestowed on a manufacturer. With statements like:

"We've found that customers really don't hold back - they give honest, candid feedback in this survey," said Andy Slawetsky, president, Industry Analysts. "The fact that Xerox has come out on top in back-to-back years proves their customers consistently experience a high level of product quality and service."

Hmmmm… proves nothing.

Another quote:

"Our Xerox color printer has surpassed our expectations," said Kim Faber, event coordinator, The Catering Company. "We have seen an increase in the number of bids we are winning due to the colorful and professional appearance of our marketing collaterals. This translates directly into more revenue for the company. The solid ink color printer was the best investment we've made in office equipment."

Wow! This must be a miracle product!

In reality, anyone could have said that about any product because it was a quote about color and design, not Xerox.

Here’s the rub.

Industry Analysts conducted extensive telephone interviews and the results are based on nearly 100 verified responses from people who use and manage color document technology within their organizations.

Do you see it? Does this designation mean anything?

How can you have “extensive telephone interviews” but not even have 100 in your sample size? How many hundreds of thousands of printers and multifunction systems have been sold this year alone? How many hundreds of millions are in use today… and your sample size is not even 100?

On top of that, who are the “Industry Analysts?” Well, the only thing I know for sure is that they are based in the same town as Xerox.

Here’s the bottom line. Any stat can say what you want it to say. Any inference can be drawn… especially when you have a small or unknown sample.

What if the “extensive telephone interviews” were conducted only in Rochester, New York where there is a high likelihood that the respondents may have a device provided from a beta or delta program or that they may feel compelled to buy from a local business? How many of the respondents even have any experience with any other device… so they couldn’t compare appropriately.

The way you use your stats says something about you. If they are that good, give the raw numbers and let people draw their own conclusions. If they aren’t that good, don’t write a press release about them.

Remember, stats never lie but liars use stats.