The numbers behind Consumer Reports’ reliability ratings

In Consumer Reports, a 3 percent problem rate can count as “much worse than average” at the system level (electrical, cooling, etc.). That’s the lowest problem rate for which they’ll assign a solid black dot, if it also happens to be 45 percent worse than the average. In the case of a 3 percent black dot, the average problem rate for the system would have to be 2 percent or lower (145 percent of 2 percent being about 3 percent).

Guess what? For 2006 and 2007 models in the current CR results, the averages are ALL 2 percent or lower. So for these model years a problem rate of 3 percent or higher at the system level WILL earn the black dot.

For the entire car, the average problem rate for a 2007 is about 18 percent. (I say “about” because this was the average in previous years. I have not been able to find this figure for the current year.) So you can calculate (within a percent or two):

“much better than average” = problem rate under 10%
“better than average” = problem rate from 10% to  14.5%
“average” = problem rate between 14.5% and 21.5%
“worse than average” = problem rate between 21.5% and 26%
“much worse than average” = problem rate over 26%

There are a few ways you can look at this:

1. “Much better than average” cars have at most 40 percent of the problem rate of “much worse than average” cars. So in percentage terms, the spread is wide.

2. But what this translates to in absolute terms is the difference between one problem for every ten (or more) cars vs. one problem for every four (or fewer) cars.

3. In other words, nearly three in four owners can report no “serious” problems, and the model can still be “much worse than average.”

4. The difference between “average” and “much worse than average” is as small as 3.5 percent, about one problem for every 30 cars.

5. The ratings in CR would be more useful if they posted the above percentatges with their results. Even better, also provide the actual problem rate for each car–a number. But they do not, feeling that people are best served by giving them only red and black dots.

It’s like providing an idiot light instead of an actual instrument.

I started TrueDelta’s research because I wanted to see the actual numbers, not just the dots. And faster.