-- and how TrueDelta will avoid them
For decades, Consumer Reports has been the best source of vehicle reliability information. But even the best is
not good enough. In at least seven ways, Consumer Reports'
data collection methods or modes of presentation mislead or underinform consumers.
In each case, TrueDelta is doing things much differently.
1. "Serious problems"
Consumer Reports' ratings are based on the number of "serious problems" reported by its members. They never precisely define the term. Instead,
Consumer Report's survey form
leaves it up to each respondent to determine which problems are serious enough to report, an invitation to bias.
In contrast, TrueDelta will report measures like "times in the shop" and "days in the shop." These mean what they seem
to mean. If a vehicle is in the shop for something other than routine maintenance or an excluded wear item (listed on
TrueDelta's survey form), that's serious enough.
2. Relative ratings
Consumer Reports rates each model relative to the average vehicle. As a result,
the absolute number of problems a vehicle will experience remains unclear. Does an "above average" vehicle "never
break?" Is a "below average" vehicle "always in the shop?"
In the absence of hard numbers, people tend to assume that the best vehicles are better than they are and that the worst
vehicles are worse than they are. I once had a vigorous discussion with the owner of a Japanese SUV. As proof of his
vehicle's superior reliability, he noted that its brand had been the highest-rated in Consumer Reports' 2005 auto
issue. His brand's nearly new cars had had eight "serious problems" per hundred
vehicles. While this was less than half the eighteen problems per hundred nearly new domestic brand vehicles, the absolute
difference was just one-tenth of a serious problem per car.
This did not--and does not--strike me as anything to get wound up over. The real problem: few
people when glancing through the magazine think about the absolute numbers behind the relative ratings.
In contrast, TrueDelta will clearly report absolute ratings.
3. Ranges
Consumer Reports' rates models on a five-point scale from "much worse than average" to "much better than
average" using their well-known red and black "blobs." In 2005 (when I first wrote this), more than half of domestic models
earned an "average" rating, while many Hondas and Toyotas earned an
"above average" rating. (With the average increasing, "much better than average" ratings have become rarer.)
"Average" means within twenty percent of the average, so 80 to 120 on an index with 100 being average. "Better than
average" ranges from 121 to 145. So if one vehicle is "average" and another is "better than average," then the
difference between them can range anywhere from a single point--totally insignificant--to 65 points--very significant.
The red and black dots appear simple to understand, but they conceal far more than the convey. As a result,
many readers of the magazine understand far less than they think they do.
In contrast, TrueDelta will clearly report the absolute differences between vehicles. For example, analysis of the data
might find that one vehicle over the first five years of ownership will take 2.3 extra trips to the shop, for a total of
3.6 extra days.
4. Only averages
Vehicle reliability has been steadily improving. Even the average eight-year-old domestic brand model was reported (on page 17 of the 2005 auto issue)
to have fewer than one-and-a-half
"serious problems" per year. Yet many people would avoid such a car because they fear it
will have "lots of problems."
While perceptions are distorted by Consumer Reports' emphasis on relative ratings, another factor is
involved: people are afraid of getting an unusually troublesome vehicle. Even if the average is the
same for two models, the chances of getting a lemon could be far higher for one than the other. People might fear
that even as the average rate of problems for domestic vehicles comes down the odds of getting a lemon remain
uncomfortably high.
Based on Consumer Reports' reported results there's no way to know one way or the other, as they only report
averages. To my knowledge, they have never discussed the odds of getting an unusually good or bad example of a
particular model.
In contrast, TrueDelta will report the odds of getting a lemon and the odds of getting a perfect car (in addition to
reporting the average number of trips to the shop and days in the shop).
5. Survey (in)frequency
Consumer Reports sends out an annual survey asking people to report problems that occurred during the entire
previous year. This is too long a period to expect people to accurately remember what happened.
In contrast, TrueDelta sends a monthly email asking people to report trips to the shop that occured the previous month.
In most cases participants will still only have to fill out one or two brief repair reports a year. So the effort will be
the same or less. But respondents' recall will be much more accurate.
6. Stale information
Consumer Reports mails out surveys each April, then first reports the results the following October.
Consequently, when a new vehicle is introduced in the fall its reliability isn't reported until a
year later. This is a long time to wait for someone interested in a hot new design; by the time its reliability
is known it will no longer be hot.
In a related issue, the vehicles reported on aren't as old as Consumer Reports suggests. For example, while "three-year-old
vehicles" are, on average, three years old at the time the auto issue appears, they were only about two years old when
the problems were reported, and only about one year old at the beginning of the period being reported upon.
In contrast, TrueDelta updates its stats quarterly, and will first report
reliability as soon as four months after a new vehicle reaches dealers. On average over the course of a year, TrueDelta's results are over eight months "fresher" than
those of Consumer Reports.
7. Fossilization
The last serious problem at least partially explains
the others: Consumer Reports, once an innovator, has ceased to innovate. They have
been asking questions and reporting results much the same way for decades. No surprise, really, as they've had no serious competition
and have been subjected to very little outside evaluation.
Want better vehicle reliability information? Participate in TrueDelta's research and
help make it happen.
Thanks for reading.
Michael Karesh, TrueDelta
First posted: September 5, 2005
Last updated: February 24, 2007
