TrueDelta

Consumer Reports announced its Annual Auto Issue today. As in past years, the press treats the reliability ratings in it as news, even though they’re the same ratings that have been around since the New Car Preview reached newsstands last October. And are based on a survey that was sent out nearly a year ago, in April 2009.

For some reason, no one ever mentions this. Certainly not CR. But not the media, either.

So we get forum posts like this one. It’s easy to understand why people, hearing there’s a new Auto Issue, think that the results in it are new. It’s just not remotely accurate.

TrueDelta recently updated its reliability stats to include owner experiences through the end of 2009. So they’re based on data that is over eight months ahead of CR’s.

When shopping for a car, do you want to know how it was faring a year ago, when it was a year younger, or how it’s been doing lately?

Car Reliability Survey results

Consumer Reports released their latest results today, based on a survey that went out last April. TrueDelta’s results are currently two months ahead of CR’s latest, and will soon be five months ahead.

Looking through the latest results, I see largely confirmation of some results we first released over a year ago, such as that for the Jaguar XF.

But I also see a couple of puzzling results. I thought CR had stopped reporting on models that were introduced around the time the survey went out, because owners haven’t had the cars long enough to provide reliable data. But they have a result for the 2010 Honda Insight. It’s much better than average–as expected for cars that were at best a few weeks old. This result appears to be correct based on our data, but this is largely dumb luck on their part.

CR also announced that the all-wheel-drive Chevrolet Traverse, GMC Acadia, and Buck Enclave “Lambda” crossovers have improved to “average,” and so can now be recommended. But not the front-wheel-drive versions–they remain “worse than average” so they are not recommended.

Now, it doesn’t make sense for one car with all the parts of another, but a few more, to be more reliable. My first thought: just another sign that their data are messy.

Then I dug deep into their results, and found a nice addition: they now state the percent difference from average. Still not an absolute repair frequency, but a step in the right direction.

FWD Lambdas: 21% worse than average

AWD Lambdas: 19% worse than average

“Average” runs from 19% below the average to 19% above the average, so the AWD crossovers just barely make the cut, while the FWDs just barely miss it.

In reality, there’s no meaningful difference between the two. But the way CR hides the facts behind its dots and recommendations will lead some people to spend the extra money for all-wheel-drive.

Operating TrueDelta, it would be in my interest to assert that Consumer Reports’ results are incorrect. But I’ve never made this argument, nor do I believe that their results are generally incorrect. Instead, I’ve repeatedly found that people’s interpretations of Consumer Reports’ results are often incorrect. A current example could have serious consquences…

In a recent Detroit Free Press article, a leading automotive journalist writes that Chrysler’s “dismal showing” in Consumer Reports’ latest reliability ratings “raises serious questions about Cerberus’ management of the automaker it acquired in 2007.” An industry consultant concurs, wondering what Chrysler gained from all of the changes it claims to have made last year to improve quality.

Both have misread Consumer Reports’ results.

It is true that Chrysler does poorly in Consumer Reports’ latest results, and worse than it did in the previous set of results. But what both the journalist and the consultant don’t realize is that the results are based on a survey conducted nearly a year ago, and are no different than the reliability results Consumer Reports first released last Fall.

Consequently, any improvements due to Cerberus’ management would not show up in these results, but would instead show up in the next set of results, in October. Note that there isn’t a single 2009 Chrysler product in the current results, and the 2008s were introduced before Cerberus could have had any impact.

The likely reason for this misperception: Consumer Reports publicizes these results as “new,” even though the reliability data on which they are based were mostly collected in April 2008.

Normally this long lag between the time Consumer Reports collects its data and the release of the Annual Auto Issue doesn’t have huge consequences. It simply results in people buying cars without knowing how these cars have been performing for the last year or so. But in Chrysler’s case, this misperception could provide fatal. These people won’t be the only ones to reach these conclusions after reading Consumer Reports’ latest press release, not by a long shot. Washington might do the same.

Unjustly so? This would depend on whether or not Cerberus has improved Chrysler’s product reliability. With prompt updates four times a year, TrueDelta’s research process could answer this question–except that few members have been buying 2009 Chryslers. Many dealers probably still have 2008s on their lots.

Even so, we do have results for the 2009 Dodge Journey, which happens to be the only new product introduced since Cerberus started making changes.

So, how is the Dodge Journey faring? Not well, with a reported repair rate of 136 repair trips per 100 cars per year, about 2.5 times the average.

This provides some evidence that Cerberus’ changes aren’t working. But this evidence is far from conclusive. The Journey was introduced just a few months after Cerberus took over. And it’s just one model. In other words, even with the most up-to-date car reliability data it’s probably too soon to render a verdict on Cerberus’ attempts to improve Chrysler’s product reliability.

We’ve been asked why it matters that TrueDelta’s car reliability information is, on average, over nine months ahead of the information provided by other sources. The plight of the domestic auto industry provides the most significant answer yet. These companies are fighting for their lives. They continue to have trouble selling cars because of lingering perceptions, sometimes well-founded, more often not, that domestic cars are unreliable. If these companies are making improvements in the reliability of their products, getting awareness of this improvement to the public up to fourteen months months earlier could make the difference between surviving, and not surviving.

While these companies don’t deserve to survive indefinitely simply because it would be costly in the short run to have them fail, they do deserve to have their products evaluated based on the most current information possible. TrueDelta’s Car Reliability Survey is uniquely designed to provide this information.

A few weeks ago it was rumored that Consumer’s Union, publisher of Consumer Reports, would purchase The Consumerist, a blog covering consumer issues and giving voice to consumer complaints. Now these rumors have proven accurate, and this purchase has been completed. 

Operating TrueDelta.com, I’ve learned quite a bit about the media. One of the most difficult lessons: journalists often won’t write about what you’re doing if it even partially competes with the activites of their employer. At least a few of them have told me they couldn’t write about TrueDelta’s unique vehicle information for this reason.

Consumer Reports is just one example. Though ostensibly putting the interests of consumers first, will they cite TrueDelta as a source of information they personally cannot yet provide? Of course not. I’ve personally been told I cannot mention the site when posting to their forums. Consumer Reports’ competitive interest comes before the interest of consumers. 

Now that they’ve bought The Consumerist, the same will likely apply to that site.

So, from where I sit media consolidation isn’t a good thing. Each media outlet has its own interests, and these come before the interests of readers. The larger these outlets get, the broader their interests get, and the more delimited their reporting gets.

Luckily, word-of-mouth remains beyond media control, and more and more people have been learning of TrueDelta from their family and friends.  

We’ve got the system in place to provide the most up-to-date, most ”real world” vehicle reliability information. Now we just need more participants. The 35,700 vehicles signed up so far are a great start. But 100,000 would be better. How to get there? Well, media coverage would help. But it has proved surprisingly difficult to obtain this coverage.

A major source of this difficulty: much of the media believes that the best possible mousetrap has already been invented as far as vehicle reliability research is concerned. So there’s no point–and no story–in what TrueDelta has achieved. As much as I’d like to “run my own race,” this belief must be directly challenged if we’re to grow like we need to grow.

Perhaps some in the media are aware that Consumer Reports could do a better job. But this has been overridden by the integrity of that organization. That Consumer Reports doesn’t accept advertising apparently places them above critique. I also hear that Consumer Reports simply has so many survey responses–they emphasize this point themselves–and they’ve been around for so long. But being big and old hasn’t placed General Motors and Ford above critique. So why should these be the end all with regard to Consumer Reports?

Consumer Reports’ refusal of advertising and other such entanglements is admirable. As is their independence and integrity. For similar reasons, TrueDelta solicits no ads from anyone with a stake in our results. (We have no control over the ads supplied by Google, and no direct contact with those advertisers.)

But should this refusal to accept advertising be everything? Shouldn’t the timliness and transparency of the information also matter? I’ve never called Consumer Reports’ integrity into question. Instead, I’ve focused on two major shortcomings that led me to develop a better research process, because I personally wanted better information.

The result: TrueDelta posts actual repair rates, not just dots, to make the size of the differences between clear. And it posts this information as much as 14 months sooner than Consumer Reports.

For example, we had initial reliability information on the Jaguar XF in August, and will have a full result in November (with a preview already available for members). Consumer Reports won’t have a result for the car until October 2009.

Consumer Reports recent release of its latest results was covered by hundreds of media outlets. None of the stories I’ve seen have stepped outside the bounds set by Consumer Reports’ press release. None have noted the age of the data on which the results are based or that only relative ratings are provided.

Many journalists enjoy nothing more than having a story first, to have a “scoop.” For some, if they’re a day late, they might as well not publish the story at all.

So I’ve often wondered why TrueDelta doesn’t receive recognition for being not just a day or two ahead of any other source of vehicle reliability information, but months ahead, even a year ahead. The message: if Consumer Reports doesn’t have a result yet, then car buyers should simply wait until they do, or simply do without reliability information when buying a car.

Does the media even help Consumer Reports in the long run by not taking a close look at its research and reporting process, and by ignoring alternatives? History makes it quite clear what happens to an organization when it is left uncritiqued and unchallenged. The outcome is rarely good for organization, much less those it serves.

At some point, we’ll get large enough that at least some media coverage will follow. Until then, TrueDelta’s growth depends on word-of-mouth. Anyone who wants the better, quicker vehicle reliability information TrueDelta’s process provides: we’re counting on you to help make it happen.

Spring is almost here, so Consumer Reports’ annual auto issue is out. A few thoughts now that I’ve had a chance to glance through the 2008 version.

First, some old news. Many people assume that because the issue is new and Consumer Reports makes such a big deal out of it that the reliability information in the issue is based on recent data and is also new. I’ve read many forum posts where the author said, “Well, the new auto issue is out, so now I can buy a car.”

I find this mildly amusing, since the reliability information in this issue is always nearly a year old, based on a survey conducted the previous spring. The results have even been available since last fall.

So why the big deal with an issue in the spring? Well, as one of my grad school advisers used to advise, “Why publish once what you can publish twice?” Or more. They initially publish the results in the fall, but only in an issue available on newsstands. This issue is then repackaged a few times. Finally, in the spring, print subscribers are finally provided with this now old information.

In general TrueDelta’s results track with those of Consumer Reports. But in some cases they differ. Sometimes this might be because we ask different questions. But another reason is that TrueDelta’s results are currently based on the year ending December 31, 2007, while theirs are based on the year ending last April or so. A lot can change with a car in eight months. With older cars, at some point they hit a point where repairs start piling up. With TrueDelta’s method, this information will be available much sooner.

One thing I’ve never accused Consumer Reports of is a bias against domestic cars. But this is the charge they hear most often. As I’ve been learning myself, people who don’t like a result love to claim its the product of bias.

Lately, Consumer Reports has been going out of its way to try to change this perception. They recently made a big deal about the high road test scores earned by the new Chevrolet Malibu and Cadillac CTS. On the cover of the issue they place a photo of a Chevrolet Silverado next to the “Top 10 Picks of the Year” blurb. (It’s the only non-Asian product among the picks.) When discussing “reliability trends,” they highlight the Ford Fusion at the top of the page, and have a section on Ford’s recent improvement. In the “vehicle ratings” section, where they provide recommendations, they have photos of six vehicles–and three of them are domestic models.

What they aren’t doing: countering any of the critiques I have made. It’s easier to counter the false accusations, after all.

One of those recommendations is iffy. Last fall they stopped recommending Toyotas before they have reliability data on the model. And yet in the new issue the 2008 Scion xB is recommended, with a predicted reliability of “above average.” That prediction is based on the totally different first-generation car. What’s the deal? Well, it seems the new xB was reviewed before the new policy went into effect, and they didn’t reverse earlier recommendations when the policy changed. The new xB got grandfathered.

This isn’t the way I would have done it, as it risks recommending a car that shouldn’t be recommended. And TrueDelta’s limited data on the new xB suggest that its repair rate is worse than average owing to a number of minor exterior trim issues.

They do have an interesting section on extended warranties that demonstrates that these warranties are rarely worth it. TrueDelta has long advised that most people will not recoup their investment in such a warranty. Consumer Reports does better, providing some actual numbers. Overall, 22 percent of those surveyed came out ahead; the other 78 percent did not. On average, the average person lost $300 on the deal.

One especially interesting number they divulged: among owners of cars that were four to six years old, only four percent had repairs totaling over $1,700 during the previous year. (They don’t explain why this number was chosen.) In other words, expensive repairs are rare.

Consumer Reports’ members are asked to only report serious problems on the survey, but since the percentage reporting such problems is way over four for cars over four years old, clearly some number well below $1,700 tends to qualify as “serious.”

Finally, there’s a blurb about reporting specific common problems in the future. They give the example of the 2004 VW Touareg, where the mosts common problem–with the tire pressure monitoring system–was reported by 19 percent of owners. My take in this particular instance: does a TPMS issue truly qualify as a “serious problem?” Generally it’s just a nuisance–the light stays on. This is the sort of problem that some owners will report but others will not given how Consumer Reports asks the question–”report only problems you considered serious.”

I must also wonder whether their data are clean enough to report problem rates for specific issues. The number of anomalies in their overall results makes this a bit dubious.

Still, it’s good to see them finally doing something new and providing additional information. We’ll have to do that much better here at TrueDelta.

In Consumer Reports, a 3 percent problem rate can count as “much worse than average” at the system level (electrical, cooling, etc.). That’s the lowest problem rate for which they’ll assign a solid black dot, if it also happens to be 45 percent worse than the average. In the case of a 3 percent black dot, the average problem rate for the system would have to be 2 percent or lower (145 percent of 2 percent being about 3 percent).

Guess what? For 2006 and 2007 models in the current CR results, the averages are ALL 2 percent or lower. So for these model years a problem rate of 3 percent or higher at the system level WILL earn the black dot.

For the entire car, the average problem rate for a 2007 is about 18 percent. (I say “about” because this was the average in previous years. I have not been able to find this figure for the current year.) So you can calculate (within a percent or two):

“much better than average” = problem rate under 10%
“better than average” = problem rate from 10% to  14.5%
“average” = problem rate between 14.5% and 21.5%
“worse than average” = problem rate between 21.5% and 26%
“much worse than average” = problem rate over 26%

There are a few ways you can look at this:

1. “Much better than average” cars have at most 40 percent of the problem rate of “much worse than average” cars. So in percentage terms, the spread is wide.

2. But what this translates to in absolute terms is the difference between one problem for every ten (or more) cars vs. one problem for every four (or fewer) cars.

3. In other words, nearly three in four owners can report no “serious” problems, and the model can still be “much worse than average.”

4. The difference between “average” and “much worse than average” is as small as 3.5 percent, about one problem for every 30 cars.

5. The ratings in CR would be more useful if they posted the above percentatges with their results. Even better, also provide the actual problem rate for each car–a number. But they do not, feeling that people are best served by giving them only red and black dots.

It’s like providing an idiot light instead of an actual instrument.

I started TrueDelta’s research because I wanted to see the actual numbers, not just the dots. And faster.

Got my latest issue of Consumer Reports today, and they tested the Toyota Highlander, Buick Enclave, Ford Taurus X, and Subaru Tribeca.

I’ve discussed before how Consumer Reports refuses to divulge how scores are calculated as a matter of “policy.” Well, while their detailed impressions of these models generally match my own, their secret scoring system yields a top place finish for the Toyota, a sizable six points ahead of both the Buick and the Ford.

I do wish they’d say how scores are calculated, because the narrative and ratings leave it unclear how the Toyota finished ahead of the others.

Noted advantages of the Highlander: acceleration, transmission behavior, fuel economy, second row comfort, access (unclear how the Highlander is better here), rearward visibility

Noted advantages of the Enclave: interior fit and finish, steering feel, third row room and comfort (big difference noted), split third row, and cargo room

The secret scoring system must weight the Highlander’s advantages more heaviy than the Enclave’s. But I suspect that many buyers have different priorities, such as a useful amount of rear seat space and cargo room, that give the Enclave the edge. The Ford also has more room for people and cargo room than the Highlander, and so could also come out ahead of the Toyota for many buyers.

Consumer Reports needs to make its scoring system transparent, so car buyers can determine whether or not they have similar priorities and, consequently, the same top pick.

TrueDelta’s results currently include all shop trips that included a successful repair (unless this repair was a simple reflash). We don’t yet distinguish between “serious” repairs and minor ones. Often this is suggested. But doing this can be very tricky.

As I first reported back in June of 2006, Consumer Reports’ method contains a serious flaw: they ask that only serious issues be reported, and then have each individual respondent decide whether or not a repair was serious enough to report. The serious issue with this: different people are going to differently evaluate the same problem.

The problem with this approach is evident just by looking at the revised list of problem categories: they now include “squeaks and rattles.” What would make a rattle a “serious issue”? Whatever the criteria, if there even are criteria by which a rattle might be deemed “serious,” such an evaluation is going to vary greatly from person to person.

Oddly, such an evaluation appears to vary depending on the powertrain a car has. If there’s one car Consumer Reports has a beyond sufficient sample size for, it’s the Toyota Camry. Yet with the 2007 Camry, “squeaks and rattles” are “much worse than average” with the Hybrid and V6, but “average” with the four-cylinder.

How might this be explained? I can only venture a guess that the highly subjective wording of their questionnaire injects so much noise into responses that even their huge sample size for the Camry cannot compensate.

In the future, TrueDelta will likely post a separate set of results that include only major repairs. But we’ll be using much more objective criteria to determine which repairs are in fact major. And I doubt any rattles will qualify.

Each year Consumer Reports posts a list of models whose reliability has declined. But in some cases this “declining reliability” is a sign that they really didn’t have enough data a year ago. Many times with models that were introduced in the winter or spring they report a “better than average” result, only to report a year later that reliability has “declined.”

This affected the GMT900s this year. It affected the Honda Ridgeline last year. And, based on TrueDelta’s data, it’ll affect the 2007 Acadia and Outlook next year.

The source of the problem is that most of their surveys are returned in April, and at that time many of these late intro models have only been owned for a few weeks–or less. People haven’t had enough time to experience problems.

TrueDelta sees a similar effect in its own results when a model is very new. When the reported trips per year is under 20 per 100 vehicles, and the average odometer reading is under 3,000, there’s a very good chance that the reported repair rate is artificially low.

This is why in the notes to the August results you’ll find:

Any result under 20 repair trips per 100 vehicles is likely a result of the small sample size or cars too new to have experienced many problems, and should be expected to increase to at least 20 in future updates.

So, when looking at TrueDelta’s results, pay attention to the average odometer reading and the average number of months per vehicle. When these are very low, there’s a fair chance that future reliability will “decline.” Not because the cars have actually gotten worse, but because earlier owners had not yet had their cars long enough to experience problems.