Consumer Reports’ recent release of its “Annual Auto Issue” has people talking about its “best picks” once again. For the second year in a row they’re all Japanese. I don’t personally have a problem with that. As long as they follow a clear procedure, I’m fine with the results.

The problem is that while they probably follow a clear procedure, to those of us on the outside this procedure isn’t clear.

Consumer Reports’ road test engineers subject every car to a thorough set of tests, and the overall scores are calculated based on the scores in these individual tests. But, unlike many car magazines that conduct comparison tests, they do not publish the formula used to sum up the test scores to arrive at an overall score.

I’ve covered this before in a piece on comparison tests. But it’s worth reiterating for a few reasons. First, we have a new “Annual Auto Issue” on newstands right now.

Second, I met some of Consumer Reports’ road test engineers at the Detroit show. Great guys, I enjoyed meeting them. I asked one of them why the formulas are never published. His response: “it’s policy.” The carved-in-stone kind. He didn’t make it, he didn’t necessarily support it personally, but as a Consumer Reports employee it was his job to follow it. Welcome to the bureaucracy.

He did say the formula differs by type of car. Which makes sense, as the same criteria and weights would not make sense for both a sports car and a minivan.

Third, someone has just asked the same question on the Consumer Reports auto forum. The moderator’s response:

Thank you for your comments. These forums are designed to help subscribers in selecting and buying a car. They are intended to be primarily peer to peer, with our Auto test experts helping out when available.

If you find errors we will be glad to look into them and make corrections, but we just don’t have the time or resources to engage in lengthy debates here. You can channel your inquiries through Customer Service. There is a link on the bottom of every forum page, and at the top of every CR on-line page. You are also welcome to visit our facilities when we hold an open house and speak directly with our test staff at that time.

This is a total cop-out. Consumer Reports employs dozens of people. TrueDelta is, aside from a bit of work I contract out, currently just me. But as members of various forums can attest, I always find the time to discuss the way TrueDelta does things. I find the time because I think it’s important, for two reasons. First, without these discussions it might not be apparent how TrueDelta produces its results. Second, I don’t know everything, and I haven’t thought of everything. Engaging in debate can teach me better ways to do things. 

Getting back to the moderator’s response, the secret nature of the formulas is not seen as an issue affecting subscribers’ ability to select and buy a car. Sorry, but it just isn’t so. If you do not know the criteria and how they were weighted, then you do not know how closely they match your own, and the results are not very useful.

To not publish the formula implies that Consumer Reports believes that there is one best set of criteria and weights, and that any reasonable consumer would share the ones Consumer Reports uses.

This is just plain wrong. There is no rational way to determine criteria and weights. These are inherently the results of personal decisions. By not publishing its formulas, Consumer Reports takes the stand that readers should let Consumer Reports decide for them what matters when choosing a car.

For the record, TrueDelta takes the opposite stance. “Best pick” lists are great for publicity. And lazy minds do want to be told what they should buy. But I just don’t believe in them. If I ever published such a list, it would be explicitly labeled “Michael Karesh’s quite personal list of the best cars.” And my reasoning would be clear.

My focus is elsewhere. I see TrueDelta as a data provider. It’s up to the buyer to decide how various bits of data and personal evaluations should be combined to arrive at an overall selection. This is more work, but, as I said in the piece referred to above, if you want the best car for you then there is no alternative.