For years the car magazines have conducted long-term road tests (links from Google). The manufacturer gives them a car to drive for 30,000 to 40,000 miles, and they write up both their impressions of what the car is like to drive and what broke. No problem with the first bit: you might learn more about a car when you drive it for a year or so. But many readers no doubt reach conclusions about a model’s reliability from these tests, and they shouldn’t.

Looking for an example to discuss, I first clicked on Car and Driver’s long-term test of the Subaru Legacy GT. I turned out to be a good one. Their Legacy had many problems during its stay, including:

• driver’s window fell off its track
• rough idle (no problem found)
• both rear wheel bearings wore out
• both headlights burned out
• misfires, required three new spark plugs and a fuel injector
• new clutch and flywheel (C&D accepts some of the blame)

From this Car and Driver concluded:

So there were a lot of headaches with our Legacy, and it suggests that if you’re looking for a sporty all-purpose sedan, this GT might wind up deep down on your wish list.

One person read this, and though now doubtful about the Legacy decided not to simply avoid the car as a result. Instead, he went to LegacyGT.com, a forum for the Subaru Legacy, and posted:  

I’m interested in purchasing a Legacy GT and saw the latest Car and Driver’s long-term test in their latest edition. The number of problems they had frightened me. Is C/D’s experience typical or are Legacys pretty reliable? Thanks.

Like the forum members who responded to this post, I’d suggest not paying much attention to Car and Driver’s conclusion. They only had experience with one car, and a sample size of one tells you virtually nothing. It’s possible to get a lemon with any car. One panel member had his 2007 Lexus ES 350 bought back after it required a new transmission, new strut, two new nav units, and an engine oil seal replacement in the first 7,000 miles. From this, should we conclude that Lexus are unreliable? Of course not–it’s just one car.

That said, responses to the forum post do suggest that the headlight bulb and wheel bearing failures are common among 2005 Legacy GTs. But this doesn’t change the fact that the long-term test isn’t a good indicator of reliability. Only with additional data sources is it possible to judge the frequency of the problems experienced with the test vehicle.

TrueDelta’s #1 challenge is obtaining sufficient sample sizes. I’m learning that the “repair trips per year” stat stabilizes once there are about 20 responses. And even 20 is far from ideal; in the future the minimum sample size will be higher.

For the 2005 Legacy and Outback, TrueDelta’s sample size for the year ending December 31, 2006, was 33 cars. The result of 0.3 repair trips per year is among the lowest. Some of the problems reported by Car and Driver don’t count as problems on TrueDelta’s survey: the spark plugs and the clutch. But still, their experience appears to have been exceptionally bad and not indicative of the typical Subaru Legacy. I would not hesistate to buy this car based on their long-term test.