TrueDelta doesn’t gather its data from a random sample of vehicle owners. Worse, many people join TrueDelta after reading about it on a forum, and many people only visit automotive forums when they have a problem with their car. For these two reasons, I’m often asked “Don’t you think your research is going to be very biased?.

The short answer: no

Maybe you were hoping for a lengthier answer? Then you’ve come to the right place. 

First of all, TrueDelta does one thing differently from every other reliability study: with the exception of the first month, it only collects data on problems going forward. So, unlike with other vehicle reliability surveys, people do not know what they will be reporting at the time they decide to participate.

For the typical model in the panel, 70 to 80 percent of panel members do not report a single repair trip in their first six months of responses. For some models this percentage is over 90.

So TrueDelta doesn’t appear to mostly be attracting people with problems to report.

Second, panel members don’t just respond once. Instead, they respond continuously, checking in at the end of each quarter. Over time brand new members will compose a smaller and smaller percentage of the total panel. Even if there is some bias in the very first month of participation, and it’s not a large amount regardless, then this bias will have an increasingly miniscule impact.

Finally, TrueDelta’s surveys clearly define which repair trips should be reported, which is essentially all of them. The survey has been designed to avoid the need for subjective evaluations. Random samples are most necessary when surveying opinions. Both Consumer Reports and J.D. Power use surveys with a much larger subjective component.

J.D. Power does use random sampling. But their response rate is typically between 20 and 25 percent. When over three-quarters of a random sample doesn’t respond, and they’re responding about events that have already happened, you must be very worried about non-response bias. Were non-responders, with no axe to grind, less likely to have problems?

TrueDelta’s survey and the process by which it is administered were not just thrown together. I put a great deal of thought into how to best overcome potential problems like bias. At the same time, I have no illusion that I already know everything there is to know, or that these methods are perfect. Instead, TrueDelta is still very young, and expects to keep learning, and keep improving, as the quarters roll by.