TrueDelta

Three months ago I changed the survey system to greatly reduce the number of emails to Car Reliability Survey participants. And I’ve been very worried about this change ever since. With fewer emails might we end up with fewer responses? Or fewer repairs being reported, and so lower quality data? These are the questions that keep me awake at night.

Well, the quarter is over, so we have our answer. The percentage of people reporting repair trips in a given month has remained around 10 percent, so the quality of the data has not been affected. The quantity, on the other hand…

For over a year the response rate has been about 36 percent, so this has been our target. Last July the response rate was 36.7 percent, and in October it was 35.7 percent, a bit short.

Last month it was 38.0 percent, a level we haven’t seen since April 2008. Among new members, the response rate was a huge 56.0 percent. Even among members who joined over three years ago–many of these email addresses are probably no longer being checked regularly–the response rate was 28.3 percent. In absolute terms, our goal last month was 14,650 responses. We ended up with 15,424.

The current results include 295 models, plus partial results for another 290. It looks like the February update will include 356 models, plus partial results for another 287. Thanks to the higher response rate, well over half of our stats will now be full results.

So now I can worry about other things…

In designing the Car Reliability Survey and the emails that support it, I’ve had to balance many factors, among them the human memory (or lack thereof), clarity, and the number of emails.

The survey has been on a rigid cadence, always covering the previous month. Emails for it went out every month, so anyone with a repair to report could report it while their memory was still fresh. But to keep the load down, if there was no repair to report a response was only needed following the end of each quarter. To make sure any repairs for the month were reported, reminders were sent to those who hadn’t recently responded.

This system worked well in many ways. The response rate has been far higher than for other surveys, and the data collected have been of very high quality.

Still, there were problems. The number of emails both made it unclear when a response was needed and clogged many inboxes. Also, some people did not want to wait as long as a month to report a repair. They wanted to report in real time, as many times a month as this required.

Starting last month, we eliminated reminder emails for anyone who responded in October. This did reduce the number of reported repairs by about 20 percent–many people need reminders.

To counteract this, the survey has been upgraded. Starting this month, it is possible to report any repair that occurred during the quarter, not just those that occurred the previous month. As a result, this month’s email will serve as a reminder email for October repairs as well as the first email for November repairs. This will probably make up for much of last month’s shortfall.

In addition, it is now also possible to report repairs that occurred during December, this month, and to respond multiple times per month. So people who want to report each repair immediately can now do so.

Much of last month’s shortfall probably involved repairs that occurred in November. With the old reminder emails, sent in the middle and at the end of the month, some people reported repairs that occurred during the current month even though the survey asked only about the previous month. The new system will eliminate this minor distortion.

Going forward, multiple reminders will only be sent to participants following the end of each quarter, the next time in January. So the third month in each quarterly cycle will serve as a sweeper to gather any data that was missed during the previous two months. By concentrating reminders in the third month, it will be clearer when a response is needed even if there have been no repairs.

A further by-product of the multi-month survey: it is now possible to close out sold vehicles entirely on the site itself, as long as there has been a recent response.

Taking all of the changes together, I expect to gather data of at least as high a quality and quantity, with fewer emails and less effort all around.

A new blog for the Land Rover Freelander charges that it’s not fair to rate the Freelander’s reliability using the same scale for all cars. The two reasons given: luxury car buyers are pickier, and luxury cars have more features.

It might or might not be true that luxury car buyers are pickier. Many people have suggested this, but I’ve never seen any data on it. One thing TrueDelta does to reduce the impact of the pickiness factor: we measure the number of successful repair trips, not the number of perceived problems.

It is certainly true that luxury cars include more parts that can break. But also note that here we report the actual repair frequencies, and not just “better than average” or “worse than average.” If a luxury model requires more repairs because it has more things that can break, then that’s just the facts. People want to know how often a model is likely to require repairs, not some stat adjusted for the number of things that can break.

The larger problem with the post: people aren’t usually going to compare the scores for a Land Rover to those of a Honda, then opt for the Honda. Instead, people considering a Land Rover are likely to compare its scores to those for competing luxury models.

Which brings us to the outright error in the post. It asserts that “other luxury brands like Mercedes-Benz, Rolls-Royce, and even Lexus also get a bad rap.” Sorry, but this just isn’t true. I’ve never seen a reliability stat for Rolls-Royce, so it seems that the author wasn’t looking at actual results when writing the post. Yet more evidence of this: unlike the larger LR3, the LR2 (Freelander2 in the UK) actually has a fairly low reported repair frequency in TrueDelta’s results. Then there’s Lexus. The great majority of Lexus models have received above average reliability scores, even when measured on the same scale as Civics and Corollas. It is more difficult to achieve top scores with a luxury model, but it is nevertheless possible.

The survey covering June ended today. July was not an easy month. The response rate was running two to three percent below the norm for much of the month. Not good, as this would have nearly cancelled out all of the people who joined in the last three months. And I’m not much for running unless it gets me somewhere.

Then, toward the end of the month, we ran a special email asking members why they hadn’t responded.

The #1 reason: the car had been sold. And so we learned that we need a more obvious way for people to report when they buy and sell cars.

Thanks largely to that email, we ended the month with over 12,000 responses, and a response rate a few tenths above the norm. Quite a comeback.

The payoff: 257 full results, and another 249 partial results, in mid-August. Including the first for a 2010 model.

Ten other models fell a single response short of the minimum. A late response might still tip them in.

Results won’t be out before mid-August because I’ve still got a lot of data to clean. When that’s done, July will finally be over. And we can then start looking forward to October…

The emails we receive suggest that many people misunderstand TrueDelta and its research process. Here I’ll attempt to address the most common misunderstandings.

Misunderstanding #1: TrueDelta wants survey responses many times a month.

Reality: We usually only need a survey response the first month you’re eligible to participate, then at the end of every quarter after that (four times a year). We send at least one email per month, though, so people with repairs to report can report them while their memories are still fresh.

If someone hasn’t responded to the survey, they are sent reminder emails. This can lead to the above misunderstanding–some people get the idea that three emails means we wanted three responses per month, when we were actually looking for–but haven’t received–about one response every three months.

Misunderstanding #2: It’s not possible to avoid the reminder emails.

Reality: It’s easy to avoid the reminder emails. First off, respond to the survey when this is needed. In the months when no response is needed, simply click on the “check in” link in the email–this takes just seconds. If you respond promptly when a response is needed, the system will “learn” that you don’t need reminding. Once this occurs–usually after two responses (so two to four months)–it’s not even necessary to click on the “check in” link.

Misunderstanding #3: Participating takes too much time.

Reality: When there’s no repair to report, responding takes the average participant about 30 seconds, and some as few as ten seconds. Even if there is a repair to report, responding takes just a couple of minutes. Other car reliability surveys are much longer.

Misunderstanding #4: The emails aren’t necessary–why not just let people report repairs as they occur?

Reality: We wish the emails were not necessary, but the response rate would be VERY low without them. Nearly all responses occur within 48 hours of an email. Once 48 hours have past, most people forget to respond, even though many intended to. So it’s also necessary to send reminders.

Misunderstanding #5: There’s no point in participating once you’ve bought a car.

Reality: The only way we can provide information when people need it is if they participate in between purchases. This is a team effort. Provide information when other members need it, and they’ll provide information when you need it. You might not need reliability information on the car you currently own, but you might later need reliability information on cars other members own.

Misunderstanding #6: It’s necessary to check your car to get an exact odometer reading.

Reality: We collect odometer readings to verify that you’re still active, and to get an idea of how many miles (or kilometers) are on the average car. It’s not important to know if the average is 11,000 or 12,000. So an approximate reading is good enough. If you’re guess is way off, we can easily correct it.

Misunderstanding #7: It’s necessary to have a repair receipt to respond to the repair survey.

Reality: The repair survey is designed to be answerable based on what you can remember. The most critical information–whether or not the car was repaired–tends to be easy to remember, especially since you’re only being asked to remember the last month or two.

Misunderstanding #8: We’re only interested in data on fairly new cars.

Reality: The Car Reliability Survey covers any car from 1995 on. The other surveys currently only cover from 2002 on, but we’ll be extending them to cover a few earlier model years.

Misunderstanding #9: We’re only interested in cars owned in the U.S.

Reality: While most members are Americans, we have thousands of Canadians and hundreds from outside North America. The Car Reliability Survey can include any car, anywhere–as long as at least 25 are signed up to participate. We’ve been adding non-US powertrains, for people in the UK and elsewhere who want to participate in the Repair History and Gas Mileage Surveys. Want more participants from where you live? Then please tell friends and family about our research.

Misunderstanding #10: We’re a big corporation collecting data from car owners so we can sell it to manufacturers.

Reality: While a few people are employed part-time on a contract basis, I’m TrueDelta’s only full-time employee. We do not sell data to the manufacturers. Instead, our focus is on providing information to members. When you respond to a survey, you’re helping other participants.

By keeping our expenses low, we’re able to do everything necessary to collect and analyze the data without charging participants any membership fee. We’re able to cover our costs through ads and affiliates–without accepting any ads directly from the car manufacturers.

The main limitation: we can’t afford to spend money on advertising and marketing. So it’s very important that members help the site grow by telling others about what we’re doing here. The more people we can get involved, the more information we can provide.

In the past, only successfully completed repairs were included in our analysis of responses to the Car Reliability Survey. This was done to provide the most precise results possible given our often small sample sizes.

Problem was, this excluded the worst problems–those that lead a person to get rid of a car–from the analysis. So, starting with this month’s survey, we’ve added “sold, traded, donated, or junked instead of repairing” to the list of problem outcomes.

I’ve also gone back over responses for the past year and recoded those that fit this new outcome. So this week’s results will include problems that resulted in the disposal of a car, even if these problems were not first repaired.

This change will generally affect only the results for old cars and for the most troublesome new cars.

Entering the second quarter of 2009, we’ve made three changes to the Car Reliability Survey.

First, major preventative maintenance, most notably timing belts and water pumps, now should be reported on the repair survey. This change has been made to equalize results between engines with timing belts and those with timing chains. Those with timing chains generally cost less to maintain, because there’s no set mileage at which timing chains must be changed. With timing belts, there is a set mileage at which they must be changed. This involves hundreds of dollars in labor, so water pumps and other relatively inexpensive items in the same area are changed at the same time. Because of such preventative maintenance, a survey that asks only about repairs would make engines with timing chains appear less reliable.

What we’ve done to equalize results up until now: water pump failures have only been included in the analysis if they occur before 70,000 miles.

Second, we’ll now ask about mods on the “no repair” survey only when a member first responds and then each January after that. Very few members have mods, and this will reduce the time and effort involved for the 95+ percent that don’t have them.

Finally, when we first created the survey we thought it would be easiest to have people round their odometer readings downward, since they’d simply have to read the numbers off their odometer–and simply drop the final three digits. But this isn’t how people are used to rounding, and quite a few probably rounded up when the last three digits were 500+. In general, it’s not a good idea to deviate from common conventions unless there a very good reason for doing so.

There isn’t a very good reason in this case. Only an attempt to simplify the survey, which didn’t actually simplify it. So the survey now simply asks that the reading be rounded to the nearest 1,000. The average readings should increase by about 500 miles as a result–not a significant change.

Nothing provides more challenges for TrueDelta’s Car Reliability Survey than email, specifically getting it to the inboxes of people who’ve signed up to participate. Too often, it gets trapped by a “bulk” or spam filter.

Some filters are better than others. The worst routinely identify TrueDelta’s email as spam, even though we send no spam and are on no blacklists. Luckily, many people with the worst filters have learned to check their spam folder often–which sort of defeats the purpose. (I receive quite a few email responses from members with [SPAM] inserted into the subject line by some indiscrimate filtering program or another.)

I had been thinking that gmail’s spam filter was perhaps the best, as it rarely had a false positive (identify non-spam as spam). We tend to have a much higher response rate with gmail users than with the users of other free email services.

But this might be changing. Lately I’ve noticed gmail putting some email from members to me into my spam folder. And, what do you know, last night while checking through the spam folder I found the email for Consumer Reports’ auto survey.

Seems this problem threatens them at least as much as it does us.

I’m recently learned that some members think we expect multiple responses to the Car Reliability Survey each month, so they haven’t been responding. Apparently, things have gotten twisted entirely around. Hopefully this post will help untwist them.

The reason for the misperception: these members have been receiving three or four emails each month that say, “Response needed even if there have been no repairs.” What hasn’t been sufficiently clear: this only happens when there hasn’t been a response in recent months (or ever). So these members have come to believe that they’d be getting just as many emails needing a response even if they had been responding to the survey.

This isn’t the case. Instead, members who consistently and promptly respond to the surveys when a response is needed get just four such emails per year. Other months they get a single email asking them to report a repair, if there was one. They never see the reminders that are sent to those who haven’t responded.

In contrast, those who don’t respond do receive the reminders, because some people forget about the intial email and need one. These bump the response rate from about 23 percent to about 36 percent.

Think of it this way: you badly need an answer to a question. So you call the person who can answer it. They don’t pick up the phone, and they don’t call you back. So you call again, and again after that, because you really need this answer. They might think you want to talk to them all the time, and they don’t have the time to talk so often, so they start ignoring your calls. In fact, you just need one quick answer. You’ve only called multiple times because they’ve never picked up the phone.

So, if it seems like you’re being asked to respond to the survey too often, it’s because you haven’t been responding at all in recent months. It’s never too late to sort this out. Respond to the survey when this is needed, and within a few months (it takes the system a little while to “learn” who doesn’t need the reminders) you’ll find yourself receiving far fewer emails.

You can contact us if you have some catching up to do.

I’ve written before about timing belts vs. timing chains. I strongly prefer the latter, because they don’t have to be replaced every 60,000 to 100,000 miles at a cost of 300 to 600 dollars (nearly all of it labor).

But TrueDelta’s repair rate analysis has disadvantaged engines with timing chains. Because most of the cost of replacing a timing belt is labor, when replacing them repair shops usually also replace other fairly inexpensive parts in the same hard-to-access area, most notably the water pump and timing belt tensioner.

As a result of this “preventative maintenance,” car engines with timing belts won’t have these parts fail nearly as often as those with timing chains.

To keep things “apples to apples” between the two engine designs, water pumps and timing belt tensioners won’t be included in the analysis retroactive to the November results. Well, unless someone else has a better solution.

Update: one change already made, another coming.

First, water pump and belt/chain tensioner failures have been reclassified as regular repairs as long as the odometer is under 70,000, since in these situations they wouldn’t have been replaced yet as part of preventative maintenance anyway.

Second, starting next month we’re probably going to start tracking timing belt replacements and “preventative maintenance.”