TrueDelta

A new blog for the Land Rover Freelander charges that it’s not fair to rate the Freelander’s reliability using the same scale for all cars. The two reasons given: luxury car buyers are pickier, and luxury cars have more features.

It might or might not be true that luxury car buyers are pickier. Many people have suggested this, but I’ve never seen any data on it. One thing TrueDelta does to reduce the impact of the pickiness factor: we measure the number of successful repair trips, not the number of perceived problems.

It is certainly true that luxury cars include more parts that can break. But also note that here we report the actual repair frequencies, and not just “better than average” or “worse than average.” If a luxury model requires more repairs because it has more things that can break, then that’s just the facts. People want to know how often a model is likely to require repairs, not some stat adjusted for the number of things that can break.

The larger problem with the post: people aren’t usually going to compare the scores for a Land Rover to those of a Honda, then opt for the Honda. Instead, people considering a Land Rover are likely to compare its scores to those for competing luxury models.

Which brings us to the outright error in the post. It asserts that “other luxury brands like Mercedes-Benz, Rolls-Royce, and even Lexus also get a bad rap.” Sorry, but this just isn’t true. I’ve never seen a reliability stat for Rolls-Royce, so it seems that the author wasn’t looking at actual results when writing the post. Yet more evidence of this: unlike the larger LR3, the LR2 (Freelander2 in the UK) actually has a fairly low reported repair frequency in TrueDelta’s results. Then there’s Lexus. The great majority of Lexus models have received above average reliability scores, even when measured on the same scale as Civics and Corollas. It is more difficult to achieve top scores with a luxury model, but it is nevertheless possible.

The survey covering June ended today. July was not an easy month. The response rate was running two to three percent below the norm for much of the month. Not good, as this would have nearly cancelled out all of the people who joined in the last three months. And I’m not much for running unless it gets me somewhere.

Then, toward the end of the month, we ran a special email asking members why they hadn’t responded.

The #1 reason: the car had been sold. And so we learned that we need a more obvious way for people to report when they buy and sell cars.

Thanks largely to that email, we ended the month with over 12,000 responses, and a response rate a few tenths above the norm. Quite a comeback.

The payoff: 257 full results, and another 249 partial results, in mid-August. Including the first for a 2010 model.

Ten other models fell a single response short of the minimum. A late response might still tip them in.

Results won’t be out before mid-August because I’ve still got a lot of data to clean. When that’s done, July will finally be over. And we can then start looking forward to October…

The emails we receive suggest that many people misunderstand TrueDelta and its research process. Here I’ll attempt to address the most common misunderstandings.

Misunderstanding #1: TrueDelta wants survey responses many times a month.

Reality: We usually only need a survey response the first month you’re eligible to participate, then at the end of every quarter after that (four times a year). We send at least one email per month, though, so people with repairs to report can report them while their memories are still fresh.

If someone hasn’t responded to the survey, they are sent reminder emails. This can lead to the above misunderstanding–some people get the idea that three emails means we wanted three responses per month, when we were actually looking for–but haven’t received–about one response every three months.

Misunderstanding #2: It’s not possible to avoid the reminder emails.

Reality: It’s easy to avoid the reminder emails. First off, respond to the survey when this is needed. In the months when no response is needed, simply click on the “check in” link in the email–this takes just seconds. If you respond promptly when a response is needed, the system will “learn” that you don’t need reminding. Once this occurs–usually after two responses (so two to four months)–it’s not even necessary to click on the “check in” link.

Misunderstanding #3: Participating takes too much time.

Reality: When there’s no repair to report, responding takes the average participant about 30 seconds, and some as few as ten seconds. Even if there is a repair to report, responding takes just a couple of minutes. Other car reliability surveys are much longer.

Misunderstanding #4: The emails aren’t necessary–why not just let people report repairs as they occur?

Reality: We wish the emails were not necessary, but the response rate would be VERY low without them. Nearly all responses occur within 48 hours of an email. Once 48 hours have past, most people forget to respond, even though many intended to. So it’s also necessary to send reminders.

Misunderstanding #5: There’s no point in participating once you’ve bought a car.

Reality: The only way we can provide information when people need it is if they participate in between purchases. This is a team effort. Provide information when other members need it, and they’ll provide information when you need it. You might not need reliability information on the car you currently own, but you might later need reliability information on cars other members own.

Misunderstanding #6: It’s necessary to check your car to get an exact odometer reading.

Reality: We collect odometer readings to verify that you’re still active, and to get an idea of how many miles (or kilometers) are on the average car. It’s not important to know if the average is 11,000 or 12,000. So an approximate reading is good enough. If you’re guess is way off, we can easily correct it.

Misunderstanding #7: It’s necessary to have a repair receipt to respond to the repair survey.

Reality: The repair survey is designed to be answerable based on what you can remember. The most critical information–whether or not the car was repaired–tends to be easy to remember, especially since you’re only being asked to remember the last month or two.

Misunderstanding #8: We’re only interested in data on fairly new cars.

Reality: The Car Reliability Survey covers any car from 1995 on. The other surveys currently only cover from 2002 on, but we’ll be extending them to cover a few earlier model years.

Misunderstanding #9: We’re only interested in cars owned in the U.S.

Reality: While most members are Americans, we have thousands of Canadians and hundreds from outside North America. The Car Reliability Survey can include any car, anywhere–as long as at least 25 are signed up to participate. We’ve been adding non-US powertrains, for people in the UK and elsewhere who want to participate in the Repair History and Gas Mileage Surveys. Want more participants from where you live? Then please tell friends and family about our research.

Misunderstanding #10: We’re a big corporation collecting data from car owners so we can sell it to manufacturers.

Reality: While a few people are employed part-time on a contract basis, I’m TrueDelta’s only full-time employee. We do not sell data to the manufacturers. Instead, our focus is on providing information to members. When you respond to a survey, you’re helping other participants.

By keeping our expenses low, we’re able to do everything necessary to collect and analyze the data without charging participants any membership fee. We’re able to cover our costs through ads and affiliates–without accepting any ads directly from the car manufacturers.

The main limitation: we can’t afford to spend money on advertising and marketing. So it’s very important that members help the site grow by telling others about what we’re doing here. The more people we can get involved, the more information we can provide.

In the past, only successfully completed repairs were included in our analysis of responses to the Car Reliability Survey. This was done to provide the most precise results possible given our often small sample sizes.

Problem was, this excluded the worst problems–those that lead a person to get rid of a car–from the analysis. So, starting with this month’s survey, we’ve added “sold, traded, donated, or junked instead of repairing” to the list of problem outcomes.

I’ve also gone back over responses for the past year and recoded those that fit this new outcome. So this week’s results will include problems that resulted in the disposal of a car, even if these problems were not first repaired.

This change will generally affect only the results for old cars and for the most troublesome new cars.

Entering the second quarter of 2009, we’ve made three changes to the Car Reliability Survey.

First, major preventative maintenance, most notably timing belts and water pumps, now should be reported on the repair survey. This change has been made to equalize results between engines with timing belts and those with timing chains. Those with timing chains generally cost less to maintain, because there’s no set mileage at which timing chains must be changed. With timing belts, there is a set mileage at which they must be changed. This involves hundreds of dollars in labor, so water pumps and other relatively inexpensive items in the same area are changed at the same time. Because of such preventative maintenance, a survey that asks only about repairs would make engines with timing chains appear less reliable.

What we’ve done to equalize results up until now: water pump failures have only been included in the analysis if they occur before 70,000 miles.

Second, we’ll now ask about mods on the “no repair” survey only when a member first responds and then each January after that. Very few members have mods, and this will reduce the time and effort involved for the 95+ percent that don’t have them.

Finally, when we first created the survey we thought it would be easiest to have people round their odometer readings downward, since they’d simply have to read the numbers off their odometer–and simply drop the final three digits. But this isn’t how people are used to rounding, and quite a few probably rounded up when the last three digits were 500+. In general, it’s not a good idea to deviate from common conventions unless there a very good reason for doing so.

There isn’t a very good reason in this case. Only an attempt to simplify the survey, which didn’t actually simplify it. So the survey now simply asks that the reading be rounded to the nearest 1,000. The average readings should increase by about 500 miles as a result–not a significant change.

Nothing provides more challenges for TrueDelta’s Car Reliability Survey than email, specifically getting it to the inboxes of people who’ve signed up to participate. Too often, it gets trapped by a “bulk” or spam filter.

Some filters are better than others. The worst routinely identify TrueDelta’s email as spam, even though we send no spam and are on no blacklists. Luckily, many people with the worst filters have learned to check their spam folder often–which sort of defeats the purpose. (I receive quite a few email responses from members with [SPAM] inserted into the subject line by some indiscrimate filtering program or another.)

I had been thinking that gmail’s spam filter was perhaps the best, as it rarely had a false positive (identify non-spam as spam). We tend to have a much higher response rate with gmail users than with the users of other free email services.

But this might be changing. Lately I’ve noticed gmail putting some email from members to me into my spam folder. And, what do you know, last night while checking through the spam folder I found the email for Consumer Reports’ auto survey.

Seems this problem threatens them at least as much as it does us.

I’m recently learned that some members think we expect multiple responses to the Car Reliability Survey each month, so they haven’t been responding. Apparently, things have gotten twisted entirely around. Hopefully this post will help untwist them.

The reason for the misperception: these members have been receiving three or four emails each month that say, “Response needed even if there have been no repairs.” What hasn’t been sufficiently clear: this only happens when there hasn’t been a response in recent months (or ever). So these members have come to believe that they’d be getting just as many emails needing a response even if they had been responding to the survey.

This isn’t the case. Instead, members who consistently and promptly respond to the surveys when a response is needed get just four such emails per year. Other months they get a single email asking them to report a repair, if there was one. They never see the reminders that are sent to those who haven’t responded.

In contrast, those who don’t respond do receive the reminders, because some people forget about the intial email and need one. These bump the response rate from about 23 percent to about 36 percent.

Think of it this way: you badly need an answer to a question. So you call the person who can answer it. They don’t pick up the phone, and they don’t call you back. So you call again, and again after that, because you really need this answer. They might think you want to talk to them all the time, and they don’t have the time to talk so often, so they start ignoring your calls. In fact, you just need one quick answer. You’ve only called multiple times because they’ve never picked up the phone.

So, if it seems like you’re being asked to respond to the survey too often, it’s because you haven’t been responding at all in recent months. It’s never too late to sort this out. Respond to the survey when this is needed, and within a few months (it takes the system a little while to “learn” who doesn’t need the reminders) you’ll find yourself receiving far fewer emails.

You can contact us if you have some catching up to do.

I’ve written before about timing belts vs. timing chains. I strongly prefer the latter, because they don’t have to be replaced every 60,000 to 100,000 miles at a cost of 300 to 600 dollars (nearly all of it labor).

But TrueDelta’s repair rate analysis has disadvantaged engines with timing chains. Because most of the cost of replacing a timing belt is labor, when replacing them repair shops usually also replace other fairly inexpensive parts in the same hard-to-access area, most notably the water pump and timing belt tensioner.

As a result of this “preventative maintenance,” car engines with timing belts won’t have these parts fail nearly as often as those with timing chains.

To keep things “apples to apples” between the two engine designs, water pumps and timing belt tensioners won’t be included in the analysis retroactive to the November results. Well, unless someone else has a better solution.

Update: one change already made, another coming.

First, water pump and belt/chain tensioner failures have been reclassified as regular repairs as long as the odometer is under 70,000, since in these situations they wouldn’t have been replaced yet as part of preventative maintenance anyway.

Second, starting next month we’re probably going to start tracking timing belt replacements and “preventative maintenance.”

After noting the dates of the Chicago Auto Show, I realized that the February results will have to be released on the 5th to avoid that media frenzy. Which will be tough, since the current survey won’t end until late at night on the 2nd.

Actually, it’s even worse. The press release firm we use requires a two-day lead time. So the release will have to go out hours after the survey ends.

So, TrueDelta’s research process will truly be put to the test the month. Usually we have about 90 percent of the analysis done by the time the survey period ends. This time it’ll have to be 99.9 percent.

Members are doing their part. Today’s email asked people to respond early in the month, so we’d have less to do later in the month. And the response rate so far is the strongest in the past two years, with 14.8 percent already responding in the first six hours.

A little perspective on growth. This month’s survey includes 25,040 cars. The survey in January 2008: 11,771. The survey in January 2007: 2,833.

If the response rate remains strong, this could be the first time we have 10,000 responses. Most likely we’ll end up between nine and ten, so this honor will fall to the April survey. But, maybe not. I’m hopeful.

Every January, April, July, and October I closely watch the responses as they come in, a flood the first couple of days, a trickle towards the end. How many people will respond this time?

Well, for October 2008 (which technically ends tomorrow morning), a bit over 7,700, up from a bit over 7,000 last time. The response rate this time: 35.4 percent, down from 36.2 percent. Given the current level of economic distress, which has many people too stressed out to participate in non-essential activities (TrueDelta isn’t putting food on the table), not bad.

It looks like we’ll have 177 model/model years in the November results, plus partial results for another 175. This is up from 149 and 152, respectively, three months ago, and about where I hoped we’d be.

And yet, despite my best efforts, we still have two models–the 2006 Acura TSX and 2007 Jeep Compass / Patriot–that are a single response short of the minimum. In both cases we have plenty of owners signed up, and 24 of them have responded. The holdouts have been informed that 24 other owners need them to do what they signed up to do, and participate in the survey. Those of you who have responded know that it takes less than a minute when there’s no repair to report (and about 90 percent of the time there isn’t).

But in these two cases that 25th participant hasn’t been forthcoming. To put it figuratively, the ball has been fumbled a yard from the goal line, but no one has picked it up and walked it over the line. They all want someone else to do it.

Part of me knows this is to be expected. But it remains among the most frustrating aspects of conducting this research. Happy about the 177 results we’ll have, but not so much as to forget the two we’ll barely miss.