Thursday, November 4, 2010

2010 CR Car Reliability Study: Some Flaws

What are the most reliable cars for 2011? Consumer Reports just published its 2010 car reliability report, along with its expected reliability ratings for 2011, and, depending upon your expectations, the surprises abound - or not.

Consumer Reports (CR)  received data from 960,000 consumers on 1.3 million vehicles. Its analysts generated expected reliability ratings for 2011, in part from CR's existing reliability records over the past three years. The results of the report were presented last week by the senior director of Consumer Reports Auto Road Test Center, David Champion.

The Consumer Reports Car Reliability Survey is traditionally the most influential reliability study for the car industry, and is widely considered to be the third most influential factor in purchasing a new car, after brand loyalty and recommendations from friends and family. It is so influential that, when Alan Mulally took over at Ford in 2006, he took his management team to the Consumer Reports test track, so as to have the management team hear first hand reliability feedback from CR (msnbc story). We have faithfully studied its recommendations, year after year, for the past 20 years. This year, however, when looking more carefully at the data used by CR, we find some areas of concern in the statistical validity of the study conclusions. In particular, it appears that there is little data available, for many suppliers, for the critical 2009 and 2010 years, clearly due to the economic crisis.

An example in kind (maybe the worst of them) is Porsche, rated #2 brand this year. The Boxter is rated #1 reliable vehicle in the whole survey, yet CR has no data for it (one entry) in 2009 and 2010. This prediction, then, is based upon 2008 data only, since CR looks at the last 3 years of data for 2011 predictions - even though there were significant changes in the 2009 model year. There is no data for the Panamera and the Cayenne, both introduced in the past 2 years. As a result, we feel that the Porsche rating is highly doubtful. While Porsche is an extreme example due to its relatively smaller size, the same problem affects other brands as well, although to a slightly lesser degree. Michael Karesh has an interesting analysis of the 2010 CR Reliability Survey in The Truth About Cars.

We therefore consider that the conclusions of the 2010 CR reliability survey have to be taken with a grain of salt. Be that as it may, here they are.

"Luxury" European cars trail the ratings

Between 45% and 75% of the models from BMW, Mercedes and Audi scored worse or much worse than average, and these brands ended up among the dregs, only ahead of Chrysler. In fact, the lowest rated car in America was the Audi A6. This disastrous result for European luxury car brands was slightly mitigated by Porsche and Volvo. Porsche, helped by having only three models tracked by CR, ended up 2nd (the Porsche Boxter was the most reliable vehicle is the survey), and Volvo ended up 8th. It is unclear what will happen to Volvo model evolution, as it was recently purchased by Chinese automaker Zhejiang Geely. Volkswagen, Audi's parent, which it not a luxury car provider, had decent scores, in particular for its Golf and Jetta models.

These results might look shocking, but they are less surprising than they look. None of the European Big Three luxury brands has ever done very well in the ratings, and they have often ranked towards the bottom, BMW typically scoring slightly better than the others. The past few years had seen some progress for these brands - the trend clearly reverted this year. Based on past performance over the last 5 years, we should not expect any of these brands to show good reliability performance in the next few years. They never have in the past except in yoyo fashion, up one year, down the next.

US car manufacturers are rapidly improving - except for Chrysler

Over the past several years, Ford had progressed in the quality curve, while GM and Chrysler were stagnant at the bottom. This years crystallizes major progress for Ford, and shows GM emerging as a significantly better reliability player.

Ford is slowly but surely closing the gap with the Japanese OEMs. As a brand, Ford ranks 10th, immediately behind Lexus, and in front of Mazda and Nissan. Approximately 90% of Ford and Lincoln models have at least average reliability. In fact, Ford ranks 1st in the highly disputed family car segment with the Fusion Hybrid. It also takes top ranking in the large SUV segment with the Flex EcoBoost. At the same time, Ford has been creeping up on the ratings for a while. We would like to see them stick to it and finally make a stable place for themselves in the top 5 - they are not there yet.

GM showed dramatic progress from last year. It now ranks ahead of Mercedes, Audi and BMW. Its main brand Chevrolet had 83% of its models rated average reliability or better, from 50% last year, while Cadillac gained 7 places over the past year. As a whole, GM had 69% of its models rated average reliability or better. Some of the progress is due to GM shuttering three brands (Saturn, Pontiac, and Hummer) and taking out quite a few low reliability models. The happy news is that some is also due to new models showing good reliability from the get go, such as the Cadillac SRX, the Chevrolet Camaro, the Chevrolet Equinox, and the Buick LaCrosse V6. We should point out that, because product development in the automotive industry is a long cycle, everything we are seeing today was done by the "Old" GM. None of the "New" GM work has seen introduction yet.

Chrysler, however, remains the lowest rated brand of all, and carries older models of poor reliability, the result of a decision in 2008 by previous management to slow down development in the face of the global automotive industry slow down. Chrysler, however, has announced that 75% of its model portfolio was new or updated in 2011. In support, Mr. Champion mentions his engineers having tried out pre-production models and being impressed by what they saw. We will have to see what the next year brings - it might be Chrysler's last chance to make good. Fiat, now in control of Chrysler, is not known for outstanding reliability. We want to see them healthy and wish them the best for 2011.

Japan is still on top - but Honda is first

Taking out the epiphenomenon of small Porsche,  the top brands, in order, Scion (Toyota), Acura (Honda), Infiniti (Nissan), Toyota, Subaru, Volvo, Lexus, and Ford. Honda, Toyota, and Subaru. When grouping by manufacturer (again excluding Porsche, now owned by VW), the top 5 are, in order, Honda, Toyota, Subaru, Volvo and Ford. The situation is inverted from what it used to be several years ago, when Toyota regularly came first and Honda second.

Among the Japanese luxury brands, Acura and Infiniti remain in the top 5, but Lexus, once the epitome of refinement, which had already come down from the top rank in the past few years, tumbles down to #9.

Despite the wide recalls, Toyota and Honda remain very well placed. One reason for it is that CR does not score recalls, but only direct problems seen by subscribers. However, there used to be very low variation in reliability between the different models from Honda and Toyota, which were of uniformly good quality. This is still true for Scion (+/- 10%), Infiniti (+/- 20%), and Acura (+/- 20%). It is less true of Honda (+/- 30%), although all of its models are still mathematically above average, and even less true of Toyota (+/-40%), where some models are mathematically below average (although they might still be binned as an average model). In other words, it is not possible any more to buy a Toyota, and to a degree a Honda, with one's eyes closed, knowing that their reliability will be way over average: some models will be much more reliable than others. Amazingly, the 2010 Prius only scores average reliability - what a disappointment!

Among the "regular" Japanese brands, Scion scores first, although this is probably due to the fact that the product line, always reliable, has not seen recent introductions (new models often go down in reliability at introduction).The next year should see new introductions, although, almost miraculously, previous Scion introductions have typically seen very high reliability from day one.

While still in #7 rank, Subaru dropped two spots from last year, a slightly worrisome trend for the traditionally non-traditional car manufacturer. Coupled with the new styling seen in recent models since 2008, which appears to have alienated some long-time Subaru buyers, this might make Subaru loyalists think twice about new purchases. Regrettably, the Impreza WRX, which was poised to replace the legendary Acura Integra of the past as the most fun practical hatchback on the market, also shows the worst reliability of all Subaru models.

Nissan is now #14 as a brand, a long term downward that has taken it from the traditional Japanese Big Three, which are now The Japanese Big Two. Nissan now scores below Volvo, Ford, Hyundai, Mazda and Kia. Variation between models is now high, at +/- 40%, and several models score mathematically below or significantly below average.

Hyundai/ Kia - the new top player

What a change in Hyundai's position in the past 10 years! From a cheap upstart with a reputation of poor reliability, it has turned into a major international car manufacturer, with two US brands, and a very good reliability ranking. All six new models in 2010 from Kia and Hyundai were rated average reliability or better, and only one vehicle overall (the Sedona) was rated below average.

What is the most reliable brand to buy?

A few years ago, there would have been an easy answer: any car from Toyota, Honda, or Nissan (in that order) would have been good enough. Today, for most brands, there is too much variation between models to be able to give a blanket endorsement. The right process to prepare for a car purchase is to research a category of cars, and evaluate, for each car, expected reliability, based on Consumer Reports and other available data. However, it is still possible to come with a small number of recommendations, combining the average score of each brand with its variation (or lack thereof) among models. Since we are not only looking at average score, but also at variation, our order is not the same as CR, even though we use their data:

  • #1 Scion  
  • #2 Acura
  • #3 Honda
  • #4 Infiniti
  • #5 Volvo

  • #6 Toyota
  • #7 Porsche
  • #8 Subaru
  • #9 Hyundai
  • #10 Kia
Amazingly,  Toyota and Subaru do not make it into our top recommendations, due to their higher variability and the fact that some of their models scored too low. We were the first ones to be shocked. The same is true for Porsche.

Not sure how to compare reliability data across sources, and go beyond Consumer Reports? Check back here on in a small number of weeks to see our 2011 Car Report along with our evaluation of sources!

Want to read more about it? Try Reuters, Detroit Free Press , Globe and Mail, LA Times, NY Times, Canadian Driver, Truth About Cars 1, Truth About Cars 2, Truth About Cars 3, CNetAutoSpies,, Car Connection, Consumerist, Washington Post, NPR, Fox News, Edmunds, Autoblog, Product Reviews, U.S. News, American Consumer News , Auto Channel, Money Watch, USA Today, Wall Street Journal, msnbc, and Canadian Press

No comments:

Post a Comment