Is IIHS testing data valuable?

From doing some research, I note that CPF has a dim opinion of Consumer Reports headlight testing.

It seems that CPF has a slightly higher opinion of IIHS' headlamp tests.

However, this part of Virgil's post from 2016 stood out to me:

No, it really doesn't. Not once you've seen the stackup of guesses, assumptions, and apparently random, pseudoscientific numbers that have gone into the test protocol.

Click to expand...

I'm looking to get a new car soon, and I live in upstate New York. Most of the highways here aren't lighted at night. I want a car with decent headlights. Can I trust IIHS testing? I'm curious as to what the assumptions and pseudoscientific numbers are in the IIHS testing protocol.

I lived right between Albany & Syracuse (exit 19A), so if I headed north - I know what you mean. If you can afford it - the newer cars with OEM LED lights really do a great job. I bought a 2012 Genesis in 2013, unfortunately it wasn't until the second upgrade package that you get the LED lights, but I got them and they do a great job.

A high-level overview of the IIHS testing protocol is thankfully publicly available.

Key data seems to be ...

• High/low beam ratings w/ 5 lux floor for illuminance
• Low beam glare rating with unspecified value
• Straightway out to 600'
• Gradual left/right curves w/ 800' radius
• Sharp left/right curves w/ 500' radius
• Visibility measurements taken at right edge of roadway
• Straightaway measurements second measurement taken on left edge
• Visibility measurements taken at 10" height
• Glare measurements taken at 3'7" height

I believe one of the main complaints about the IIHS testing protocols was that they didn't aim the headlights, using factory/dealer settings as-is. The rationale was that most drivers don't adjust their vehicle's headlights so this testing was "real world" for most of the driving public.

Of course, the problem with this approach was that it leaves much performance to chance - are the lights aimed too high? Too low? Were otherwise excellent headlights rated poorly because of a bad factory/dealer adjustment? It might reflect the reality most drivers live with, but it also introduces a wildcard to the results - one that's easily fixable, oftentimes with a Philips screwdriver, ten minutes, and no more involvement than popping the hood and cranking the light until it's aimed.

We don't know why they chose 5 lux as their threshhold value. Nor did they asses more granular beam distribution issues although the "around curve" performance at least hints at beam width. Given that IIHS is the Insurance Institute for Highway Safety I guess their interest is in coarse long-distance lighting and their idea of a "gradual" and "sharp" curves will be limited to highway notions. This doesn't speak to city driving scenarios where the curves can be sharper and there's often a jumble of street lights to muddy the waters of headlight performance.

One gets the feeling that goniophotometer readings at 2 distances could extrapolate the same outcomes that IIHS produced with a big test track while also producing far more granular results covering covering far more scenarios.

This is all very interesting to me. Not only from a practical standpoint, as in, what car and headlight package should I get so I can safely drive on these unlit NY highways, but also from a technical standpoint. I appreciate your thorough reply. It has the engineer in me excited!

idleprocess said:

I believe one of the main complaints about the IIHS testing protocols was that they didn't aim the headlights, using factory/dealer settings as-is. The rationale was that most drivers don't adjust their vehicle's headlights so this testing was "real world" for most of the driving public.

Of course, the problem with this approach was that it leaves much performance to chance - are the lights aimed too high? Too low? Were otherwise excellent headlights rated poorly because of a bad factory/dealer adjustment? It might reflect the reality most drivers live with, but it also introduces a wildcard to the results - one that's easily fixable, oftentimes with a Philips screwdriver, ten minutes, and no more involvement than popping the hood and cranking the light until it's aimed.

Click to expand...

idleprocess said:

One gets the feeling that goniophotometer readings at 2 distances could extrapolate the same outcomes that IIHS produced with a big test track while also producing far more granular results covering far more scenarios.

Click to expand...

I remember when IIHS first came out with its testing results, the organization bashed existing headlight standards and regulations for having little bearing on real-world results precisely because these results were obtained in a lab and not on the road. As you said, people don't aim their lights, at least not here in the US.

http://www.iihs.org/iihs/news/desktopnews/most-small-suv-headlights-rate-poor-in-iihs-evaluations

Headlight performance in today's vehicles varies widely. Government standards are based on laboratory tests, which don't accurately gauge performance in real-world driving. The issue merits attention because about half of traffic deaths occur either in the dark or around dawn or dusk.

Click to expand...

Interestingly enough, the NHTSA, or "the government," I remember, is considering implementing a similar headlight testing protocol. And it's a "real-life" protocol rather than a lab protocol, which, to me, lends credence to the fact that IIHS chose to do track tests rather than lab tests.

http://www.nhtsa.gov/images/5stars/nhtsa-5stars-rfc-notice.pdf

idleprocess said:

We don't know why they chose 5 lux as their threshhold value. Nor did they asses more granular beam distribution issues although the "around curve" performance at least hints at beam width. Given that IIHS is the Insurance Institute for Highway Safety I guess their interest is in coarse long-distance lighting and their idea of a "gradual" and "sharp" curves will be limited to highway notions. This doesn't speak to city driving scenarios where the curves can be sharper and there's often a jumble of street lights to muddy the waters of headlight performance.

Click to expand...

I know I read a rationale for the 5 lux figure before. I forgot what it was though, but I finally found this article:

http://www.iihs.org/iihs/sr/statusreport/article/51/3/1

For comparison, a full moon on a cloudless night illuminates the ground below to about 1 lux. Three lux is typically enough to make out low-contrast objects, but 5 lux can be more accurately measured and therefore works better as a threshold for the test.

Click to expand...

Does 3 lux make any sense as a threshold? Is 3 lux a value commonly cited in academic literature as a "threshold"?

Interestingly enough, the NHTSA also seems to like the 3 lux threshold. Does this lend credence to the 5 lux figure used by the IIHS? From the NHTSA paper linked above:

The level of performance would be established based on the lower beam headlighting system's ability to provide 3.000 lux of light to each of the five detectors. If all five detectors are illuminated to at least 3.000 lux and the glare detector is illuminated at less than 0.634 lux.

Click to expand...

Also interesting is the government's ability to measure headlight performance to the ten-thousandths place while IIHS can only afford to measure headlight performance to a whole number greater than 3 (they couldn't even measure 3, so they chose to measure 5). :twothumbs

the newer cars with OEM LED lights really do a great job

Click to expand...

Not necessarily. LED light sources don't automatically mean good headlight performance. There are (still) good, bad, lousy and outstanding headlamps of just about every technological description.

this part of Virgil's post from 2016 stood out to me

Click to expand...

Read the entire thread, not just that one quote from pretty far down the rabbit hole.

The IIHS tests aren't garbage or anything, it's just that they don't test what they claim to be testing. They claim to be testing the headlight performance of whatever year, make, model, trim of vehicle is named at the top of the page. But what they're actually testing is the headlight performance of the one example of that vehicle they happen to have gotten. That's because they don't aim the headlamps correctly before doing the test. Their explanation is reasonable -- most headlamps don't get checked or properly adjusted ever in their life -- and the results they're getting demonstrate that headlamp aim is crucial and needs to be getting a lot more attention than it does. In response the industry is working to get more careful about how headlamps are aimed on new vehicles, and maybe that effort will eventually filter down to the dealer level and it might stop being so close to impossible to get your headlamps checked and aimed correctly in this country.

But it really is either/or: either IIHS aims the headlamps correctly and then the results they get are indicative of the headlight performance one can expect on that kind of car with that headlamp configuration aimed correctly, or IIHS doesn't aim the headlamps and then the results they get are indicative of the luck of the draw on that particular, specific vehicle they tested…and no other.

I'm curious as to what the assumptions and pseudoscientific numbers are in the IIHS testing protocol.

Click to expand...

I have not seen any "pseudoscientific numbers" in the IIHS test protocol or results. What "pseudoscience" do you think you see?

NHTSA, or "the government," I remember, is considering implementing a similar headlight testing protocol

Click to expand...

Similar idea, but it's actually quite different in its crucial details.

And it's a "real-life" protocol

Click to expand...

Well, no, not really. It's a static test, not a dynamic test like the IIHS protocol.

Given that IIHS is the Insurance Institute for Highway Safety I guess their interest is in coarse long-distance lighting

Click to expand...

That is not even slightly correct. The "highway" in IIHS's name refers to public roads of every kind, not highways versus city streets.

-Virgil- said:

or IIHS doesn't aim the headlamps and then the results they get are indicative of the luck of the draw on that particular, specific vehicle they tested…and no other.

Click to expand...

Is it really a matter of the "luck of the draw"? Do manufacturers just slap the headlights into the car any which way and torque the attachment bolts and aiming screws to arbitrary values, or do they aim for some level of consistency?

Granted, I don't know the answer, but I'm thinking that there's some level of consistency at the manufacturing level that makes the results for each vehicle/trim level generalizable.

-Virgil- said:

I have not seen any "pseudoscientific numbers" in the IIHS test protocol or results. What "pseudoscience" do you think you see?

Click to expand...

I don't know what "pseudoscientific numbers" are in the IIHS test protocol or results. Those are your words, and I was hoping you could shed some light on what you wrote. I read the entire thread looking for examples of these "pseudoscientific numbers" and I couldn't find any.

Ls400 said:

Is it really a matter of the "luck of the draw"?

Click to expand...

Yes. Test ten different cars, and you'll get ten different results. In response to the IIHS results the automakers are working on doing a better, more consistent job of aiming headlamps before the cars leave the factory, though effective aim will change with suspension settling (cars strapped down for transportation from the factory, then driven around, etc), and there's still the tendency of dealer service departments to do a sloppy, quick job of headlamp "aiming" as a part of pre-sale preparation, if they touch it at all.

I don't know what "pseudoscientific numbers" are in the IIHS test protocol or results. Those are your words

Click to expand...

Oh, so they are -- sorry, I guess I should re-read my old posts from time to time What I meant by that was basically the same thing I said here in this thread earlier: not aiming before the test means the tests do not (cannot) tell us much about the headlight performance on a particular kind of car...they can only tell us about the headlight performance on a particular example of a particular kind of car.

The tests aren't garbage and the results aren't useless, they just do something other than what the IIHS promotes them as doing. I think the IIHS didn't go to misrepresent their activities, I think they made a decision about what kind of publicity would put the tests in the news, and "XX% of all the vehicles we tested have inadequate headlamps!" is a better fit for that goal than something like "There's a significant range of effective headlight performance depending on aim angle variance".

-Virgil- said:

Not necessarily. LED light sources don't automatically mean good headlight performance. There are (still) good, bad, lousy and outstanding headlamps of just about every technological description.

Click to expand...

Man...., okay mine self level when I start the car - and at night I can clearly see the horizontal "cut-off" at the top when it does that (my car is pointing slightly downhill). I find my LEDs are quite superior to my previous car (last model year Bonneville). I cannot really comment on the multitude of rental cars I get a year since they are usually driven in well lit areas.

Since I've had a beer (or two) I'll expound on my recent rental car experience in a new thread in the cafe. Bottom line - check the tires - particularly in an area where snow tires are mandatory in the winter and they are swapping tires/wheels.

nfetterly said:

Man...., okay mine self level when I start the car

Click to expand...

That's good, but it doesn't mean they aim themselves correctly, it just means they self-compensate for whatever cargo or passenger load is in the rear of the car. The lamps could very easily still be misaimed.

at night I can clearly see the horizontal "cut-off" at the top

Click to expand...

Contrary to mythology, a sharp cutoff line at the top of the low beam doesn't necessarily mean it's a good headlamp. There are advantages and disadvantages to a sharp cutoff, but the amount of light and how it's distributed below the cutoff (sharp, soft, or nonexistent) is much more important.

I find my LEDs are quite superior

Click to expand...

They very well might be, depending on the particulars. But be careful not to mistake subjective impressions for facts; human eyes don't do a good job of accurately determining how much light there is or how well they're doing.

Does anyone have an opinion about the 3 lux "floor" that is referenced by both the IIHS and NHTSA?

My opinion: if the goal is to describe what the driver can see by the headlights, 3 lux might be a more reasonable threshold than 1 lux and 5 lux might be a more realistic threshold than 3 lux, but it depends on where we're trying to see stuff. Way down the road in the distance, I don't think it's going to make a very big difference whether there's 1, 3 or 5 lx on the target; assuming the target is 25 meters away, that's 625, 1875, or 3125 candela -- not a lot of light at that distance. But if we're talking about peripheral/lateral stuff at closer distances, then 1875 or 3125 candela is a useful amount of light, and a reasonable argument can be made that the 625 cd (1 lux) level is useful at closer distances; that's roughly the amount of light you might get from a cornering light with an 1156 or similar bulb.