In this thread: we cannot stop the error analysis.
As a direct sensor reading, lux can be close to fact, but the least light pollution or smudge on your sensor and you're off. Now we're calling the lux meter itself into question. Is it calibrated? How long ago? Is it possibly damaged?
As a calculated and experimental value, lumens are a bit tricky to pin down within +-5% even for the big boys with pocket protectors, graphing calculators, spreadsheets, and no date on Friday night. So your lux meter is fine, but is your calibration factor off? Is there a new problem with the gear that formerly worked?
Again, the multiple samples are needed to make the correction factor acurate as all factory lights will be off their ANSI spec for one reason or another. Some manufacturers are very close and some are off due to lesser manufacturing tolerances. I need the correction factor to be accurate as possible in order to conver the lux reading from the sensor into lumens. I've read numerous reviews from respected members and they've found some lights 15% lower than advertised. I have to go by the advertised figure therefore the more samples the less my correction factor is off.
Oh yes, "small sample size error" can really throw things out of whack. There is never any harm in comparing 10 samples of the same thing. That sounds more like science to me.
Not just manufacturing tolerances, but different batteries can have huge impacts as well. To say a light has "one output and only one measurement is required" is not correct.
An example from our illustrious Vinh could be the many times where he has compared lux and lumens
with the same flashlight but different batteries. We've seen raw max/min spreads in excess of 25%! I'd post a link, but this comes up so frequently and lux/lumens are said so frequently in the V54 forum that it's actually kind of hard to search for an example.
This is a factor, particularly on current boosted lights where max/turbo is direct drive and the light becomes limited by resistances of the system and what the battery is capable of providing. In certain such lights, a protected battery that trips at 5 amps or a battery with poor maximum drain is going to get functionally stomped by something like the Sony VTC5 (30 amp continuous) and you'd never know without taking multiple data points and scratching your head over the "error bars." What was the cause of the discrepancy?
https://en.wikipedia.org/wiki/Error_bar
Error bars are a graphical representation of
the variability of data and are used on graphs to indicate the error, or uncertainty in a reported measurement. They give a general idea of how precise a measurement is, or conversely, how far from the reported value the true (error free) value might be. Error bars often represent one
standard deviation of uncertainty, one
standard error, or a certain
confidence interval (e.g., a 95% interval). These quantities are not the same and so the measure selected should be stated explicitly in the graph or supporting text.
Error bars can be used to compare visually two quantities if various other conditions hold. This can determine whether differences are
statistically significant. Error bars can also suggest
goodness of fit of a given function, i.e., how well the function describes the data.
Scientific papers in the experimental sciences are expected to include error bars on all graphs, though the practice differs somewhat between sciences, and each journal will have its own
house style.
When I was partying my brains out in engineering college, error analysis was a freshman first semester sort of topic. My first reaction to error analysis math was "WTH is this???" After I wrapped my head around it, I saw that no engineering can be complete without it.
Hate it if you want, but you cannot escape error analysis. Just one measurement is the Diet Coke of analysis. Just one calorie, not enough.
We are not at complete odds. What we are saying is that, if an ANSI laboratory tested a single lower mode (so batt. condition wasn't a factor) from one of your lights, and said that output is exactly X, then you be done and have your calibration light. The problem is knowing what true ANSI is, and all we are saying is that, testing more and more lights will not get you any closer to ANSI, and in fact may take you further and further away as, logically, on average, the industry is predisposed to exaggeration.
Of course, there is no certainty with any of this, but am now very comfortable with my scale, and IMHO if I where to use some sort of regression analysis from my whole collection, I'd be much worse off.
Part of error analysis is being able to throw out obviously erroneous data, including bogus manufacturer claims.
I see what you're saying about using false maker claims as part of calibration. IMHO, one of the hardest things about calibration scales as we're discussing is getting started with valid data. Using bogus data to calibrate will definitely cause issues, I agree with you completely sir!
