After a recent magazine article Jim and I had a pretty good rant about torch testing, wondering why reviewers didn’t seem to do any testing of their own. Instead simply trusting the marketing focused specs published by the manufacturers. It is frustrating to see several brands of torches which use the same batteries and run the similar LED’s, and thus produce a similar lumen output to the Gamma being quoted as able to last 50 hours longer! Taking these manufacturers stated figures without comparative testing definitely shows big imbalances in expected performance.
The reason for this imbalances is simple, where do you draw the line for burntime? Some manufacturers do not consider their torches to have depleted until their output drops to 0.25 Lux (a similar level to that of moonlight) at a distance of 2m in front of you. This seems disingenuous; surely you buy a 50 lumen headtorch with a burntime of 55 hours on full power.. 50 lumens of power. In reality how much of this 55 hours is at 50 lumens and how much will be just enough to even read a book by!? As the author of flashlightreview.com points out:
“I would expect that a consumer would buy a light which has output that matches their needs and when it drops below 50% of that output, the light is no longer meeting their original needs and the batteries will have to be replaced. The additional 10, 100, 1000 (or whatever) hours of runtime are essentially useless for meeting the needs of that consumer.”.
We’re not photonics boffins in any way and have never professed to be. Our knowledge of torches up to this point is little more than our experience of using them. We work closely with our suppliers and entrust their knowledge of electronics to develop products we feel will work well for climbers, walkers and mountaineers. When they tell us the torch they are making for us has a 70 lumen output we don’t have the lab equipment or expertise to say otherwise. We approach product testing from a different angle; we get out there and hammer them in real life; we all used the Gamma extensively from everything from car tinkering to long night rides before we launched it as a live product on our website. We think it’s a great bit of kit and we are very proud of it.
The same torch and same camera settings showing full power and 0.25 Lux at 2m, not much use for navigation!
Our original idea was to have our own little ‘Grouptest,’ in house to benchmark our torches against a few others. The plan was to measure the brightness of the torches, ideally in lumens to see if they live up to the claims, and take a photo of each firing down a calibrated ‘range’ to give a good idea of how much useful light is put out by each.
After stumbling across Flashlightreview.com we saw a way that we could on a shoestring (the ‘real’ equipment costs tens of thousands) take an objective look at torches, compare them like for like rather than simply accepting the marketing hype, develop a way of generating a real life indicator of how our torches perform, allowing users to make a properly informed decision about the products.
There is a standard for measuring the performance of torches (ANSI / NEMA FL 1-2009) which deals with beam distance, beam intensity, run time, light output, impact resistance and water penetration ratings. Surely standardized measurements will mean that data will be available which can be compared to give a fair like for like comparison, problem solved! However this standard is voluntary and because the tests are purely lab based the figures can be easily skewed. Before I talk about this further here is a quick note on the units commonly used when discussing light outputs:
The unit of luminous flux, this is the measure of the total light output by a light source in every direction, corrected to ignore wavelengths not perceived by the human eye. Basically the total amount of visual light generated by a source.
The unit of luminous intensity, or the light coming out of a source. Supersedes the now obsolete ‘Candlepower’ which measured luminance relative to the output of a candle of specific size and composition. As this is a measure of the intensity of light as it leaves the source it is not affected by distance.
The unit of illuminance, a measure of the intensity of light hitting a surface at a given distance. Equivalent to Watts per square meter but corrected according to the human perception. Basically how much a torch (or other source) can light up a 1m2 area at whatever distance. Usually quoted in the format ‘514 lux at 10m.’
So back to the standard, most manufacturers use four commonly quoted stats to describe the output of torches. Covered in the ANSI standard they are; beam distance, burntime, peak beam intensity and light output.
Beam Distance / throw
The ANSI / NEMA FL 1-2009 standard defines beam distance as the maximum distance that is projected by a torch producing a throw of a minimum of 0.25 lux (moonlight level,) measured in the brightest part of the beam.
This seems sensible, but consider; the 0.25 lux is partially absorbed and has to travel the whole distance back to your eyes (0.25 lux at a distance of 100m on a light-absorbing object such as a tree is far too little to observe anything in real life) so the figure is essentially meaningless! Other factors such as a light mist (which is nearly always present at night,) has an enormous influence on the distance of the beam, also the brightest part of the beam determines the specification which can be a very small spot in the beam.
In the ANSI / NEMA standard, the run time is measured using the batteries that are sold with the torch or those that are recommended by the manufacturer. The run time cutoff is defined as the moment when the light output of the torch is only 10% of the maximum light output. As this is a voluntary standard, several leading torch manufacturers choose to ignore this part and instead apply the ‘moonlight standard’ which defines the cutoff as an output of 0.25 Lux (moonlight intensity) at 2m. We feel this gives misleadingly burntime statistics.
Peak Beam Intensity
This measurement, defined as ‘the maximum luminous intensity of the source,’ is the intensity of light leaving the source at it’s brightest point. Basically how bright the brightest part of the beam is at it’s brightest point. The point in question can be extremely small so this does not take into account how the beam will spread or give an indication of the ‘usability’ of the light.
Lumens light output
The light output in lumens is a measurement for the total quantity of light produced by a torch, irrespective of how it is focused or where it is focussed. The standard prescribes how the light output must be measured but the light output is measured within the first 2 minutes with new batteries. We know torches fade as the batteries deplete, and there is no requirement to measure or state light output as this happens.
Two 120 lumen lights, one has a tight, narrow 'spot' beam, one wide and 'floody.'
While it is a step in the right direction these ‘standard’ measurements can be misleading to consumers. The individual numbers alone do not tell the whole story. An extreme way to think of this is would be to imagine a laser beam, massive lumen output, massive peak beam intensity, massive throw, but completely useless for walking around in the dark. Few if any give any information as to how the torch will perform in real life situations. Even if this voluntary standard is adopted and applied religiously across the industry the numbers it generates are at best irrelevant to use in outdoor pursuits and potentially very misleading.
So unconvinced by the ANSI / NEMA standards I broke out the cardboard and sticky backed plastic and Blue Petered up our own version of an integrating sphere based on the setup proposed by Quickbeam. The basic plan was to design an enclosed space with a window for a light meter which will allow a torch to be shone in, working on the assumption that as the light would be reflected around the inside of the box and the reading on the the light meter would give a good indication of the relative total output of the torches.
Lightbox mk.1 under construction & experimenting different with torch angles and positions
A few versions of the box and tweaks to the method were made before we came up with a method and setup that gave consistent results at around the expected levels. We were unsure whether it would be best to shine the torch straight down into the box and take that reading, or whether it was better to shine the torch in at various angles and take the maximum reading. After graphing both methods we discovered that holding the torch statically, pointing straight down gave the most consistent results and did not seem to be affected by how focused or ‘floody’ the beam was.
A similar process was needed to decide what to line the box with; the first version was lined with reflective silver vinyl, but this favored more focussed torches (a torch with variable focus was used to check this,) on the silver lined box a more focussed beam gave a far higher total output than the same torch on flood mode. A second box was lined with satin white vinyl, this seemed to remove this issue giving consistent results for the adjustable torches across the beam types.
The final setup;
A small box, lined with white satin vinyl inside and silver vinyl outside to prevent torch light escaping or ambient light entering the box
A light meter fixed to the box with a hole cut to allow the sensor to pick up light from inside
A baffle to prevent torches from shining directly onto the sensor.
A foam jig for each torch to hold them horizontal and with the lens level with the ‘roof’ of the box. The jig also prevented ambient light entering around the torch.
The Final Set-Up Used
Using the setup described above we spot tested a selection of torches, all using brand new Duracell Alkaline batteries to give us a multiplier like Quickbeam used. I then ran a test over time checking the light output every 1/2 hr until the Gamma was pretty much dead. Plotting the results of the first test I noticed that the maximum output dropped considerably during the first 30 minutes to an hour. I showed this to Jim and the rest of the team and we were all shocked, what was going on. We re-tested different Gamma and the same depletion happened. We repeated it again with a leading and highly respected outdoor torch manufacturers unit and the same happened. After noticing this reduction of output I adjusted the measurement intervals to record more accurately this early power drop off. Every five minutes for the first half hour and every half hour up to four hours. I then plotted a burntime graph of overall relative output against time for each torch I had tested. Again each test was conducted with brand new Duracell Alkaline batteries.
The results look shocking. The initial test of the torches we had in the warehouse (some of ours, some other manufacturers’ and some prototypes from our supplier) showed that the output of the torches drops at a dizzying rate, with the output of all but one dropping 25% or more in the first 30 minutes.
percentage of power lost over time
Although these results look shocking we would be confident to take any of them on to the hill. So what is going on here and what can we draw from these results. I think the first thing we can draw from this is that most of us need far less lumens than we think we need. LED torches have revolutionised outdoor lighting and this is still rapidly developing. There is no doubt the battery life of LED units is much greater than traditional bulb torches with all the torches giving an adequate light supply for most outdoor pursuits users for several hours. There is a definite benefit from running a torch on lower settings when ever possible as the relative percentage drop of light output is much less. Try to think of the high power setting as a boost function used for short periods.
We have a raft of heath robinson style tests we would like to complete which will be my first task of 2011. These, together with real world experiences should hopefully give us a better understanding of what we should expect from our lighting. I will publish all of the experiments I conduct so they may be repeated by anyone who choses to do so.