Pretty much forever, photographers have been arguing over numbers.
Back in the film days, we argued about granularity (Velvia was a 9 while Ektachrome was an 11). With printing of images, we continue to argue about Dmax (print density; e.g. most matte prints only get to about 1.7, glossy tends to be about 2). In the early days of digital on the Internet, the big number was the standard deviation of mid-tone gray (higher is noisier). These days, we have DxOMark, Imatest, and other test suites providing lots of numbers to argue about.
I've written this before, but it seems as if it needs repeating and amplification: I'm a fan of testing the gear that comes into my office. Almost everything gets a basic Imatest suite of tests thrown at it. What I'm not a fan of is arguing that any one number, or even groups of numbers, makes a product better than another.
Numerology has a long history in evaluating consumer products. Back in the high fidelity audio days everyone measured frequency range, power, distortion, and a host of other supposedly critical numbers. Notice how we don't do that anymore? In the CD and MP3 eras we moved on. In many ways CD and MP3 tracks test worse than LPs and tape did. With MP3s in particular, there's compression at work, and a sophisticated compression that tries to hide data that isn't generally audible so as to promote the data that you can hear. An MP3 output from most players would test terrible against my old reel-to-reel tape gear. But it doesn't matter. Those numbers don't matter much any more.
Or maybe automobiles? How fast does the car go from 0 to 60? How many feet does it take to stop it? How many G's does it manage in circles you and I would never attempt? Automobile magazines and reviews are filled with numbers. But frankly, the numbers you probably mostly base your purchase upon are price and miles per gallon, which represents an ongoing price.
Here's the thing: early in a technology's history, the numbers often do describe critical differences between products. Nascent technology starts to mature and gets better as it does. At some point, generally when a product category has become completely mainstream and ubiquitous and stays so, the numbers start to matter less from a practical standpoint. All the products do 90% or more of what you need and the actual measured differences between products becomes smaller and less useful in making decisions about them. Price starts to become more of a factor than performance as products move towards commodity.
That doesn't stop marketing departments from trumpeting numbers, though (6! 12!,16! 24! 36!). Nor does it stop the truly serious from testing and announcing other numbers (80! 83! 89! 94! 95!). Obviously the 16mp camera that scores a DxOMark of 80 must be worse than the 36mp camera that scored a 95, right?
Photo taken with a crappy 6/50 camera ;~)
Sure, just like 420hp 0-60 in 5.5 seconds is better than 273hp 0 to 60 in 7.6 seconds (Porsche Cayenne versus Mazda CX-9). Of course, one costs about US$100,000 more than the other (as tested). Can you guess which one?
Here's the thing that people don't ask: does the number in question make any practical difference to my needs? Or better still: for 99% (or some other well considered percentage) of my work what's the number I actually need?
Marketing departments never want us to ask that question. Bigger is always better, higher is always better, or sometimes lower is always better, depending upon the number in question. Always. You'd be a fool to buy something whose number isn't "the best." I pity the fool who won't buy the best.
Which brings me to why I measure my equipment: I want to know what it can actually do, where its limits are, where I can ignore differences and where I have to pay attention to differences. For example, on that last one, after some careful measuring I determined that my D3s could be shot at base ISO to ISO 800 without anything changing in my overall perception of the images or my processing of them. It didn't matter if I shot at ISO 200, 400, or 800. For my work, the results were as close to the same as to make no difference to me. At the far end, the question is where the results started to severely change how I had to deal with both the image taking and the image processing. As it turns out: up to ISO 800 I didn't care, from ISO 800 to ISO 6400 I would tend to try for a lower ISO but wouldn't be afraid to use a higher one if it was necessary, above 6400 I had to really need the shot because of the compromises that started to come into play.
Great. Most of my work is at ISO 3200 or less, so the D3s was not a problematic camera for me in any sense. I could ignore the numbers until they got above 800, pay more attention to them progressively until they hit 6400, and then stop and seriously consider what I was doing and why.
Pros work a little differently than amateurs. First, we get paid more for higher quality and more creativity. A camera that is not restrictive to either of those things is more enticing than a camera that puts some restrictions into place. Second, we have to stay ahead of the masses. If we're selling gallery prints or stock, larger allows us to stand out a bit more from Joe and Jill Weekender. Third, we tend to have to work faster than you think. Most photographers actually spend very little of their time photographing and post processing, as we're too busy with the marketing, sales, and promotion aspects of our businesses. Thus, anything that causes us to have to work more diligently or changes or workflow towards slower is a real issue. It used to be that we needed lots of massive, time-consuming lights for studio shooting. Now, not so much. I've moved mostly to continuous LED lighting (partly because I need it for video shoots) and my cameras don't need as much light as before to get noise-free results.
I would argue that most amateurs can often get around certain numbers and limitations because they're not under the constraints that pros are. I'll give you an example. Someone recently sent me an example of the "horrendous" banding in the D7100 images when you underexpose significantly and then bring deep shadow exposures up by large amounts. First, that's not likely something a pro would do: they'd be exposing right and lighting the shadows well if they had any option to do so at all. But that's besides the point. The point was, yes, there was very visible banding with color noise in the sample. Curious, I brought it into Photoshop, ran Topaz DeNoise, adjusted the parameters, especially the Debanding control, and found that I could remove all of the banding. I did have some vague blotchiness to the darkest colors, but I suspect I could have dealt with that, too. Time heals all wounds. To "solve" Nikon's supposed banding problem I just had to spend more time working with the image.
In other words, who cares what the actual "number" for noise was in the D7100 image? I could make it "better." Indeed, I could make it much, much better. And of course, this was at the pixel level, which isn't the way I'd be outputting the image. At 300 dpi my fixes were invisible to the eye. In an 600x800 Web version, ditto. So why are we obsessing about how many stops we can recover shadow information from without visible noise? It's a non-number. A number that has little overall implication about whether you should buy the camera or not.
I'd argue--indeed I've been arguing for some time with many people who email with the "what camera should I buy" question--that how you respond to the controls of the camera may be more important now than any number you can cite. A picture is a precise moment (or moments) in time. If you miss that moment, you get a lesser picture. If you're fiddling with controls trying to change something, you'll miss the moment. If you can't see well through the viewfinder, you can't frame the shot. If you can't hold the camera steady, you've got even bigger problems. If the focus system fights you, you won't get the picture. If the camera's so precious or so big that you always carry it in a pack on your back, I hope you're not thinking that you're going to capture spontaneous images.
Even pros think this way. One reason why we like the big body designs is that they've been well-designed to our hands and shooting needs. We have dedicated buttons for everything (and in the Nikon pro world, you theoretically can change most critical controls without ever taking your eye from the viewfinder or your index finger off the shutter release). The big batteries last forever and the dual card slots can hold a ton of images, so we're not constantly having to stop to change things; we just shoot and shoot and shoot. I even go so far as to try to always have my main and backup body the same. This was great in the D3s/D3x era: I could move from camera to camera without ever having to think about the camera itself (other than ISO, basically). It's not so great in the D4/D800E era, when the cameras just aren't the same and I do have to think about which one's in my hands. Even if it were the same 36mp sensor, I want a D4x. Likewise, even if it were the same 16mp D4 sensor, I want a D800h.
But let's back up a bit and hit numbers from another angle. Let's talk lenses for a moment. The general question I get is "how sharp is it?" Okay, you want an MTF number, is that it? Good luck. Which camera? What aperture? What focal length? What target? What target distance?
Oh, so we get a range of numbers when we use something like Imatest to check out our lenses. Interesting. Let me ask you this: what's more important, some single number or knowledge of the full range as it applies to your camera?
There's some voodoo underneath the skin here we need to talk about. One thing I'm having increasing problems with and one reason why I haven't reviewed any F-mount lenses lately, is that I'm finding that I'm getting variable results. My favorite example of this is the 20mm f/2.8. It tested very poorly on early DX cameras. It still tests less than well on current DX cameras, but it tests better, all else equalized, on some current DX cameras than the older ones. But it tests better still on some FX cameras. What?
With lenses things are compacted by a second optical system. What system is that? A glass surface transition followed by UVIR filtration, followed by horizontal displacement (AA), followed by vertical displacement, followed by another flat glass surface transition, followed by air space, followed by microlenses, followed by Bayer filtration, sometimes followed by a photosite tunnel. Where's the focus plane in that concoction? And what happens to light after it passes that point? Turns out, it makes small, but sometimes tangible differences.
That's why I turn the lens question completely around these days. Given that a lens is reasonably competent at what it purports to do (e.g. has a respectable level of sharpness and a minimal number of artifacts or distortions), I choose a lens on how I need to use it. For example, while the 24-70mm f/2.8 tests a bit better than the 24-120mm f/4 at a number of things, the real question is how I'm using the lens. If I need true versatility and may be shooting a number of shots at f/8 or f/11 on my D800, then the Nikon-supplied MTF numbers aren't totally relevant. I'm into the diffraction zone at those apertures, and 70mm cropped is never going to be as good as 120mm not cropped. On the other hand, if I'm in that low-light gym shooting sports, f/2.8 is the only number I really need to pay attention to (indeed, I'm balancing the f/2.8 of the zoom against the f/1.4 of some primes and deciding that, again, versatility wins over "the number").
But don't get me wrong. I've tested my gear so that I know what to expect from any combination I'm using. I know what the "best aperture" is on each of my lenses on each of my cameras because I tested that. Indeed, I try to do something that none of the published lens tests on the Internet ever seem to do: test at close, medium, and far distances. Most of the test results you see on the Web are at what I'd call medium close distances. That might be right for event shooters, it might not be right for wildlife shooters. (Aside: do you know how big the test target is for 14mm at "distance" ? ;~).
Thus, numbers do come into play in my choices when shooting, but there's never a single test number that I'd ever consider in buying equipment. Okay, maybe that's too strongly stated: I suppose if I saw a test that said the MTF of lens A was 300 and the MTF of lens B was 800 at a critical aperture/focal length, I'd fairly quickly start dismissing lens A. (Warning: those numbers might be on a camera that has a top MTF capability of 800 or one that tops out at 2000 or 4000. Does that make a difference in what you think?)
Amusingly, no sooner do we get all excited about something and a trend comes along that destroys that. Lenses have to be sharp, right? Why did the LensBaby become popular, then? Or using a shift/tilt lens to isolate focus and blur the rest of the scene? Colors have to be accurate, right? Then what are all those Instagram effects all about?
Ultimately, the real issue is whether a photograph communicates to and resonates with a viewer, not how your gear measures on some arbitrary tests. Tests help you understand what you gear can and can't do, and help you optimize your shooting choices, but tests are not photographs. Moreover, there's not a single number, or even a small set of numbers, that can fully inform a buying decision, IMHO. Not even close. So why everyone is so obsessed with obtaining these magic numbers that have no clear meaning, I don't know.
He who dies with the highest MTF wins. Sounds sort of lame, doesn't it? (Why do I get the feeling I'm going to see a t-shirt with that on it soon? ;~) She who makes a great photograph that impresses everyone who sees it is the real winner. No number can express that.
