You'll notice a few things about my lens reviews, in particular the lack of measurement numbers or test chart samples.
Testing lenses has always been tough. These days it's compounded by a wide range of new problems caused by constant iteration of cameras and tweaking of the package on top of the sensor itself.
A common chart used by Imatest to test lenses
I've always believed in using test suites on both cameras and lenses. That tells me a lot about how the products function, but it's generally not enough to say how they perform in photographic situations. That's one of the reasons why I don't publish numbers from my testing. The other reason is that the numbers often can't be compared, as a lot of folk using Internet sources are trying to do.
What do I mean by that?
Well, take Imatest, which a lot of us use for a fairly broad suite of excellent tests. The same lens tested on a D70, D700, and D7000 will produce different numbers. Indeed, sometimes you see differences between something as simple as the change in image sensor filtration of the D800 and D800E. No sooner do you test with 16mp sensors than 24mp sensors come along. Then comes 36mp. Then 45mp. Do you continue testing only with 16mp sensors and reporting only those numbers? Do you go back and retest everything again with the new sensor?
I should also note that Imatest's designer, Norman Koren, has written a huge amount of documentation on how to interpret test numbers and what the various things Imatest does mean. See the Image Quality section under About to get to just some of what he's written. Also see Norman's own site. Prepare to be overwhelmed.
Good sites that report numbers tell you what they tested with. That way you can at least try to compare apples with apples. But the problem just keeps compounding with new digital gear. In DX we went from 2.5 to 3 to 6 to 12 to 16 to 24mp in just 12 years with Nikon bodies. In FX we went from 12 to 16 to 24 to 36mp to 45mp in ten years.
Tests that reported numbers a few years ago are not particularly helpful today. For a site like this one, which tries to keep lens information and reviews available over long periods of time, number reporting becomes a very clear problem that, to do even close to right, would require me to be constantly doing full retests of everything every time a new camera came out, and then publishing dense tables of data for a broad range of gear (50mm on D300, 50mm on D7000, 50mm on D7100, 50mm on D500, etc.). Maybe if you paid me more money for access to this site I'd be able to hire a staff to do that. Doh! The site's free! Dagnabit.
Instead, I've opted to report my findings more anecdotally, though I've tried to keep my standards consistent. If I write "excellent in the center" in 2014 it should mean the same thing in 2017 and the same thing as in 2010. Of course, there's a problem, isn't there? I have to periodically go back through the lens reviews in order to update them to current cameras.
A good case in point for this is the 18-200mm DX lens. Back when it came out in 2005, we had 4mp and 6mp DX bodies as state of the art. The lens seemed to perform quite well on those cameras, and I pronounced it a very good lens. When the D300 came out, I went back and retested it. Okay, maybe it's just a good lens. When the 16mp and 24mp cameras came out, I tested it once again and found its acuity to be no better than average for a super zoom.
Not all of the lens reviews that I've done in the past 20 years have been fully vetted for current cameras yet, which is why they don't appear on this site (I've removed them from the old bythom.com site where they first appeared). As I get time to completely redo an old, now deprecated review to current standards, I bring it over to this site. Also, I'm committing to revisit all the reviews on this site and update them if necessary every time we get a substantive change at the sensor.
I do provide some numbers where relevant, by the way. For example, linear distortion and vignetting don't tend to be sensor-impacted (though vignetting can be as we move a lens between different mounts with differing flange distances). But MTF and sharpness numbers? No way. The bar keeps moving so you can't really tend to relate one reported value to another.
Okay, so why don't I show example images of test charts any more?
Same answer, but with a kicker: do you actually use your camera to shoot test charts at a fixed distance?
You see, test charts are flat. The things you photograph aren't. While some macro and prime lenses are corrected to render flat objects flat, in general that's not the case with most lenses. More often than not they have small amounts (and sometimes large amounts) of field curvature in their optical designs. If you shoot a flat test chart with a lens not corrected for flat fields, you get variable results from center to corner. (That's true under all cases. If you focus in the center and check the corner, field curvature is the variable. If you focus in the corner and check the corner, how the lens elements are aligned at the focus position is different than they would be for center focus, another variable.)
Worse still, test charts tend to be photographed only at one distance, and in general I find that to be a fairly close distance (particularly with wide angle lenses, otherwise you'd need enormous charts ;~). You really need to examine a lens thoroughly at its entire range of distances. Often I find that a lens might perform really well at distance, not so well close up, or vice versa.
Is this lens sharp? USAF test chart from center of lens (left) to corner (right).
Here's the detail of center (left) and corner (right). Still not sure? Look at the innermost set of the three line spiral. On the left, even with JPEG compression and image size reduction you should see that the lines still look like lines further down the spiral than for the sample on the right. But how far should the corner hold up to the center? This is the 70-200mm f/2.8 (version II) at 105mm and f/16, by the way. Does that change your thinking?
Maybe the results at f/2.8 will change your opinion. Oh, wait, we're still looking at scaled results (technically, this site uses an image processor that rescales images to your display size so that it works on phones as well as computers, but that's another story), so let's just look at the corner at f/2.8:
Okay, you should now see there's some blur (and color, indicative of chromatic aberration issues). But we're also looking at a 211k segment of a 24mp image. On my display this little patch is now 3x the size it would print at (300 dpi). Again, this site is set to resize to your device, which means that any JPEG images on this site may be resized without you knowing it, and that’ll impact whether you think something is sharp or not ;~).
More importantly, and as I hope that little example helped show, another thing I'm finding more and more as we get into higher and higher resolutions is that most people aren't well enough trained to actually see the things that are visible in any images, not able to interpret what they do see correctly, plus the JPEG compression we use on the Internet to keep pages from taking forever to load also tends to get in the way of seeing anything useful at the granular level we're now up to with high-density image sensors. Heck, even how I (or you) would process a raw file can mask important things, too. For instance, did I sharpen those examples I just showed? (No, though I moved the white and black points, which increases contrast.)
One additional thing that comes into play is sample variation. One good rule of thumb is that the more expensive the lens the less likely there's sample variation that is significant. But no two lenses are going to be perfectly the same. Where I have doubts about a lens, I'll examine a second or even third sample (I always state how many samples I’ve used in the review). I also tend to see a lot of the main Nikon-related lenses over and over at workshops, so with many lenses, such as the 70-200mm f/2.8 or 200-400mm f/4, I've developed a pretty good sense of how the lens should perform. Yet many tests where you see numbers or charts posted on the Internet are from single sample tests on flat charts.
I would tend to say that Zeiss lenses have very little sample variation. I can't recall the last time I saw a Zeiss lens that differed from my test sample in any significant way. The same is mostly true of Nikkors, though at the low cost end of the spectrum (e.g. kit lenses and very low cost lenses) I've seen some sample variation. Low price lenses from third parties have shown up with lots of sample variation; in the two decades I've been reviewing Nikon-related gear, I've had to test two or three samples most often with the under US$300 third party lenses.
I've mentioned before that I know someone in NYC that is stunningly good at recognizing the lens that was used by just looking at an image (both in its entirety and at the pixel level). He's edited so many files from so many pros and he's trained himself to recognize certain clues. I tried to get him to teach me to do the same thing and it only partially took. If you give me a choice of a few lenses the sample could be from, I can sometimes identify it properly (more often than chance would suggest). But he can do it 9 times out of 10 without even knowing which camera was used or even what aperture (he'll supply that information, too ;~). It's uncanny.
But my point is this: we don't have a strong program of training photographers what to look for (though it wouldn't hurt you to try to make it through Norman's articles). It would take me a book just to describe all the things that I look when evaluating a lens, and how I go about distinguishing them. I'm not infallible, but I believe I'm pretty good, mostly due to lots of practice and some very good teachers along the way. Over time, I hope I can share some more of my knowledge with you so that you can get better, too.
Where there's something clear that I can show via a JPEG on the Internet, I will do so. But if you're expecting sample JPEGs that you can do A B comparisons with, you're not going to get them here. If you really think that's a useful thing, and you think that a static, one distance, flat subject shot is going to tell you something useful, there are other places on the Internet to see that.
This is a long-winded way of saying "believe what I write, not what an Internet JPEG shows."
Finally, I keep my ratings for a large group of "current" lenses on the main review page. I put them there rather than on the individual reviews for two reasons: first, it's easier for me to go in and maintain as the state-of-the-art changes; and second, it's easier for you to compare what I think about more than one lens simultaneously.
