I’ve been reviewing products for almost 50 years now (digital cameras for 25). I’ve written review guidelines for multiple national publications, I’ve helped train many, many reviewers.
One problem with reviews is remaining consistent in a changing environment.
Let’s start with something I don’t do: assign numbers (or even report many from the extensive tests I do in the studio with test charts and scenes). Numbers tend to be meaningless, and they tend to be overprecise.
For instance, one large camera review Web site uses ratings from 0 (I think) to 100%. Looking at most of their recent reviews, most products seem to be getting an overall rating in the 80’s, with a few sneaking into the 90’s and a few dropping into the 70’s. Has a camera gotten a 100%? Could it? Has any camera gotten a 30%? Could it? And what’s this a percentage of?
Statisticians argue about how much differentiation most people can usefully make, because it is important for accuracy in surveys when you’re trying to ascertain how much someone does/doesn’t agree/disagree with something. Many believe you can only discriminate between a maximum of five levels, most of the rest would say that you can discriminate with ratings from 1 to 7. So an 83 out of a 100 is meaningless discrimination. It gets very difficult to be consistent when you rate one thing 83 and another 84.
Yet review after review, in both magazines and the Internet, seems to try to reduce everything to one large number that implies really good ability to discriminate between very small things.
You’ll note that I have four basic “ratings” in my reviews: (1) no recommendation; (2) recommended (with reservations); (3) recommended; and (4) highly recommended. I’m pretty sure that my highly recommended products are clearly better than my recommended ones, and so on. Within one of those four ratings, you need to pay attention to your needs and biases, not mine.
I also use star ratings on some of my pages, from 1 to 5. This rating system is a little different than you’d expect: 3 stars means “as you’d expect”, or average. Fewer stars means “less than you’d expect” and more stars means “more than you’d expect. This is based upon price/positioning of the product. It’s tough for a camera like the D6 to get 5 stars in performance, for instance, because you’re expecting high performance. The Canon 1DX Mark III should get more stars than the Nikon D6 because of the Canon’s infinite buffer and faster frame rates, for instance.
But even with just those four things I conclude a review with and the five star rating system I use, things change with time. A camera that I might say is recommended in 2012 and got 4 stars then may not get my recommendation today (though it might, since cameras have been quite good for some time now). And it might not deserve 4 stars any more due to expectations at that price point and product position having changed. Thus, I’ve added a set of years during which my recommendation applies (e.g. Recommended (2019, 2020)). I’m in the process of going through and updating my recommendations and ratings on both sansmirror and dslrbodies this month. I have to do that every year, basically, as what would be highly recommended today is different than what I highly recommended five years ago (or even last year).
Meanwhile, you’re staring at an 88% from site X. Oh, did I mention that they then add a gold, silver, or bronze “rating”? And that a product with a lesser number can get a higher metal rating? What the heck is that all about, and how can you be even close to consistent with their system? But more importantly, this constantly confuses the readers of that site and provokes the question “Why did the X only get a Y?”
Likewise, I try to use a set of specific words in my lens optical descriptions: poor, fair, good, very good, excellent, superb. That last word has changed from time to time—I’ve sometimes used “exceptional”, for example—but it has always meant the same thing: a lens optical trait that is clearly in the top league and probably state-of-the-art. “Poor” means just that: a real problem, and a problem I’d likely recommend everyone avoid. “Fair” and “Good” are not necessarily problems, but you need to be aware that there are issues that are (often) visible. The last three words indicate some level of excellence, something generally above average expectations. “Very good” is at the bottom of that, while “Superb” is at the top.
With lenses, too, I’ve had to go back from time to time and adjust my wording. The recent RF, Z, and FE lenses I’ve used are breaking through barriers and providing performance in one or more aspects unlike anything we’ve seen before. What I thought was an superb corner performance for a 20mm focal length in the past—and again, that would be way above average expected performance—is no longer true. (Foreshadowing: that Sony 20mm f/1.8G is really, really good.)
One of the reasons why I haven’t reposted some old reviews (e.g. Nikon 70-200mm f/2.8G I and II), is that I have to do a lot of adjusting for current standards.
I haven’t done this word fixing or reposting for lens reviews yet, but I intend to do so sometime this summer.
