Excellent question for which there isn't a good answer. Funny thing is, older Nikon lenses do seem to have very good tripod collars. It's only been in recent times that we've seen slop in the connection between the collar and the lens that impacts results. Perhaps Nikon thinks that having VR on most of these lenses now obviates the need for a tight, locking collar. If they think that, they are wrong. Very wrong. The final answer is this: they can make a stable tripod collar, they just aren't.
For some lenses, you can find collars made by third parties that are better. Two sites to look at for better collars are Really Right Stuff and Kirk Enterprises. They aren't produced for all lenses (partly because it requires that the lens have a removable collar in the first place), but in my experience the third party solutions are always better when they exist for a lens.
Update: the most recent Nikkors—e.g. the 200-500mm f/5.6E and 70-200mm f/2.8E FL—seem to be going back the other direction and getting slightly better tripod collars than we had for the decade at the start of the century.