Lens Testing

You've mentioned in several podcasts and a few blogs that you test all your lenses to determine where they perform best. How do you go about doing your testing?

— Jim

   

The trick is to use a testing system that is consistent and comparable. You want any lens you test to result in data that is meaningful when compared to a different lens similarly tested. This is precisely why I don't rely exclusively on "real world" testing as some have suggested. Just photographing, say, a hillside of trees to see if a lens is "sharp" can't possibly give you comparable data to some other lens photographing a different hillside of different trees at a different time of year under different lighting conditions. Worth doing, but not an objective test.

For four decades now I've use the PLI (Photo Lab Index) test chart which, unfortunately has been out of print for years and is no longer available. Not to worry, there are others that are just as good. And, I'm including a link to a Photoshop reproduction of the test patch below. Here is what the chart looks like.

The trick with this chart — and I assume other charts are used similarly — is to photograph it from a precisely measured distance. This precise distance times the focal length of any lens allows direct comparison between lenses of different focal lengths that are meaningful. Even zoom lenses can be tested with comparable results.

The important target pattern in the chart is a series of lines that are pre-sized to be photographed at this specific distance — btw, exactly 26 times the actual focal length of the lens, not the digital 35mm equivalent BS which we see so often these days. Here is what it looks like.

You can download as a high-resolution PSD file of this test patch with this link (16 mb PSD file). At 26 times the focal length of your lens, the smallest lines are the equivalent of 56 lines per milllimeter, the next patch 40 lines per millimeter, and so on.

This basic pattern is reproduced, as you can see in the full chart above, in four colors (RGBK) in several locations near the center and corners. This well-thought-out design allows for a number of tests to be done with minimal effort. It tests for resolving power, spherical aberration, chromatic aberration, curvilinear distortion, curvature of field, lateral color, coma, and most importantly astigmatism.

Photograph this test pattern on film and look at the results with a magnifier. Or, photograph it with a digital camera and just look at the images in Photoshop. View cameras or 35mm, small-sensor digital cameras or full-frame digital cameras. They are all comparable as long as the test pattern is photographed at precisely 26 times the actual focal length of the lens.

Here is an image of the results of photographing this test pattern with a Fuji s602Z digital camera I owned a number of years back. I shot this using f/5.6. First, note that the resolution is good at 28 lines per millimeter, but non-existent at 56 lines per millimeter. At 40 lpm there is some odd mosaic aberration happening and some astigmatism (lines in one direction looking sharper than in the other direction). Lots of purple blooming can be seen, too. All in all, not very impressive, but the camera was good enough to use for some learning my way around the digital workflow and to play with. I longed for something better for serious use.

   

When I purchased my Sony DSC-1, I did the test as I had always done, but could not determine the resolution of the camera because it resolved the full 56 lines with ease! Yikes, it exceeded the chart!! Needless to say, I was excited. In order to find out how far this camera/lens could resolve, I doubled the distance and photographed it at 52 times the focal length of the lens. At this new distance, the smallest patch measures 112 lines per millimeter, the next patch would actually be 80 lpm, and so on. Here is the result of photographing at double the recommended distance, again using f/5.6.

I'd reach its resolving limit which appears to be where the chart reads "40" (barely) but is actually 80 because of the doubled distance. Notice also there is some astigmatism, and that odd mosaic pattern cropping up again beyond the "40." At least I eliminated the purple blooming by using camera RAW chromatic aberration adjustments. Very nice results and this camera was a joy to use. Superb resolution and tack sharp results.

But, things change. I was really torn about letting this camera go, but I wanted some of the new features that were available in the Panasonic G1. So, I purchased one, but held onto the Sony until I could compare the lenses.

Here is the same test with my current camera, the Panasonic G1, similar focal length, also at f/5.6. Again, the Panasonic resolves 80 lpm (notice I fixed the numbers on the chart), but at 112 lpm shows no resolution and some mosaic patterning. Using this objective testing, I determined that the Panasonic 14-45mm zoom lens was just as sharp as the Sony DSC-R1 with its superb Carl Zeiss lens I'd been so happy with. In fact, the Panasonic lens was even better. The Sony had a little corner fall off in resolution due to spherical aberration. The Panasonic has none. I sold the Sony and have a new standard bearer in the Panasonic G1. No speculation here; facts and test results.

You can see how this testing helps in the comparison of camera to camera, lens to lens. The Fuji s602Z could see 28 lines per millimeter whereas the Sony DSC-R1 and the Panasonic G1 (14-45mm zoom kit lens) can see 80 lpm. Wow, what an improvement.

Typically, with every lens I buy I run the full range of apertures to see how they perform. This allows me to compare each aperture to see which is the sharpest. I test the corners to see if there is fall off and how much. I check for chromatic aberration — which I never saw in film lenses, but routinely see in digital cameras due, I think, to the sensor as much as to the lens. In short, after testing I know precisely how each lens compares to others I've owned and tested and also know how best to use each lens to maximize its performance.

Just for comparison, here is an example of results from my old Fuji 105mm view camera lens photographed at f/5.6 onto AgfaPan AP100 film, developed in HC-110. To view this test result, I photographed the negative through the microscope with a digital camera and inverted the tones so it shows black lines against a white background. This is an accurate representation of the actual film results — 28 lines per millimeter resolving with ease, but no resolution at 40. Actual results would be somewhere in between, say 36 lines or so. Notice the visible grain in the image, too.

Clearly, the Panasonic G1 camera with its 14-45mm zoom lens will resolve a lot more detail than either the Fuji digital camera or the view camera lens I used back in my film days. Knowing this as objective results of testing gives me a lot of confidence using this new camera.

Well, back to the use of the PLI chart. Notice it has a black, blue, red, and green patch in the center. These are all repeated near the edges. This allows me to test for chromatic aberration (center) and spherical aberration (edges). That each set of patches has lines going in opposite directions allows me to test for astigmatism. The long lines around the outside of the chart allow for testing of curvilinear distortion. Of course, use a good, solid tripod and measure carefully.

Even though this chart is no longer available, I suppose you could make one using the PSD file I've provided. Print out multiple copies of this test patch in black, red, green, and blue (assuming you have a color printer) and photograph at the precise distance of your choice. (The PSD file has layers with the different numbers for 26X and 52X use.) It's a good start. Here is a PDF I've used in my workshops that has the basic instructions on what to test and how to do it as well as some background on the various aberrations.

Comments

[Douglas R Winn]

Hi Brooks:

Your read my mind....I was going to ask you about this very subject.

Before everything went digital in the camera world, I used Pentax 645 NII and Pentax 67 II cameras with a full batch of lenses for both systems. Now I have $15K of glass and gear that is worthless...except Pentax has the new K-7 body at 14.6 MP, shake resistant and sensor cleaning, and I can mount all of my glass on the camera [I actually did this with my first digital camera, the *istD, with very good results, but Pentax didn't move past 7 MP, so I moved on to Canon]. The K-7 is also very small and light, and being a little crippled, that is a good thing for me. So, I'm going to test my lenses using your method and see what comes out of it. Thanks.

[Svein]

Thanks for an interesting blog, both about Lens Testing and other posts.

I did a search on the net for lens test charts a while ago, and there are several do it yourself charts available for download.
Check Noman Koren: http://www.normankoren.com/Tutorials/MTF5.html
and Cornell:
http://www.graphics.cornell.edu/~westin/misc/res-chart.html

I've got one question, and that's about camera-to-chart distance. Photgraphinig a fairly small chart usually mean using a fairly short distance (compared to a landscape shot) if you want to get both center and corner in one shot. Have you done any testing to see if distance influence lens performance?

[Janne]

Of course, while the per-unit resolution of the LF lens is lower then the small-camera lens, the total image resolution is still much higher; the whole LF negative would have about 12 times more data (around 3.5 times the linear resolution) than the shot from the 4/3 camera. When and whether you'd need such high resolution images is another question of course.

It's worth keeping in mind, though, when comparing different formats.

[Michael Newsom]

Hey Brooks, Thanks for the testing idea. I am curious though about the test of the large format lens. When you photograph the negative using a digital camera and the microscope, are you not getting the resolution of the digital camera lens and the microscope combined? And is it not affected by reflections from the film. I think projecting a positive film image at the same size as the test pattern would be more accurate, Best wishes, Michael Newsom

[Theno23]

So, I don't quite follow how this methodology let's you compare lenses between formats.

It's clearly a great way to compare different lenses on (say) different 135 cameras, but if you compare your G1 and 5x4 at the same diffstance for the same focal length, then how do you compare resolution at the same LPM? The difference is that there's a lot more M in the 5x4.

[Brooks Jensen]

Michael,
Yes, by rephotographing the camera negative through the microscope I've introduced the capabilities of the microscopes lenses and the digital camera lenses, but these don't affect the results because the magnification is so larger. By focusing on the grain in the film, I am sure the image is sharp. By making it large enough, I know I am far beyond the resolution limits of the microscope lenses. The fact that we can see the grain gives us confidence that we could see the lines if they were resolved by the original camera lens.
Brooks

[Brooks Jensen]

Janne,
Yes, there is no doubt that the larger the negative, the more detail and information is captured. This leads us to the discussion of enlargement factor I've mentioned elsewhere. In short, the larger negative will make a larger print — which doesn't take much math to figure out! It's not the TOTAL resolution that I'm interested in with these tests, but rather the lens to lens comparison.
This also explains, btw, the issue I mentioned in http://technology.lenswork.com/2010/01/empty-magnification-part-2.html. The greater resolution of the digital camera means that I can make a relatively larger enlargement from it and still show detail without empty magnification.
All this supports one of the earliest lessons I learned, "Make the largest negative you must, but the smallest you need."
Brooks

[Brooks Jensen]

Theno23,
Here's the way I think of this. Imagine you and I are right next to each other, tripods ready, yours mounting a 4x5 and mine mounting a digital camera. We compose the same images of the same subject so we see the same thing on our viewfinders and make proper exposures. We now both have the same image to print for similar photographs. However, because your camera uses larger film, the image you've captured is larger than mine by a huge factor. That larger image will contain more detail even though your lpm might be lower. (You also had to use a much longer focal length lens to cover the film area, but remember our image compositions are comparable.) Because your "system" (long lens plus big film) captures much more detail, you'll be able to make a larger print with smoother tones than I can with my relatively smaller "system" (short lens plus small sensor) even though mine might mathematically capture more lpm. What counts in enlarging is the total detail available in the film/file. Make sense?
Where my camera makes sense is in making smaller prints. In, for example, an 8x10" print, our results might look very comparable because either camera will provide all the detail our eyes can see. As we increase the size of the enlargements, once we cross the threshold the larger negative will toast the smaller sensor. The same can be said when comparing a 4x5 to an 8x10" camera. There simply is no substitute for a larger negative when you need it. The question that always dictates the choice of equipment is How large of a print will you be making?
Brooks

[Bernard]

How do you ensure that the camera is parallel to the test target? This would be the most difficult part for me.

Bernard

[Brooks Jensen]

First, the longer the lens the bigger the room you'll need. I have the chart affixed to a flat piece of melamine board which I attach to the wall roughly waist high. I use a bubble level in the hot shoe of the camera to make sure the camera is pointing level on the X and Y axis. I then measure up from the floor to the center of the chart and adjust the tripod height so the center of the lens is at that same height. The final step is to adjust the distance from the center of the target to the camera so it is precisely the correct distance — either 26X or 52X the focal length of the lens. For consistency, I always measure to the iris diaphragm position in the lens rather than to the film plane.

I suspect that Ctein's comments are correct that even a few millimeters variance in getting all this rectilinear could mar the results on the edges. But, if you do the test at the recommended distances, you'll see that the chart is actually pretty small and in the center of the field of view. The instructions on the chart state that to test for spherical distortion, it is necessary to move the camera closer so the chart fills the frame. This makes the numbers meaningless, but the relative sharpness of the center spots compared to those on the outer corners is quite evident.

I like Ctein's suggestion to fix the focus and move the camera in and out by a cm or two to test absolute focusing. Neat suggestion.

[MartSharm@yahoo.com]

Brooks,

A very interesting test, although I'm not going to carry it out any time soon. I must say I think your astigmatism test (which you deem most important) could be a lot more sensitive, and in its present form can actually miss the presence of astigmatism. Let me explain.

You have parallel lines oriented (approximately) at 45 and 135 degrees on your test. If you have a lens with astigmatism close to the 90 or 180 degree meridian, this will give equal blur to both sets of lines, and the astigmatism will go undetected.

I would recommend increasing the number of line orientations. You have eight sets of the "mini-test" in each diagonal of the large test. Perhaps each mini-test could be rotated by 11.25 degrees from its neighbour, thus covering all possible astigmatism axes in the eight tests, allowing you to accurately detect astigmatism axis and magnitude, and prevent anything slipping through the net.

[Brooks Jensen]

I suppose you're right about adding additional angles for a more sophisticated analysis of astigmatism. Typically, I've found the test as is provides me enough data for using my lenses more intelligently for my needs — i.e., which apertures to avoid and when. Perhaps I could tweak another half a stop or so by refining the test, but in all honesty it tries my patience as it is. I'm always itching to get the tests done so I can go out photographing! Shooting test charts is not my idea of a good time, albeit a necessary evil to gain the necessary knowledge about how to use a lens to its best abilities.

[Bernard]

YOu do ensure that the camera is positioned at the correct distance and height, and the correct vertical orientation. But how do you ensure that it is not slightly deviating to the left or right? This would be crucial to comparing performance on the sides (decentered lens).

Bernard

[Brooks Jensen]

Oops. Sorry. I forgot this part. Very crucial. Use geometry. I measure a spot equidistant on the wall left and right of the center of the target. Then I measure from the left spot diagonally to the camera. Then measure from the right spot diagonally to the camera. These hypotenuse (Gesundheit!) measurements should be equal if the camera is directly in front of the chart and not shifted a little left or right of dead center. Make sense?

[Bernard]

Yes it does, thanks. But I was not clear... what I meant was getting the sensor (or film) plane parallel to the target (or the target wall).

[Brooks Jensen]

Well, if the wall is straight (think bubble level here) and the bubble level on the camera is level, wouldn't the film plane be parallel to the chart? Assuming the film plane is perfectly perpendicular to the earth.

And as far as left and right, if the hypotenuse method I mentioned above is use, then simply point the camera directly at the chart so it's centered in the field of view and I would think this will make the film plane parallel to the chart on the that axis. I'd hope!

[Ken]

There is a fairly trivial way to ensure the target is perfectly parallel to the sensor plane. The procedure is very simple and is as follows:

1. Set the camera and target at the appropriate distances. To make the adjustments easy get the camera and the target at the same height and level the camera. The positioning doesn't need to be precise (that's the next step) but the better you get it now the easier the adjustments will be.

2. Place a small flat mirror on the center of the target.

3. Adjust the target such that the center of the target (where the mirror now is) is in the exact center of the viewfinder and through the viewfinder you see the reflection of the center of the lens in the center of the mirror.

4. Your done, fire away.

This optical orientation is far more accurate than anything involving tape measures and bubble levels.