Tuesday, September 18, 2018

Towards a computer controlled macro/microscopic imager system

Note: this is a work in progress...





I need a digital system for macro/micro photography and video that can be fully computer controlled. This robotic microscope will be integrated in an instrument that will observe a great number of sand particles automatically, unsupervised, day in, day out.

Requirements:
  • high-resolution RGB still-image output 
  • live video output, minimally HD 1920x1080, ideally UHD 3840x2160
  • field of view of a few millimeters
  • high optical and mechanical quality
  • precise focus adjustment, for auto-focus and stacking purposes
  • computer control of all available camera and processing parameters
  • reliable 24/7 operation
  • compatible with Linux and OS X operating systems

It doesn't look like such a camera is available on the market (yet)! The bulk of today's digital micro photography is done with regular microscopes that are equipped with a camera sensor inserted into either a trinocular or the eyepiece tube. Sometimes a DSLR camera is with a suitable adapter is mounted and the image of the microscope lens is projected directly onto the sensor. Some use a compact camera to take pictures through the eyepiece, with or without an adapter. 

!   D o   I t   Y o u r s e l f   !


What would be more appropriate and potentially satisfying than designing and making a custom macro/microscope, using a mix of readily available quality parts and some of our very own fabrications? If you already have some usable parts, it is worthwhile to experiment with them and compare the results of different setups. 

Ingredients:
  • high quality objective
  • camera or sensor module
  • (perhaps) eyepiece
  • (perhaps) reduction lens
  • (perhaps) tube lens
  • chassis
  • motorized focusing mechanism

- -  the objective

There are two possible sources for high quality macro/micro lenses: microscopes and enlargers. Enlarger objectives are usually designed to work with relatively large fields of view, for instance 36x24 or 60x60 mm, so they would be naturally suited when working with large image sensors. Microscope objectives are designed for smaller fields of view, and thus more suitable for smaller sensors, in theory. 

There are two types of microscope objectives: those with finite and those with infinite optics.
Finite optics: the objective projects a real image at a finite distance, usually 160 mm. Such objectives can be used to form an image directly onto the sensor. By varying the distance to the object, the distance to the projected image, and thus its size and magnification varies as well.
Infinite optics: the objective projects a virtual image at an infinite distance. A secondary lens, usually 200mm, is needed to transform the virtual image into a real image that can be projected onto a sensor. The secondary lens is usually called the 'tube lens'. Varying the focal length of the tube lens varies the size of the projected image and its magnification. One can insert filters, polarisers, irises and other devices between an infinite objective and the tube lens, where the rays are parallel, without introducing aberrations.

Nikon and Mitutoyo microscope objectives are designed to correct all aberrations in the objective itself. This makes them especially suitable for setups where the image is projected directly onto the sensor.
The objectives of Olympus, Leitz, Zeiss are designed to correct the aberrations in cooperation with the eyepiece and/or tube lens. Using such an objective on its own will lead to sub-optimal aberration correction.   

In the domain of macro and microphotography we are dealing with very shallow depths of field. But even if we observe a flat object, which should theoretically be in focus everywhere, we still see the focus distance vary, it is different in the centre than at the edges of the view. This phenomenon is called field curvature. A flat object is deformed into a dish like shape. Lenses that are designed to exhibit minimal field curvature usually have 'p' or 'plan' in their name, meaning planar.


Nikon Plan Apo 4/0.16 160/-

The good old Nikon Plan Apo 4/0.16 is known to deliver excellent image quality.  Sourced on eBay for €310. Resolution, sharpness and contrast of this apochromatic objective are outstanding. The field curvature is minimal, the loss of sharpness near the edges is modest. This microscope objective has finite optics, and is designed to work optimally with a 160 mm tube. The mounting thread is standard 20.2 mm RMS (Royal Microscope Society).
Experimentation with this objective shows that the working distance is short, depending on the magnification that is desired, but usually below 10 mm. This is not a problem with flat backlit objects, it might be an issue when observing larger objects with lighting from above.




Nikon Plan 4x/0.10 ∞/-

The modern Nikon Plan 4x/0.10 ∞/- microscope objective is not a very high-end part but I sourced it anyway on eBay for €90 to use for experiments with an infinity optics objective. It is designed to be used with a tube lens with a focal distance of 200 mm. The mounting thread is 25 mm, which is significantly wider than the old standard 20.2 mm RMS thread, making this objective feel very sturdy.
Experimentation with this objective reveals that a tube lens is not necessary to project a real image with these optics, even though it has the 'infinity' label. The working distance is large, still over 50 mm at the higher magnifications, but the effective magnification is lower.





Rodenstock Rodagon 1:4 f=60mm

The Rodenstock Rodagon 1:4 f=60mm is an objective for enlargers. The magnification range is 4-22x (10x optimum). I salvaged this one out of a 35mm slide duplicator. Enlarger objectives are designed to exhibit minimal field curvature and offer good sharpness up to the edges of the view, both highly desirable qualities for any imager system.  This objective is said to be of superior quality.  



- -  the sensor module

There exist a great many sensors for microscope photography, often called eyepiece cameras. Most of them have a C-mount, and are coupled with the microscope using an adapter. Some are inserted into the tube instead of the eyepiece, so that the microscope objective projects its image onto the sensor. Others are mounted on top of a trinocular. It is common to use a reduction lens which makes the projected image smaller, so that small sensors are used more efficiently.  Maximal resolutions are modest, usually up to 8 MP. 
Many of these devices look exactly the same, except for the sticker. I suspect they all come from the same factory in China somewhere. The high-end microscope brands like Olympus, Nikon, Leica and Zeiss make their own sensor modules which are usually very expensive. Almost all these digital microscope imaging sensors work with proprietary protocols and if they come with an application, it is usually for Microsoft Windows only. This is of little use when you require full control over the device in your own application.
This led me to research other options: minimal but potentially more flexible solutions that are standards compliant and implemented with current technology, at reasonable prices.




ELP-SUSB1080P01

ELP-SUSB1080P01

The ELP-SUSB1080P01 is a USB 3 camera module for €50. It is equipped with a 2 MP Sony IMX291 sensor, size 1/2.8" (5.63x3.17 mm). Available resolutions are 1920x1080, 1280x720 and 640x480. Note: the lower resolution outputs are cropped from the native full HD frame.
The device supports the USB UVC (USB Video Class) protocol, and is thus compatible with all operating systems. No proprietary drivers needed, good!
The module is available with a variety of lenses. Mine is equipped with an extreme wide-angle lens. The nice thing about this module is that the lens can be screwed out completely,  so that it can be used as a sensor-only device as well. It should be possible to mount other lenses, given a suitable adapter. The screw thread is 12 mm,  fine pitch. The collar into which the lens is screwed is cylindrical, so it will be easy to clamp-fit a simple adapter of my own making. Ideal for experimentation!
Bad news: the frame rate can become irregular and slow down a lot when moving the camera around. It is unclear why this happens. Maybe a flaw in the connector? When the module rests somewhere stable, the frame rate is consistently high, even when the image it sees is changing rapidly.
After some experimentation with this camera module i concluded  that a resolution of 1920x1080 is going to be insufficient for serious macro/microscopy work. 


See3Cam_CU135

E-Con systems See3CAM_CU135

The See3CAM_CU135 is a USB 3 camera module for €150 (lens excluded). It is equipped with a 13 MP Aptina AR1335 sensor, size 1/3" (4.8x3.6 mm). Available resolutions are 4208x3120, 4096x2160, 3840x2160, 1920x1440, 1920x1080, 1280x960, 1280x720, 640x480. Output formats are MJPEG or UYVY (uncompressed).  The device supports the USB UVC protocol and uses some extensions for a few hardware specific features. Documentation and an open source viewer implementation is provided, so accessing the extensions should be doable.
As with the ELP-SUSB1080P01, the lens can be screwed out completely, revealing the bare sensor. This is a highly interesting device, but i expect to see a lot of noise because the sensor is so tiny. However, I did see some review videos,  and was quite impressed by the output quality. Definitely worth experimenting with, so i got myself one. First impressions: the noise level is quite ok, especially after turning the sharpness control to zero. No frame rate slowdown when moving the module wildly or even when shaking it. But... when looking at the 4K images at 100% zoom, there is no truly impressive sharpness anywhere. This might be because of the mediocre optical quality of the 12mm lens that i tried. The best output quality is when the 3840x2160 frames are down-sampled to 1920x1080.


- -  the camera

A regular consumer camera is probably the easiest and most cost efficient way to acquire an integrated package with good quality optics, sensors, image processors, and what not. I suspect it will be hard, if not impossible to find modules with sensors as large as are common in regular digital cameras. Large sensors are less susceptible to noise, and thus better in low light conditions, which will be prevalent in microscopy. 

Features of the ideal camera:
  • compact
  • affordable (€500 max for this project)
  • removable lens, so it can be used in 'sensor only' mode
  • powered and controlled over USB (preferably USB 3 SuperSpeed)
  • real-time 4K video 
  • live view
  • clean HDMI output 
There is currently no camera on the market that has all these features combined. A few models come close and might be usable. The biggest problem is finding out whether a camera can be fully controlled by a computer.

These are the most promising cameras available in 2018:

Canon EOS M100

Canon EOS M100
24 MP APS-C (22.3 x 14.9 mm) sensor, max resolution 6000 x 4000.
Video: HD,  output on HDMI.
USB 2.0 (no charging), Wi-Fi 802.11b/g/n with NFC.
€319
The battery has to be removed from the camera to be charged in an external charger. The Canon ACK-E12 AC-adapter can be used to power the camera continuously. €110. Cheaper no-name alternatives can be found.



Panasonic Lumix DMC-GX85

Panasonic Lumix DMC-GX85
16 MP Four Thirds (17.3 x 13 mm) sensor, max resolution 4592 x 3448.
Video: HD and UHD, output on HDMI.
USB 2.0, Wi-Fi 802.11b/g/n with NFC.
€449



Sony a5100

Sony a5100
24 MP APS-C (23.5 x 15.6 mm), max resolution 6000 x 4000.
Video: HD,  output on HDMI.
USB 2.0 (+ charging), Wi-Fi 802.11b/g/n with NFC.
€350
Power is not supplied during shooting/playback if the camera is charged via the USB connector. To supply power to the camera during shooting/playback, use the AC-PW20 AC Adapter (sold separately). The official Sony Adapter is expensive, there are cheaper alternatives:
https://www.amazon.com/AC-PW20-Adapter-Replacement-NP-FW50-DSC-RX10/dp/B076HSGG3V



Most modern high-end cameras will support tethered shooting, where they will show a live preview image on a computer screen or tablet and where all settings are under software control.  Note that 'Live View' resolutions are fairly low, with sizes ranging from 640x480 to 1024x680. If one needs more, it is possible to capture the HDMI output of the camera and get uncompressed 1080p frames. If the frames need not be processed, the camera can be plugged into a HDMI monitor directly. This should be highly convenient for manual work with the digital macro/microscope.

Manufacturers don't state which standard protocols they support, if any. I have the impression that most manufacturers prefer to develop their own communications protocol instead of adhering to an existing industry standard. Note that all the modern cameras have Wi-Fi, and each brand has a proprietary app for remote control.  Sony offers a "Camera Remote API".  This SDK for wireless control of Sony cameras is available for download at https://developer.sony.com/develop/cameras.

The open source gphoto2 library is able to control many cameras, be it via standard protocols or reverse engineered proprietary protocols.  libgphoto2 supports the Sony a5100 camera,
including Live View.  The Canon M100 is not listed as supported by libgphoto2, but the M10 is, and the author of libgphoto2 thinks the M100 will very likely work as well. But it is uncertain if Live View will be available. The Panasonic cameras are supported via PTP only in the most basic form. I have not been able to find evidence of remote capture or even Live View working...


I have decided to buy the Sony a5100 body for experiments, after playing with it in a store. I could remove the lens and see what happens when making a photo. By default you can't. The camera warns that no lens is recognised, and hints at enabling 'shooting without lens' in the menu. Then it works. The mechanical shutter is quite loud, and can't be turned off (only the more expensive a6300 and a6500 models allow that). Fortunately the shutter vibrations have no impact on the sharpness of the photos taken, even when used in a high magnification setup. I have found a €10 USB powered dummy battery at ebay. I have found a €12 Sony E-mount to Nikon mount adapter, with a full metal bayonet. I transformed this part into a generic adapter with a flange that allows other hardware to be fixed with M2 screws.
Computer control of the a5100 is working well with the recent versions of libgphoto. All camera settings are accessible. Live View is up and running in 1024x680 @ 25 Hz. 

UPDATE
The Sony a5100 and Canon EOS M100 turn out to have a mechanical shutter that is rated for 100000 activations. 100000 sounds like a lot, but when the camera is used for machine vision tasks, taking photos every few seconds, the shutter will be worn out in a few days already! The more expensive models, like the Sony a6300, offer a 'silent shutter' mode of operation which is purely electronic: no moving parts, and thus no wear. The a6300 camera is somewhat bulkier and heavier than the a5100. It is drop-in compatible though.

Certain types of lighting produce artefacts when using the electronic shutter. Unfortunately the LED lights used in the micro setup fall in this category: with most shutter speeds you see a soft horizontal banding pattern in the background. Fortunately this is not the case with shutter speeds of 1/50 and 1/100 seconds. The camera supports many ISO settings so it is no problem to find the ideal exposure using either 1/50 or 1/100.   

Sony a6300

Sony a6300
24 MP APS-C (23.5 x 15.6 mm), max resolution 6000 x 4000.
Video: HD, UHD,  output on HDMI.
USB 2.0 (+ charging), Wi-Fi 802.11b/g/n with NFC.
€740


- - - -    i m a g i n g   m e t h o d s


There are basically two approaches to capture macro/micro images:
  • Afocal photography, also known as afocal projection is a method of photography where a camera with objective is mounted over the eyepiece of a microscope or telescope, with the camera lens taking the place of the human eye.
  • Focal photography, also known as focal projection, is a method of photography where the real image produced by a microscope or telescope objective is projected directly onto an imaging sensor.

I favour the most direct approach. Focal projection,  using a minimal number of lenses which are of maximal quality. Theoretically this should give the best images, and it is also the most flexible and modular way to design and assemble your own imager system.

A combination of a some items out of my collection of salvaged parts resulted in the first iteration of a generic macro/micro imager. The bellows system uses a simple method of mounting cylindrical plates to its front and backside. It is straightforward to create an adapter plate for each objective and sensor to test. All permutations can then be tried easily. Changing the bellows length changes the magnification of the system. All in all this a very convenient platform for experimentation.

The experimental setup, shown here with the See3CAM_CU135 and Nikon Plan 4x mounted.

This combination turns out to produce the highest quality images so far:
Sony a5100 + Nikon Plan Apo 4/0.16

- - - -    e x p e r i m e n t a l   r e s u l t s

On all the following test photos, the subject was a piece of graph paper with a 1.0 millimeter grid. This makes assessing and comparing the magnifications of each combination of image sensor and lens easy. The paper fibers and ink pattern provide enough features to get an impression of sharpness and contrast of the lens that has been used. 

Rodagon 60mm, the ideal diaphragm setting?

The Rodagon is the only lens to be tested that has a variable diaphragm. The first experiment serves to compare the photos taken with 3 settings: F4.0, F5.6 and F8.0. The three test photos were taken with the the Sony a5100 camera. For each diaphragm setting a composite image is shown, containing a 256x256 center crop at the left, and a 256x256 corner crop at the right. These crops are shown here at 1:1 scale and are just a tiny part of the 6000x4000 full resolution photos.


Rodagon 60mm @ F4.0  (left: center, right: corner)
Rodagon 60mm @ F5.6  (left: center, right: corner)
Rodagon 60mm @ F8.0  (left: center, right: corner)


Clearly the sharpness decreases in the center, and increases in the corner, when using a smaller diaphragm opening. A compromise as always... Note that part of the blurriness of the corners is caused by field curvature: it is slightly out of focus. The best setting? F5.6 will be used in all further experiments. 


See3CAM_CU135 + Nikon Plan Apo 4/0.16

All photos taken with the 4.8x3.6 mm Aptina sensor have high magnification and lack in sharpness. Such a small sensor needs to be used with lenses that are specially designed to project a relatively small image.



See3CAM_CU135 + Nikon Plan Apo 4/0.16 @ 100mm
See3CAM_CU135 + Nikon Plan Apo 4/0.16 @ 100mm  (left: center, right: corner)

Field of view: 1.8 x 1.3 mm. Spatial resolution: 0.43 μm per pixel.



See3CAM_CU135 + Nikon Plan Apo 4/0.16 @ 120mm
See3CAM_CU135 + Nikon Plan Apo 4/0.16 @ 120mm  (left: center, right: corner)

Field of view: 1.5 x 1.2 mm. Spatial resolution: 0.36 μm per pixel.




See3CAM_CU135 + Nikon Plan Apo 4/0.16 @ 140mm
See3CAM_CU135 + Nikon Plan Apo 4/0.16 @ 140mm  (left: center, right: corner)

Field of view: 1.3 x 1.0 mm. Spatial resolution: 0.31 μm per pixel.




See3CAM_CU135 + Nikon Plan 4x/0.10

All photos taken with the 4.8x3.6 mm Aptina sensor have high magnification and lack in sharpness. Such a small sensor needs to be used with lenses that are specially designed to project a relatively small image.


See3CAM_CU135 + Nikon Plan 4x/0.10 @ 60mm
See3CAM_CU135 + Nikon Plan 4x/0.10 @ 60mm  (left: center, right: corner)

Field of view: 5.3 x 4.0 mm. Spatial resolution: 1.26 μm per pixel.




See3CAM_CU135 + Nikon Plan 4x/0.10 @ 80mm
See3CAM_CU135 + Nikon Plan 4x/0.10 @ 80mm  (left: center, right: corner)

Field of view: 3.8 x 2.9 mm. Spatial resolution: 0.90 μm per pixel.




See3CAM_CU135 + Nikon Plan 4x/0.10 @ 100mm
See3CAM_CU135 + Nikon Plan 4x/0.10 @ 100mm  (left: center, right: corner)

Field of view: 2.8 x 2.1 mm. Spatial resolution: 0.67 μm per pixel.




See3CAM_CU135 + Rodagon 60mm

All photos taken with the 4.8x3.6 mm Aptina sensor have high magnification and lack in sharpness. Such a small sensor needs to be used with lenses that are specially designed to project a relatively small image.

See3CAM_CU135 + Rodagon @ 100mm
See3CAM_CU135 + Rodagon @ 100mm  (left: center, right: corner)

Field of view: 5.8 x 4.3 mm. Spatial resolution: 1.38 μm per pixel.




See3CAM_CU135 + Rodagon @ 120mm
See3CAM_CU135 + Rodagon @ 120m (left: center, right: corner)



Field of view: 4.0 x  3.0 mm. Spatial resolution: 0.95 μm per pixel.




Sony a5100 + Nikon Plan Apo 4/0.16

The Nikon Plan Apo 4/0.16 has turned out to be the best lens in my collection for this purpose.

Sony a5100 + Nikon Plan Apo 4/0.16 @ 130mm
Sony a5100 + Nikon Plan Apo 4/0.16 @ 130mm  (left: center, right: corner)

Field of view: 7.0 x 4.6 mm. Spatial resolution: 1.17 μm per pixel.
The lens is used well below its designed projection distance here. Vignetting is obvious, the field curvature and edge sharpness are very bad, and even in the center, the sharpness is disappointing.




Sony a5100 + Nikon Plan Apo 4/0.16 @ 150mm
Sony a5100 + Nikon Plan Apo 4/0.16 @ 150mm  (left: center, right: corner)

Field of view: 6.0 x 4.0 mm. Spatial resolution: 1.00 μm per pixel.
The lens is used at the designed projection distance here. Optimal sharpness is present in the center, but it decreases rapidly toward the edges. Some signs of vignetting are still visible in the corners.




Sony a5100 + Nikon Plan Apo 4/0.16 @ 160mm
Sony a5100 + Nikon Plan Apo 4/0.16 @ 160mm  (left: center, right: corner)

Field of view: 5.4 x  3.6 mm. Spatial resolution: 0.90 μm per pixel.
From a projection distance of 160 and up, all vignetting effects have disappeared. The center is still very sharp and the edges are acceptable.




Sony a5100 + Nikon Plan Apo 4/0.16 @ 180mm
Sony a5100 + Nikon Plan Apo 4/0.16 @ 180mm  (left: center, right: corner)

Field of view: 5.2 x 3.5 mm. Spatial resolution: 0.87 μm per pixel.
A very even image, center sharpness is still impressive.







Sony a5100 + Nikon Plan Apo 4/0.16 @ 200mm

Sony a5100 + Nikon Plan Apo 4/0.16 @ 200mm  (left: center, right: corner)

Field of view: 4.5 x 3.0 mm. Spatial resolution: 0.77 μm per pixel.
A very even image, center sharpness is still quite good. The corners are doing quite well. Perhaps the best distance to use in the final instrument.




Sony a5100 + Nikon Plan Apo 4/0.16 @ 210mm
Sony a5100 + Nikon Plan Apo 4/0.16 @ 210mm  (left: center, right: corner)

Field of view: 4.3 x 2.9 mm. Spatial resolution: 0.72 μm per pixel.
This is the longest distance i could test on the bellows setup and the extender for the lens. Sharpness is
still usable, on the edge of becoming fuzzy though.




Sony a5100 + Nikon Plan 4x/0.10

Compared to the Nikon Plan Apo 4 lens, this infinity lens produces a much smaller magnification.


Sony a5100 + Nikon Plan 4x/0.10 @ 140mm
Sony a5100 + Nikon Plan 4x/0.10 @ 140mm  (left: center, right: corner)

Field of view: 10.2 x 6.8 mm. Spatial resolution: 1.70 μm per pixel.






Sony a5100 + Nikon Plan 4x/0.10 @ 160mm
Sony a5100 + Nikon Plan 4x/0.10 @ 160mm  (left: center, right: corner)

Field of view: 8.4 x  5.6 mm. Spatial resolution: 1.40 μm per pixel.




Sony a5100 + Rodagon 60mm

The Rodagon lens is excellent if one needs lower magnifications. The image is very even, sharpness is good, and there is virtually no distortion.


Sony a5100 + Rodagon @ 140mm
Sony a5100 + Rodagon @ 140mm  (left: center, right: corner)



Field of view: 17.4 x 11.6 mm. Spatial resolution: 2.90 μm per pixel.




Sony a5100 + Rodagon @ 160mm
Sony a5100 + Rodagon @ 160mm  (left: center, right: corner)

Field of view: 13.3 x 8.9 mm. Spatial resolution: 2.22 μm per pixel.



- - - -    N u m e r i c a l   r e s u l t s


It is interesting to see the JPEG file sizes of the original captures. The larger the file size, the more information it contains.The effectiveness of the JPEG compression algorithm can be used as measure of complexity of the image. Sharp images contain many details and thus make the image more complex. An overall sharpness quality score can be computed by encoding all test shots with the same JPEG compression factor, after resizing the original images to have the same number of pixels.

    6000 x 4000 resized to 3000 x 2000 (6000000 pixels)
    4208 x 3120 resized to 2844 x 2110 (6000840 pixels)
    JPEG quality 96%

The resized test photos sorted from larger to smaller .jpg file, in bytes:

2640560  a5100_Rodagon-140mm-f5.6.jpg
2449088  a5100_Rodagon-160mm-f5.6.jpg
2435815  cu135_Rodagon-100mm-f5.6.jpg
2387791  a5100_NikonPlanApo4-150mm.jpg
2375228  a5100_NikonPlanApo4-210mm.jpg
2275505  a5100_NikonPlanApo4-200mm.jpg
2269918  cu135_NikonPlan4-80mm.jpg
2264491  cu135_NikonPlanApo4-120mm.jpg
2239599  cu135_NikonPlanApo4-140mm.jpg
2232573  cu135_NikonPlan4-60mm.jpg
2222341  a5100_NikonPlanApo4-180mm.jpg
2207270  a5100_NikonPlan4-140mm.jpg
2194019  cu135_Rodagon-120mm-f5.6.jpg
2178010  a5100_NikonPlanApo4-160mm.jpg
2159283  a5100_NikonPlan4-160mm.jpg
2148042  cu135_NikonPlan4-100mm.jpg
2029100  cu135_NikonPlanApo4-100mm.jpg
1749841  a5100_NikonPlanApo4-130mm.jpg


Conclusion: the Sony a5100 + Nikon Plan Apo 4/0.16 @ 210mm gives the best result for macro work at higher magnifications. For macro work at lower magnifications, the Rodagon 60mm is the ideal objective.
Small sensors will only produce good results with optics specifically designed for them.



Note: this is a work in progress...

No comments:

Post a Comment

Finding the faces