Method Of Fabricating A Multi-aperture System For Foveated Imaging And Corresponding Multi-aperture System

Abstract

multi-aperture system for foveated imaging as well as to a corresponding multi-aperture system. The method comprises the steps of: providing an image sensor; and forming, by means of a 3D-printing technique, at least two imaging elements directly on the image sensor such that the imaging elements have a common focal plane located on the image sensor and each imaging element has a different focal length and field of view.

Claims

1. A method of fabricating a multi-aperture system for foveated imaging, comprising the steps of: providing an image sensor; and forming, by means of a 3D-printing technique, at least two imaging elements directly on the image sensor such that the imaging elements have a common focal plane located on the image sensor and each of the at least two imaging elements has a different focal length and field of view.

2. The method according to claim 1, wherein the forming the at least two imaging elements is performed in one single 3D-printing step.

3. The method according to claim 1, wherein the forming the at least two imaging elements comprises forming at least one discharge opening in each imaging element so that an unexposed photoresist is able to discharge after the 3D-printing.

4. The method according to claim 1, wherein the forming the at least two imaging elements comprises forming at least two multiple-lens objectives.

5. The method according claim 1, wherein the method comprises a pre-processing of the image sensor before the at least two imaging elements are formed, and wherein the pre-processing comprises a removal of functional elements that are disposed on the image sensor.

6. The method according to claim 1, wherein the at least two imaging elements are formed such that each of the at least two imaging elements has dimensions of less than 800 m, and/or wherein the at least two imaging elements are formed such that each of the at least two imaging elements occupies an area on the image sensor that is smaller than *400.sup.2 m.sup.2.

7. The method according to claim 1, wherein the forming the at least two imaging elements comprises forming four imaging elements that are arranged in a 22 arrangement, and wherein the four imaging elements are preferably formed such that they occupy a total area on the image sensor less than 10001000 m.sup.2.

8. The method according to claim 1, wherein the image sensor is a CCD-sensor or a CMOS-sensor.

9. A multi-aperture system for foveated imaging comprising: an image sensor; at least two imaging elements which are directly formed on the image sensor by means of a 3D-printing technique, wherein the at least two imaging elements have a common focal plane located on the image sensor and each imaging element has a different focal length and field of view.

10. The multi-aperture system according to claim 9, wherein the image sensor is a CCD-sensor or a CMOS-sensor.

11. The multi-aperture system according to claim 9, wherein each imaging element comprises at least one discharge opening so that an unexposed photoresist is able to discharge after the 3D-printing.

12. The multi-aperture system according to any one of claims 9, wherein each of the at least two imaging elements has dimensions of less than 800 m, and/or wherein each of the at least two imaging elements occupies an area on the image sensor that is smaller than *400.sup.2 m.sup.2.

13. The multi-aperture system according to claim 9, wherein the at least two imaging elements comprise four imaging elements arranged in a 22 arrangement, wherein the four imaging elements preferably occupy a total area on the image sensor of less than 10001000 m.sup.2.

14. The multi-aperture system according to claim 9, further comprising a processor for processing individual images captured by the at least two imaging elements.

15. The multi-aperture system according to claim 14, wherein the processor is configured to process the individual images and generate a resulting image such that in a central area of the resulting image the information density is higher compared to a peripheral area of the resulting image.

Description

BRIEF DESCRIPTION OF THE FIGURES

[0041] The above and other objects, features and advantages of the present invention will become more apparent upon reading of the following description of preferred embodiments and accompanying drawings. Other features and advantages of the subject-matter described herein will be apparent from the description and the drawings and from the claims. It should be understood that even though embodiments are separately described, single features and functionalities thereof may be combined without prejudice to additional embodiments. The present disclosure is illustrated by way of example and not limited by the accompanying figures.

[0042] Preferred embodiments of the present invention are exemplarily described regarding the following figures:

[0043] FIG. 1a shows a schematic representation of a multi-aperture system for foveated imaging according to an example.

[0044] FIG. 1b shows a schematic representation of an exemplary fusion of pixel information to create a foveated image.

[0045] FIG. 2a is a sectional view of a schematic representation of an imaging element according to an example.

[0046] FIG. 2b shows a schematic representation of four imaging elements arranged in a 2x2 arrangement.

[0047] FIG. 3 shows on the left hand side a photograph of an image sensor with imaging elements directly printed onto the chip. On the right hand side, details of one lens group with four imaging elements forming one camera for foveated imaging are shown.

[0048] FIG. 4 shows a measured MTF contrast in object space as a function of angular resolution for four imaging elements, each having a different field of view.

[0049] FIG. 5 is a photograph showing a comparison of simulation and measurement for foveated imaging systems fabricated by the method according to the present invention. In particular, FIG. 5(a) shows the imaging through a single compound lens with 70 field of view. FIG. 5(b) shows foveated images for four different lenses with fields of view of 20, 40, 60 and 70. The measurement for (a) and (b) was done on a glass substrate. FIG. 5(c) shows the same as FIG. 5(a) but simulated and measured on a CMOS image sensor with 1.41.4 m.sup.2 pixel size. FIG. 5(d) shows foveated results from the CMOS image sensor.

[0050] FIG. 6 is a photograph showing a comparison of the 70 FOV image with its foveated equivalents after 3D-printing on the chip. In particular, FIG. 6(a) shows a comparison of the test picture Lena, taken with the eagle-eye camera and FIG. 6(b) shows the imaging performance for a Siemens star test target.

[0051] FIG. 7 shows the development cycle of different lens systems with fields of view (FOV) varying between 20 and 70. The process chain can be separated (from top to bottom) into optical design, mechanical design, 3D-printing, and measurement of the imaging performance using an USAF 1951 test target.

[0052] It is noted that FIGS. 3 and 5 to 7 include photographs which are helpful to illustrate and understand the present invention. These figures cannot be presented otherwise, e.g. as line drawings.

DETAILED DESCRIPTION OF THE FIGURES

[0053] The following detailed description relates to exemplary embodiments of the present invention. Other embodiments of the invention are possible within the scope of the invention as defined by the appended claims. Throughout the figures, same reference signs are used for the same or similar elements.

[0054] FIG. 1a shows a schematic representation of a multi-aperture system 100 for foveated imaging according to an example. The multi-aperture system comprises a CMOS image sensor 1 and four different imaging elements 11, 12, 13 and 14 which are directly formed on the image sensor 1 by means of a 3D-printing technique. The imaging elements 11, 12, 13 and 14 have a common focal plane located on the image sensor 1 and each imaging element has a different field of view (FOV). In the example shown, imaging element 11 is a doublet objective comprising two lenses 11a and 11b and having a FOV of 70 as well as an equivalent focal lengths for 35 mm film of f=31 mm. Imaging element 12 is a doublet objective comprising two lenses 12a and 12b and having a FOV of 60 as well as an equivalent focal lengths for 35 mm film of f=38 mm. Imaging element 13 is a doublet objective comprising two lenses 13a and 13b and having a FOV of 40 as well as an equivalent focal lengths for 35 mm film of f=60 mm. And imaging element 14 is a doublet objective comprising two lenses 14a and 14b and having a FOV of 20 as well as an equivalent focal lengths for 35 mm film of f=123 mm. Accordingly, the multi-aperture system 100 comprising four different compound lenses, i.e. imaging elements 11, 12, 13 and 14, on the same CMOS image sensor 1 combine different field of views and focus lengths in one single system.

[0055] Due to the different fields of view, each imaging element 11, 12, 13 and 14 generates an image with a different information density or resolution. This is indicated in FIG. 1a by a distinct pattern drawn on the image sensor 1 for each imaging element. Imaging element 11 has the lowest resolution (indicated by a white pattern) and imaging element 14 has the highest resolution (indicated by a black pattern).

[0056] FIG. 1b shows a schematic representation of how the pixel information is subsequently fused to form a foveated image. Each imaging element 11, 12, 13 and 14 creates an image of the same lateral size on the image sensor 1. However, due to the varying fields of view, the telephoto lens (20 FOV) magnifies a small central section of the object field of the 70 lens. Appropriate scaling and overlaying of the images thus lead to the foveated image with increasing level of detail towards the center of the image.

[0057] As illustrated in FIG. 1b by means of different patterns which correspond to the patterns shown in FIG. 1a, the center of the foveated image has the highest resolution obtained by imaging element 14, while the outer or peripheral region of the foveated image has the lowest resolution obtained by imaging element 11. From the center to the outer region of the foveated image the resolution decreases in accordance with the resolutions of imaging element 13 and imaging element 12.

[0058] FIG. 2a is a sectional view of a schematic representation of an imaging element 11 according to an example. Imaging element 11 is a doublet lens system or a doublet objective comprising a first lens 11a and a second lens 11b. The individual freeform surfaces with higher-order aspherical corrections are clearly visible. Further, imaging element 11 comprises discharge openings 15 so that an unexposed photoresist is able to discharge after the 3D-printing.

[0059] FIG. 2b shows a schematic representation of the multi-aperture system 100 comprising an image sensor 1, e.g. a CMOS image sensor, and four 3D-printed imaging elements 11, 12, 13 and 14, arranged in a 22 arrangement on the image sensor. In this preferred arrangement the imaging elements are arranged on the image sensor 1 to occupy an area with a rectangular or quadratic shape. While imaging elements 11 and 12 are arranged in a first row, imaging elements 13 and 14 are arranged in a second row, which is parallel to the first row. Such an arrangement is space-saving, especially when using small rectangular or quadratic image sensors. In particular, the multi-aperture design combines four aberration corrected air-spaced doublet objectives 11 to 14 with different focal lengths and a common focal plane which is situated on the image sensor 1. Particularly beneficial is the abilitiy to create aspherical freeform surfaces by means of a 3D-printing, which is heavily utilized in the lens design.

[0060] FIG. 3 depicts the image sensor 1 after 3D-printing multiple groups of the foveated imaging systems directly on the chip. The left hand side of FIG. 3 shows a photograph of image sensor 1 with a plurality of imaging elements directly printed onto the chip. On the right hand side of FIG. 3, details of one lens group with four imaging elements 11, 12, 13 and 14 forming one camera 100 for foveated imaging are shown.

[0061] FIG. 3 shows the sensor with 9 groups of the same four objectives 11 to 14. Each group forms its own foveated camera 100 and occupies a surface area of less than 300300 m.sup.2. Preferably, the system is designed such that the four separate imaging elements 11 to 14 are closely merged into one single object which is then 3D-printed in one single step.

[0062] The highly miniaturized camera 100, mimicking the natural vision of predators, is fabricated by directly 3D-printing the different multi-lens objectives 11 to 14 onto the CMOS image sensor 1. Preferably, the system combines four printed doublet lenses with different focal lengths (equivalent to f=31 to 123 mm for 35 mm film) in a 22 arrangement to achieve a full field of view of 70 with an increasing angular resolution of up to 2 cycles/degree field of view in the center of the image. The footprint of the optics on the chip is below 300300 m.sup.2, while their height is <200 m. Since the four imaging elements 11 to 14 are printed in one single step without the necessity for any further assembling or alignment, this approach allows for fast design iterations and can lead to a plethora of different miniaturized multi-aperture imaging systems with applications in fields such as endoscopy, optical metrology, optical sensing, or security.

[0063] FIG. 4 compares the normalized MTF contrast as a function of angular resolution in object space for the four different imaging elements after measurement with the knife edge method. Curve 20 corresponds to 20 FOV, curve 21 corresponds to 40 FOV, curve 22 corresponds to 60 FOV and curve 23 corresponds to 70 FOV. As expected, systems with longer focal lengths and smaller FOVs exhibit higher object space contrast at higher resolutions due to their telephoto zoom factor. The dashed vertical lines indicate the theoretical resolution limit due to the pixel pitch of the imaging sensor. Dashed line 20a corresponds to 20 FOV, dashed line 21a corresponds to 40 FOV, dashed line 22a corresponds to 60 FOV and dashed line 23a corresponds to 70 FOV. All systems deliver more than 10% of contrast at the physical limits of the sensor. It is noted that the data does not include the effects of the CMOS image sensor and is obtained by knife edge MTF measurements of the samples printed on a glass slide.

[0064] FIG. 5 depicts the simulated and measured results for four cases. FIG. 5a exhibits a comparison between measured and simulated imaging performance of imaging element 11 with 70 FOV printed on a glass slide and imaged through a microscope. Since the simulated results do not include surface imperfections and scattering, a smaller overall contrast is the most striking difference. FIG. 5b compares the foveated images of imaging elements 11 to 14. Both results exhibit visible improvement in level of detail for the central part. If the pixelation effects of the image sensor are taken into account, all images loose information. FIG. 5c displays the simulation results of imaging element 11 if a pixel size of 1.4 m is assumed and compares it to the data attained on the chip. Due to the fact that the chip does not perfectly transfer the image, there is a notable difference in image quality. This effect can be explained by the chip plane being not perfectly aligned with the focal plane and due to the fact that the micro lens array on the chip, important for the imaging perfomance, was removed prior to 3D-printing. The resulting foveated images are shown in FIG. 5d. As in FIGS. 5a to 5c, the simulation result is shown on the top while the measurement result is shown on the bottom. As can be seen in FIG. 5d compared to FIGS. 5a to 5c, the imaging resolution is considerably increased towards the center of the images. Although the measured image does not completely achieve the quality of the simulated one, significant improvement is visible. A further improvement of the image quality and an increase of the information content in a smoothly varying fashion may be achieved by using more advanced image fusion algorithms.

[0065] To further demonstrate the potential of the present invention, the test image Lena as well as a Siemens star have been used as targets, as illustrated in FIG. 6. In both examples, the center of the original images contains more information than the outer parts. Using the foveated approach, the totally available bandwidth is optimized such that the important parts (center) are transferred with a higher spatial bandwidth (resolution) compared to the less important outer areas in the pictures. The results confirm strikingly that the multi-aperture system according to the present invention delivers superior images.

[0066] The employed 3D-printing technology is almost unrestricted in terms of fabrication limitations. This offers high degrees of freedom and unique opportunities for the optical design. However, finding the optimum system becomes more difficult, since the parameter space is much less constrained as compared to many classical design problems. Due to the mature one-step fabrication process, the challenges of the development are, in comparison to competing manufacturing methods - thus shifted from technology towards the optical design.

[0067] To ensure an efficient use of the available space, four different two-lens systems (imaging elements) are designed with full FOVs of 70, 60, 40, and 20. The numbers have been chosen based on the achievable performance in previous experiments and such that each lens contributes to the foveated image with similarly sized sections of the object space. Table 1 shows an overview of the resulting parameters. It is noted that these parameters are only presented as examples. Since the lens stacks and support material are all fully transparent, it is important to keep the aperture stop on the front surface during design. Otherwise, light refracted and reflected by the support structures would negatively influence the imaging performance. Buried apertures inside the lenses are not possible until now because absorptive layers can not be implemented by femtosecond 3D-printing. Due to the scaling laws of optical systems, small f-numbers can be easily achieved. The aperture diameter may be 100 m for all lenses. As a restriction, the image circle diameter may be set to 90 m.

TABLE-US-00001 TABLE 1 Selected parameters of the designed lens systems Lens 1 Lens 2 Lens 3 Lens 4 FOV 70 60 40 20 visible object 2.75 m 1.73 m 0.84 m 0.36 m diameter at 1 m distance focal length 64.6 m 78.3 m 123.9 m 252.2 m f number 0.7 0.8 1.2 2.6 hyperfocal 6.4 mm 7.2 mm 8.1 mm 7.3 mm distance Fresnel number at 70 58 36.7 18 = 550 nm 35 mm equivalent 31 mm 38 mm 60 mm 123 mm focal length

[0068] Before simulation and optmization, it is important to determine the best suited method. The Fresnel numbers of all systems indicate that diffraction does not significantly influence the simulation results. Therefore, geometric optics and standard raytracing can be used to design the different lenses. As an example, the commercial raytracing software ZEMAX may be used. Since the fabrication method by 3D-printing poses no restrictions for the surface shape, aspheric interfaces up to 10.sup.th order are used. As refractive medium, the photoresist IP-S of the company Nanoscribe GmbH, Eggenstein-Leopoldshafen, Germany has been implemented based on previously measured dispersion data. After local and global optimization, the resulting designs reveal diffraction limited performance (Strehl ratio <0.8) for most of the lenses and field angles. The raytracing design has been finalized polychromatically with a direct optimisation of the modulation transfer function (MTF) which includes diffraction effects at the apertures.

[0069] Compared to conventional single-interface microlenses, the close stacking of two elements offers significant advantages and is crucial for the imaging performance. On the one hand, pupil positions and focal lengths can be changed independently which allows for real telephoto and retrofocus systems. On the other hand, aberrations such as field curvature, astigmatism, spherical aberration, and distortion can be corrected effectively.

[0070] After the optical design, the final results are transferred to a computer aided design

[0071] (CAD) software. In terms of support structure design it is important to find a good trade-off between rigidity and later developability of the inner surfaces. So far, the best results have been achieved with open designs based on pillars, as shown in FIG. 3. All of the lens fixtures have an outer diameter of 120 m.

[0072] FIG. 7 shows the different stages of the development process. To measure the imaging performance, samples have been 3D-printed onto glass substrates as well as onto a CMOS imaging sensor (Omnivision 5647). This chip offers a pixel pitch of 1.4 m which results in single images with 3240 pixels. Using a state-of-the-art sensor with 1.12 m pixel pitch would increase this number to 5071 pixels.

[0073] To improve the adhesion of the lenses, the color filter and microlens array on the sensor had to be removed before the 3D-printing.

[0074] To charaterize the optical performance without pixelation effects, the four different compound lenses are printed onto glass slides. Since the lenses have been designed for imaging from infinity and their focal lengths are smaller than 260 m, the hyperfocal distance is about 8 mm and objects further away always remain focused. To assess the imaging quality, the intermediate image formed by the lenses is reimaged with an aberration corrected microscope. Measurements of the modulation transfer function (MTF) based on imaging a knife edge were performed in the same way as described before.

[0075] The foveated camera performance has been evaluated after 3D-printing on the image chip. The sensor device has been placed in 70 mm distance from a target which consists of different patterns printed onto white paper. The target has been illuminated from the backside with an incoherent white light source. The image data from the chip was then read out directly. It has to be noted that the chip and the read out software automatically performed some operations with the images such as color balance or base contrast adjustment. However, there were no edge enhancment algorithms used which would have skewed the displayed results. Due to their different f-numbers, all lenses lead to a different image brightness. To compensate for this effect, the illumination optics have been adjusted such that approximately the same optical power is transferred to the image for all four imaging elements.

[0076] In summary, the present disclosure demonstrates direct 3D-printing of varying complex multi-component imaging systems onto a chip to form a multi-aperture camera. In particular, four different air-spaced doublet lenses to obtain a foveated imaging system with a FOV of 70 and angular resolutions of >2 cycles/degree in the center of the image are combined. Only the chip dimensions and pixel size limit the overall systems dimensions and the optical performance at the moment. Thus, devices can become smaller than 300300200 m.sup.3 in volume and at the same time transfer images with higher resolution. The present invention thus provides improved imaging systems in terms of miniaturization, functionality and imaging quality as compared to the state of the art.

LIST OF REFERENCE NUMERALS

[0077] 1 image sensor (CMOS- or CCD-sensor)

[0078] 11 imaging element (3D-printed)

[0079] 11a lens

[0080] 11b lens

[0081] 12 imaging element (3D-printed)

[0082] 12a lens

[0083] 12b lens

[0084] 13 imaging element (3D-printed)

[0085] 13a lens

[0086] 13b lens

[0087] 14 imaging element (3D-printed)

[0088] 14a lens

[0089] 14b lens

[0090] 15 discharge opening

[0091] 20 curve of MTF contrast for a 20 FOV imaging element

[0092] 20a theoretical resolution limit due to pixel pitch for a 20 FOV imaging element

[0093] 21 curve of MTF contrast for a 40 FOV imaging element

[0094] 21a theoretical resolution limit due to pixel pitch for a 40 FOV imaging element

[0095] 22 curve of MTF contrast for a 60 FOV imaging element

[0096] 22a theoretical resolution limit due to pixel pitch for a 60 FOV imaging element

[0097] 23 curve of MTF contrast for a 70 FOV imaging element

[0098] 23a theoretical resolution limit due to pixel pitch for a 70 FOV imaging element

[0099] 100 multi-aperture system or camera for foveated imaging/foveated imaging system