Dual aperture zoom digital camera
11470257 · 2022-10-11
Assignee
Inventors
- Gal Shabtay (Tel Aviv, IL)
- Ephraim Goldenberg (Ashdod, IL)
- Oded Gigushinski (Herzlia, IL)
- Noy Cohen (Tel Aviv, IL)
Cpc classification
H04N23/45
ELECTRICITY
H04N23/667
ELECTRICITY
H04N23/00
ELECTRICITY
H04N23/67
ELECTRICITY
H04N23/58
ELECTRICITY
H04N23/69
ELECTRICITY
G02B27/0075
PHYSICS
International classification
G02B27/00
PHYSICS
Abstract
A dual-aperture zoom digital camera operable in both still and video modes. The camera includes Wide and Tele imaging sections with respective lens/sensor combinations and image signal processors and a camera controller operatively coupled to the Wide and Tele imaging sections. The Wide and Tele imaging sections provide respective image data. The controller is configured to combine in still mode at least some of the Wide and Tele image data to provide a fused output image from a particular point of view, and to provide without fusion continuous zoom video mode output images, each output image having a given output resolution, wherein the video mode output images are provided with a smooth transition when switching between a lower zoom factor (ZF) value and a higher ZF value or vice versa, and wherein at the lower ZF the output resolution is determined by the Wide sensor while at the higher ZF value the output resolution is determined by the Tele sensor.
Claims
1. A camera, comprising: a Wide imaging section that includes a fixed focal length Wide lens with a Wide field of view (FOV) and a Wide sensor, the Wide imaging section operative to provide Wide image data of an object or scene; a Tele imaging section that includes a fixed focal length Tele lens with a Tele FOV that is narrower than the Wide FOV and a Tele sensor, the Tele imaging section operative to provide Tele image data of the object or scene; and a camera controller operatively coupled to the Wide and Tele imaging sections, the camera controller configured to provide continuous zoom video output images of the object or scene with a smooth transition when switching between a lower zoom factor (ZF) value and a higher ZF value or vice versa, each output image having a respective output resolution, wherein at the lower ZF value the output resolution is determined by the Wide sensor, wherein at the higher ZF value the output resolution is determined by the Tele sensor, and wherein the smooth transition is performed by position matching between the video output images of the Wide imaging section and the Tele imaging section in a region of interest (ROI) so as to overcome object-distance-dependent position shifting.
2. The camera of claim 1, wherein the ROI is a region on which both cameras are focused.
3. A mobile device comprising the camera of claim 2.
4. The mobile device of claim 3, wherein the mobile device is a smartphone.
5. The camera of claim 1, wherein a ratio of a total track length (TTL) to effective focal length (EFL) of the Tele lens is smaller than 1.
6. The camera of claim 1, wherein the position matching includes registration performed between the Wide image data and the Tele image data.
7. The camera of claim 1, wherein the camera controller is further configured to provide the continuous zoom video output images of the object or scene without fusion.
8. The camera of claim 1, wherein the camera controller includes a user control module for receiving user inputs and a sensor control module for configuring each sensor to acquire the Wide and Tele image data based on the user input, wherein the user input includes a zoom factor and a camera mode.
9. The camera of claim 1, wherein the camera controller configuration to provide video output images with a smooth transition when switching between a lower ZF value and a higher ZF value or vice versa includes a configuration that uses information either from the Wide sensor or from the Tele sensor.
10. The camera of claim 1, wherein the camera controller configuration to provide video output images with a smooth transition when switching between a lower ZF value and a higher ZF value or vice versa includes a configuration that uses at a high ZF secondary information from the Wide camera and uses at a low ZF secondary information from the Tele camera.
11. The camera of claim 1, wherein the position matching between the video output images is provided by the camera controller being further configured to perform a registration between the Wide and Tele image data to output a transformation coefficient and to resample the Tele image data or Wide image data according to the transformation coefficient to reduce crossover changes in the image during crossover from one imaging section to the other.
12. A mobile device comprising the camera of claim 1.
13. The mobile device of claim 12, wherein the mobile device is a smartphone.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
(1) Non-limiting examples of embodiments disclosed herein are described below with reference to figures attached hereto that are listed following this paragraph. The drawings and descriptions are meant to illuminate and clarify embodiments disclosed herein, and should not be considered limiting in any way.
(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9)
(10)
(11)
DETAILED DESCRIPTION
(12)
(13) Sensor control module 116 is connected to the two sub-cameras and to the user control module 118 and used to choose, according to the zoom factor, which of the sensors is operational and to control the exposure mechanism and the sensor readout. Mode choice function 120 is used for choosing capture/video modes. ROI function 122 is used to choose a region of interest. As used herein, “ROI” is a user defined as a sub-region of the image that may be exemplarily 4% or less of the image area. The ROI is the region on which both sub-cameras are focused on. Zoom factor function 124 is used to choose a zoom factor. Video processing module 126 is connected to mode choice function 120 and used for video processing. Still processing module 128 is connected to the mode choice function 120 and used for high image quality still mode images. The video processing module is applied when the user desires to shoot in video mode. The capture processing module is applied when the user wishes to shoot still pictures.
(14)
(15) Following is a detailed description and examples of different methods of use of camera 100.
(16) Design for Continuous and Smooth Zoom in Video Mode
(17) In an embodiment, in order to reach high quality continuous and smooth optical zooming in video camera mode while reaching real optical zoom using fixed focal length sub-cameras, the system is designed according to the following rules (Equations 1-3):
Tan(FOV.sub.Wide)/Tan(FOV.sub.Tele)=PL.sub.Wide/PL.sub.video (1)
where Tan refers to “tangent”, while FOV.sub.Wide and FOV.sub.Tele refer respectively to the Wide and Tele lens fields of view (in degrees). As used herein, the FOV is measured from the center axis to the corner of the sensor (i.e. half the angle of the normal definition). PL.sub.Wide and PL.sub.video refer respectively to the “in-line” (i.e. in a line) number of Wide sensor pixels and in-line number of output video format pixels. The ratio PL.sub.Wide/PL.sub.video is called an “oversampling ratio”. For example, in order to get full and continuous optical zoom experience with a 12 Mp sensor (sensor dimensions 4000×3000) and a required 1080p (dimension 1920×1080) video format, the FOV ratio should be 4000/1920=2.083. Moreover, if the Wide lens FOV is given as FOV.sub.Wide=37.5.sup.0, the required Tele lens FOV is 20.2.sup.0 The zoom switching point is set according to the ratio between sensor pixels in-line and the number of pixels in-line in the video format and defined as:
Z.sub.switch=PL.sub.Wide/PL.sub.video (2)
Maximum optical zoom is reached according to the following formula:
Z.sub.max=Tan(FOV.sub.Wide)/Tan(FOV.sub.Tele)*PL.sub.Tele/PL.sub.video (3)
For Example: For the Configuration Defined Above and Assuming PL.sub.Tele=4000 and PL.sub.video=1920, Z.sub.max=4.35.
(18) In an embodiment, the sensor control module has a setting that depends on the Wide and Tele FOVs and on a sensor oversampling ratio, the setting used in the configuration of each sensor. For example, when using a 4000×3000 sensor and when outputting a 1920×1080 image, the oversampling ratio is 4000/1920=2.0833.
(19) In an embodiment, the Wide and Tele FOVs and the oversampling ratio satisfy the condition
0.8*PL.sub.Wide/PL.sub.video<Tan(FOV.sub.Wide)/Tan(FOV.sub.Tele)<1.2*PL.sub.Wide/PL.sub.video. (4)
Still Mode Operation/Function
(20) In still camera mode, the obtained image is fused from information obtained by both sub-cameras at all zoom levels, see
(21)
(22) Specifically with reference to
(23) In another embodiment, the camera controller synchronizes the Wide and Tele sensors so that for both sensors the rolling shutter starts at the same time.
(24) The exposure times applied to the two sensors could be different, for example in order to reach same image intensity using different F # and different pixel size for the Tele and Wide systems. In this case, the relative exposure time may be configured according to the formula below:
ET.sub.Tele=ET.sub.Wide.Math.(F#.sub.Tele/F#.sub.Wide).sup.2.Math.(Pixel size.sub.Wide/Pixel size.sub.Tele).sup.2 (5)
Other exposure time ratios may be applied to achieve wide dynamic range and improved SNR. Fusing two images with different intensities will result in wide dynamic range image.
(25) In more detail with reference to
Y.sub.Wide start=½.Math.PC.sub.Wide(1−1/ZF)
Y.sub.Wide end=½.Math.PC.sub.Wide(1+1/ZF)
where PC is the number of pixels in a column, and Y is the row number 2) Cropping index Tele sensor:
If ZF>Tan(FOV.sub.Wide)/Tan(FOV.sub.Tele), then
Y.sub.Tele start=½.Math.PC.sub.Tele(1−(1/ZF).Math.Tan(FOV.sub.Tele)/Tan(FOV.sub.Wide))
Y.sub.Tele end=½.Math.PC.sub.Tele(1+(1/ZF).Math.Tan(FOV.sub.Tele)/Tan(FOV.sub.Wide))
If ZF<Tan(FOV.sub.Wide)/Tan(FOV.sub.Tele), then
Y.sub.Tele start−=0
Y.sub.Tele end=PC.sub.Tele
This will result in an exposure start time of the Tele sensor with a delay of (in numbers of lines, relative to the Wide sensor start time):
(1−ZF/((Tan(FOV.sub.Wide)/Tan(FOV.sub.Tele))).Math.1/(2.Math.FPS) (6)
where FPS is the sensor's frame per second configuration. In cases where ZF>Tan (FOV.sub.Wide)/Tan(FOV.sub.Tele), no delay will be introduced between Tele and Wide exposure starting point. For example, for a case where Tan(FOV.sub.Wide)/Tan(FOV.sub.Tele)=2 and ZF=1, the Tele image first pixel is exposed ¼.Math.(1/FPS) second after the Wide image first pixel was exposed.
(26) After applying the cropping according to the required zoom factor, the sensor rolling shutter time and the vertical blank should be configured in order to satisfy the equation to keep the same frame rate:
VB.sub.Wide+RST.sub.Wide=VB.sub.Tele+RST.sub.Tele (7)
(27)
(28) Generally, working with a dual-sensor system requires multiplying the bandwidth to the following block, for example the ISP. For example, using 12 Mp working at 30 fps, 10 bit per pixel requires working at 3.6 Gbit/sec. In this example, supporting this bandwidth requires 4 lanes from each sensor to the respective following ISP in the processing chain. Therefore, working with two sensors requires double bandwidth (7.2 Gbit/sec) and 8 lanes connected to the respective following blocks. The bandwidth can be reduced by configuring and synchronizing the two sensors. Consequently, the number of lanes can be half that of a conventional configuration (3.6 Gbit/sec).
(29)
(30)
(31) To reduce processing time and power, steps 506, 508, 510, 512 could be bypassed by not fusing the images in non-focused areas. In this case, all steps specified above should be applied on focused areas only. Since the Tele optical system will introduce shallower depth of field than the Wide optical system, defocused areas will suffer from lower contrast in the Tele system.
(32) Zoom-In and Zoom-Out in Still Camera Mode
(33) We define the following: TFOV=tan(camera FOV/2). “Low ZF” refers to all ZF that comply with ZF<Wide TFOV/Tele TFOV. “High ZF” refers to all ZF that comply with ZF>Wide TFOV/Tele TFOV. “ZFT” refers to a ZF that complies with ZF=Wide TFOV/Tele TFOV. In one embodiment, zoom-in and zoom-out in still mode is performed as follows:
(34) Zoom-in: at low ZF up to slightly above ZFT, the output image is a digitally zoomed, Wide fusion output. For the up-transfer ZF, the Tele image is shifted and corrected by global registration (GR) to achieve smooth transition. Then, the output is transformed to a Tele fusion output. For higher (than the up-transfer) ZF, the output is the Tele fusion output digitally zoomed.
(35) Zoom-out: at high ZF down to slightly below ZFT, the output image is a digitally zoomed, Tele fusion output. For the down-transfer ZF, the Wide image is shifted and corrected by GR to achieve smooth transition. Then, the output is transformed to a Wide fusion output. For lower (than the down-transfer) ZF, the output is basically the down-transfer ZF output digitally zoomed but with gradually smaller Wide shift correction, until for ZF=1 the output is the unchanged Wide camera output.
(36) In another embodiment, zoom-in and zoom-out in still mode is performed as follows:
(37) Zoom-in: at low ZF up to slightly above ZFT, the output image is a digitally zoomed, Wide fusion output. For the up-transfer ZF and above, the output image is the Tele fusion output.
(38) Zoom-out: at high ZF down to slightly below ZFT, the output image is a digitally zoomed, Tele fusion output. For the down-transfer ZF and below, the output image is the Wide fusion output.
(39) Video Mode Operation/Function
(40) Smooth Transition
(41) When a dual-aperture camera switches the camera output between sub-cameras or points of view, a user will normally see a “jump” (discontinuous) image change. However, a change in the zoom factor for the same camera and POV is viewed as a continuous change. A “smooth transition” is a transition between cameras or POVs that minimizes the jump effect. This may include matching the position, scale, brightness and color of the output image before and after the transition. However, an entire image position matching between the sub-camera outputs is in many cases impossible, because parallax causes the position shift to be dependent on the object distance. Therefore, in a smooth transition as disclosed herein, the position matching is achieved only in the ROI region while scale brightness and color are matched for the entire output image area.
(42) Zoom-In and Zoom-Out in Video Mode
(43) In video mode, sensor oversampling is used to enable continuous and smooth zoom experience. Processing is applied to eliminate the changes in the image during crossover from one sub-camera to the other. Zoom from 1 to Z.sub.switch is performed using the Wide sensor only. From Z.sub.switch and on, it is performed mainly by the Tele sensor. To prevent “jumps” (roughness in the image), switching to the Tele image is done using a zoom factor which is a bit higher (Z.sub.switch+ΔZoom) than Z.sub.switch. ΔZoom is determined according to the system's properties and is different for cases where zoom-in is applied and cases where zoom-out is applied (ΔZoom.sub.in≠ΔZoom.sub.out). This is done to prevent residual jumps artifacts to be visible at a certain zoom factor. The switching between sensors, for an increasing zoom and for decreasing zoom, is done on a different zoom factor.
(44) The zoom video mode operation includes two stages: (1) sensor control and configuration, and (2) image processing. In the range from 1 to Z.sub.switch, only the Wide sensor is operational, hence, power can be supplied only to this sensor. Similar conditions hold for a Wide AF mechanism. From Z.sub.switch+ΔZoom to Z.sub.max only the Tele sensor is operational, hence, power is supplied only to this sensor. Similarly, only the Tele sensor is operational and power is supplied only to it for a Tele AF mechanism. Another option is that the Tele sensor is operational and the Wide sensor is working in low frame rate. From Z.sub.switch to Z.sub.switch+ΔZoom, both sensors are operational.
(45) Zoom-in: at low ZF up to slightly above ZFT, the output image is the digitally zoomed, unchanged Wide camera output. For the up-transfer ZF, the output is a transformed Tele sub-camera output, where the transformation is performed by a global registration (GR) algorithm to achieve smooth transition. For higher (than the up-transfer), the output is the transfer ZF output digitally zoomed.
(46) Zoom-out: at high ZF down to slightly below ZFT, the output image is the digitally zoomed transformed Tele camera output. For the down-transfer ZF, the output is a shifted Wide camera output, where the Wide shift correction is performed by the GR algorithm to achieve smooth transition, i.e. with no jump in the ROI region. For lower (than the down-transfer) ZF, the output is basically the down-transfer ZF output digitally zoomed but with gradually smaller Wide shift correction, until for ZF=1 the output is the unchanged Wide camera output.
(47)
(48) In more detail, for ZF range 1:Z.sub.switch, for ZF<Z.sub.switch, the Wide image data is transferred to the ISP in step 612 and resampled in step 614. For ZF range=Z.sub.switch:Z.sub.switch+ΔZoom.sub.in, both sensors are operational and the zoom image is generated from the Wide sensor. The color balance is calculated for both images according to a given ROI. In addition, for a given ROI, registration is performed between the Wide and Tele images to output a transformation coefficient. The transformation coefficient is used to set an AF position. The transformation coefficient includes the translation between matching points in the two images. This translation can be measured in a number of pixels. Different translations will result in a different number of pixel movements between matching points in the images. This movement can be translated into depth and the depth can be translated into an AF position. This enables to set the AF position by only analyzing two images (Wide & Tele). The result is fast focusing.
(49) Both color balance ratios and transformation coefficient are used in the ISP step. In parallel, the Wide image is processed to provide a processed image, followed by resampling. For ZF range=Z.sub.switch+ΔZoom.sub.in:Z.sub.max and for Zoom factor>Z.sub.switch, ΔZoom.sub.in, the color balance calculated previously is now applied on the Tele image. The Tele image data is transferred to the ISP in step 612 and resampled in step 614. To eliminate crossover artifacts and to enable smooth transition to the Tele image, the processed Tele image is resampled according to the transformation coefficient, the requested ZF (obtained from zoom function 124) and the output video resolution (for example 1080p).
(50)
(51) Optical Design
(52) Additional optical design considerations were taken into account to enable reaching optical zoom resolution using small total track length (TTL). These considerations refer to the Tele lens. In an embodiment, the camera is “thin” (see also
(53) In another embodiment of a lens block in a thin camera, shown in
(54) In conclusion, dual aperture optical zoom digital cameras and associate methods disclosed herein reduce the amount of processing resources, lower frame rate requirements, reduce power consumption, remove parallax artifacts and provide continuous focus (or provide loss of focus) when changing from Wide to Tele in video mode. They provide a dramatic reduction of the disparity range and avoid false registration in capture mode. They reduce image intensity differences and enable work with a single sensor bandwidth instead of two, as in known cameras.
(55) All patent applications mentioned in this specification are herein incorporated in their entirety by reference into the specification, to the same extent as if each individual patent application was specifically and individually indicated to be incorporated herein by reference. In addition, citation or identification of any reference in this application shall not be construed as an admission that such reference is available as prior art to the present disclosure.
(56) While this disclosure has been described in terms of certain embodiments and generally associated methods, alterations and permutations of the embodiments and methods will be apparent to those skilled in the art. The disclosure is to be understood as not limited by the specific embodiments described herein, but only by the scope of the appended claims.