DUAL-APERTURE ZOOM DIGITAL CAMERA WITH AUTOMATIC ADJUSTABLE TELE FIELD OF VIEW

Abstract

Digital camera comprising an upright Wide camera configured to provide a Wide image with a Wide image resolution and a folded Tele camera configured to provide a Tele image with a Tele image resolution higher than the Wide image resolution, the Wide and Tele cameras having respective Wide and Tele fields of view FOV.sub.W and FOV.sub.T and respective Wide and Tele image sensors, the digital camera further comprising a rotating OPFE operative to provide a folded optical path between an object or scene and the Tele image sensor, wherein rotation of the OPFE moves FOV.sub.T relative to FOV.sub.W. In some embodiments, a rectangular FOV.sub.T is orthogonal to a rectangular FOV.sub.W. When included in a host device having a user interface that displays FOV.sub.T within FOV.sub.W, the user interface may be used to position FOV.sub.T relative to FOV.sub.W, scan FOV.sub.T across FOV.sub.W and acquire, store and display separate Wide and Tele images, composite Wide plus Tele images and stitched Tele images. The positioning of FOV.sub.T within FOV.sub.W, can be done automatically (autonomously) by continuously tracking an object of interest.

Claims

1. A mobile device, comprising: a screen on a front side of the mobile device; an upright Wide camera having a first field-of-view (FOV1), a first resolution, and a first image sensor that lies in a plane parallel to the screen; and a folded Tele camera having a second field-of-view (FOV2), a second resolution, a Tele lens module, an optical folding element (OPFE), and a second image sensor, wherein FOV2<FOV1, wherein the OPFE is configured to receive light from a first direction and redirect the light to a second direction different from the first direction, and wherein the OPFE has an angle α with the screen that fulfills 30≤α<45 degrees.

2. The mobile device of claim 1, wherein 30≤α<40.

3. The mobile device of claim 2, 30≤α<35.

4. The mobile device of claim 1, wherein the folded Tele camera is an auto-focus camera.

5. The mobile device of claim 4, wherein the OPFE is a prism.

6. The mobile device of claim 5, wherein the first resolution is different from the second resolution.

7. The mobile device of claim 6, wherein the upright Wide camera and the folded Tele camera are disposed on a back side of the mobile device, the back side being opposite to the front side.

8. The mobile device of claim 7, wherein the mobile device is configured to display a first icon and a second icon on the screen, wherein the mobile device is configured to display a first scene captured by the upright Wide camera when the first icon is selected, and wherein the mobile device is configured to display a second scene captured by the folded Tele camera when the second icon is selected.

9. The mobile device of claim 8, wherein the first and second icons are displayed along a same axis on one side of the screen.

10. The mobile device of claim 9, further comprising a front camera disposed on the front side of the mobile device, wherein the mobile device is configured to display a third icon, and wherein the mobile device is configured to display a third scene captured by the front camera when the third icon is selected.

11. The mobile device of claim 10, wherein a ratio FOV2/FOV1 is between ¼ and ¾.

12. The mobile device of claim 11, wherein the mobile device is a smartphone.

13. A mobile device, comprising: a screen on a front side of the mobile device; an upright Wide camera having a first field-of-view (FOV1), a first resolution, and a first image sensor that lies in a plane parallel to the screen; and a folded Tele camera having a second field-of-view (FOV2), a second resolution, a Tele lens module, an optical folding element (OPFE), and a second image sensor, wherein FOV2<FOV1, wherein the OPFE is configured to receive light from a first direction and redirect the light to a second direction different from the first direction, and wherein an angle β between the first direction and the second direction is greater than 90 degrees.

14. The mobile device of claim 13, wherein β>95 degrees.

15. The mobile device of claim 13, wherein β>100 degrees.

16. The mobile device of claim 13, wherein β is in the range of 90 degrees to 105 degrees.

17. The mobile device of claim 13, wherein the folded Tele camera is an auto-focus camera.

18. The mobile device of claim 17, wherein the OPFE is a prism.

19. The mobile device of claim 18, wherein the first resolution is different from the second resolution.

20. The mobile device of claim 19, wherein the upright Wide camera and the folded Tele camera are disposed on a back side of the mobile device, the back side being opposite to the front side.

21. The mobile device of claim 20, wherein the mobile device is configured to display a first icon and a second icon on the screen, wherein the mobile device is configured to display a first scene captured by the upright Wide camera when the first icon is selected, and wherein the mobile device is configured to display a second scene captured by the folded Tele camera when the second icon is selected.

22. The mobile device of claim 21, wherein the first and second icons are displayed along a same axis on one side of the screen.

23. The mobile device of claim 22, also comprising a front camera disposed on the front side of the mobile device, wherein the mobile device is configured to display a third icon, and wherein the mobile device is configured to display a third scene captured by the front camera when the third icon is selected.

24. The mobile device of claim 23, wherein a ratio FOV2/FOV1 is between ¼ and ¾.

25. The mobile device of claim 24, wherein the mobile device is a smartphone.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Non-limiting examples of embodiments disclosed herein are described below with reference to figures attached hereto that are listed following this paragraph. Identical structures, elements or parts that appear in more than one figure are generally labeled with a same numeral in all the figures in which they appear. The drawings and descriptions are meant to illuminate and clarify embodiments disclosed herein, and should not be considered limiting in any way.

[0023] FIG. 1A shows schematically an image reflecting the Wide FOV of a scene;

[0024] FIG. 1B shows schematically an image reflecting the Tele FOV of the scene in FIG. 1A;

[0025] FIG. 2A shows schematically a dual-aperture camera comprising a first upright camera and a second folded camera, with a prism folding an optical path to the folded camera in a zero position;

[0026] FIG. 2B shows a composite image obtained with a dual-aperture camera as in FIG. 2A, with the Tele FOV resulting from the zero position of the prism;

[0027] FIG. 3 shows schematically an arrangement enabling tilt of a prism in a folded optical path;

[0028] FIG. 4A shows schematically the dual-aperture camera of FIG. 2A with the prism rotated to a first position;

[0029] FIG. 4B shows a composite image obtained with a dual-aperture camera as in FIG. 4A, with the Tele FOV resulting from the first prism position;

[0030] FIG. 5A shows schematically the dual-aperture camera of FIG. 2A with the prism rotated to a second position;

[0031] FIG. 5B shows a composite image obtained with a dual-aperture camera as in FIG. 5A, with the Tele FOV resulting from the second prism position;

[0032] FIG. 6A shows schematically a dual-aperture camera comprising a first upright camera with a sensor rotated by 90 degrees relative to that in the camera of FIG. 2A and with a prism folding an optical path to a folded camera in a zero position;

[0033] FIG. 6B shows a composite image obtained with a dual-aperture camera as in FIG. 6A, with the Tele FOV resulting from the zero position of the prism;

[0034] FIG. 7A shows schematically the dual-aperture camera of FIG. 6A with the prism rotated to a first position;

[0035] FIG. 7B shows a composite image obtained with a dual-aperture camera as in FIG. 7A, with the Tele FOV resulting from the first prism position;

[0036] FIG. 8A shows schematically the dual-aperture camera of FIG. 6A with the prism rotated to a second position;

[0037] FIG. 8B shows a composite image obtained with a dual-aperture camera as in FIG. 8A, with the Tele FOV resulting from the second prism position;

[0038] FIG. 9A shows a smartphone and user interface incorporating a dual-aperture camera with Tele FOV positioning capability disclosed herein in a zero, centered position on the smartphone screen;

[0039] FIG. 9B shows the smartphone and user interface of FIG. 9A with the Tele FOV moved to a down position within the screen;

[0040] FIG. 9C shows the smartphone and user interface of FIG. 9A with the Tele FOV moved to an up position within the screen;

[0041] FIG. 9D shows the smartphone and user interface of FIG. 9A with the Tele FOV rotated by 90 degrees relative to the Wide FOV and moved to a right position within the screen;

[0042] FIG. 9E shows the smartphone and user interface of FIG. 9A with the Tele FOV rotated by 90 degrees relative to the Wide FOV and moved to a left position within the screen;

[0043] FIG. 10 shows a smartphone and user interface as in FIGS. 9D or 9E used in a scanning mode of the Tele FOV.

[0044] FIG. 11A shows a smartphone and user interface incorporating a dual-aperture camera with automatic Tele FOV tracking disclosed herein with a first Tele FOV position on the smartphone screen;

[0045] FIG. 11B shows the smartphone and user interface of FIG. 11A with a second Tele FOV position on the smartphone screen;

[0046] FIG. 11C shows the smartphone and user interface of FIG. 11A with a third Tele FOV position on the smartphone screen.

DETAILED DESCRIPTION

[0047] FIG. 1A shows schematically an image (or frame of a video stream) 102 reflecting the Wide FOV of a scene 106. FIG. 1B shows schematically an image (or frame of a video stream) 104 reflecting the Tele FOV of scene 106. Only part of the scene of FIG. 1A is seen in FIG. 1B. The two images or video streams are obtained simultaneously with a dual-aperture camera having an upright Wide camera and a folded Tele camera of the type for example disclosed in U.S. patent application Ser. No. 14/455,906. The two cameras may, for example, be two back cameras included in a smartphone or in another personal communication device.

[0048] FIG. 2A shows schematically a dual-aperture camera 200 comprising a first upright camera 202 and a second folded camera 204. An XYZ coordinate system as shown is used in the description of this and all following “camera” drawings. For example, upright camera 202 is a Wide camera that includes a Wide lens assembly (or simply “lens”) 206 with an optical axis 207, and a Wide sensor 208. For example, folded camera 204 is a Tele camera that includes a Tele lens 210 with an optical axis 211, and a Tele sensor 214. An OPFE (e.g. prism) 212 may be part of the folded Tele camera or may be a separate element that folds an optical path parallel to axis 207 into an optical path parallel to axis 211 into the Tele lens. The Wide and Tele sensors lie in respective orthogonal planes, respectively the XZ plane and the XY plane. For example, the Wide and Tele sensors are substantially rectangular with respective height (H) and width (W) dimensions. Thus, Wide sensor 208 has a height H.sub.W and a width W.sub.W and Tele sensor 214 has a height H.sub.T and a width W.sub.T. The H/W ratio in both sensors is typically (although not necessarily) 9/16 or ¾. In the folded camera, positioning of sensor 214 with its height in the Y direction is advantageous in that it allows a smaller host device (e.g. a smartphone) thickness. In camera 200 in FIG. 2A (as well as cameras 200 in FIGS. 3, 4A and 5A), the width dimensions of sensors 208 and 214 are parallel to each other and to the X-axis.

[0049] The operation of a camera such as camera 200 is described in more detail in U.S. patent application Ser. No. 14/455,906. In particular, the prism can perform a tilt (rotation) movement 216 around the X-axis as shown. The rotation motion may be caused by a stepping motor 302 shown schematically in FIG. 3. The prism rotation angle may range between −15 degrees and +15 degrees around a “zero” position (see below), corresponding to ±30 degrees FOV.sub.T shift. Cameras 202 and 204 are used for example to take respectively the Wide and Tele images of FIGS. 1A and 1B.

[0050] FIG. 2B shows an image 230 identical with Wide image 102 with a frame 232 that indicates the position of the Tele image FOV. The camera can have high resolution in this framed FOV either by fusing the Wide and Tele images or by capturing and saving the Tele image. For reference purposes, the position of the prism that causes the Tele FOV and resulting image to be centric to the Wide FOV and Wide image is called here for example and in a non-limiting way a “zero” position of the prism.

[0051] The rotation of prism 212 around the X-axis moves the Tele FOV relative to the Wide FOV, causing other portions of scene 106 to become a “Tele image” with higher resolution. Thus, FIG. 4A shows prism 212 rotated counter-clockwise (as indicated by a curved arrow 402 when viewed in the —X direction) around the X-axis from its zero position to a new, first position. The counter-clockwise rotation causes the Tele FOV, indicated by frame 232, to move to a new, “down” position relative to the Wide FOV indicated by frame 230. FIG. 5A shows prism 212 rotated clockwise (as indicated by a curved arrow 502 when viewed in the —X direction) around the X-axis from its zero position to a new, second position. The clockwise rotation causes the Tele FOV, indicated by frame 232, to move to another new, “up” position relative to the Wide FOV indicated by frame 230. While FIGS. 4A, 4B, 5A and 5B show two discrete prism rotation movements and two discrete Tele FOV positions relative to the Wide FOV, there is clearly an entire range of practically continuous positions that the Tele FOV may occupy, depending on the degree and direction of prism rotation.

[0052] As mentioned, in FIGS. 2A, 3, 4A and 5A, both Wide sensor 208 and Tele sensor 214 are positioned with their longer side (“width”) in the X direction. Their shorter side (“height”) is in the Z direction for sensor 208 and in the Y direction for sensor 214. FIG. 6A shows schematically a dual-aperture camera 200′ comprising an upright camera 206 with a Wide sensor 208′ rotated by 90 degrees relative to sensor 208 in camera 206 of FIG. 2A. As also shown in FIGS. 7A and 8A, Wide sensor 208′ now has its height in the X direction and its width in the Z direction. Note that W.sub.W is now also orthogonal to the H.sub.T. This 90 degree in-plane rotation of the Wide sensor provides certain advantages in terms of the positioning and movement of FOV.sub.T relative to FOV.sub.W and consequently in terms of the capture, processing and display of Tele and Wide images. One major advantage is that a larger percentage of FOV.sub.W can have high resolution by setting a prism rotation angle. FIG. 6B shows schematically the position of FOV.sub.T 232′ (now rotated by 90 degrees relative to FOV.sub.W 230) in a centered position caused by a zero position of prism 212, after the camera itself was rotated by 90 degrees (not shown) compared to the camera of FIG. 2B.

[0053] FIG. 7A shows schematically the dual-aperture camera of FIG. 6A with prism 212 rotated counter-clockwise (as indicated by a curved arrow 702 when viewed in the —X direction) around the X-axis from its zero position to a new position. As shown in FIG. 7B, this prism rotation causes Tele FOV 232′ to move to a “right” position relative to the Wide FOV. FIG. 8A shows prism 212 rotated clockwise (as indicated by a curved arrow 802 when viewed in the −X direction) around the X-axis from its zero position to a second position, As shown in FIG. 8B, this prism rotation causes Tele FOV 232′ to move to a “left” position relative to the Wide FOV. While FIGS. 7A, 7B, 8A and 8B show two discrete prism rotation movements and two discrete Tele FOV positions relative to the Wide FOV, there is clearly an entire range of practically continuous positions that the Tele FOV may occupy, depending on the degree and direction of prism rotation.

[0054] Note that a similar FOV.sub.T relative positioning effect to that described above may be obtained by in-plane rotating Tele sensor 214 by 90 degrees and by leaving Wide sensor 208 unchanged from its original position in FIG. 2A. However, the positioning of sensor 214 with W.sub.T in the Y direction may disadvantageously increase a camera height (and therefore a host device thickness).

[0055] When a dual-aperture camera described above is included for example in a smartphone, the Tele image (i.e. the part of scene 106 viewed and acquired by the Tele camera) is bound by a frame 932 visible on the smartphone screen. FIG. 9A shows an exemplary smartphone 900 that includes on a back side (not shown) a Wide camera and a Tele camera as in FIGS. 2A or 6A. The Wide and Tele cameras have known fields of view with a known ratio M=FOV.sub.T/FOV.sub.W between them. In general, M may have any value between ¼ and ¾. For example, M may have values of ½, 9/16 or ¾. Consequently, frame 932 includes almost exactly the image seen by FOV.sub.T and has a size that is a fraction M of the entire screen (which includes the image seen by FOV.sub.W). Note that for camera configuration as the one shown in FIG. 6A and for Wide and Tele image sensors with 4:3 aspect ratio, selecting M=¾ will result in FOV.sub.T that will overlap the short dimension of FOV.sub.W in its entirety and will enable complete scanning of the Wide FOV by tilting the prism. The same argument is applicable for image sensors having 16:9 aspect ratio with M= 9/16.

[0056] In still mode, scene 106 is acquired by both cameras, with the Wide camera providing the entire image seen (i.e. the Wide image) and the Tele camera providing the part of scene 106 bound by frame 932. Smartphone 900 further includes, on a front side opposite to the back side, a screen or display 902 displaying a view of scene 106. Screen 902 may display icons or text “A”, “B”, “C”, etc., that provide indications and/or are selectable to perform various operations of the phone and/or the cameras. Such icons or text may be indicative of flash setting, video or stills selection, back or front camera selection, etc. The square boxes surrounding “A”, “B” and “C” are merely illustrative and may have different shape or be removed altogether in some cases. Note that the fact that only three icons are shown is not meant to be limiting, and that more or fewer icons may be displayed and/or selectable at any time during or prior to image acquisition by the cameras and/or during display of acquired images. In an embodiment of the dual-aperture camera as in FIG. 2A, the “zero” position of the prism provides the composite image seen in FIG. 9A, where frame 932 is centered on the screen.

[0057] In various embodiments and as described in more detail in PCT/IB2016/056060, smartphone 900 may have a user interface that includes a single camera icon (or “button”) 908 and a “two-camera” button 910. The two-camera button may appear on the screen when the FOV of the scene (FOV.sub.scene) is greater or equal to FOV.sub.T. As described in detail in PCT/IB2016/056060, the user interface displays visually the almost exact FOV.sub.T and enables simple acquisition of the image within FOV.sub.T, thereby providing a Tele image with the highest resolution enabled by the Tele camera. The user interface also enables simultaneous acquisition (with a single input through the user interface, i.e. using two-camera button 910) of a Wide image and a Tele image. Image fusion of the Wide and Tele images or video streams can take place on the capturing device or in a cloud environment.

[0058] The present inventors have determined that, advantageously, a user interface as described above can be used to “drag” frame 932 (and FOV.sub.T) on screen 902 to bring different parts of the scene into FOV.sub.T. That is, the dragging of frame 932 is translated into rotation of the “folded” path prism, such that the higher resolution Tele image “moves” to different parts of the scene. The dragging may be performed by a firm touch of the screen by a finger 920 and movement of the finger across the screen. In FIG. 9A, the finger touches the screen in the general area of “zero” positioned (centered) frame 932. The finger may drag the frame (and FOV.sub.T) to a “down” position as in FIG. 9B, to an “up” position as in FIG. 9C, or to any intermediate position (not shown) between the down and up positions. The touch and drag actions are relayed to a camera controller (not shown), which, in turn, controls the prism movement. The dragging of frame 932 to the down position in FIG. 9B, indicated schematically by an arrow 934, is equivalent to rotation of the prism to its position in FIG. 4A. The dragging of frame 932 to the up position in FIG. 9C is equivalent to rotation of the prism to its position in FIG. 5A. Similarly, with a dual-aperture camera with a 90 degree rotated Wide sensor as in FIGS. 6A-8A, a frame 932′ (now also rotated by 90 degrees relative to frame 932 in FIGS. 9A-9C) may be dragged from a zero position to a “right” position as in FIG. 9D or a “left” position as in FIG. 9E. The dragging of frame 932′ to the right position in FIG. 9D is equivalent to rotation of the prism to its position in FIG. 7A. The dragging of frame 932′ to the up position in FIG. 9E is equivalent to rotation of the prism to its position in FIG. 8A.

[0059] As described in detail in PCT/IB2016/056060, in terms of image acquisition, a user may press two-camera button 910 to simultaneously acquire two images, the Wide image of scene 106 at its respective (lower) image resolution and the Tele image of region (frame) 932 (or 932′) at its respective (higher) image resolution. The two images may be stored in an on-board storage (such as “camera roll” in an iPhone) and may be displayed or downloaded for further use as known in the art. The user may press single camera button 908 to acquire only the Wide image, which can further be stored, displayed and downloaded for further use. The user may choose for display on screen 902 only the Tele image by, for example, double-tapping or pressing at any point on the screen within frame 932 (or 932′). This action leads to display of the Tele image on the entire screen. The Tele image (only) can then be chosen for acquisition by pressing on single camera button 908. The acquired Tele image can then be stored, displayed and downloaded for further use as above. The two images can also be fused (on the camera hosting device or in a cloud) into a composite image with a portion marked by a frame 932 or 932′ formed by the higher-resolution Tele image, and with a peripheral portion formed by a peripheral portion of the relatively lower resolution Wide image.

[0060] Clearly, frame 932′ (and the Tele FOV) may be dragged to any intermediate position (not shown) between the right and left positions. In other words, the Tele FOV may be moved laterally on the screen to a number of partially overlapping or non-overlapping (but touching) positions, from a right-most position (at a right screen edge 940) to a left-most position (at a left screen edge 942) or vice-versa, and an entire image of the scene may be “stitched” together from the partially overlapping or non-overlapping Tele images. For example, as shown in FIG. 10, the ratio M of the Tele and Wide FOVs may be such that screen 902 includes, and is substantially covered by, four adjacent frames 932′a-d. Note that the screen may include fewer adjacent frames than the number required to substantially fill in the entire screen. For example, in an embodiment, there may be only two or three adjacent frames (instead of the four shown). An optional “Scan” icon or button 1002 may then be used to scan (i.e. move the FOV.sub.T frame) automatically from a right-most position on the screen (i.e. from frame 932′a) to a left-most position (i.e. to frame 932′d) or vice-versa. Alternatively, a “Scan” command may be given by voice control. The scan function, similarly to the dragging action, provides partially overlapping or non-overlapping Tele images that can be stitched together into a high-resolution Tele image of the entire scene. A scan command, whether through icon 1002 or vocally, may also lead to acquisition of each Tele image defined by a frame 932′ and to its storage for further processing. A Wide image can be captured during the scan function, allowing further processing that may include image fusion of the Wide image with the various Tele images (or the stitched Tele images) to form a high resolution image with a FOV that is larger than FOV.sub.T.

[0061] Note that the direction of prism rotation and the consequent movement of FOV.sub.T relative to FOV.sub.W on a smartphone (or any other host device) screen depends on the geometry of the assembly of the dual-aperture camera in the host device. The description above relates to one particular such geometry. In a different geometry, the prism rotation directions and the resulting FOV.sub.T movement may be in opposite directions to those described above.

[0062] The devices, used interface and associated methods disclosed above may be used for automatic movement or “automatic adjustment” of the Tele FOV for e.g. tracking a subject in an autonomous manner. We refer to a camera mode that performs automatic Tele FOV movement to track an object or subject of interest as “autonomous Tele FOV tracking”. The autonomous Tele FOV movement is in response to recognition (through e.g. the smartphone camera) of the object or subject of interest, and the Tele image focuses on and displays the object or subject of interest. The object recognition may be performed using any of the methods known in the art. An example of autonomous Tele FOV tracking is shown in FIGS. 11A-11C.

[0063] FIG. 11A shows a smartphone and user interface incorporating a dual-aperture camera with automatic Tele FOV tracking disclosed herein with a first Tele FOV position on the smartphone screen. FIG. 11B shows the smartphone and user interface of FIG. 11A with a second Tele FOV position on the smartphone screen. FIG. 11C shows the smartphone and user interface of FIG. 11A with a third Tele FOV position on the smartphone screen. In each of these figures, the object of interest is a runner 1102. The decision to track the runner is taken either by the user (e.g., by touching the runner's image) or automatically (e.g., using face detection). It is assumed that the Tele camera can change its FOV by tilting the prism to track the object of interest. Ideally, the camera will track the object such that it is as close as possible to the center of the adjustable Tele FOV as seen in 1032a, 1032b and 1032c.

[0064] While the smartphone shown in FIGS. 11A-11C displays icons 908, 910 and 1002, one or more other icons (not shown) may replace and/or be used in addition to or instead of icons 908, 910 and 1002 during operation in the Tele FOV tracking mode.

[0065] Wide and Tele images and/or video streams can be recorded during the automatic tracking mode and fused together to form a composite image or composite video stream. This fusion can be applied on the camera hosting device or alternatively, Wide and Tele images or video streams can be uploaded to the cloud for applying this fusion operation. Each composite image may also be formed with FOV.sub.W by scanning with the Tele camera, stitching a plurality of Tele images to provide a “stitched” Tele image, then fusing the stitched Tele image with the Wide image. This is advantageous in that the Wide image captures the entire scene simultaneously, while the Tele images to be stitched together are consecutive, so one can overcome motion or occlusions in the scene if required. The stitching of the Tele images and/or the fusion of the stitched Tele image with the Wide image may also be performed in a cloud.

[0066] While this disclosure has been described in terms of certain embodiments and generally associated methods, alterations and permutations of the embodiments and methods will be apparent to those skilled in the art. The disclosure is to be understood as not limited by the specific embodiments described herein, but only by the scope of the appended claims.