Abstract
An apparatus, system and method controls and interacts within an interaction volume within a height over the coordinate plane of a computer such as a computer screen, interactive whiteboard, horizontal interaction surface, video/web-conference system, document camera, rear-projection screen, digital signage surface, television screen or gaming device, to provide pointing, hovering, selecting, tapping, gesturing, scaling, drawing, writing and erasing, using one or more interacting objects, for example, fingers, hands, feet, and other objects, for example, pens, brushes, wipers and even more specialized tools. The apparatus and method be used together with, or even be integrated into, data projectors of all types and its fixtures/stands, and used together with flat screens to render display systems interactive. The apparatus has a single camera covering the interaction volume from either a very short distance or from a larger distance to determine the lateral positions and to capture the pose of the interacting object(s).
Claims
1-20. (canceled)
21. An apparatus for determining a position or posture or both of an object, wherein the object is in whole or partly located within an interaction volume with a certain height range over an interaction surface, the apparatus comprising: a mirror arrangement configured to produce a view of the interaction volume, the mirror arrangement being comprising one or more mirror sections; a camera configured to see the object through the mirror arrangement and directly without use of the mirror arrangement, and to capture a picture including the object and the interaction surface, the camera being arranged such that a camera's field-of-view includes both the interaction volume and the mirror arrangement; and a computational unit for computation of the position or posture or both of the object based on information from the camera, wherein the mirror arrangement, where the one or more mirror sections comprises at least one off-axis concave parabolic optical mirror element at a plane including the interaction surface and where the off-axis concave parabolic optical mirror element is arranged with its focal point at an entrance pupil of the camera and its axis parallel with the interaction surface, produces the view of the interaction volume with constant magnification of the height of the interaction volume by the off-axis concave parabolic optical mirror element along its axis, and the computational unit determines the position or posture or both of the object based on information of the picture from the camera.
22. The apparatus according to claim 21, wherein a second object is within the camera's field-of-view, where the posture of the second object is determined.
23. The apparatus according to claim 21, wherein the off-axis concave parabolic optical mirror element comprises Fresnel like mirror element providing off-axis concave parabolic mirror function.
24. The apparatus according to claim 21, wherein the off-axis concave parabolic optical mirror element comprises a mirror element and a lens element arranged in combination for providing the off-axis concave parabolic mirror function.
25. The apparatus according to claim 21, wherein the off-axis concave parabolic optical mirror element comprises a reflective surface where reflection is provided either by a metalized plastics material film, metalized plastics material injection-moulded parts, by total internal reflection or by total internal reflection combined with metallizing.
26. The apparatus according to claim 21, wherein the off-axis concave parabolic optical mirror element comprises a layer of plastics material or a special coating or both which selectively stops or passes light within given wavelength ranges allowing, the mirror element to be functional in the near infrared light with reduced reflections of visual light.
27. The apparatus according to claim 21, wherein the off-axis concave parabolic optical mirror element is adapted to be arranged in an exterior of a periphery of the interaction surface.
28. The apparatus according to claim 21, wherein the off-axis concave parabolic optical mirror element is arranged in a straight moulding along an exterior of an edge of the interaction surface.
29. The apparatus according to claim 21, wherein the off-axis concave parabolic optical mirror element is distributed in a semi-circular shape adapted to be arranged at a wall or table mount.
30. The apparatus according to claim 21, wherein the mirror sections are arranged for providing multiple views of the object.
31. The apparatus according to claim 30, wherein the mirror elements are arranged in a mosaic structure for reducing shading and enhancing mirror-to-pixel mapping characteristics.
32. The apparatus according to claim 21, the camera comprising two or more cameras, wherein the cameras are arranged to provide multiple views of the object, and in areas of direct line of sight from the cameras to avoid shading.
33. The apparatus according to claim 21, wherein the camera is arranged with a bi-focal lens to magnify the view of the mirror arrangement.
34. The apparatus according to claim 21, wherein the camera comprises at least one optical filter to block out or pass light at a selected wavelength such that unwanted light is stopped while allowing light in the wavelength range to pass.
35. The apparatus according to claim 21, wherein the camera comprises at least one selectable optical filter for selectively blocking out or passing light at different wavelength ranges.
36. The apparatus according to claim 21, comprising an illumination arrangement arranged to provide illumination of at least one part of the interaction volume with visual or near infrared light or both, directly or indirectly via the mirror arrangement.
37. The apparatus according to claim 36, wherein the illumination arrangement is controlled to turn the illumination on and off or to provide flashing within an active exposure period of the camera to freeze motions of the object.
38. The apparatus according to claim 36, wherein the illumination arrangement is arranged in a proximity of the camera's entrance pupil, namely close to the focal point of the off-axis concave parabolic optical mirror element, and illuminates indirectly through the mirror arrangement such that the illumination from the illumination arrangement is spread in the interaction volume with rays parallel to the interaction surface.
39. The apparatus according to claim 36, further comprising: a separate, second mirror arrangement arranged to contribute to illuminating the interaction volume.
40. The apparatus according to claim 36, further comprising: a separate, second illumination arrangement arranged to contribute to illuminating the interaction volume.
41. The apparatus according to claim 36, wherein the illumination arrangement is operable to provide for direct illumination and for indirect illumination through a mirror arrangement and wherein the direct and indirect illumination is controlled separately to improve detection of the object.
42. The apparatus according to claim 36, wherein the illumination arrangement is operable to change an appearance of the object, by projecting colored or flashing illumination or both.
43. The apparatus according to claim 21, further comprising: additional curved or flat mirror elements adapted to provide spatial information when observed from the camera.
44. The apparatus according to claim 21, wherein the mirror arrangement comprises two mirror sections arranged at a distance to allow finding the position or the posture or both of the object by triangulation.
45. The apparatus according to claim 21, wherein lens optics of the camera is separated for direct view and view through the at least one off-axis concave parabolic optical mirror element by utilizing one or more separate sensors.
46. An interaction system for providing interactive use of an object in an interaction surface, the interaction system comprising: an apparatus for determining position or posture or both according to claim 21; and a presentation device arranged to present images at the interaction surface.
47. The interaction system according to claim 46, further comprising: a front-projection screen as the interaction surface, wherein the camera and a projector as the presentation device are arranged on the same side of the front-projection screen as the interaction volume.
48. The interaction system according to claim 46, further comprising: a semi-transparent rear-projection screen as the interaction surface, wherein the camera and a projector as the presentation device are arranged on an opposite side of the semi-transparent rear-projection screen to the interaction volume.
49. The interaction system according to claim 46, further comprising: a semi-transparent flat screen as the interaction surface, wherein the camera is arranged on an opposite side of the semi-transparent flat screen to the interaction volume.
50. The interaction system according to claim 46, wherein the interaction surface is arranged at a wall, a table or a handheld device.
51. The interaction system according to claim 46, further comprising: an attachment device for the presentation device, wherein the mirror arrangement is arranged in connection with the attachment means such that near optimal positioning of different components of the system is facilitated.
52. A method of determining a position or posture or both of an object, wherein the object is in whole or partly located within an interaction volume with a certain height range over an interaction surface, the method comprising: reflecting radiation from the object within the interaction volume using a mirror arrangement comprising at least one off-axis concave parabolic optical mirror element at a plane including the interaction surface; producing a view of the interaction volume with constant magnification of the height for each parabolic optical mirror element along its axis by the off-axis concave parabolic optical mirror element arranged with its focal point at an entrance pupil of a camera and its axis parallel with the interaction surface; recording reflected radiation by the camera arranged such that a camera's field-of-view includes both the interaction volume and the mirror arrangement; transferring information from the camera to a computing unit; and computing the position or posture or both of the object based on information from the camera.
53. A method of calibration and control in the height over the interaction surface for precise touch and hovering information comprising: the method for determining the position or posture or both according to claim 52, and further comprising: placing a semi-transparent three-dimensional pattern test object on the interaction surface; highlighting the interaction surface in circular areas one by one; observing the semi-transparent three-dimensional pattern test object directly and seen from the side through the mirror arrangement by the camera; identifying pattern of the semi-transparent three-dimensional pattern test object; calibrating and mapping from interaction surface coordinates to display coordinates or calibrating the height measuring or both; and determining thresholds for touch and hovering.
54. A method for finding for an object, a distance to the interaction surface, three dimensional coordinates and touch and hovering status, comprising: the method for determining the position or posture or both according to claim 52, and further comprising: performing a standard image acquisition and feature extraction; finding solid angles which a tip of the object at the camera's entrance pupil in a front view and in a mirror viewpoint; finding the distance between the object and the interaction surface by using a direct linear model if the mirror is a parabola or else a parabolic approximation model; finding three-dimensional coordinates of the object based on the solid angles and the distance to the interaction surface; and finding hover or touch status of the object by comparing the distance to the interaction surface with threshold values.
55. A method of speeding-up a computation and a search for tracking an object in an interaction volume, comprising: the method for determining the position or posture or both of the object according to claim 32, and further comprising: performing a standard image acquisition and feature extraction within a sub-image including the mirror arrangement; finding a distance between the object and the interaction surface and an effective observation angle of parabolic mirror element along the interaction volume; finding a straight line in the interaction volume representing all the possible X-Y positions of the object; finding a corresponding two-dimensional trajectory in a pixel array of the camera; traversing the trajectory with a certain pathwidth by an edge detector and finding a candidate of the object at X-Y position in the pixel array; performing detailed edge detection or template matching to find an accurate array X-Y position in the pixel array for the candidate of the object; finding a corresponding X-Y position, when Z is known; and reporting one or more of X,Y,Z and touch and hover information to a computer.
Description
DESCRIPTION OF THE DRAWINGS
[0089] The invention is herein described, by way of examples only, with reference to accompanying drawings, wherein:
[0090] FIG. 1 is an illustration of an exemplary configuration according to a preferred embodiment of the present invention, wherein a mirror arrangement of off-axis substantially parabolic elements is organized in a semi-circular manner around a short-throw projector mount;
[0091] FIG. 2 is a presentation of a configuration as provided in FIG. 1 in a side view;
[0092] FIG. 1B is an illustration of an exemplary configuration according to a preferred embodiment of the present invention, wherein a mirror arrangement of off-axis substantially parabolic elements is organized in a semi-circular shape over the flat screen;
[0093] FIG. 2B is an illustration of a configuration as provided in FIG. 1B in a side view;
[0094] FIG. 3 is an illustration of an exemplary configuration according to a preferred embodiment of the present invention, wherein a mirror arrangement of off-axis substantially parabolic elements is organized along a straight moulding just above a projector display area;
[0095] FIG. 4 is a presentation of a configuration as provided in FIG. 3 in a side view;
[0096] FIG. 3B is an illustration of an exemplary configuration according to a preferred embodiment of the present invention, wherein a mirror arrangement of off-axis substantially parabolic elements is organized along a straight moulding just above a flat display area;
[0097] FIG. 4B is a presentation of a configuration as provided in FIG. 3B in a side view;
[0098] FIG. 5 is an illustration of an exemplary configuration according to a preferred embodiment of the present invention, wherein a mirror arrangement of off-axis substantially parabolic elements is organized to avoid obstacles like, for example, a short-throw projector chassis or a mount, to dispose the mirror elements in areas of direct line-of-sight from a camera disposed outside a display area;
[0099] FIG. 6 is a presentation of a configuration as provided in FIG. 5 in a side view;
[0100] FIG. 7 is an illustration of an exemplary configuration according to a preferred embodiment of the present invention, wherein a mirror arrangement of off-axis substantially parabolic elements is organized in a semi-circular shape on a table close to a projector and a camera mount;
[0101] FIG. 8 is a presentation of a configuration as provided in FIG. 7 in a side view;
[0102] FIG. 7B is an illustration of an exemplary configuration according to a preferred embodiment of the present invention, wherein a mirror arrangement of off-axis substantially parabolic elements is organized on a table close to a camera mount and a flat screen;
[0103] FIG. 8B is a presentation of a configuration as provided in FIG. 7B in a side view;
[0104] FIG. 9 is an illustration of an exemplary configuration according to a preferred embodiment of the present invention, wherein the mirror arrangement of off-axis substantially parabolic elements is organized organized along a straight moulding just above projector display area for a rear-projection system;
[0105] FIG. 10 is a presentation of a configuration as provided in FIG. 9 in a side view;
[0106] FIG. 9B is an illustration of an exemplary configuration according to a preferred embodiment of the present invention, wherein a mirror arrangement of off-axis substantially parabolic elements is organized along a straight moulding just above a display area for a transparent screen (e.g. OLED) system;
[0107] FIG. 10B is a presentation of a configuration as provided in FIG. 9B in a side view;
[0108] FIG. 11 is an illustration of an exemplary configuration according to a preferred embodiment of the present invention, wherein a mirror arrangement of off-axis substantially parabolic elements is organized along a straight moulding just above a top side of a projector display area for a rear-projection system mounted in a table;
[0109] FIG. 12 is a presentation of a configuration as provided in FIG. 11 in a side view;
[0110] FIG. 11B is an illustration of an exemplary configuration according to a preferred embodiment of the present invention, where a mirror arrangement of off-axis substantially parabolic elements is organized along a straight moulding just above a display area for a transparent screen (e.g. OLED) mounted in a table;
[0111] FIG. 12B is a presentation of a configuration as provided in FIG. 11B in a side view;
[0112] FIG. 13 is illustration of an exemplary configuration according to a preferred embodiment of the present invention, wherein a mirror arrangement of off-axis substantially parabolic elements is organized along a circular shape, for example, above a top side of a projector display area for a rear-projection system mounted in a table, or organized in elements in areas of direct line-of-sight from a camera to avoid obstacles but outside the display area;
[0113] FIG. 14 is a presentation of a configuration as provided in FIG. 13 in a side view;
[0114] FIG. 13B is an illustration of an exemplary configuration according to a preferred embodiment of the present invention, wherein a mirror arrangement of off-axis substantially parabolic elements is organized along a circular shape, for example, above a top side of a transparent display (e.g. OLED) screen mounted in a table, or organized in elements in areas of direct line-of-sight from a camera to avoid obstacles but outside the display area.
[0115] FIG. 14B is a presentation of a configuration as provided in FIG. 13B in a side view;
[0116] FIG. 15 is an illustration of an exemplary configuration according to a preferred embodiment of the present invention, wherein a mirror arrangement of off-axis substantially parabolic elements is organized along a circular shape, for example, above a top side of a projector display area for a wall-mounted rear-projection system, or organized in elements in areas of direct line-of-sight from a camera to avoid obstacles but outside the display area;
[0117] FIG. 16 is a presentation of a configuration as provided in FIG. 15 in a side view;
[0118] FIG. 15B is an illustration of an exemplary configuration according to a preferred embodiment of the present invention, wherein a mirror arrangement of off-axis substantially parabolic elements is organized along a circular shape, for example, above a top side of a transparent display (e.g. OLED) screen mounted on a wall, or organized in elements in areas of direct line-of-sight from a camera to avoid obstacles but outside the display area;
[0119] FIG. 16B is a presentation of a configuration as provided in FIG. 15B in a side view;
[0120] FIG. 17 is an illustration of an exemplary configuration according to a preferred embodiment of the present invention, wherein a mirror arrangement of off-axis substantially parabolic elements is organized along a straight moulding just above a display area for a transparent screen (e.g. OLED) mounted in a handheld device;
[0121] FIG. 18 is an illustration of typical camera images for some exemplary configurations according to preferred embodiments of the present invention, wherein the mirror arrangement of off-axis substantially parabolic elements is organized in various different ways;
[0122] FIG. 19A is an illustration of a parabola and an off-axis segment;
[0123] FIG. 19B to 19F are illustrations of exemplary configurations of off-axis concave substantially parabolic elements, and also illustrations of some manufacturing limitations;
[0124] FIG. 20 is an illustration of exemplary configurations of mirror elements according to preferred embodiments of the present invention;
[0125] FIG. 21A is a flow diagram illustrating an exemplary methodology that facilitates finding fingers' distance to a surface, finding fingers' three dimensional coordinates within a volume, and a touch and hovering status of the fingers;
[0126] FIG. 21B is a flow diagram illustrating a speed-up methodology for finding an object;
[0127] FIG. 22 is an illustration of exemplary methodologies that facilitate calibration of a camera to a display screen, according to a preferred embodiment of the present invention;
[0128] FIG. 23 is an illustration of exemplary configurations of a pen with tracking patterns, a mirror with localization control patterns, and a coordinate plane with localization control patterns;
[0129] FIG. 24 is an illustration of exemplary configurations of providing a controlled background for imaging and for measuring by using a small moulding or list along one or more edges of a coordinate plane;
[0130] FIG. 25 is an illustration of exemplary configurations of a mirror arrangement of off-axis substantially parabolic elements at a coordinate plane combined with additional curved or flat mirror elements further outside the coordinate plane, for providing spatial information, observed by using a camera;
[0131] FIG. 26 is an illustration of exemplary configurations of a mirror arrangement of off-axis substantially parabolic elements at a coordinate plane for an observation of an object's height relative to a coordinate plane, combined with a separate apparatus for illumination;
[0132] FIG. 27 is a schematic illustration of a system comprising a display, a cooperating computer and an apparatus according to the present invention;
[0133] FIG. 28 is an illustration of an exemplary configuration according to a preferred embodiment of the present invention, wherein a direct view and a mirror view are captured by two cooperating separated image sensors with optics to optimize each view for low manufacturing cost, miniaturization and simple set-up; and
[0134] FIG. 29 is a schematic illustration of an exemplary configuration of a mirror arrangement of two sections consisting of off-axis substantially parabolic mirror elements M1 and M2 which can be observed by a camera and an object P which is located in an interaction volume.
DESCRIPTION OF EMBODIMENTS OF THE INVENTION
[0135] The present invention pertains to an apparatus, a system and a method for a camera-based computer input device for man-machine interaction. Moreover, the present invention also concerns apparatus for implementing such systems and executing such methods.
[0136] Before explaining at least one embodiment of the invention in detail, it is to be understood that the invention is not limited in its application to the details of construction and arrangements of components set forth in the following description or illustrated in the drawings. The invention is capable of being implemented by way of other embodiments or of being practiced or carried out in various ways. Moreover, it is to be understood that phraseology and terminology employed herein are for the purpose of description and should not be regarded as being limiting.
[0137] The principles and operation of the interaction input device apparatus, system and method, according to the present invention, may be better understood with reference to the drawings and their accompanying descriptions.
[0138] Firstly, a principle of an interaction device and an interaction system will be described. Thereafter, a detailed description of some preferred embodiments will be described together with their detailed system operation principles.
[0139] The principle of the interaction apparatus and interaction system is described by referring to an exemplary configuration as illustrated in FIG. 1 and FIG. 2 which schematically depict a hardware configuration of a preferred embodiment of the present invention, as seen in perspective and from side view. The hardware components of this embodiment are a short-throw data projector 3 placed along with a camera 5 and an illuminant 6 on a wall mount 4.
[0140] The appearance and the practical implementation of the wall mount 4 can vary significantly, but a main purpose of it is to dispose one or more of the short-throw projector 3, the camera 5 and the illuminant 6 in a proper distance to a screen and to mount on the wall 11 preferably above a displayed picture 12. The displayed picture 12 also represents the coordinate plane 12 and is preferably projected onto a smooth and white surface suitable for projection, pen operation and touching, while in the case of using a flat display the interaction surface 12 is the display itself, optionally protected with a special transparent material in typically glass or plastics material for protection, such that it is robust for pen and touch operation. The data projector 3 has a field-of-view 9 and is operable to project the displayed picture 12, as represented by the solid line rectangle within an interaction volume 1.
[0141] An object 2 which, for example, is the user's finger and/or hand can interact with a computer or similar within the interaction volume 1 limited by a particular height over the coordinate plane. There is included a mirror arrangement 7 of at least one off-axis substantially parabolic element outside the interaction volume 1 with its axis parallel to the coordinate plane and its parabolic focal point at the camera's entrance pupil to provide a constant magnification of the volume's height dimension along its axis.
[0142] The camera 5 has a field-of-view 8 which includes the interaction volume 1 and the mirror arrangement 7, such that the object's coordinates and the object's hover height can be calculated and/or its hovering condition and/or touch condition and/or the posture characteristics can be derived based on a single image processed by the computational unit, and/or the object's movement and/or the object's gestures can be further calculated based on a sequence of images processed by the computational unit, where the computational unit typically, but not necessarily, is embedded into the camera 5. The camera 5 may have optical filters to selectively block out light of different wavelength ranges, for example, to reduce the influence of daylight and light from the display. The camera 5 may also be equipped with a bi-focal lens to magnify the mirror arrangement 7 on the expense of its surroundings thus increasing the resolution of the imaging of the mirror arrangement 7 in the camera 5 sensor pixel array.
[0143] The computational unit has communication means, for example a microcontroller, for transferring the coordinates and the other interaction data to a computer, by, for example, using some serial bus standard and circuits (like USB) or by using wireless communication protocols and devices.
[0144] The illuminant 6 can be directive and switchable, thus illuminating the object 2 either directly or through the mirror arrangement 7, such that a most appropriate illumination can be selected for the lateral positioning and the hover height determination, respectively. For lateral positioning of the object, illumination through the mirror arrangement 7 may be preferable because of the formation of a mainly constant height field of light rays parallel to the plane which will illuminate the object from the side when entering the interaction volume 1 and thus also providing some contouring of the object 2 when observed directly from the camera 5. In contradistinction, for the determination of hover height, direct illumination may be more attractive (than illumination through the mirror arrangement 7) thus separating the optical paths for the illuminant 6 and the camera 5, to maximize the signal-to-noise ratio, and further providing some contouring of the object 2 when observed through the mirror arrangement 7. In some exemplary configurations, the sideway illumination can also be done by a substantially similar mirror arrangement, which is separated from the mirror arrangement 7 adapted to be optimized for the observation to get the best signal-to-noise ratio for the combination of sideway illumination and sideway observation.
[0145] In all the exemplary configurations and preferred embodiments according to the present invention, there may further be included at least one outer shield or chassis, omitted here for clarity of the figures, but which may enclose one or more of the hardware components: the projector 3, the camera 5 (including the computational unit and communication means), the illuminant 6, the wall mount 4, the mirror arrangement 7 and the display and coordinate plane 12. The purpose for the outer shield or chassis may, for example, be to make the interaction system robust, maintenance-free, dustproof, user-friendly, safer, easier to manufacture, simpler to install, and to present the system with a professional look according to some given principles and elements of design.
[0146] Referring further to FIG. 1 and FIG. 2, the mirror arrangement 7 of off-axis substantially parabolic elements is in this exemplary configuration disposed in a mainly semi-circular curvature above the coordinate plane and display 12 preferably either mounted on the wall 11, on the projector mount 4 or on the surface extending the coordinate plane and display 12. In this preferred embodiment the mirror arrangement 7 may be an integral part of the wall mount or an integral part of the complete interactive whiteboard. The mirror arrangement 7 may also be included in a retrofit kit for upgrading an existing whiteboard or short throw projector installation to become touch-sensitive.
[0147] Referring to FIG. 1B and FIG. 2B, the configuration illustrated here is similar to that as described above for FIG. 1 and FIG. 2, except that the projector 3 and the projector display surface 12 is replaced by a flat screen (LCD, plasma, OLED, rear-projection etc.) for the display 12.
[0148] Further referring to FIG. 1B and FIG. 2B, a stand-alone configuration without any display 12 may be utilized for capturing, for example, precisely strokes from a chalk and sponge and finger touch on a traditional blackboard, while captured results are stored in a computer and the input or some interpretation of the input is shown by its normal computer screen or by a connected display or a projector for the reference of the user or/and the audience.
[0149] Referring to FIG. 3 and FIG. 4, the mirror arrangement 7 comprises off-axis substantially parabolic elements placed along a straight line outside one edge, preferably an upper edge, of the display and coordinate plane 12. The same properties and functions as described for FIG. 1 and FIG. 2 pertain except for a difference regarding the physical appearance of the mirror arrangement 7.
[0150] Referring to FIG. 3B and FIG. 4B, the configuration is similar as described above for FIG. 3 and FIG. 4, except that the projector 3 and the projector display surface 12 are replaced by a flat screen (LCD, plasma, OLED, rear-projection etc.) for the display 12.
[0151] Referring to FIG. 5 and FIG. 6, the mirror arrangement 7 comprises off-axis substantially parabolic elements placed in areas of direct line-of sight from the camera 5 to avoid obstacles due to, for example, the projector 3 chassis or wall mount 4, while being outside the display and coordinate plane 12. The same properties and functions as described for FIG. 1 and FIG. 2 pertain except in respect of the physical appearance of the mirror arrangement 7. In some configurations, the projector 3 and the projector display surface 12 are replaced by a flat screen (LCD, plasma, OLED, rear-projection etc.) for the display 12.
[0152] Referring to FIG. 7 and FIG. 8, the same properties and functions as described for FIG. 1 and FIG. 2 pertain except that the system is not mounted for vertical use on a wall but rather mounted for horizontal use on a table surface 12.
[0153] Referring to FIG. 7B and FIG. 8B, the configuration is similar as described above for FIG. 7 and FIG. 8, except that the projector 3 and the projector display surface 12 are replaced by a flat screen (LCD, plasma, OLED, rear-projection etc.) for the display 12.
[0154] Referring to FIG. 9 and FIG. 10, the same properties and functions as described for FIG. 3 and FIG. 4 pertain except that the system now is adapted for a semi-transparent rear-projection screen 12, such that the camera 5, the illuminant 6, the projector 3 and the wall mount 4 are behind the wall 11, whereas the mirror arrangement 7 of off-axis substantially parabolic elements along a straight moulding is mounted above the projection screen 12 on the wall to observe the interaction volume 1 at a certain given height over the display and coordinate plane 12.
[0155] Referring to FIG. 9B and FIG. 10B, the configuration is similar as described above for FIG. 9 and FIG. 10, except that the projector 3 and the projector display surface 12 are replaced by a semi-transparent flat screen (OLED etc.) for the display 12.
[0156] Referring to FIG. 11 and FIG. 12, the same properties and functions as described for FIG. 9 and FIG. 10 pertain except that the system is not mounted for vertical use on a wall but rather mounted for horizontal use on a table surface 12.
[0157] Referring to FIG. 11B and FIG. 12B, the configuration is similar as described above for FIG. 11 and FIG. 12, except that the projector 3 and the projector display surface 12 are replaced by a semi-transparent flat screen (OLED etc.) for the display 12.
[0158] Referring to FIG. 13 and FIG. 14, the same properties and functions as described for FIG. 11 and FIG. 12 pertain except that the mirror arrangement 7 of off-axis substantially parabolic elements is organized along a circular shape, for example, above a top side of the projector display area for a rear-projection system mounted in a table, or organized in elements in areas of direct line-of-sight from the camera to avoid obstacles but outside the display area.
[0159] Referring to FIG. 13B and FIG. 14B, the configuration is similar as described above for FIG. 13 and FIG. 14, except that the projector 3 and the projector display surface 12 are replaced by a semi-transparent flat screen (OLED etc.) for the display 12.
[0160] Referring to FIG. 15 and FIG. 16, the same properties and functions as described for FIG. 9 and FIG. 16 pertain except that the mirror arrangement 7 of off-axis substantially parabolic elements is organized along a circular shape, for example, above the top side of the projector display area 12, or organized in elements in areas of direct line-of-sight from the camera 5 to avoid obstacles but outside the display area 12.
[0161] Referring to FIG. 15B and FIG. 16B, the configuration is similar as described above for FIG. 15 and FIG. 16, except that the projector 3 and the projector display surface 12 are replaced by a semi-transparent flat screen (OLED etc.) for the display 12.
[0162] Referring to FIG. 17, the same properties and functions as described for FIG. 9B, FIG. 10B, FIG. 11B and FIG. 12B pertain except that the interactive system is adapted to be mounted in a handheld device.
[0163] Referring to FIG. 18, typical images for some exemplary configurations according to the preferred embodiments of the present invention are illustrated, wherein the mirror arrangement 7 of off-axis substantially parabolic elements is organized (a) along a circular shape as in FIG. 1, FIG. 2, FIG. 1B, FIG. 2B, FIG. 7, FIG. 8, FIG. 7B, FIG. 8B; (b) along a straight moulding parallel to an edge of the coordinate plane 12 as in FIG. 3, FIG. 4, FIG. 3B, FIG. 4B; (c) along elements in areas of direct line-of-sight from the camera 5 to avoid obstacles as in FIG. 5 and FIG. 6; (d) along two, three or four straight mouldings parallel to the edges of the coordinate plane 12 which may provide multiple views of the object 2; (e) along a straight long moulding parallel to the upper edge of a very wide coordinate plane 12 covered by the viewpoints of several cameras 5; (f) along one or more elements in areas of direct line-of-sight from the cameras 5 to avoid obstacles and which may provide multiple views of the object 2. This configuration may also be applicable in interactive signage and in interactive posters in exhibitions and museums, where several interactive areas or islands may be established between areas with, for example, three-dimensional structures with informational content which the user can interact with.
[0164] Referring to FIG. 19, a parabola with focal point
[00001]
described by the equation
[00002]
and an example of an off-axis substantially parabolic element (above the hatched area and inside the dashed oval) is shown.
[0165] Now, example numerical values will be provided for a semi-circular mirror arrangement 7 of parabolic elements for a camera 5 with entrance pupil placed x=510 mm away from the display 12, and with an outer radius of R=150 mm, and a height of H=50 mm (meaning that an interaction volume 1 with height 50 mm can be observed through the mirror arrangement 7). The focal point is
[00003]
[0166] The distance R-r from the outer radius as a function of the actual height h of the parabolic element surface, where R is outer radius and r is actual radius, can be found for some height h values, as following:
TABLE-US-00001 h 50 40 30 20 10 0 R-r ≈63.55 ≈51.36 ≈38.91 ≈26.20 ≈13.23 ≈0
[0167] Referring to FIG. 19B, an exemplary mirror arrangement 7 is shown relating to the above numerical example, wherein the off-axis concave parabolic elements are arranged in sector of 176° of a circle with outer radius of 150 mm. The mirror arrangement 7 is of height 0-50 mm, while the overall height of the unit is 60 mm. The part can be moulded in ABS plastic and metalized by aluminium and protected by a thin polymer layer to avoid degradation by oxidation. Alternatively, a sheet of metalized plastics material can be glued to the part, but then the correct double-curved surface is not feasible to form.
[0168] Referring to FIG. 19C, the shape of a sheet of metalized plastics material for the exemplary mirror arrangement 7 as described in FIG. 19B and related to the above numerical example.
[0169] Referring to FIG. 19D, a perspective drawing of the exemplary mirror arrangement 7 as described in FIG. 19A, 19B and 19C is shown. The mirror arrangement 7 is adapted to be placed directly on the surface extending the coordinate plane 12 or at the same level mounted on the wall 11 or the wall mount 4.
[0170] Referring to FIG. 19E, an exemplary mirror arrangement 7 may be designed which due to some manufacturing limitations in a given case only allow the mirror surface to be single curved. FIG. 19E is an illustration the different shape of the ideal off-axis parabolic function and this linearized off-axis parabolic function. The slope for the single curved surface is adapted to be almost correct at height=0, meaning that the reading of the “final touch” at h=0 will be rather correct, For the mirror arrangement 7 with ideal parabolic function, the reading through the mirror of the object's height over the coordinate plane 11 will be directly a linear function and independent upon the actual (X,Y) location in the interaction volume 1, while for the mirror arrangement 7 using such a manufactured non-ideal parabolic function the reading of object's height will have to be corrected by a (X,Y) location dependent error term, for example, implemented by a look-up table.
[0171] Referring to FIG. 19F, an exemplary mirror arrangement 7 is designed which due to some manufacturing limitations, for example, is restricted to have two single curved surfaces, namely the two linear sections in order to approximate the off-axis concave ideal parabolic shape. The figure illustrates the difference in shape between the ideal off-axis parabolic function and the off-axis substantially parabolic function having two linearized sections. These shape artifacts will distort the image of the object, since the deflection angles are not correct. In general, it is feasible due to, for example, manufacturing limitations to utilize different linearized, segmented or other approximated functions to approximate the ideal off-axis concave parabolic function as, for example, of FIG. 19A, and such resulting off-axis concave substantially parabolic element can provide sufficient image quality for observing the object and determining the object's hover height with a sufficient accuracy according to given system requirements, adapted well to the sensor's finite image resolution and the camera's given lens quality.
[0172] Referring to FIG. 20, exemplary configurations of the mirror elements according to a preferred embodiment of the present invention: (a) a mosaic of small off-axis substantially parabolic mirror segments; (b) mirror-like metalized plastics material films glued to a base; (c) mirror by utilizing total internal reflection in glass or plastics material; (d) mirror by utilizing total internal reflection in glass or plastics material and using metallization for protection and extension of the mirror function for smaller angles than the critical angle for total-internal reflection; (e) mirror by utilizing a flat mirror and one or more Fresnel lenses for providing the required curvature for the off-axis substantially parabolic function when the camera is in front of screen (front-projection); (f) mirror by utilizing a flat mirror and one or more Fresnel lenses for providing the required curvature for the off-axis substantially parabolic function when the camera is behind the screen (rear-projection or “looking through” transparent flat screen, for example, (OLED); (g) mirror by utilizing Fresnel-like segments for the off-axis substantially parabolic function equivalently with (e) and (f);
[0173] Referring to FIG. 21A, a flow diagram provides a illustration of an exemplary methodology that facilitates finding fingers' distance to surface, finding fingers' three dimensional coordinates within volume, and the touch and hovering status. The off-axis substantially parabolic mirror elements represent an alternative viewpoint for observing the objects, and the mirror elements explicitly represent the hover level or height or the orthogonal distance Z of the object above the interaction surface within the interaction volume. Simple image acquisition and feature extraction as depicted in box FIG. 21A can find the candidate object positions within the two regions of interest in the camera image array, namely within the direct, or synonymously the front, viewpoint and the mirror viewpoint. For each view a solid angle which the candidate object subtends at the camera's entrance pupil can be found. In the mirror view, the height Z over the interaction surface (12) is found explicitly and the correspondence problem related to match one or more points in the three dimensional space by two observation and image processing of two different two-dimensional views will be substantially simplified.
[0174] FIG. 21B is a flow diagram illustrating a speed-up methodology for finding an object. In this example the mirror arrangement 7 is a semi-circular off-axis substantially parabolic mirror section as, for example, is illustrated in FIG. 19B to 19D, and with a typical image FIG. 18A, where an object is seen both through the mirror and directly.
[0175] The height Z and the angle AZIMUTH for an object (2) can be observed by the camera through the mirror representing a straight line trajectory in the coordinate system of the interaction volume (1). This straight line in the three-dimensional interaction volume (1) represents all the possible (X-Y) positions the object (2) can have for the given Z and AZIMUTH. This three-dimensional trajectory is by the coordinate transformation for the lens mapped to a two-dimensional trajectory in the camera pixel array which for example can be found by a look-up table, and this trajectory can be traversed starting from the end closest to the mirror and with a certain pathwidth given in number of pixels an edge detector algorithm can find a candidate object. Then detailed sub-pixel edge detection or template matching can be performed to find the pixel position (x-y) with higher accuracy, and then transformed by an inverse coordinate transformation by, for example, a look-up table, the candidate object's coordinates with high accuracy (X-Y) in the surface volume coordinates are calculated. Finally, after this search algorithm, the X,Y,Z and posture information can be reported as described.
[0176] Compared to a full search algorithm in the two-dimensional pixel array with a edge-detector algorithm, which is computational complexity is proportional with the size of the array of interest covering the interaction volume (1), the described algorithm is much less complex, and is substantially proportional with the length of the diagonal of the array, such that the speed-up factor may be substantial, in the range of 100×-1000×, dependent on the resolution of the sensor and the area of interest.
[0177] Referring to FIG. 22, exemplary methodologies are given that facilitate calibration of the camera to the display screen, according to a preferred embodiment of the present invention, wherein (a) is a standard manual calibration approach where crosses are presented on the display screen and an operator uses a pen or the finger to touch each cross in a given sequence; (b) is a automatic calibration approach using patterns like in the inventions WO2001N000369/U.S. Pat. No. 7,083,100B2 and/or WO2006135241A1/US2009040195A1 to identify the different calibration points, these inventions being hereby incorporated by reference; (c) is a semi-automatic calibration approach using patterns like in (b) first to identify the different calibration points, then presenting a set of white circular discs on a black background in given locations in which the operator disposes in a given sequence a semitransparent cylinder with internal opaque or reflective material, such that the touch detection limits can be set or controlled.
[0178] Referring to FIG. 23, exemplary configurations of (a) a pen with tracking patterns 13; (b) a mirror with localization control patterns 13; and (c) a coordinate plane with localization control patterns 13; used together with the present invention, are shown. The patterns may be, for example, patterns used for identification and tracking of objects as in WO2001N000369/U.S. Pat. No. 7,083,100B2 and/or WO2006135241A1/US2009040195A1, hereby incorporated by reference. Referring further to FIG. 23, using such patterns and pattern recognition, the pen input can be distinguished from other interaction input devices like a human finger, such that dual-mode input systems can easily be implemented by the present invention and the actual referred inventions. Referring further to FIG. 23, the interaction surface and the mirror can also be equipped with such patterns, such that automatic control, calibration and self-adjusting set-up can be realized by utilizing the present invention with the other referred inventions.
[0179] Referring to FIG. 24, exemplary configurations of providing a controlled background for the imaging and measurements by using a small moulding or list 15 along one or more edges of the coordinate plane, typically being white, black or having a retro-reflective optical property 14 in the actual near-infrared wavelength range. In this example, the moulding/list is also serving as a pen shelf 15 beneath the coordinate plane.
[0180] Referring to FIG. 25, exemplary configurations of a mirror arrangement of off-axis substantially parabolic elements at the coordinate plane are shown adapted to detect the object's height above the plane, while additional curved or flat mirror elements 16 further outside the off-axis substantially parabolic mirror elements are adapted to provide spatial information of the scene when these mirrors are observed from the camera's viewpoint. This exemplary configuration can enhance the ability to follow and determine the posture and gestures of objects 2 also outside the interaction volume 1 by observing the objects 2 in the mirrors 16. Also, in a more advanced human-computer interaction scenario, the user gestures and behavior can be analyzed by observing the direct view and the view in the mirrors 16 to forecast new interaction events. The three-dimensional position and posture of the object 2 can also be estimated.
[0181] Referring to FIG. 26, exemplary configurations of a mirror arrangement of off axis substantially parabolic elements at the coordinate plane are shown for the observation of object's height relative to coordinate plane, combined with other illumination apparatus 17 for providing illumination, such that the mirror arrangement 7 itself for the observation the object's height over the coordinate plane are less exposed to the direct illumination, thus reducing unwanted reflections of the optical interfaces and by that increasing the signal-to-noise ratio of the measurements.
[0182] Referring to FIG. 27, a system is shown comprising a display 12, a cooperating computer 18 and the apparatus 19 according to the present invention, and the communication means 20 between the cooperating computer and the display 12 and the communication means 21 between the cooperating computer and the present apparatus 19 according to the present invention. The communication means 20 is optionally implemented as a wireless data link and/or a direct cable-connected link and/or an optically modulated link.
[0183] Referring to FIG. 28, shows an exemplary configuration according to a preferred embodiment of the present invention, where the direct view and the mirror view are captured by two cooperating, separated image sensors 23 and 24, respectively, with separate optics to optimize each view for low cost, miniaturization and simple set-up, and connected through, for example, a high speed serial link 22. The dashed line 10 indicates that one or more of the different components may be enclosed by a chassis 10. A separated illumination unit 17 as shown in FIG. 26 may also be included in such chassis 10. However, the components can also be separated and be modular for retrofitting an existing projector installation to make it interactive or, for example, upgrade a pen-based interactive whiteboard to be touch-sensitive. Optionally, lens optics are used which are best suited for the two separate views, and then executing the same computations on the pair of images by the computational unit. The speed-up scheme described in FIG. 21B for the present invention will also apply with same speed-up potential in such dual sensor/lens configuration.
[0184] Referring to FIG. 29, a redundant scheme for finding the interaction object (2) and touch and hovering state in case of occlusion in the direct camera view, is inspired by the speed-up procedure described in FIG. 21B applied on, for example, two mirror arrangements 7: mirror M1 and mirror M2, wherein a distance between the mirrors M1 and M2 is a baseline L as shown. Correspondingly to methods in FIG. 21B, one may find the azimuth α and height Z1 for object P by observing the mirror M1 and the azimuth β and height Z2 for an object P by observing the mirror M2, and by triangulation finding the object position (X-Y) in the interaction surface 12 or interaction volume 1.
[0185] The two mirrors M1 and M2 are located with a distance L apart, i.e. the baseline is L. Then the distance d from baseline of length L to the target P is:
[00004]
[0186] The distance d can also be expressed as:
[00005]
[0187] The X and Y coordinates can be simply derived by simple trigonometric calculations.
[0188] By coordinate transformation or by a look-up table the corresponding sensor image (x-y) position can be found and a detailed image analysis can be done locally in the image in a neighborhood around the (x-y) position to get a more accurate positioning, which by coordinate transformation or look-up table can be transformed to a corresponding accurate (X-Y) position in the interaction surface (12) or interaction volume (1) coordinate system.
[0189] Modifications to embodiments of the invention described in the foregoing are possible without departing from the scope of the invention as defined by the accompanying claims. Expressions such as “including”, “comprising”, “incorporating”, “consisting of”, “have”, “is” used to describe and claim the present invention are intended to be construed in a non-exclusive manner, namely allowing for items, components or elements not explicitly described also to be present. Reference to the singular is also to be construed to relate to the plural. Numerals included within parentheses in the accompanying claims are intended to assist understanding of the claims and should not be construed in any way to limit subject matter claimed by these claims.
