GESTURE RECOGNITION

Abstract

Disclosed herein is a detector for gesture detection including an illumination source configured for projecting an illumination pattern including a plurality of illumination features on an area including an object, where the object includes at least partially a human hand.

Claims

1. A detector for gesture detection comprising: at least one illumination source configured for projecting at least one illumination pattern comprising a plurality of illumination features on at least one area comprising at least one object, wherein the object comprises at least partially at least one human hand; at least one optical sensor having at least one light-sensitive area, wherein the optical sensor is configured for determining at least one image of the area, wherein the image comprises a plurality of reflection features generated by the area in response to illumination by the illumination features; and at least one evaluation device, wherein the evaluation device is configured for determining at least one depth map of the area by determining at least one depth information for each of the reflection features, wherein the evaluation device is configured for finding the object by identifying the reflection features which were generated by illuminating biological tissue, wherein the evaluation device is configured for determining at least one reflection beam profile of each of the reflection features, wherein the evaluation device is configured for identifying a reflection feature as to be generated by illuminating biological tissue in case its reflection beam profile fulfills at least one predetermined or predefined criterion, wherein the predetermined or predefined criterion is or comprise at least one predetermined or predefined value and/or threshold and/or threshold range referring to a material property, wherein the evaluation device is configured for identifying the reflection feature as to be background otherwise, wherein the evaluation device is configured for segmenting the image of the area by using at least one segmentation algorithm, wherein the reflection features identified as to be generated by illuminating biological tissue are used as seed points and the reflection features identified as background are used as background seed points for the segmentation algorithm, wherein the evaluation device is configured for determining position and/or orientation of the object in space considering the segmented image and the depth map.

2. The detector according to the claim 1, wherein the evaluation device is configured for identifying image coordinates of palm and finger in the segmented image, wherein the evaluation device is configured for determining at least one three-dimensional finger vector considering image coordinates of palm and finger and the depth map.

3. The detector according to claim 1, wherein the evaluation device is configured for determining at least one hand pose or gesture from the position and/or the orientation of the object in space.

4. The detector according to claim 1, wherein the segmentation algorithm is based on energy or cost functions.

5. The detector according to claim 1, wherein segmentation of the image is driven by color homogeneity and edge indicators, wherein the seed points constitute edge- and color homogeneity criterions.

6. The detector according to claim 1, wherein the evaluation device is configured for comparing the reflection beam profile of each of the reflection features with at least one predetermined and/or prerecorded and/or predefined beam profile.

7. The detector according to the claim 6, wherein the comparison comprises overlaying the reflection beam profile and the predetermined and/or prerecorded and/or predefined beam profile such that their centers of intensity match, wherein the comparison comprises determining a deviation between the reflection beam profile and the predetermined and/or prerecorded and/or predefined beam profile, wherein the evaluation device is configured for comparing the determined deviation with at least one threshold, wherein in case the determined deviation is below and/or equal the threshold the reflection feature is indicated as biological tissue.

8. The detector according to claim 1, wherein the evaluation device is configured for determining the depth information for each of the reflection features by one or more of the techniques selected from the group consisting of depth-from-photon-ratio, structured light, beam profile analysis, time-of-flight, shape-from-motion, depth-from-focus, triangulation, depth-from-defocus, and stereo sensors.

9. The detector according to claim 1, wherein the evaluation device is configured for determining the depth information for each of the reflection features by using depth-from-photon-ratio technique, wherein the evaluation device is configured for determining at least one first area and at least one second area of a beam profile of at least one of the reflection features, wherein the evaluation device is configured for integrating the first area and the second area, wherein the evaluation device is configured to derive a quotient Q by one or more technique selected from the group consisting of dividing the integrated first area and the integrated second area, dividing multiples of the integrated first area and the integrated second area, and dividing linear combinations of the integrated first area and the integrated second area.

10. The detector according to claim 9, wherein the first area of the reflection beam profile comprises essentially edge information of the reflection beam profile and the second area of the reflection beam profile comprises essentially center information of the reflection beam profile, and/or wherein the first area of the reflection beam profile comprises essentially information about a left part of the reflection beam profile and the second area of the reflection beam profile comprises essentially information about a right part of the reflection beam profile.

11. The detector according to claim 9, wherein the evaluation device is configured for deriving the quotient Q by $Q=\frac{\int {\int }_{A1}E\left(x,y\right)dxdy}{\int {\int }_{A2}E\left(x,y\right)dxdy}$ wherein x and y are transversal coordinates, A1 and A2 are the first and second area of the reflection beam profile, respectively, and E(x,y) denotes the reflection beam profile.

12. The detector according to claim 1, wherein the illumination source is configured for generating the at least one illumination pattern in the near infrared region (NIR).

13. The detector according to claim 1, wherein the optical sensor comprises at least one CMOS sensor.

14. The method for gesture detection, wherein at least one detector according to the claim 1 is used, wherein the method comprises the following steps: a) projecting at least one illumination pattern comprising a plurality of illumination features on at least one area comprising at least one object, wherein the object comprises at least partially at least one human hand; b) determining at least one image of the area using at least one optical sensor having at least one light-sensitive area, wherein the image comprises a plurality of reflection features generated by the area in response to illumination by the illumination features; c) determining at least one depth map of the area by determining at least one depth information for each of the reflection features by using at least one evaluation device; d) finding the object by using the evaluation device by identifying the reflection features which were generated by illuminating biological tissue, wherein at least one reflection beam profile of each of the reflection features is determined, wherein a reflection feature is identified as to be generated by illuminating biological tissue in case its reflection beam profile fulfills at least one predetermined or predefined criterion, wherein the predetermined or predefined criterion is or comprise at least one predetermined or predefined value and/or threshold and/or threshold range referring to a material property, wherein the reflection feature otherwise is identified as background; e) segmenting the image of the area by using the evaluation device by using at least one segmentation algorithm, wherein the reflection features identified as to be generated by illuminating biological tissue are used as seed points and the reflection features identified as background are used as background seed points for the segmentation algorithm; and f) determining position and/or orientation of the object in space considering the segmented image and the depth map by using the evaluation device.

15. A method of using the detector according to claim 1, the method comprising using the detector for a purpose selected from the group consisting of: driver monitoring; in-cabin surveillance; gesture tracking; a security application; a safety application; a human-machine interface application; an information technology application; an agriculture application; a crop protection application; a medical application; a maintenance application; and a cosmetics application.

16. The detector according to claim 1, wherein the segmentation algorithm is based on energy or cost functions selected from the group consisting of graph cut, level-set, fast marching, Markov random field approaches, and combinations thereof.

Description

BRIEF DESCRIPTION OF THE FIGURES

[0166] Further optional details and features of the invention are evident from the description of preferred exemplary embodiments which follows in conjunction with the dependent claims. In this context, the particular features may be implemented in an isolated fashion or in combination with other features. The invention is not restricted to the exemplary embodiments. The exemplary embodiments are shown schematically in the figures. Identical reference numerals in the individual figures refer to identical elements or elements with identical function, or elements which correspond to one another with regard to their functions.

[0167] Specifically, in the figures:

[0168] FIG. 1 shows an embodiments of a detector according to the present invention; and

[0169] FIGS. 2A to 2C show pose detection according to the present invention; and

DETAILED DESCRIPTION OF THE EMBODIMENTS

[0170] FIG. 1 shows in a highly schematic fashion an embodiment of a detector 110 for gesture detection. The gesture may comprise body gesture such as of at least one part of the body, in particular hand gesture. The gesture may be static or dynamic. The gesture may comprise movement of at least one part of a hand, such as of a finger, movement of one or both hands, face, or other parts of the body. The gesture detection may comprise determining presence or absence of a gesture and/or gesture recognition. The gesture recognition may comprise interpreting of human gestures via mathematical algorithms.

[0171] The detector 110 comprises at least one illumination source 112 configured for projecting at least one illumination pattern comprising a plurality of illumination features on at least one area 114 comprising at least one object 116. The object 116 comprises at least partially at least one human hand.

[0172] The illumination source 112 may be configured for providing the illumination pattern for illumination of the area 114. The illumination source 112 may be adapted to directly or indirectly illuminating the area 114, wherein the illumination pattern is reflected or scattered by surfaces of the area 114 and, thereby, is at least partially directed towards at least one optical sensor 118. The illumination source 112 may be configured for illuminating the area 114, for example, by directing a light beam towards the area 114, which reflects the light beam. The illumination source 112 may be configured for generating an illuminating light beam for illuminating the area 114.

[0173] The area may comprise the at least one object 116 and a surrounding environment. For example, the object may be at least one object selected from the group consisting of: a scene, a human such as a human, wood, carpet, foam, an animal such as a cow, a plant, a piece of tissue, a metal, a toy, a metallic object, a beverage, a food such as a fruit, meat, fish, a dish, a cosmetics product, an applied cosmetics product, cloth, fur, hair, a maintenance product, a plant, a body, a part of a body, organic material, inorganic material, a reflective material, a screen, a display, a wall, a sheet of paper, such as photograph. The object 116 may comprise at least one surface on which the illumination pattern is projected. The surface may be adapted to at least partially reflect the illumination pattern back towards the detector 110. The object 116 may, specifically, be a human body or at least one part of a human body such as at least one arm, at least one hand, at least one finger or a face. The object 116 comprises at least partially at least one human hand. The object 116 may be a human hand and/or at least one finger and/or at least one part of a palm.

[0174] The illumination source 112 may comprise at least one light source. The illumination source 112 may comprise a plurality of light sources. The illumination source 112 may comprise an artificial illumination source, in particular at least one laser source and/or at least one incandescent lamp and/or at least one semiconductor light source, for example, at least one light-emitting diode, in particular an organic and/or inorganic light-emitting diode. As an example, the light emitted by the illumination source may have a wavelength of 300 to 1100 nm, especially 500 to 1100 nm. Additionally or alternatively, light in the infrared spectral range may be used, such as in the range of 780 nm to 3.0 μm. Specifically, the light in the part of the near infrared region where silicon photodiodes are applicable specifically in the range of 700 nm to 1100 nm may be used. The illumination source may be configured for generating the at least one illumination pattern in the infrared region. Using light in the near infrared region allows that light is not or only weakly detected by human eyes and is still detectable by silicon sensors, in particular standard silicon sensors.

[0175] The illumination source 112 may be configured for emitting light at a single wavelength. Specifically, the wavelength may be in the near infrared region. In other embodiments, the illumination source 112 may be configured for emitting light with a plurality of wavelengths allowing additional measurements in other wavelengths channels.

[0176] The illumination source 112 may be or may comprise at least one multiple beam light source. For example, the illumination source 112 may comprise at least one laser source and one or more diffractive optical elements (DOEs). Specifically, the illumination source 112 may comprise at least one laser and/or laser source. Various types of lasers may be employed, such as semiconductor lasers, double heterostructure lasers, external cavity lasers, separate confinement heterostructure lasers, quantum cascade lasers, distributed bragg reflector lasers, polariton lasers, hybrid silicon lasers, extended cavity diode lasers, quantum dot lasers, volume Bragg grating lasers, Indium Arsenide lasers, transistor lasers, diode pumped lasers, distributed feedback lasers, quantum well lasers, interband cascade lasers, Gallium Arsenide lasers, semiconductor ring laser, extended cavity diode lasers, or vertical cavity surface-emitting lasers. Additionally or alternatively, non-laser light sources may be used, such as LEDs and/or light bulbs. The illumination source 112 may comprise one or more diffractive optical elements (DOEs) adapted to generate the illumination pattern. For example, the illumination source 112 may be adapted to generate and/or to project a cloud of points, for example the illumination source may comprise one or more of at least one digital light processing projector, at least one LCoS projector, at least one spatial light modulator; at least one diffractive optical element; at least one array of light emitting diodes; at least one array of laser light sources. On account of their generally defined beam profiles and other properties of handleability, the use of at least one laser source as the illumination source is particularly preferred. The illumination source 112 may be integrated into a housing 120 of the detector 110.

[0177] The light beam or light beams generated by the illumination source 112 generally may propagate parallel to an optical axis 122 or tilted with respect to the optical axis 112, e.g. including an angle with the optical axis. The detector 110 may be configured such that the light beam or light beams propagates from the detector 110 towards the area 114 along an optical axis of the detector 110. For this purpose, the detector 110 may comprise at least one reflective element, preferably at least one prism, for deflecting the illuminating light beam onto the optical axis 122. As an example, the light beam or light beams, such as the laser light beam, and the optical axis 122 may include an angle of less than 10°, preferably less than 5° or even less than 2°. Other embodiments, however, are feasible. Further, the light beam or light beams may be on the optical axis or off the optical axis. As an example, the light beam or light beams may be parallel to the optical axis 122 having a distance of less 10 than 10 mm to the optical axis, preferably less than 5 mm to the optical axis or even less than 1 mm to the optical axis or may even coincide with the optical axis.

[0178] The illumination pattern may comprise at least one illumination feature adapted to illuminate at least one part of the area 114. The illumination pattern may comprise a single illumination feature. The illumination pattern may comprise a plurality of illumination features. The illumination pattern may be selected from the group consisting of: at least one point pattern; at least one line pattern; at least one stripe pattern; at least one checkerboard pattern; at least one pattern comprising an arrangement of periodic or non periodic features. The illumination pattern may comprise regular and/or constant and/or periodic pattern such as a triangular pattern, a rectangular pattern, a hexagonal pattern or a pattern comprising further convex tilings. The illumination pattern may exhibit the at least one illumination feature selected from the group consisting of: at least one point; at least one line; at least two lines such as parallel or crossing lines; at least one point and one line; at least one arrangement of periodic or non-periodic feature; at least one arbitrary shaped featured. The illumination pattern may comprise at least one pattern selected from the group consisting of: at least one point pattern, in particular a pseudo-random point pattern; a random point pattern or a quasi random pattern; at least one Sobol pattern; at least one quasiperiodic pattern; at least one pattern comprising at least one pre-known feature at least one regular pattern; at least one triangular pattern; at least one hexagonal pattern; at least one rectangular pattern at least one pattern comprising convex uniform tilings; at least one line pattern comprising at least one line; at least one line pattern comprising at least two lines such as parallel or crossing lines. For example, the illumination source 112 may be adapted to generate and/or to project a cloud of points. The illumination source 112 may comprise the at least one light projector adapted to generate a cloud of points such that the illumination pattern may comprise a plurality of point pattern. The illumination source 112 may comprise at least one mask adapted to generate the illumination pattern from at least one light beam generated by the illumination source 112.

[0179] A distance between two features of the illumination pattern and/or an area of the at least one illumination feature may depend on the circle of confusion in the image. As outlined above, the illumination source 112 may comprise the at least one light source configured for generating the at least one illumination pattern. Specifically, the illumination source comprises at least one laser source and/or at least one laser diode which is designated for generating laser radiation. The illumination source 112 may comprise the at least one diffractive optical element (DOE). The detector 110 may comprise at least one point projector, such as the at least one laser source and the DOE, adapted to project at least one periodic point pattern.

[0180] The illumination source 112 may illuminate the at least one object 116 with the illumination pattern. The illumination pattern may comprise a plurality of points. These points are illustrated as light beam 124 emerging from the illumination source 112.

[0181] The detector 110 comprises the at least one optical sensor 118 having at least one light-sensitive area 126. The optical sensor 118 is configured for determining at least one image 128 of the area 114. An embodiment of an image is shown in FIG. 2A. The image 128 comprises a plurality of reflection features 130 generated by the area 114 in response to illumination by the illumination features. The detector 110 may comprise a plurality of optical sensors 118 each having a light sensitive area 126. Preferably, the light sensitive area 126 may be oriented essentially perpendicular to the optical axis 122 of the detector 110.

[0182] The optical sensor 118 specifically may be or may comprise at least one photodetector, preferably inorganic photodetectors, more preferably inorganic semiconductor photodetectors, most preferably silicon photodetectors. Specifically, the optical sensor 118 may be sensitive in the infrared spectral range. The optical sensor 118 may comprise at least one sensor element comprising a matrix of pixels. All pixels of the matrix or at least a group of the optical sensors of the matrix specifically may be identical. Groups of identical pixels of the matrix specifically may be provided for different spectral ranges, or all pixels may be identical in terms of spectral sensitivity. Further, the pixels may be identical in size and/or with regard to their electronic or optoelectronic properties. Specifically, the optical sensor 118 may be or may comprise at least one inorganic photodiode which are sensitive in the infrared spectral range, preferably in the range of 700 nm to 3.0 micrometers. Specifically, the optical sensor 118 may be sensitive in the part of the near infrared region where silicon photodiodes are applicable specifically in the range of 700 nm to 1100 nm. Infrared optical sensors which may be used for optical sensors may be commercially available infrared optical sensors, such as infrared optical sensors commercially available under the brand name Hertzstueck™ from TrinamiX™ GmbH, D-67056 Ludwigshafen am Rhein, Germany. Thus, as an example, the optical sensor 118 may comprise at least one optical sensor of an intrinsic photovoltaic type, more preferably at least one semiconductor photodiode selected from the group consisting of: a Ge photodiode, an InGaAs photodiode, an extended InGaAs photodiode, an InAs photodiode, an InSb photodiode, a HgCdTe photodiode. Additionally or alternatively, the optical sensor 118 may comprise at least one optical sensor of an extrinsic photovoltaic type, more preferably at least one semiconductor photodiode selected from the group consisting of: a Ge:Au photodiode, a Ge:Hg photodiode, a Ge:Cu photodiode, a Ge:Zn photodiode, a Si:Ga photodiode, a Si:As photodiode. Additionally or alternatively, the optical sensor 118 may comprise at least one photoconductive sensor such as a PbS or PbSe sensor, a bolometer, preferably a bolometer selected from the group consisting of a VO bolometer and an amorphous Si bolometer.

[0183] The optical sensor 118 may be sensitive in one or more of the ultraviolet, the visible or the infrared spectral range. Specifically, the optical sensor 118 may be sensitive in the visible spectral range from 500 nm to 780 nm, most preferably at 650 nm to 750 nm or at 690 nm to 700 nm. Specifically, the optical sensor 118 may be sensitive in the near infrared region. Specifically, the optical sensor 118 may be sensitive in the part of the near infrared region where silicon photodiodes are applicable specifically in the range of 700 nm to 1000 nm. The optical sensor 118, specifically, may be sensitive in the infrared spectral range, specifically in the range of 780 nm to 3.0 micrometers. For example, the optical sensor 118 each, independently, may be or may comprise at least one element selected from the group consisting of a photodiode, a photocell, a photoconductor, a phototransistor or any combination thereof. For example, the optical sensor 118 may be or may comprise at least one element selected from the group consisting of a CCD sensor element, a CMOS sensor element, a photodiode, a photocell, a photoconductor, a phototransistor or any combination thereof. Any other type of photosensitive element may be used. The photosensitive element generally may fully or partially be made of inorganic materials and/or may fully or partially be made of organic materials. Most commonly, one or more photodiodes may be used, such as commercially available photodiodes, e.g. inorganic semiconductor photodiodes.

[0184] The optical sensor 118 may comprise at least one sensor element comprising a matrix of pixels. Thus, as an example, the optical sensor 118 may be part of or constitute a pixelated optical device. For example, the optical sensor 118 may be and/or may comprise at least one CCD and/or CMOS device. As an example, the optical sensor 118 may be part of or constitute at least one CCD and/or CMOS device having a matrix of pixels, each pixel forming a light-sensitive area. The sensor element may be formed as a unitary, single device or as a combination of several devices. The sensor element comprises a matrix of optical sensors. The sensor element may comprise at least one CMOS sensor. The matrix may be composed of independent pixels such as of independent optical sensors. Thus, a matrix of inorganic photodiodes may be composed. Alternatively, however, a commercially available matrix may be used, such as one or more of a CCD detector, such as a CCD detector chip, and/or a CMOS detector, such as a CMOS detector chip. Thus, generally, the sensor element may be and/or may comprise at least one CCD and/or CMOS device and/or the optical sensors may form a sensor array or may be part of a sensor array, such as the above-mentioned matrix. Thus, as an example, the sensor element may comprise an array of pixels, such as a rectangular array, having m rows and n columns, with m, n, independently, being positive integers. Preferably, more than one column and more than one row is given, i.e. n>1, m>1. Thus, as an example, n may be 2 to 16 or higher and m may be 2 to 16 or higher. Preferably, the ratio of the number of rows and the number of columns is close to 1. As an example, n and m may be selected such that 0.3≤m/n≤3, such as by choosing m/n=1:1, 4:3, 16:9 or similar. As an example, the array may be a square array, having an equal number of rows and columns, such as by choosing m=2, n=2 or m=3, n=3 or the like.

[0185] The matrix may be composed of independent pixels such as of independent optical sensors. Thus, a matrix of inorganic photodiodes may be composed. Alternatively, however, a commercially available matrix may be used, such as one or more of a CCD detector, such as a CCD detector chip, and/or a CMOS detector, such as a CMOS detector chip. Thus, generally, the optical sensor may be and/or may comprise at least one CCD and/or CMOS device and/or the optical sensors of the detector may form a sensor array or may be part of a sensor array, such as the above-mentioned matrix.

[0186] The matrix specifically may be a rectangular matrix having at least one row, preferably a plurality of rows, and a plurality of columns. As an example, the rows and columns may be oriented essentially perpendicular. As used herein, the term “essentially perpendicular” refers to the condition of a perpendicular orientation, with a tolerance of e.g. ±20° or less, preferably a tolerance of ±10° or less, more preferably a tolerance of ±5° or less. Similarly, the term “essentially parallel” refers to the condition of a parallel orientation, with a tolerance of e.g. ±20° or less, preferably a tolerance of ±10° or less, more preferably a tolerance of ±5° or less. Thus, as an example, tolerances of less than 20°, specifically less than 10° or even less than 5°, may be acceptable. In order to provide a wide range of view, the matrix specifically may have at least 10 rows, preferably at least 500 rows, more preferably at least 1000 rows. Similarly, the matrix may have at least 10 columns, preferably at least 500 columns, more preferably at least 1000 columns. The matrix may comprise at least 50 optical sensors, preferably at least 100000 optical sensors, more preferably at least 5000000 optical sensors. The matrix may comprise a number of pixels in a multi-mega pixel range. Other embodiments, however, are feasible. Thus, in setups in which an axial rotational symmetry is to be expected, circular arrangements or concentric arrangements of the optical sensors of the matrix, which may also be referred to as pixels, may be preferred.

[0187] Thus, as an example, the sensor element may be part of or constitute a pixelated optical device. For example, the sensor element may be and/or may comprise at least one CCD and/or CMOS device. As an example, the sensor element may be part of or constitute at least one CCD and/or CMOS device having a matrix of pixels, each pixel forming a light-sensitive area. The sensor element may employ a rolling shutter or global shutter method to read out the matrix of optical sensors.

[0188] The detector 110 further may comprise at least one transfer device 132. The detector 110 may further comprise one or more additional elements such as one or more additional optical elements. The detector 110 may comprise at least one optical element selected from the group consisting of: transfer device, such as at least one lens and/or at least one lens system, at least one diffractive optical element. The transfer device 132 may be adapted to guide the light beam onto the optical sensor. The transfer device 132 specifically may comprise one or more of: at least one lens, for example at least one lens selected from the group consisting of at least one focus-tunable lens, at least one aspheric lens, at least one spheric lens, at least one Fresnel lens; at least one diffractive optical element; at least one concave mirror; at least one beam deflection element, preferably at least one mirror; at least one beam splitting element, preferably at least one of a beam splitting cube or a beam splitting mirror; at least one multi-lens system. The focal length constitutes a measure of an ability of the transfer device 132 to converge an impinging light beam. Thus, the transfer device 132 may comprise one or more imaging elements which can have the effect of a converging lens. By way of example, the transfer device 132 can have one or more lenses, in particular one or more refractive lenses, and/or one or more convex mirrors. In this example, the focal length may be defined as a distance from the center of the thin refractive lens to the principal focal points of the thin lens. For a converging thin refractive lens, such as a convex or biconvex thin lens, the focal length may be considered as being positive and may provide the distance at which a beam of collimated light impinging the thin lens as the transfer device may be focused into a single spot. Additionally, the transfer device 132 can comprise at least one wavelength-selective element, for example at least one optical filter. Additionally, the transfer device can be designed to impress a predefined beam profile on the electromagnetic radiation, for example, at the location of the sensor region and in particular the sensor area. The abovementioned optional embodiments of the transfer device 132 can, in principle, be realized individually or in any desired combination.

[0189] The transfer device 132 may have an optical axis. In particular, the detector 110 and the transfer device 132 have a common optical axis 122. The optical axis 122 may be a line of symmetry of the optical setup of the detector 110. The transfer device 132 may constitute a coordinate system 134, wherein a longitudinal coordinate is a coordinate along the optical axis 122 and wherein d is a spatial offset from the optical axis 122. The coordinate system 134 may be a polar coordinate system in which the optical axis of the transfer device 132 forms a z-axis and in which a distance from the z-axis and a polar angle may be used as additional coordinates. A direction parallel or antiparallel to the z-axis may be considered a longitudinal direction, and a coordinate along the z-axis may be considered a longitudinal coordinate. Any direction perpendicular to the z-axis may be considered a transversal direction, and the polar coordinate and/or the polar angle may be considered a transversal coordinate.

[0190] The optical sensor 118 is configured for determining the at least one image 128 of the area 114. The image 128 comprises a plurality of reflection features 130 generated by the area in response to illumination by the illumination features. The image 128 may be at least one two-dimensional image. The image 128 may be an RGB (red green blue) image.

[0191] The detector comprises the at least one evaluation device 136. The evaluation device 136 is configured for evaluating the image 128. The evaluation device 136 may comprise at least one data processing device and, more preferably, by using at least one processor and/or at least one application-specific integrated circuit. Thus, as an example, the at least one evaluation device 136 may comprise at least one data processing device having a software code stored thereon comprising a number of computer commands. The evaluation device 136 may provide one or more hardware elements for performing one or more of the named operations and/or may provide one or more processors with software running thereon for performing one or more of the named operations. Operations, including evaluating the images. Specifically the determining a beam profile and indication of the surface, may be performed by the at least one evaluation device 136. Thus, as an example, one or more instructions may be implemented in software and/or hardware. Thus, as an example, the evaluation device 136 may comprise one or more programmable devices such as one or more computers, application-specific integrated circuits (ASICs), Digital Signal Processors (DSPs), or Field Programmable Gate Arrays (FPGAs) which are configured to perform the above-mentioned evaluation. Additionally or alternatively, however, the evaluation device 136 may also fully or partially be embodied by hardware.

[0192] The evaluation device 136 and the detector 110 may fully or partially be integrated into a single device. Thus, generally, the evaluation device 136 also may form part of the detector 110. Alternatively, the evaluation device 136 and the detector 110 may fully or partially be embodied as separate devices. The detector 110 may comprise further components.

[0193] The evaluation device 136 is configured for determining at least one depth map of the area 114 by determining at least one depth information for each of the reflection features 130. The evaluation device 136 may be configured for determining the depth information for each of the reflection features 130 by one or more of the following techniques: depth-from-photon-ratio, structured light, beam profile analysis, time-of-flight, shape-from-motion, depth-from-focus, triangulation, depth-from-defocus, stereo sensors. The evaluation device 136 may be configured for considering the depth information for segmenting the object 116, in particular a hand region in the image 128, from the background in terms of the depth map.

[0194] For example, the evaluation device 136 may be configured for determining the depth information for each of the reflection features 130 by using depth-from-photon-ratio technique. Each of the reflection features 130 may comprise at least one beam profile, also denoted reflection beam profile. The beam profile may be selected from the group consisting of a trapezoid beam profile; a triangle beam profile; a conical beam profile and a linear combination of Gaussian beam profiles. The evaluation device 136 may be configured for determining depth information for each of the reflection features 130 by analysis of their beam profiles.

[0195] The evaluation device 136 may be configured for determining at least one longitudinal coordinate z.sub.DPR for each of the reflection features 130 by analysis of their beam profiles. For example, the analysis of the beam profile may comprise at least one of a histogram analysis step, a calculation of a difference measure, application of a neural network, application of a machine learning algorithm. The evaluation device 136 may be configured for symmetrizing and/or for normalizing and/or for filtering the beam profile, in particular to remove noise or asymmetries from recording under larger angles, recording edges or the like. The evaluation device 136 may filter the beam profile by removing high spatial frequencies such as by spatial frequency analysis and/or median filtering or the like. Summarization may be performed by center of intensity of the light spot and averaging all intensities at the same distance to the center. The evaluation device 136 may be configured for normalizing the beam profile to a maximum intensity, in particular to account for intensity differences due to the recorded distance. The evaluation device 136 may be configured for removing influences from background light from the beam profile, for example, by an imaging without illumination.

[0196] The evaluation device 136 may be configured for determining the longitudinal coordinate z.sub.DPR for each of the reflection features by using depth-from-photon-ratio technique. With respect to depth-from-photon-ratio (DPR) technique reference is made to WO 2018/091649 A1, WO 2018/091638 A1 and WO 2018/091640 A1, the full content of which is included by reference. The evaluation device 136 may be configured for executing at least one depth-from-photon-ratio algorithm which computes distances for all reflection features 130 with zero order and higher order.

[0197] The evaluation of the image 128 comprises identifying the reflection features 130 of the image 128. The evaluation device 136 may be configured for performing at least one image analysis and/or image processing in order to identify the reflection features 130. The image analysis and/or image processing may use at least one feature detection algorithm. The image analysis and/or image processing may comprise one or more of the following: a filtering; a selection of at least one region of interest; a formation of a difference image between an image created by the sensor signals and at least one offset; an inversion of sensor signals by inverting an image created by the sensor signals; a formation of a difference image between an image created by the sensor signals at different times; a background correction; a decomposition into color channels; a decomposition into hue; saturation; and brightness channels; a frequency decomposition; a singular value decomposition; applying a blob detector; applying a corner detector; applying a Determinant of Hessian filter; applying a principle curvature-based region detector; applying a maximally stable extremal regions detector; applying a generalized Hough-transformation; applying a ridge detector; applying an affine invariant feature detector; applying an affine-adapted interest point operator; applying a Harris affine region detector; applying a Hessian affine region detector; applying a scale-invariant feature transform; applying a scale-space extrema detector; applying a local feature detector; applying speeded up robust features algorithm; applying a gradient location and orientation histogram algorithm; applying a histogram of oriented gradients descriptor; applying a Deriche edge detector; applying a differential edge detector; applying a spatio-temporal interest point detector; applying a Moravec corner detector; applying a Canny edge detector; applying a Laplacian of Gaussian filter; applying a Difference of Gaussian filter; applying a Sobel operator; applying a Laplace operator; applying a Scharr operator; applying a Prewitt operator; applying a Roberts operator; applying a Kirsch operator; applying a high-pass filter; applying a low-pass filter; applying a Fourier transformation; applying a Radon-transformation; applying a Hough-transformation; applying a wavelet-transformation; a thresholding; creating a binary image. The region of interest may be determined manually by a user or may be determined automatically, such as by recognizing a feature within the image 128 generated by the optical sensor 118.

[0198] For example, the illumination source 112 may be configured for generating and/or projecting a cloud of points such that a plurality of illuminated regions is generated on the optical sensor 118, for example the CMOS detector. Additionally, disturbances may be present on the optical sensor 118 such as disturbances due to speckles and/or extraneous light and/or multiple reflections. The evaluation device 136 may be adapted to determine at least one region of interest, for example one or more pixels illuminated by the light beam which are used for determination of the longitudinal coordinate of the object. For example, the evaluation device 136 may be adapted to perform a filtering method, for example, a blob-analysis and/or an edge filter and/or object recognition method.

[0199] The evaluation device 136 may be configured for performing at least one image correction. The image correction may comprise at least one background subtraction. The evaluation device 136 may be adapted to remove influences from background light from the beam profile, for example, by an imaging without further illumination.

[0200] The evaluation device 136 is configured for finding the object 116 by identifying the reflection features 130 which were generated by illuminating biological tissue. The evaluation device 136 is configured for determining the at least one reflection beam profile of each of the reflection features 130. The evaluation device 136 is configured for identifying a reflection feature 130 as to be generated by illuminating biological tissue in case its reflection beam profile fulfills at least one predetermined or predefined criterion. The evaluation device 136 is configured for identifying the reflection feature 130 as to be background otherwise.

[0201] The detector 110 may be a device for detection, in particular optical detection, of biological tissue, in particular of human skin. The identification of being generated by biological tissue may comprise determining and/or validating whether a surface to be examined or under test is or comprises biological tissue, in particular human skin, and/or to distinguish biological tissue, in particular human skin, from other tissues, in particular other surfaces, and/or distinguishing different types of biological tissue such as distinguishing different types of human tissue e.g. muscle, fat, organs, or the like. For example, the biological tissue may be or may comprise human tissue or parts thereof such as skin, hair, muscle, fat, organs, or the like. For example, the biological tissue may be or may comprise animal tissue or a part thereof such as skin, fur, muscle, fat, organs, or the like. For example, the biological tissue may be or may comprise plant tissue or a part thereof. The detector 110 may be adapted to distinguish animal tissue or parts thereof from one or more of inorganic tissue, metal surfaces, plastics surfaces, for example of farming machines or milking machines. The detector 110 may be adapted to distinguish plant tissue or parts thereof from one or more of inorganic tissue, metal surfaces, plastics surfaces, for example of farming machines. The detector 110 may be adapted to distinguish food and/or beverage from dish and/or glasses. The detector 110 may be adapted to distinguish different types of food such as a fruit, meat, and fish. The detector 110 may be adapted to distinguish a cosmetics product and/or, an applied cosmetics product from human skin. The detector 110 may be adapted to distinguish human skin from foam, paper, wood, a display, a screen. The detector 110 may be adapted to distinguish human skin from cloth. The detector 110 may be adapted to distinguish a maintenance product from material of machine components such metal components etc. The detector 110 may be adapted to distinguish organic material from inorganic material. The detector 110 may be adapted to distinguish human biological tissue from surfaces of artificial or non-living objects. The detector 110 may be used, in particular, for non-therapeutic and non-diagnostic applications.

[0202] The predetermined or predefined criterion may be or may comprise at least one property and/or value suitable to distinguish biological tissue, in particular human skin, from other materials. The predetermined or predefined criterion may be or may comprise at least one predetermined or predefined value and/or threshold and/or threshold range referring to a material property. The reflection feature 130 may be indicated as to be generated by biological tissue in case the reflection beam profile fulfills at least one predetermined or predefined criterion. The indication may be an arbitrary indication such as an electronic signal and/or at least one visual or acoustic indication.

[0203] The evaluation device 136 may be configured for determining at least one material property m of the object by evaluating the beam profile of the reflection feature. The material property may be or may comprise at least one arbitrary property of the material configured for characterizing and/or identification and/or classification of the material. For example, the material property may be a property selected from the group consisting of: roughness, penetration depth of light into the material, a property characterizing the material as biological or non-biological material, a reflectivity, a specular reflectivity, a diffuse reflectivity, a surface property, a measure for translucence, a scattering, specifically a back-scattering behavior or the like. The at least one material property may be a property selected from the group consisting of: a scattering coefficient, a translucency, a transparency, a deviation from a Lambertian surface reflection, a speckle, and the like. The identifying at least one material property may comprise one or more of determining and assigning the material property to the object. The detector 110 may comprise at least one database comprising a list and/or table, such as a lookup list or a lookup table, of predefined and/or predetermined material properties. The list and/or table of material properties may be determined and/or generated by performing at least one test measurement using the detector according to the present invention, for example by performing material tests using samples having known material properties. The list and/or table of material properties may be determined and/or generated at the manufacturer site and/or by the user of the detector 110. The material property may additionally be assigned to a material classifier such as one or more of a material name, a material group such as biological or non-biological material, translucent or non-translucent materials, metal or non-metal, skin or non-skin, fur or non-fur, carpet or non-carpet, reflective or non-reflective, specular reflective or non-specular reflective, foam or non-foam, hair or non-hair, roughness groups or the like. The detector 110 may comprise at least one database comprising a list and/or table comprising the material properties and associated material name and/or material group.

[0204] For example, without wishing to be bound by this theory, human skin may have a reflection profile, also denoted back scattering profile, comprising parts generated by back reflection of the surface, denoted as surface reflection, and parts generated by very diffuse reflection from light penetrating the skin, denoted as diffuse part of the back reflection. With respect to reflection profile of human skin reference is made to “Lasertechnik in der Medizin: Grundlagen, Systeme, Anwendungen”, “Wirkung von Laserstrahlung auf Gewebe”, 1991, pages 10 171 to 266, Jürgen Eichler, Theo Seiler, Springer Verlag, ISBN 0939-0979. The surface reflection of the skin may increase with the wavelength increasing towards the near infrared. Further, the penetration depth may increase with increasing wavelength from visible to near infrared. The diffuse part of the back reflection may increase with penetrating depth of the light. These properties may be used to distinguish skin from other materials, by analyzing the back scattering profile.

[0205] Specifically, the evaluation device 136 may be configured for comparing the reflection beam profile with at least one predetermined and/or prerecorded and/or predefined beam profile. The predetermined and/or prerecorded and/or predefined beam profile may be stored in a table or a lookup table and may be determined e.g. empirically, and may, as an example, be stored in at least one data storage device of the detector. For example, the predetermined and/or prerecorded and/or predefined beam profile may be determined during initial start-up of a mobile device comprising the detector. For example, the predetermined and/or prerecorded and/or predefined beam profile may be stored in at least one data storage device of the mobile device, e.g. by software, specifically by the app downloaded from an app store or the like. The reflection feature 130 may be identified as to be generated by biological tissue in case the reflection beam profile and the predetermined and/or prerecorded and/or predefined beam profile are identical. The comparison may comprise overlaying the reflection beam profile and the predetermined or predefined beam profile such that their centers of intensity match. The comparison may comprise determining a deviation, e.g. a sum of squared point to point distances, between the reflection beam profile and the predetermined and/or prerecorded and/or predefined beam profile. The evaluation device 136 may be adapted to compare the determined deviation with at least one threshold, wherein in case the determined deviation is below and/or equal the threshold the surface is indicated as biological tissue and/or the detection of biological tissue is confirmed. The threshold value may be stored in a table or a lookup table and may be determined e.g. empirically and may, as an example, be stored in at least one data storage device of the detector 110.

[0206] Additionally or alternatively, for identification if the reflection feature 130 was generated by biological tissue, the evaluation device 136 may be configured for applying at least one image filter to the image 128 of the area 114. The evaluation device 136 may be configured for determining at least one material feature φ.sub.2m by applying at least one material dependent image filter ϕ.sub.2 to the image. The material feature may be or may comprise at least one information about the at least one material property of the surface of the area 114 having generated the reflection feature 130.

[0207] The material dependent image filter may be at least one filter selected from the group consisting of: a luminance filter; a spot shape filter; a squared norm gradient; a standard deviation; a smoothness filter such as a Gaussian filter or median filter; a grey-level-occurrence-based contrast filter; a grey-level-occurrence-based energy filter; a grey-level-occurrence-based homogeneity filter; a grey-level-occurrence-based dissimilarity filter; a Law's energy filter; a threshold area filter; or a linear combination thereof; or a further material dependent image filter ϕ.sub.2other which correlates to one or more of the luminance filter, the spot shape filter, the squared norm gradient, the standard deviation, the smoothness filter, the grey-level-occurrence-based energy filter, the grey-level-occurrence-based homogeneity filter, the grey-level-occurrence-based dissimilarity filter, the Law's energy filter, or the threshold area filter, or a linear combination thereof by |ρ.sub.ϕ2other,ϕm|≥0.40 with ϕ.sub.m being one of the luminance filter, the spot shape filter, the squared norm gradient, the standard deviation, the smoothness filter, the grey-level-occurrence-based energy filter, the grey-level-occurrence-based homogeneity filter, the grey-level-occurrence-based dissimilarity filter, the Law's energy filter, or the threshold area filter, or a linear combination thereof. The further material dependent image filter ϕ.sub.2other may correlate to one or more of the material dependent image filters ϕ.sub.m by |ρ.sub.ϕ2other,ϕm|≥0.60, preferably by |ρ.sub.ϕ2other,ϕm|≥0.80.

[0208] The evaluation device 136 may be configured for using at least one predetermined relationship between the material feature φ.sub.2m and the material property of the surface having generated the reflection feature 130 for determining the material property of the surface having generated the reflection feature 130. The predetermined relationship may be one or more of an empirical relationship, a semi-empiric relationship and an analytically derived relationship. The evaluation device 136 may comprise at least one data storage device for storing the predetermined relationship, such as a lookup list or a lookup table.

[0209] The evaluation device 136 is configured for identifying a reflection feature 130 as to be generated by illuminating biological tissue in case its corresponding material property fulfills the at least one predetermined or predefined criterion. The reflection feature 130 may be identified as to be generated by biological tissue in case the material property indicates “biological tissue”.

[0210] The reflection feature 130 may be identified as to be generated by biological tissue in case the material property is below or equal at least one threshold or range, wherein in case the determined deviation is below and/or equal the threshold the reflection feature is identified as to be generated by biological tissue and/or the detection of biological tissue is confirmed. At least one threshold value and/or range may be stored in a table or a lookup table and may be determined e.g. empirically and may, as an example, be stored in at least one data storage device of the detector. The evaluation device 136 is configured for identifying the reflection feature as to be background otherwise. Thus, the evaluation device 136 may be configured for assigning each projected spot with a depth information and a material property, e.g. skin yes or no.

[0211] The material property may be determined by evaluating φ.sub.2m subsequently after determining of the longitudinal coordinate z such that the information about the longitudinal coordinate z can be considered for evaluating of φ.sub.2m.

[0212] The evaluation device 136 is configured for segmenting the image 128 of the area 114 by using at least one segmentation algorithm. The segment may comprise a set of pixels. The segmenting may comprise a process of partitioning the image 128 into multiple segments. The segmentation may comprise assigning at least one label to every pixel of the image 128 such that pixels with the same label share at least one characteristic. The labels may be assigned under a predefined target. The segmentation may be a binary segmentation. The binary segmentation may comprise labeling the pixels of the image 128 as “skin-pixels” 138 and “background pixel” 140. All non-skin pixels may be considered as background. Skin-pixels may be considered as foreground seeds, also denoted seed points, serving as input for an image-based segmentation algorithm. With respect to image segmenting reference is made to https://en.wikipedia.org/wiki/Image_segmentation. The segmentation algorithm may comprise at least one region growing image segmentation method.

[0213] The segmentation algorithm may be configured for fusioning and/or considering a-priory knowledge. Specifically, the segmentation algorithm may be configured for fusioning and/or considering the material information determined from the beam profile analysis. This may allow to obtain or filter position data for motions of visible skin only. In contrast, in conventional setups, a recognition and separation of e.g. what is a finger/arm and what is not, needs to be done first. In addition to using the material information, the evaluation device 136 may be configured for considering the depth information for segmenting the object, in particular a hand region in the image, from the background in terms of the depth map.

[0214] The segmentation algorithm may be based on energy or cost functions such as one or more of graph cut, level-set, fast marching and Markov random field approaches. Segmentation of the image may be driven by color homogeneity and edge indicators, wherein the seed points constitute edge- and color homogeneity criterions. The reflection features 130 identified as to be generated by illuminating biological tissue are used as seed points and the reflection features identified as background are used as background seed points for the segmentation algorithm. These seed points may constitute, edge- and color homogeneity criterions, and thus provide an appropriate initialization of the segmentation of the target: “Which color and/or reflection and/or appearance has the skin in the image?”.

[0215] For example, graph cut segmentation may be used. Graph cut segmentation may be configured for fusioning the a-priory knowledge as it additionally provides real-time capability when combining it with pre-clustering algorithms such as superpixels, see R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and S. Susstrunk, “Slic superpixels compared to state-of-the-art superpixel methods,” IEEE TPAMI, vol. 34, no. 11, 2012. For example, a modified lazy snapping graph cut may be used. With respect to modified lazy snapping graph cut reference is made to Li, Yin and Sun, Jian and Tang, Chi-Keung and Shum, Heung-Yeung, “Lazy Snapping”, ACM Trans. Graph., vol. 23, no. 3, 2004. The Gibbs energy term may comprise an additional term to incorporate depth information for the graph cut segmentation routine. Instead of watershed-based pre-segmentation as proposed in Li, Yin and Sun, Jian and Tang, Chi-Keung and Shum, Heung-Yeung, “Lazy Snapping”, ACM Trans, a superpixel clustering may be used to ensure real-time capability.

[0216] The image 128 before the segmentation is shown in FIG. 2A. Each projected spot is assigned with a depth and a material property (skin yes/no). To obtain the binary segmentation illustrated in FIG. 2B, all non-skin points are considered as background, skin-points are considered as foreground seeds serving as input for an image-based segmentation routine. In this example, a modified lazy snapping graph cut was used. The Gibbs energy term includes an additional term to incorporate depth information for the graph cut segmentation routine. Furthermore, a superpixel clustering was used to ensure real-time capability.

[0217] The evaluation device 136 is configured for determining position and/or orientation of the object 116 in space considering the segmented image and the depth map. The evaluation device 136 may be configured for determining at least one hand pose or gesture from the position and/or the orientation of the object 116 in space. The evaluation device 136 may be configured for recognition of gestures from the segmented image, in particular from the parts of the segmented image labeled as object 116. The evaluation device 136 may be configured for identifying image coordinates of palm and finger in the segmented image. Palm and finger identification may be obtained from the segmentation. The evaluation device 136 may be configured for extracting features such as color and/or brightness and/or gradient values from the parts of the segmented image labeled as object. For example, palm and finger tips may be detected using standard OpenCV routines.

[0218] The evaluation device 136 may be configured for determining at least one three-dimensional finger vector 141 considering image coordinates of palm 142, such as of a center point 144 of the palm, and finger tip 146 and the depth map. FIG. 2C shows an embodiment of the 3D finger vector 140. The 3D finger vector 140 may specify the position and orientation of the finger in space. The evaluation device 136 may be configured for determining the 3D finger vector 140 from the image 128 and depth map. The 3D finger vector 140 may provide the basis for hand pose estimation and scene interpretation. For the recognition of hand gestures, the evaluation device 136 may be configured for using at least one classifier such as a hidden Markov model (HMM), a support vector machine (SVM), a conditional random field (CRF) and the like. The classifier may be configured for discriminating hand gestures.

[0219] The use of the beam profile analysis sensor fusion concept according to the present invention have the advantages for providing robust detection results, no need for complex model assumptions, a single sensor concept, easy integration and real-time capability.

LIST OF REFERENCE NUMBERS

[0220] 110 detector [0221] 112 illumination source [0222] 114 area [0223] 116 object [0224] 118 optical sensor [0225] 120 housing [0226] 122 optical axis [0227] 124 Light beam [0228] 126 light-sensitive area [0229] 128 image [0230] 130 Reflection feature [0231] 132 transfer device [0232] 134 coordinate system [0233] 136 evaluation device [0234] 138 skin-pixels [0235] 140 background pixel [0236] 141 three-dimensional finger vector [0237] 142 palm [0238] 144 center point [0239] 146 finger tip