Device and method for touchless palm print acquisition

Abstract

The present invention describes a device and method for touchless palmprint acquisition. The method can be applied to a terminal device (including but not limited to mobile phones, tablet computers and other portable devices) with a display screen and a front-facing camera. The camera is used to capture images of palms. The screen is used as a supplementary light source. When lighting condition in the palm area is undesirable, the screen changes its display to improve the lighting condition inside the palm, which increases the recognizability of the palmprint. The palmprint acquisition device and method described in the present invention can be directly applied to existing terminal devices without requiring any additional hardware. By adopting a variety of ways to change the display of the built-in screen of the terminal device, the lighting condition of the palm area will be improved and the recognizability of the palmprint will be increased.

Claims

1. A method for touchless acquisition of a palmprint using a terminal device with a display screen and a front-facing camera, comprising the steps of: displaying a static visual cue on the display screen with which a user is prompted to align an image of the user's palm, the static visual cue formed from a boundary of a change in color and/or transparency of a display screen image; capturing palm images using the front-facing camera; displaying a dynamic visual cue on the display screen, corresponding to a detected position of a user's palm in the palm images, the dynamic visual cue forming an area with a change in color and/or transparency of the display screen image relative to portions of the display screen outside the dynamic visual cue; and when a lighting condition is undesirable, changing the display on portions of the display screen outside the static visual cue and dynamic visual cue as a supplementary light source to improve the lighting condition in a palm area in order to increase the recognizability of the palmprint; wherein the step of changing the display comprises one or multiple of the following: (1) displaying a high-brightness background; and/or (2) displaying a blue or purple background.

2. The method of claim 1, further comprising displaying real-time feedback on the screen of the terminal device to guide the user to place the palm in a suitable position.

3. The method of claim 1, further comprising determining whether the lighting condition in the palm area is undesirable using one or more of the criteria as follows: (a) using a built-in ambient light sensor of the terminal device to measure the light intensity of the environment whereby when the measured light intensity is lower than or equal to a first preset value, it is determined that the lighting condition is undesirable; (b) analyzing the grayscale intensity of the captured image inside the palm area, whereby when the intensity is lower than or equal to a second preset value, it is determined that the lighting condition is undesirable; (c) analyzing the image contrast of the captured image inside the palm area, whereby when the contrast is lower than or equal to a third preset value, it is determined that the lighting condition is undesirable.

4. The method of claim 2, further comprising displaying a background simultaneously with the real-time feedback but on a different layer or in different subareas of the display screen.

5. The method of claim 2, wherein the real-time feedback comprises a real-time video displaying the subareas of the video captured by the front-facing camera corresponding to areas within the static visual cue and/or dynamic visual cue.

6. The method of claim 1, wherein the static visual cue and the dynamic visual cue each consist of one or more of: arrows, wireframes, filled figures and text.

7. The method of claim 1, in which the step of displaying a dynamic visual cue comprises: detecting whether a palm is present in the captured palm image, and if so, locating a detected palm area in the image, and visualizing the detected palm area as a dynamic visual cue on the screen.

8. The method of claim 5, wherein the step of changing the display on portions of the display screen outside the static visual cue and variable positioning cue comprises setting a background to be translucent and placed in front of real-time video captured by the camera corresponding to areas outside of the static visual cue and dynamic visual cue.

9. The method of claim 4, wherein a border between the background and the real-time video is used as the static visual cue.

10. A method for touchless acquisition of facial images using a terminal device with a display screen and a front-facing camera, comprising the steps of: displaying a static visual cue on the display screen with which a user is prompted to align an image of the user's face, the static visual cue formed from a boundary of a change in color and/or transparency of a display screen image; capturing facial images using the front-facing camera; displaying a dynamic visual cue on the display screen, corresponding to a detected position of a user's face in the facial images, the dynamic visual cue forming an area with a change in color and/or transparency of the display screen image relative to portions of the display screen outside the dynamic visual cue; and when a lighting condition is undesirable, changing the display on portions of the display screen outside the static visual cue and dynamic visual cue as a supplementary light source to improve the lighting condition in a face area in order to increase the recognizability of the facial image; wherein the step of changing the display comprises one or more of the following: (1) displaying a high-brightness background; and/or (2) displaying a blue or purple background.

11. A method for touchless acquisition of eye pattern images using a terminal device with a display screen and a front-facing camera, comprising the steps of: displaying a static visual cue on the display screen with which a user is prompted to align an image of the user's eye, the static visual cue formed from a boundary of a change in color and/or transparency of a display screen image; capturing eye pattern images using the front-facing camera; displaying a dynamic visual cue on the display screen, corresponding to a detected position of a user's eye in the eye pattern images, the dynamic visual cue forming an area with a change in color and/or transparency of the display screen image relative to portions of the display screen outside the dynamic visual cue; and when a lighting condition is undesirable, changing the display on portions of the display screen outside the static visual cue and dynamic visual cue as a supplementary light source to improve the lighting condition in an eye area in order to increase the recognizability of the eye pattern image; wherein the step of changing the display comprises one or more of the following: (1) displaying a high-brightness background; and/or (2) displaying a blue or purple background.

12. A device for touchless acquisition of a palmprint comprising a terminal device with a display screen and a front-facing camera, wherein: the terminal is configured to display on the display screen a static visual cue with which a user is prompted to align an image of the user's palm, the static visual cue formed from a boundary of a change in color and/or transparency of a display screen image; the front-facing camera is used to capture palm images of a palm and display on the display screen a dynamic visual cue corresponding to a detected position of the palm in the palm images; and the screen configured to adjust portions of a display thereon outside of the static visual cue and dynamic visual cue when a lighting condition is undesirable to improve the lighting condition of a palm area in order to increase recognizability of the palmprint within the palm images, by displaying a high-brightness background; and/or (2) displaying a blue or purple background.

13. The device of claim 12, further comprising a distance sensor that measures a distance between the palm and the front-facing camera.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) FIG. 1 shows a flow chart of touchless palmprint recognition using the presented palmprint acquisition method.

(2) FIG. 2 shows a diagram of touchless palmprint acquisition on a terminal device.

(3) FIG. 3 illustrates a display screen used in an embodiment of the touchless palmprint acquisition method.

(4) FIG. 3a illustrates one example of real-time feedback;

(5) FIG. 3b illustrates the second example of real-time feedback;

(6) FIG. 3c illustrates the third example of real-time feedback.

DETAILED DESCRIPTION OF THE DRAWINGS

(7) The specific embodiments of the present invention are described in detail below with reference to the drawings. However, it should be understood that the present invention is not limited to such embodiments described below, and the technical idea of the present invention can be implemented in combination with other well-known technologies or other technologies having the same functions as those of well-known technologies.

(8) In the description of the following specific embodiments, in order to clearly show the structure and working mode of the present invention, many descriptions involve directional words such as “front”, “back”, “left”, “right”, “outer” “ ”, “inner”, “outward”, “inward”, “axial”, “radial” etc. These directional should be understood as convenient terms, and should not be understood as limiting words.

Explanation of Terms

(9) High-brightness background: refers to a background with a certain brightness, i.e., the brightness of the screen display that is sufficient to illuminate the palm area for palm image acquisition. In the present invention, it can be achieved by increasing the brightness level of the screen and/or by displaying a high-brightness background on it. The high-brightness background described in the present invention can emit light that can improve the contrast of the palmprint image, such as blue light, violet light, or composite light containing blue and violet spectrums. In particular, the high-brightness background contains light having a wavelength between 400 nm and 480 nm. This high-brightness background facilitates the acquisition of high contrast palmprint instead of beautifying the image. This is fundamentally different from the prior art that uses a supplementary light source that is white, yellowish or reddish, etc.

(10) First preset value: in the present invention, the first preset value refers to a preset light intensity value. When the ambient light intensity of the environment is lower than or equal to the first preset value, the lighting condition is regarded as undesirable. A person skilled in the art can select the first preset value according to the camera configuration of the terminal device and the ambient light intensity. Herein the present invention does not make special restrictions. For instance, assuming that the minimum light intensity required for palmprint recognition is 10 lux, and the light intensity of the environment is less than 10 lux, the brightness level of the display screen can be changed to increase the light intensity of the palm area to be above 10 lux in order to obtain a clear palmprint image. In another example, an ambient light sensor is used to determine whether the light intensity of the environment meets the demand. If not, a high-brightness background is displayed, and the brightness level of the screen can be adjusted at the same time or later.

(11) Second preset value: in the present invention, the second preset value refers to a preset grayscale intensity value of the image. When the grayscale intensity of the palm area in the captured image is lower than or equal to the second preset value, the lighting condition is regarded as undesirable. The second preset value can be configured according to the target false rejection rate (FRR) and false acceptance rate (FAR) of palmprint recognition, the scene of palmprint acquisition, the chip specs of the terminal device, etc.

(12) Third preset value: in the present invention, the third preset value refers to a preset image contrast value. When the contrast value of the palm area in the acquired image is lower than or equal to the third preset value, the lighting condition is regarded as undesirable. The contrast value measures the difference of the grayscale intensity between the brightest and the darkest regions in the palm area of the image. The larger the difference, the greater the contrast. The third preset value can be configured according to the false rejection rate (FRR) and false acceptance rate (FAR) of palmprint recognition, the scene of palmprint acquisition, the chip specs of the terminal device, etc.

(13) Preferably, the comparison between the ambient light intensity of the environment and the first preset value, the grayscale intensity of the palm area in the image and the second preset value, the contrast value of the palm area in the image and the third preset value can be performed simultaneously. For example, the screen display can be changed when the grayscale intensity of the palm area is too low, or when the grayscale intensity is high enough but the contrast value is too low.

(14) Real-time video: in the present invention, the real-time video is a sequence of images captured using the front-facing camera. The displayed part can be the whole image or certain subareas of the image. The video can be captured continuously without being manually triggered by the user. When it is determined that the lighting condition is undesirable, the display can be continuously adjusted according to the live image of the real-time video. For example, the screen brightness can be gradually increased until the palm region is bright enough.

(15) Real-time feedback: in the present invention, the real-time feedback refers to the area on the display that is used to display guidance on palm placement, to view the real-time video of the palm, to indicate the current position of the detected palm, or to indicate the desired direction to move the palm. Feedback includes static visual cues, real-time video, dynamic visual cues, etc. These three components can be displayed alone or simultaneously. In the present invention the background and the real-time video can be displayed on different layers or in different areas of the screen depending on the requirements from the user or interface design.

(16) The specific embodiments of the present invention will be described in detail below in conjunction with FIGS. 1-3

Embodiment One

(17) FIG. 1 shows the workflow of palmprint recognition using the touchless palmprint acquisition method provided by the present invention. The processes include image acquisition, palm detection, image analysis and processing, and palmprint recognition.

(18) The user places the palm in front of the terminal device (including but not limited to mobile phones, tablet computers and other portable devices) with a screen and a front-facing camera. The camera captures camera image 101 frame by frame to obtain the user's palmprint. The palm detection module 102 detects whether the palm is present (P103) in the camera image 101. If the palm is not present, the module reports no palm detected (P115). If the palm is present, the module locates the palm area by detecting the position and size of the palm inside the image (P104), provides feedback (P116) and determines whether the detected palm meet the requirements for palmprint recognition (P105). For instance, if part of the palm is missing from the image or the palm area does not contain enough resolution, the feedback on adjusting the palm position is provided (P117). If the requirements are satisfied, an image with the palm area labelled is generated (106) and passed to the image analysis and processing module 107.

(19) In the above process, the touchless palmprint acquisition method provided by the present invention can be applied to terminal devices with a display screen and a front-facing camera. Without any additional hardware (including but not limited to an external infrared camera) palm detection can be performed by analyzing the image captured by the built-in front-facing camera in an open space without controlled lighting. Palm detection determines whether the palm is present in the camera image in real time. When the palm is present, its area in the camera image is accurately located and fed back to the user through the display in real time (P116). If the current palm position is undesirable (too far, or outside the camera field of view, etc.), a visual cue can be displayed to indicate the desired direction to move the palm to adjust its position (P117).

(20) The workflow of the image analysis and processing module 107 is as follows: first, the image quality is analyzed (P108) to determine whether the palm-labelled image 106 meets the requirements of palmprint recognition, such as whether the grayscale intensity and/or contrast are sufficient. If not, the feedback on adjusting camera parameters is provided (P118). If yes, the module continues to perform image preprocessing (P109) that consists of operations including but not limited to noise removal and image enhancement. The output preprocessed image 110 is then transmitted to the palmprint recognition module 111, which consists of the palmprint registration module 112 and the palmprint matching module 113.

(21) Palmprint registration module 112 performs palmprint registration. It creates a palmprint template 114 from the preprocessed image 110, saves the palmprint template 114 and reports a successful registration (P119). Palmprint matching module 113 verifies user identity by matching the palmprint template created from newly captured preprocessed image 110 with one or multiple palmprint templates 114 that have been registered in the system previously. Finally, the matching result is returned (120).

(22) The various modules, processes and feedback of palmprint acquisition for palmprint recognition described above are only exemplary descriptions. Those skilled in the art may appropriately replace or remove certain steps as needed to achieve the same effect.

Embodiment Two

(23) FIG. 2 shows a diagram of touchless palmprint acquisition using a terminal device. The terminal device with a screen and a front-facing camera consists of: one or more screens 201, one or more cameras 202 for capturing images, a terminal device main board 203, a trusted execution environment 206. The terminal device main board 203 includes a device processor 204 and a device storage 205. The trusted execution environment 206 includes an arithmetic module 207 and a storage module 208.

(24) Preferably, the terminal device with a screen and a front-facing camera may further comprise an ambient light sensor 213, a distance sensor 214, a motion sensor 215, and a flash light 216, wherein the ambient light sensor 213, the distance sensor 214, the motion sensor 215, the flash light 216 can all be connected to the device processor 204 (including but not limited to, central processing unit CPU and graphics processor GPU) through the terminal device main board 203, and the device storage 205 is used to store palmprint templates 114. The palmprint recognition module 209 runs on the terminal device main board 203 and includes a palm detection module 210, an image analysis and processing module 211, and a palmprint template registration and matching core module 212. The palmprint recognition module 209 is connected with peripheral hardware such as the camera 202 and screen 201. The palmprint recognition module 209 not only takes images and sensor inputs, but also controls and adjusts the parameters of peripheral hardware as well as feeds the processing results back to the user through the display screen.

(25) Preferably, the distance between the palm and the camera can be measured by a distance sensor 214. When the measured distance is less than a preset threshold, due to the limited camera field of view, only part of the palm is contained in the image. In this case, the palm detection module 102 may not be called. Feedback can be displayed through the screen 201 to instruct the user to check the current position of the palm. If the palm is too close to or outside the camera field of view, the user can adjust the palm position so that the entire palm or the majority part of the palm is visible in the camera image.

(26) The palmprint recognition module 209 is embedded in the device processor 204 and device storage 205 of the terminal device with a display and a front-facing camera.

(27) Preferably, when a trusted execution environment 206 is available in the terminal device, palmprint template registration and matching core module 212 may run completely or partially in the trusted execution environment 206. In the present invention, when the lighting condition is undesirable, the brightness level of the screen 201 can be adjusted to be high enough and/or display a high-brightness background for light filling.

Embodiment Three

(28) FIG. 3 illustrates a screen display used by the touchless palmprint acquisition method provided by the present invention. The screen 201 of the terminal device displays real-time feedback to the user to guide palm placement.

(29) FIG. 3a shows an example of real-time feedback that includes a static visual cue 33 for guiding the user to place the palm at a suitable position. The static visual cue 33 can include but is not limited to arrows, wireframes, filled figures, text, etc. Preferably, the static visual cue 33 may be a palm-shaped frame, or a geometric frame such as a rectangle, circle, oval, etc. such that the user can adjust and place the palm within the displayed frame. Preferably, the static visual cue 33 may be a palm-shaped translucent filled image. Preferably, a text description can be displayed to instruct the user to use the border between the static visual cue 33 and the background 32 as a guide to place the palm inside the static visual cue 33, or make the borders of the palm and the static visual cue 33 substantially coincide.

(30) Preferably, when the transparency of the background 32 is not uniform across different areas, the boundary of the changes in transparency can be used as a static visual cue 33. Preferably, the boundary of the changes of the transparency can be set as palm-shaped, rectangular, circular, elliptical, etc.

(31) FIG. 3b shows the second example of real-time feedback that includes a real-time video 31 displaying the whole or certain subareas of the images obtained by the front-facing camera. The user may use the real-time video 31 as feedback and together with static visual cue 33 to place the palm in a suitable position. Preferably, the background 32 is set to be translucent and be placed in front of the real-time video 31. The transparency of the background 32 may or may not be uniform in different areas.

(32) FIG. 3c shows the third example of real-time feedback that includes one or multiple dynamic visual cues 34 for guiding the positioning and/or the desired direction to move the palm. The dynamic visual cues 34 include but are not limited to arrows, wireframes, filled maps, and/or texts.

(33) Preferably, the dynamic visual cue 34 may include a palm-shaped frame, or a geometric-shaped frame (including but not limited to a rectangle, a circle, an ellipse, etc.). The dynamic visual cue can prompt the user whether the palm is detected, and if yes, the position of the detected palm.

(34) Preferably, the dynamic visual 34 may include a translucent filled image that is palm-shaped or geometric-shaped (including but not limited to rectangular, circular, elliptical, etc.). The dynamic visual cue can prompt the user whether the palm is detected, and if yes, the position of the detected palm.

(35) Preferably, the dynamic visual cue 34 may include an arrow or an arrow animation for guiding the user to move the palm in a specific direction, and/or the distance to be moved.

(36) Preferably, the dynamic visual cue 34 may include a text description for guiding the user to move the palm in a specific direction, and/or the distance to be moved. For example, the text can prompt the user to “please move the palm closer to the camera”.

(37) Preferably, the transparency of the background 32 and the real-time video area can be set to 0 and 1 respectively, i.e., the screen displays a high-brightness background in most areas, and displays the real-time video 31 in openings that are fully transparent. At the same time, the boundary of the above-mentioned opening (fully transparent area) can be used as a static visual cue 33 for guiding the user to place the palm.

(38) Preferably, the background 32 is arranged in a grid format with interleaving transparent and opaque areas. The real-time video 31 is displayed through the transparent areas of the grid.

Embodiment Four

(39) On a terminal device that uses the palmprint acquisition method provided by the present invention, if the light intensity of the environment is sufficient, the front-facing camera directly acquires the palmprint image. If the light intensity of the environment is insufficient and/or if there is directional light coming from the back or side of the palm which makes the lighting condition in the palm area undesirable, the captured camera image will have a low grayscale intensity or low contrast inside the palm area, and therefore will not meet the requirements of palmprint recognition. In this case, the screen 201 changes its display and use the screen light to improve the lighting conditions in the palm area in order to increase the recognizability of the palmprints.

(40) The present invention exemplarily provides the following scenarios on how to determine whether the lighting condition is undesirable:

(41) (a) The terminal device is equipped with an ambient light sensor. When the light intensity of the environment measured by the ambient light sensor is lower than or equal to a first preset value, it is determined that the lighting condition is undesirable;

(42) (b) The image analysis and processing module 107 analyzes the grayscale intensity of the captured camera image in the palm area. When the grayscale intensity is lower than or equal to a second preset value, it is determined that the lighting condition is undesirable; or

(43) (c) The image analysis and processing module 107 analyzes the contrast of the captured camera image in the palm area. When the contrast is lower than or equal to a third preset value, it is determined that the lighting condition is undesirable.

(44) When any of the above situations occurs, the lighting condition is undesirable for palmprint recognition. To improve the lighting condition, the brightness level of the screen 201 can be tuned higher, and a high-brightness background can be displayed, or a blue or purple background can be displayed for supplementary lighting. Preferably, the background 32 may be set to blue, that is, the light emitted by the screen is mainly in the blue spectrum. When palm is illuminated by blue light, its palmprint texture has a higher contrast in the camera image.

(45) Compared with the prior art, the touchless palmprint acquisition device and method presented in this invention have the following advantages:

(46) (1) The palmprint acquisition method of the present invention can be customized into an independent touchless palmprint recognition device, which is directly applicable to a terminal device using the built-in display screen, camera, ambient light sensor, distance sensors, processor chips etc. The hardware realizes palmprint detection, acquisition, identification, matching and feedback, and depending on the lighting condition, can provide supplementary lighting using the terminal device's own display screen;

(47) (2) The palmprint acquisition method of the present invention facilitates a better human-computer interaction. The screen has areas to display the palmprint image captured by the front-facing camera in real time, as well as static and/or dynamic visual cues to guide the user to adjust palm position, distance or posture.

(48) (3) The palmprint acquisition method of the present invention adopts a variety of ways to change the display of the screen. Using the display screen as a supplementary light source, the lighting condition of the palm area will be improved and the recognizability of the palmprint will be increased.

(49) (4) The palmprint acquisition method of the present invention makes the identification process more efficient, easier to operate, and lowers the requirements on the lighting environment. It is applicable to not only authentication and identification on a personal mobile device, but also to multi-person authentication on a terminal in public places.

(50) Unless otherwise specified, the qualifiers similar to “first” and “second” in this article do not refer to chronological order, quantity, or importance, but are merely intended to distinguish one feature of the present technical solution from another feature of the present technical solution. Similarly, the qualifiers similar to “one” in this article do not refer to the number, but describe a technical feature that has not appeared in the foregoing.

(51) While certain embodiments of the invention have been described herein in detail for purposes of clarity and understanding, the foregoing description and figures merely explain and illustrate the present invention and the present invention is not limited thereto. It will be appreciated that those skilled in the art, having the present disclosure before them, will be able to make modifications and variations to that disclosed herein without departing from the scope of the invention or appended claims.