VISION CORRECTION DISPLAY DEVICE, EYE-TRACKING SYSTEM AND METHOD TO COMPENSATE FOR VISUAL IMPAIRMENTS

Abstract

The invention relates to a display device for displaying an image for a user having a visual impairment. The display device comprises a pixelated display, a processor, an eye-tracking system and a microlens array. The invention is also related to a method for displaying an image for a user having a visual impairment on a display device. The invention is further related to a user having a visual impairment and a computer program for performing the method of the invention.

Claims

1. Display device (1) adapted for displaying an image (2) for a user having a visual impairment, the display device (1) comprising: a pixelated display (3) comprising an array of pixels (4), each pixel comprising a number of subpixels, and wherein the pixelated display (3) is operable to display the image (2) to an eye (5) of the user located at a viewing location relative to a position of the pixelated display (3); a processor (6) for operating the pixels (4) of the pixelated display (3) to display the image (2) on the pixelated display (3) wherein the processor (6) is operable to store information on the eye (5) of the user focal length of the eye; an eye-tracking system (7) adapted for determining changes in the viewing location of the eye (5) of the user relative to the position of the pixelated display (3), wherein, preferably, the processor is configured for determining the change in viewing location of the eye of the user based on the change in viewing location detected by the eye-tracking system; and a microlens array (8) provided on the pixelated display (3), wherein the microlens array (8) comprises microlenses (9) that are arranged in a two-dimensional plane oriented parallel to the pixelated display (3), wherein the microlens array (8) covers, preferably completely covers, the pixelated display (3), and wherein the microlens array (8) is configured to project light emitted by the pixelated display (3) towards the eye (5) of the user in the viewing location, characterized in that the processor (6) is configured for determining a first location of a virtual image plane (10), corresponding to a first viewing location of the eye (5) of the user, by comparing the focal length of the eye of the user to the first viewing location, wherein a virtual image (11) is located on the virtual image plane (10) which virtual image (11) corresponds to an image (2) to be displayed on the pixelated display (3), wherein the virtual image (11) determines values of virtual pixels (12) of the virtual image plane (10), wherein the processor (6) is configured for relating a subpixel (13) of the pixelated display (3) to a corresponding first virtual pixel (14) of the virtual image plane (10) by determining between the corresponding subpixel (13), the microlens array (8), the retina of the eye (5) of the user, and the first virtual pixel (14) a light path (15) of a light ray emitted by the corresponding subpixel (13), wherein the processor (6) is further configured for determining a first value of the corresponding subpixel (13) of the pixelated display (3) by comparing the value of the corresponding subpixel (13) to the value of the first virtual pixel (14), wherein the processor (6) is configured to operate the corresponding subpixel (13) of the display (3) based on the respective first value when the eye (5) of the user is in the first viewing location, wherein the processor (6) is configured for determining a second location of the virtual image plane (10), corresponding to a second viewing location of the eye (5) of the user, when the eye tracking system (7) determines a change in the viewing location of the eye (5) of the user from the first viewing location to the second viewing location, wherein the processor (6) is further configured to relate the corresponding subpixel (13) of the pixelated display (3) to a second virtual pixel (16) of the virtual image plane (10) when the virtual image plane (10) is in the second location, wherein the processor (6) is further configured for determining a second value of the corresponding subpixel (13) of the pixelated display (3) by comparing the value of the corresponding subpixel (13) to a value of the second virtual pixel (16); and wherein the processor (6) is configured to operate the corresponding subpixel (13) of the display (3) based on the second value when the eye (5) of the user is in the second viewing location.

2. Display device (1) according to claim 1, wherein the processor (6) is configured for relating the corresponding subpixel (13) to the second virtual pixel (16) of the virtual image plane (10) by: determining, when the virtual image plane (10) is in the first location, a first pixel location relative to the pixelated display (3) of the corresponding first virtual pixel (14); comparing, when the virtual image plane (10) is in the second location and for one or more of the virtual pixels (12), a location of each of the virtual pixels (12) to the first pixel location; and relating the corresponding subpixel (13) of the pixelated display (3) to the virtual pixel (16) whose location corresponds to the first pixel location when the virtual image plane (10) is in the second location.

3. Display device according to claim 2, wherein the processor (6) is configured for determining: a parallel component of the first pixel location parallel to the pixelated display (3); a perpendicular component of the first pixel location perpendicular to the pixelated display (3);parallel components of the locations of each of the virtual pixels (12) when the virtual image plane (10) is in the second location; and perpendicular components of the locations of each of the virtual pixels (12) when the virtual image plane (10) is in the second location, and wherein the processor (6) is configured for, when the perpendicular components are equal, relating the corresponding subpixel (13) of the pixelated display (3) to the virtual pixel (16) whose parallel component is equal to the parallel component of the first pixel location.

4. Display device (1) according to claim 3, wherein the processor (6) is configured for relating the corresponding subpixel (13) of the pixelated display (3) to the virtual pixel (16) whose parallel component is equal to the parallel component of the first pixel location when the eye-tracking system (7) does not detect a change of the eye (5) of the user in a perpendicular direction relative to the pixelated display (3).

5. Display device (1) according to one or more of the preceding claims, wherein the display device (1) is adapted to provide a feedback, e.g. a haptic feedback, to the user, e.g. prompting the user to change the position of the pixelated display (3).

6. Display device (1) according to one or more of the preceding claims, wherein the eye-tracking system (7) is further adapted for tracking a gaze of the eye (5) of the user for determining at which part of the pixelated display (3) the user is gazing.

7. Display device (1) according to claim 6, wherein the corresponding subpixel (13) is located in the part of the pixelated display (3) at which the user is gazing.

8. Display device (1) according to one or more of the preceding claims, wherein a microlens width and/or a microlens height is equal or smaller than a pitch of 8 subpixels (13) of the pixelated display (3).

9. Display device (1) according to one or more of the preceding claims, wherein the display device (1) is further provided with an optical gap layer (17), which is provided between the pixelated display (3) and the microlens array (8), wherein the optical gap layer (17) defines an optical gap (18) between the pixelated display (3) and the microlens array (8).

10. Display device (1) according to claim 9, wherein an optical gap layer is present, wherein for each lens of the microlens array, the relation:
smaller(microlens width,microlens height)/optical gappupil diameterpoint projection diameter)/object distance is minimized, wherein the smaller of the corresponding microlens width and microlens height is denoted by smaller(microlens width, microlens height), an estimate for the pupil size of the user is denote by pupil diameter, a size of a projection of an arbitrary point within a subpixel on the plane in which the pupil of the eye lies is denoted by point projection diameter, and a distance from the pixelated display to the eye of the user is denoted by object distance.

11. Display device (1) according to one or more of the claims 9-10, wherein the focal length of the microlenses (9) in the microlens array (8) is equal to the optical gap (18).

12. Display device (1) according to one or more of the claims 9-11, wherein the optical gap (18) comprises a secondary microlens array.

13. Display device (1) according to one or more of the preceding claims, wherein the microlenses (9) are rectangular microlenses (9).

14. Display device (1) according to one or more of the preceding claims, wherein the microlenses (9) are square microlenses (9).

15. Display device (1) according to one or more of the preceding claims, wherein the display device (1) further comprises a protection layer (19) which is placed on top of the microlens array (8).

16. Display device (1) according to one or more of the preceding claims, wherein the pixelated display (3) is provided in a smart phone case.

17. Method for displaying an image (2) for a user having a visual impairment wherein use is made of a display device (1) according to one or more of the preceding claims, wherein the display device (1) comprises: a pixelated display (3) comprising an array of pixels (4), each pixel comprising a number of subpixels, and wherein the pixelated display (3) is operable to display the image (2) to an eye (5) of the user located at a viewing location relative to a position of the pixelated display (3); a processor (6) for operating the pixels (4) of the pixelated display (3) to display the image (2) on the pixelated display (3) wherein the processor (6) is operable to store information on the eye (5) of the user focal length of the eye; an eye-tracking system (7) adapted for determining changes in the viewing location of the eye (5) of the user relative to the position of the pixelated display (3), wherein, preferably, the processor is configured for determining the change in viewing location of the eye of the user based on the change in viewing location detected by the eye-tracking system; and a microlens array (8) provided on the pixelated display (3), wherein the microlens array (8) comprises microlenses (9) that are arranged in a two-dimensional plane oriented parallel to the pixelated display (3), wherein the microlens array (8) covers, preferably completely covers, the pixelated display (3), and wherein the microlens array (8) is configured to project light emitted by the pixelated display (3) towards the eye (5) of the user in the viewing location, wherein the processor (6) is configured for determining a first location of a virtual image plane (10), corresponding to a first viewing location of the eye (5) of the user, by comparing the focal length of the eye of the user to the first viewing location, wherein a virtual image (11) is located on the virtual image plane (10) which virtual image (11) corresponds to an image (2) to be displayed on the pixelated display (3), wherein the virtual image (11) determines values of virtual pixels (12) of the virtual image plane (10), wherein the processor (6) is configured for relating a subpixel (13) of the pixelated display (3) to a corresponding first virtual pixel (14) of the virtual image plane (10) by determining between the corresponding subpixel (13), the microlens array (8), the retina of the eye (5) of the user, and the first virtual pixel (14) a light path (15) of a light ray emitted by the corresponding subpixel (13), wherein the processor (6) is further configured for determining a first value of the corresponding subpixel (13) of the pixelated display (3) by comparing the value of the corresponding subpixel (13) to the value of the first virtual pixel (14), wherein the processor (6) is configured to operate the corresponding subpixel (13) of the display (3) based on the respective first value when the eye (5) of the user is in the first viewing location, wherein the processor (6) is configured for determining a second location of the virtual image plane (10), corresponding to a second viewing location of the eye (5) of the user, when the eye tracking system (7) determines a change in the viewing location of the eye (5) of the user from the first viewing location to the second viewing location, wherein the processor (6) is further configured to relate the corresponding subpixel (13) of the pixelated display (3) to a second virtual pixel (16) of the virtual image plane (10) when the virtual image plane (10) is in the second location, wherein the processor (6) is further configured for determining a second value of the corresponding subpixel (13) of the pixelated display (3) by comparing the value of the corresponding subpixel (13) to a value of the second virtual pixel (16); and wherein the processor (6) is configured to operate the corresponding subpixel (13) of the display (3) based on the second value when the eye (5) of the user is in the second viewing location.

18. Method according to claim 17, wherein the method comprises: determining a first location of a virtual image plane (10) by comparing the focal length of the eye (5) of the user to the viewing location, wherein a virtual image (11) is located on the virtual image plane (10) which virtual image (11) corresponds to an image (2) to be displayed on the pixelated display (3), wherein the virtual image (11) determines values of virtual pixels (12) of the virtual image plane (10); relating a subpixel (13) of the pixelated display (3) to a corresponding first virtual pixel (14) of the virtual image plane (10) by determining between the corresponding subpixel (13), the microlens array (8), the retina of the eye (5) of the user, and the first virtual pixel (14) a light path (15) of a light ray emitted by the corresponding subpixel (13); determining a first value of the corresponding subpixel (13) of the pixelated display (3) by comparing the value of the corresponding subpixel (13) to the value of the first virtual pixel (14); operating the corresponding subpixel (13) of the display (3) based on the respective first value when the eye (5) of the user is in the first viewing location; determining a second location of the virtual image plane (10), corresponding to a second viewing location of the eye (5) of the user, based on a change in the viewing location of the eye (5) of the user determined by the eye-tracking system (7); relating the corresponding subpixel (13) of the pixelated display (3) to a second virtual pixel (16) of the virtual image plane (10) when the virtual image plane (10) is in the second location; determining a second value of the corresponding subpixel (13) of the pixelated display (3) by comparing the value of the corresponding subpixel (13) to a value of the second virtual pixel (16); and operating the corresponding subpixel (13) of the display (3) based on the second value when the eye (5) of the user is in the second viewing location.

19. Method according to claim 18, wherein the method further comprises: determining a first pixel location relative to the pixelated display (3) of the corresponding first virtual pixel (14) with the virtual image plane (10) is in the first location; comparing, when the virtual image plane (10) is in the second location and for one or more of the virtual pixels (12), a location of each of the virtual pixels (12) to the first pixel location; and relating the corresponding subpixel (13) of the pixelated display (3) to the virtual pixel (16) whose location corresponds to the first pixel location when the virtual image plane (10) is in the second location.

20. Method according to one or more of the claims 17-19, wherein the method comprises: providing a feedback, e.g. a haptic feedback, to the user, e.g. prompting the user to change the position of the pixelated display (3).

21. Method according to one or more of the claims 17-20, wherein the method comprises: tracking the location of the eye (5) of the user by the eye-tracking system (7) in a direction perpendicular to the pixelated display; changing the first location of the virtual image plane (10) based on a change in a location of the eye (5) of the user in the direction perpendicular to the pixelated display (3).

22. Method according to one or more of the claims 17-21, wherein the method comprises: tracking a gaze of the eye (5) of the user by the eye-tracking system (7) for determining at which part of the pixelated display (3) the user is looking; wherein the corresponding subpixel (13) is located in the part of the pixelated display (3) at which the user is gazing.

23. Processor (6) for operating a display device (1) adapted for displaying an image (2) for a user having a visual impairment according to one or more of the claims 1-16.

24. Computer program for performing the method according to claims 17-22.

25. Smart phone case comprising a display device (1) according to one or more of the claims 1-16 and adapted for operating the display device (1) according to the method according to one or more of the claims 17-22.

Description

[0106] FIG. 1 depicts a display device comprising a pixelated display;

[0107] FIG. 2 depicts a side view of a pixelated display covered by a microlens array;

[0108] FIG. 3 depicts a front view of a pixelated display schematically showing a number of pixels and subpixels;

[0109] FIG. 4 depicts a schematic view of a light path between the pixelated display and the virtual image plane behind the eye of the user;

[0110] FIG. 5 depicts a schematic view of a light path between the pixelated display, the eye of the user and the virtual image plane behind the pixelated display;

[0111] FIG. 6 depicts a schematic view of the eye of the user in a first viewing location and the eye of the user in a second viewing location;

[0112] FIG. 7 depicts a flow chart of the method for displaying an image on the pixelated display; and

[0113] FIG. 8 depicts a flow chart of a method for determining a second virtual pixel.

[0114] FIG. 1 depicts a display device 1 comprising a pixelate display 3. The pixelated display 3 comprises a number of pixels 4 which are schematically shown in FIG. 1. By emitting light at certain frequencies the pixels 4 may render an image 2 on the pixelated display 4. Each pixel 4 of the pixelated display 3 comprises a number, for example three, subpixels 13 which are not shown in FIG. 1.

[0115] The display device 1 further comprises a processor 6 for operating the pixels 4, e.g. by operating the subpixels 13. By operating the pixels 4 the image 2 may be displayed on the pixelated display 3. The processor is further operable to store information on the eye 5 of the user such as the focal length of the eye 5 of the user, the size of the eye 5 of the user, or the size of the pupil of the eye 5. This information may then be used by the processor 6 to determine values for the pixels 4, e.g. for the subpixels 13, to render the image 2 in a way that appears sharp to the user having the visual impairment.

[0116] The display device 1 further comprises an eye tracking system 7 adapted for determining changes in the viewing location of the eye 5 of the user relative to the position of the pixelated display 3. In other words the eye tracking system 7 tracks movement of the eye 5 of the user relative to the pixelated display 3.

[0117] The display device 1 of FIG. 1 is depicted wherein the pixelated display 3, the processor 6 and the eye-tracking system 7 all form part of the same apparatus. This is not necessarily the case, for example the processor 6 may be in a different location and communicate with the eye-tracking system 7 and the pixelated display 3 remotely, e.g. via internet or Bluetooth.

[0118] FIG. 2 depicts a side view of a pixelated display 3 covered by a microlens array 8. The microlens array 8 is separated from the pixelated display 3 by an optical gap layer 17 which defines an optical gap 18 between the pixelated display 3 and the microlens array.

[0119] The microlens array 8 comprises a number of microlenses 9. The microlenses 9 of FIG. 3 are depicted as plano-convex. However other types of microlenses 9 may be used. The microlenses 9 may vary in shape, microlens width and microlens height. In embodiments the focal length of the microlenses 9 is substantially equal to the optical gap layer 17 such that the pixelated display 3 lies at the focal length of the microlens array 8. In embodiments the microlens array 8 may be covered by a protection layer 19 such as a translucent plastic layer to protect the microlenses 9 from e.g. scratches and dust.

[0120] FIG. 3 depicts a front view of a pixelated display 3 schematically showing a number of pixels 4 and subpixels 13. Each pixel 4 of the pixelated display 3 in FIG. 3 comprises three subpixels 13. For example each subpixel corresponds to another color channel such as red, green and blue. By emitting light with different values from these subpixels 13 the pixels 4 may appear to emit light at any wavelength. Thus this allows an image 2 to be displayed on the pixelated display 3.

[0121] FIG. 4 depicts a schematic view of a light path 15 between the pixelated display 3 and the virtual image plane 10 behind the eye 5 of the user. The pixelated display 3 comprises a pixel 4 which comprises a number of subpixels 13. One of the subpixels 13 emits a light ray which travels along the light path 15 through the microlens array 8 and towards the eye 5 of the user. The processor 6 determines the light path 15 from the pixelated display 3 through the eye 5 of the user to the virtual image plane 10 using a technique such as ray tracing.

[0122] The virtual image plane 10 comprises a number of virtual pixels 12 whose values are determined by a virtual image 11 which is placed on the virtual image plane 10. The location of the virtual image plane 10 is determined by the processor 6 by comparing the focal length of the eye 5 of the user to the location wherein the eye 5 is, e.g. the first viewing location.

[0123] The light path 15 crosses the virtual image plane 10 in one of the virtual pixels 12. This may determine which of the virtual pixels 12 is the first virtual pixel 14. The value of the first virtual pixel 14 determines a value of the corresponding subpixel 13 when the eye of the user 5 is in the corresponding viewing location, e.g. in the first viewing location.

[0124] By determining a light path 15 for multiple subpixels 13 a value of each of these subpixels is determined by corresponding first virtual pixels 14. In this way a corrected image 2 may be displayed on the pixelated display 3 by the display device 1 which appears in focus to the user, e.g. based on the focal length of the eye 5 of the user, e.g. which focal length is effected by the visual impairment of the user.

[0125] FIG. 5 depicts a schematic view of a light path 15 between the pixelated display 3, the eye of the user and the virtual image plane 10 which is located behind the pixelated display 3. This embodiment shows an alternative method of finding a first virtual pixel 14 corresponding to a subpixel 13.

[0126] In FIG. 5 the virtual image plane 10 is located behind the pixelated display 3. In this case a light path 15 is determined by the processor 6 from the virtual image plane 10, through the pixelated display 3 and the microlens array 8, to the eye 5 of the user. This allows a corresponding virtual pixel 12, e.g. a corresponding first virtual pixel 14 if the eye 5 of the user is in the first viewing location, to be found for each subpixel 13 of the pixelated display 3.

[0127] FIG. 6 depicts a schematic view of the eye 5 of the user in a first viewing location and the eye 5 of the user in a second viewing location. The change in location of the eye 5, 5 has been detected by the eye-tracking system 7, and a second virtual pixel 16 is related to the corresponding subpixel 13 by the processor 6. In an embodiment of the invention the second virtual pixel 16 is related to the corresponding subpixel 13 by a light path 15. The light path 15 is between the subpixel 13 and the virtual image plane 10 in the second location thereof.

[0128] The processor 6 is configured for determining a second value of the corresponding subpixel 13 of the pixelated display 3 by comparing the value of the corresponding subpixel 13 to a value of the second virtual pixel 16. The processor 6 is further configured to operate the corresponding subpixel 13 of the display 3 based on the second value when the eye 5 of the user is in the second viewing location.

[0129] In another embodiment the second virtual pixel 16 is not determined by relating the second virtual pixel 16 to the subpixel 13 by a light path 15. In this embodiment a first pixel location of the first virtual pixel 12 is determined when the virtual image plane 10 is in the first location. A location of the virtual pixels 12 is compared to this first pixel location when the virtual image plane 10 is in the second location. The subpixel 13 of the display 3 is then related to the virtual pixel 16 whose location corresponds to the first pixel location when the virtual image plane 10 is in the second location.

[0130] Advantageously, in this embodiment it is not necessary to recalculate light paths 15 of light rays emitted by the subpixels 13 of the pixelate display. This reduces computational intensity significantly. As a result the display device 1 may determine a corrected image 2 faster so that the user experience is not affected negatively when the eye of the user moves relative to the pixelated display 3.

[0131] FIG. 7 depicts a flow chart of the method for displaying an image 2 on the pixelated display 3. The first step 101 of the method is to determine a first location of the virtual image plane 10 by comparing the focal length of the eye 5 of the user to the viewing location. A virtual image 11 may be located on the virtual image plane 10. The virtual image 11 corresponds to an image 2 to be displayed on the pixelated display 3 and the virtual image 11 determines values of virtual pixels 12 of the virtual image plane 10.

[0132] The second step 102 of the method relates a subpixel 13 of the pixelated display 3 to a corresponding first virtual pixel 12 of the virtual image plane 10 by determining between the corresponding subpixel 13, the microlens array 8, the retina of the eye 5 of the user, and the first virtual pixel 14 a light path 15 of a light ray emitted by the corresponding subpixel.

[0133] The third step 103 of the method is determining a first value of the corresponding subpixel 13 of the pixelated display 3 by comparing the value of the corresponding subpixel 13 to the value of the first virtual pixel 14.

[0134] The fourth step 104 of the method is operating the corresponding subpixel 13 of the display 3 based on the respective first value when the eye 5 of the user is in the first viewing location.

[0135] The fifth step 105 of the method is detecting a change in location of the eye 5 of the user by the eye-tracking system 7.

[0136] The sixth step 106 of the method is determining a second location of the virtual image plane corresponding to a second viewing location of the eye 5 of the user, based on the change in the viewing location of the eye 5 of the user determined by the eye-tracking system 7. Wherein the second viewing location of the eye 5 of the user is different from the first viewing location.

[0137] The seventh step 107 of the method is relating the corresponding subpixel 13 of the pixelated display to a second virtual pixel 16 of the virtual image plane 10 when the virtual image plane is in the second location.

[0138] The eight step 108 of the method is determining a second value of the corresponding subpixel 13 of the pixelated display 3 by comparing the value of the corresponding subpixel 13 to a value of the second virtual pixel 16.

[0139] The ninth step 109 of the method is operating the corresponding subpixel 13 of the display 3 based on the second value when the eye 5 of the user is in the second viewing location.

[0140] This method the user to move relative to the display device 1 without the user having a negative impact on the viewing experience as a result of viewing the corrected image 2 from different angles. The user may experience a negative impact on the viewing experience because the ability of the user to view the image 2 in focus is reduced depending on the viewing angle.

[0141] In embodiments wherein method steps 107-109 are repeated for each subpixel 13 of the pixelated display 3 an image 2 may be displayed on the pixelated display 3 that appears in focus for the user when the eye 5 of the user is in the second viewing location. The display device 1 thus has compensated the image 2 displayed on the pixelated display 3 for the new location of the eye 5 of the user. The invention thus allows an image 2 to be compensated when the user moves to a second viewing location.

[0142] By repeating the method steps 105-109 when the eye 5 of the user moves to a third viewing location the image 2 will continue to appear in focus as viewed from the third viewing location. The invention thus allows the image 2 displayed on the pixelated display 3 to appear in focus irrespective of the viewing location and irrespective of movement of the eye 5 of the user relative to the pixelated display 3. Thus the viewing experience is not negatively affected by movement relative to the pixelated display 3.

[0143] FIG. 8 depicts a flow chart of a method for determining a second virtual pixel 16 which may be performed while performing method steps 101-107. The first step 110 is determining a first pixel location relative to the pixelated display 3 of the corresponding first virtual pixel 14 when the virtual image plane 10 is in the first location.

[0144] The second method step 111 is determining a location of one or more of the virtual pixels 12 when the virtual image plane 10 is in the second location.

[0145] The third method step 112 is comparing, when the virtual image plane 10 is in the second location and for one or more of the virtual pixels 12, a location of each of the virtual pixels 12 to the first pixel location.

[0146] The fourth method step 113 is relating the corresponding subpixel 13 of the pixelated display 3 to the virtual pixel 16 whose location corresponds to the first pixel location when the virtual image plane 10 is in the second location.