Display apparatus
11699371 · 2023-07-11
Assignee
Inventors
- Franz Josef Maier (Poertschach am Woerthersee, AT)
- Manuel Dorfmeister (Wiesen, AT)
- Joerg REITTERER (Brunn Am Gebirge, AT)
- Louahab NOUI (East Sussex, GB)
Cpc classification
G09G2320/0247
PHYSICS
G02B2027/0147
PHYSICS
G09G2320/0233
PHYSICS
G09G2320/0242
PHYSICS
G09G2340/0435
PHYSICS
G09G2320/0261
PHYSICS
International classification
Abstract
A display apparatus comprises a mirror assembly, a first mirror of the mirror assembly oscillating about a first axis upon excitation by a first excitation signal and the first or a second mirror of the mirror assembly oscillating about a second axis upon excitation by a second excitation signal, a light source projecting a light beam onto the mirror assembly for deflection by the mirror assembly towards an image area, the light source being controlled according to pixels of image frames, a gaze tracker detecting a user's region of interest, ROI, within the image area, and a controller modulating one of the excitation signals by a first modulation signal which is dependent on the ROI detected by the gaze tracker.
Claims
1. A display apparatus, comprising: a mirror assembly, wherein a first mirror of the mirror assembly is configured to oscillate about a first axis upon excitation by a first excitation signal of a first frequency and wherein the first or a second mirror of the mirror assembly is configured to oscillate about a second axis upon excitation by a second excitation signal of a second frequency; a light source configured to project a light beam onto the mirror assembly for deflection by the mirror assembly towards an image area, the light source having an input via which the light source can be controlled according to pixels of image frames to be displayed on the image area with a frame rate, each pixel of the image area being hit by the light beam with a per-pixel refresh rate; a gaze tracker configured to detect a user's region of interest, ROI, within the image area by tracking a user's gaze; and a controller connected to the gaze tracker and configured to modulate one of the first and second excitation signals by a first modulation signal which is dependent on the ROI detected by the gaze tracker.
2. The display apparatus according to claim 1, wherein the dependency of the first modulation signal on the ROI is such that the per-pixel refresh rate is higher in the ROI than outside of the ROI.
3. The display apparatus according to claim 1, wherein the frequency of the first modulation signal is a one- or morefold of the frame rate.
4. The display apparatus according to claim 1, wherein the controller is configured to decrease the intensity of the light beam for a pixel in the ROI when the amount of time the light beam spends in that pixel during a frame increases, and vice versa.
5. The display apparatus according to claim 1, wherein the controller is configured to amplitude-modulate said one of the first and second excitation signals by the first modulation signal.
6. The display apparatus according to claim 1, wherein the gaze tracker is configured to detect the ROI by predicting the ROI from an analysis of a past track of the user's gaze on the image area.
7. The display apparatus according to claim 6, wherein the gaze tracker is configured to determine a duration of a past saccade of the user's gaze from the analysis, and wherein the controller is configured to complete a change of the first modulation signal from one ROI to another ROI within that duration.
8. The display apparatus according to claim 1, wherein the display apparatus is configured to be head-mounted and the gaze tracker is an eye tracker.
9. The display apparatus according to claim 1, wherein the controller has a memory with a look-up table which stores, for each one of a set of different ROIs within the image area, a respective first modulation signal dependent on that ROI, wherein the controller is configured to retrieve the first modulation signal dependent on the detected ROI from the look-up table.
10. The display apparatus according to claim 9, wherein the controller is configured to modulate the other one of the first and second excitation signals by a second modulation signal which is dependent on the ROI detected by the gaze tracker, and wherein the look-up table stores, for each one of a set of different ROIs within the image area, a respective second modulation signal dependent on that ROI, wherein the controller is configured to retrieve also the second modulation signal dependent on the detected ROI from the look-up table.
11. The display apparatus according to claim 1, wherein the controller is configured to modulate the other one of the first and second excitation signals by a second modulation signal which is dependent on the ROI detected by the gaze tracker.
12. The display apparatus according to claim 11, wherein the dependency of the second modulation signal on the ROI is such that the per-pixel refresh rate is higher in the ROI than outside of the ROI.
13. The display apparatus according to claim 11, wherein at least one of the first and second modulation signals is a triangular or saw-tooth signal with an offset, wherein the slopes of the saw-teeth or triangles and the offset depend on the detected ROI.
14. The display apparatus according to claim 11, wherein at least one of the first and second modulation signals is a sequence of sine halves with an offset, wherein the amplitudes of the sine halves and the offset depend on the detected ROT.
15. The display apparatus according to claim 11, wherein at least one of the first and second modulation signals is a repetition of a step-like function comprised of a first and a last section with high slope, a middle section with low slope, and an offset, wherein the respective slopes and lengths of the sections and the offset depend on the detected ROI.
16. The display apparatus according to claim 11, wherein at least one of the first and second modulation signals is a repetition of a step function comprised of at least two sections of different respective constant values, wherein the respective values and lengths of the sections depend on the detected ROI.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
(1) The disclosed subject matter will now be described by means of exemplary embodiments thereof with reference to the enclosed drawings, in which show:
(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9)
(10)
(11)
(12)
(13)
(14)
DETAILED DESCRIPTION
(15)
(16) In the example of
(17) Instead of the semi-transparent combiner 6 the display apparatus 1 could be used with any other image area 4, e.g., a conventional reflective projection screen such as a miniature screen mounted on the frame of virtual reality (VR) glasses, or a projection wall or a movie screen, for example when the display apparatus 1 is used as a miniature (or full-scale) video beamer. The display apparatus 1 could even be used to directly project the image frames 2 into the user's eye 5, i.e., use the retina of the eye 5 as image area 4, optionally with suitable optics therebetween.
(18) The display apparatus 1 comprises a light source 11 which emits a collimated light beam 12. The light source 11 can be of any kind including gas lasers, fibre lasers, semiconductor lasers etc. For miniaturisation the light source 11 may employ LEDs, micro LEDs or laser diodes, e.g., edge-emitting laser diodes or surface-emitting laser diodes. For colour pixels 3, the light source 11 may be a polychromatic light source 11, e.g., a set of laser diodes of three primary colour which emit a light beam 12 comprised of three different wavelengths for colour perception.
(19) The light beam 12 carries the image frames 2 in a time-multiplexed manner, i.e., the intensity (luminance) and/or colour values of the pixels 3 one after the other in the sequence the pixels 3 are “painted” on the image area 4, when the light beam 12 is moved along a trajectory over the image area 4 as will be explained further on. To control the intensity and/or colour of the light beam 12 pixelwise, the light source 11 has a control input 13 (
(20) To move (scan) the light beam 12 along its a trajectory over the image area 4, the display apparatus 1 comprises a mirror assembly 16, here: one single micro-electro-mechanical-system (MEMS) mirror, downstream of the light source 11 in the path of the light beam 12. The mirror assembly 16 deflects the light beam 12 into subsequent directions (angles) towards the image area 4. Optionally, additional optics or waveguides can be interposed in the path of the light beam 12 from the light source 11 via the mirror assembly 16 to the image area.
(21) As shown in
(22) To induce the oscillation of the mirror 17 about the first axis 19 a first actuator 21 acts between the mirror 17 and the support 18. The actuator 21 may be a coil attached to the mirror 17 and lying in a magnetic field of the support 18, through which coil a first excitation signal S.sub.1 (here: an excitation current) is passed. For inducing the oscillation of the mirror 17 about the second axis 20 a second actuator 22 acts between the mirror 17 and the support 18, e.g., also a coil, through which a second excitation signal S.sub.2 is passed. The excitation signals S.sub.1, S.sub.2 are obtained from signal generators 23, 24 which may be external or internal to the display apparatus 1 and may be a part of the MEMS mirror 16 or the controller 14. Instead of electromagnetic actuators 21, 22 with coils any other type of actuators for driving the oscillations of the mirror 17 about the two axes 19, 20 can be used, e.g., electrostatic, piezoelectric, electrothermal or magnetostrictive actuators.
(23) The frequencies f.sub.1 and f.sub.2 of the two excitation signals S.sub.1 and S.sub.2 are chosen such that the mirror 17 oscillates about each axis 19, 20 at—or nearly at—the resonance frequency of the respective articulation of the mirror 17 on the support 18 (or a multiple thereof, e.g., a harmonic frequency of higher order). The resonance frequency or natural harmonics about the respective axis 19, 20 is defined, i.a., by the mass distribution of the mirror 17 about that axis 19, 20, the spring forces and frictional resistances of the articulations of the mirror 17 about that axis 19, 20, and the magnetic, electrostatic, etc. counterforces of the actuators 21, 22. By oscillating the mirror 17 about the axes 19, 20 at—or in the vicinity of—its resonance frequency about the respective axis 19, 20 a large amplitude of the mirror movement (a large angular sway) can be achieved with small excitation signals S.sub.1, S.sub.2, i.e., of low power or low amplitude, which allows to use particularly small actuators with small moving masses and high resonance frequencies.
(24) To excite and maintain the resonant oscillations of the mirror 17 about the axes 19, 20 the excitation signals S.sub.1, S.sub.2 can be of any form, e.g., pulse signals which trigger the mirror oscillations every oscillation period, every other oscillation period or even more seldomly. However, usually the frequencies f.sub.1, f.sub.2 of the excitation signals S.sub.1, S.sub.2 will be the same as the oscillation frequencies of the mirror 17 about the axes 19, 20, and most commonly sinusoidal excitation signal S.sub.1, S.sub.2 will be used, as shown in
(25) The frequencies f.sub.1 and f.sub.2 of the excitation signals S.sub.1 and S.sub.2 are chosen such that the trajectory of the light beam 12 on the image area 4 is a Lissajous figure which covers the entire image area 4 so that each pixel 3 of the image area 4 is hit at least once by the light beam 12 per image frame 2, e.g., during the period 1/f.sub.fr, of the frames. Such a “complex” or “dense” Lissajous figure can be achieved when the frequencies f.sub.1, f.sub.2 are greater than the frame rate f.sub.fr, e.g., greater than 1 kHz or tens of kHz, and the beginnings B of their respective oscillation periods T.sub.i=1/f.sub.1, T.sub.j=1/f.sub.2 (i=1, 2, j=1, 2, see
(26) Alternatively, instead of the single mirror 17 oscillating about two axes 19, 20, the mirror assembly 16 could comprise two mirrors (not shown) each of which oscillates about a respective one of the (e.g. perpendicular) axes 19, 20 in dependence on the respective excitation signal S.sub.1, S.sub.2 for successive deflection of the light beam 12. Of course, any of the embodiments described herein may be carried out for this variant as well.
(27)
(28) By varying the amplitude of the oscillations of the mirror 17 about the two axes 19, 20 and hence the current maximum size of the trajectory L while it is drawn to “build-up” a frame, as it is shown in
(29) The amplitude of oscillations of the mirror 17 about the axes 19, 20 can be altered in different ways, for example, by changing the amplitude of the excitation signals S.sub.1, S.sub.2; by moving the frequencies f.sub.1, f.sub.2 of the excitation signals S.sub.1, S.sub.2 further away from the respective resonance frequency of the mirror 17 about the respective axis 19, 20, which leads to a drop of the oscillation amplitude from its maximum at resonance; by reducing the pulsewidth of a pulsed excitation signal S.sub.1, S.sub.2; etc. In general, the amplitude of the mirror oscillation about any of the two axes 19, 20 can be varied by amplitude modulation, frequency modulation, pulsewidth modulation or phase modulation of the respective excitation signal S.sub.1, S.sub.2 with a respective modulation signal M.sub.1, M.sub.2.
(30)
(31) The modulation of the excitation signals S.sub.1, S.sub.2 by the modulation signals M.sub.1, M.sub.2 is used in the display apparatus 1 to adapt the local per-pixel refresh rate RR.sub.xy in a user's region of interest, ROI, 26 (
(32) The gaze tracker 28 can work according to any principle known in the art, e.g., by eye-attached tracking with special contact lenses worn by the user which have embedded mirrors or sensors, or by optical tracking of corneal or retinal reflections of visible or invisible light rays. Most commonly, the gaze trackers 28 will be implemented optically, e.g., by means of a camera directed at the user's eye or eyes to view and track the gaze 27. Such a video camera can be used both for eye tracking (when head-mounted) or gaze tracking from a stationary point in the environment 10 when it views and analyses both eye movement and head movement.
(33) The detection of the user's ROI 26 on the image area 4 is used to move the area 25 of increased per-pixel refresh rate RR.sub.xy, achieved by the current modulation of the excitation signals S.sub.1, S.sub.2 applied, either into (or over) the ROI 26 or outside of the ROI 26. The latter embodiment can be useful to avoid flickering in the periphery of the user's field of view where the rods of the retina are highly responsive to fast light changes whereas the cones in the middle or fovea of the retina, which are used to look (gaze) at the ROI 26, are much less susceptible to the perception of flickering.
(34) The former embodiment where the region 25 of increased per-pixel refresh rate RR.sub.xy is laid into or over the ROI 26 avoids flickering just there where the user looks to and can be particularly useful in AR applications where image content is selectively displayed as an overlay in ROI 26. This embodiment is shown in
(35) In a first variant, shown in
(36) As can be seen from
(37) In a second variant shown by the sequence of
(38) From an analysis of the past track 29 of the gaze 27 the gaze tracker 28 can then predict the current ROI 26 for a frame 2 to display, and the controller 14 can—even pre-emptively—change the modulation signals M.sub.1, M.sub.2 so that the ROI 26 will always be hit or covered by a region 25 of increased per-pixel refresh rate RR.sub.xy. The gaze tracker 28 can even predict a next saccade x.sub.i+1 from an analysis of the track 29, particularly from a past sequence of fixations p.sub.i and saccades x.sub.i, in order to adjust the display apparatus 1 for the next saccade x.sub.i+1 of the user's gaze 27. When predicting the ROI 26, the gaze tracker 28 can not only predict the location of the ROI 26 but optionally also the size of the ROI 26. For instance, the size of the ROI 26 can be determined in dependence on a calculated location prediction uncertainty, e.g., in order to have a larger size of the ROI 26 in case of a higher location prediction uncertainty.
(39) Furthermore, the gaze tracker 28 can be configured to determine a duration (or average duration) d.sub.i of one (or more) past saccades x.sub.i, and the controller 14 can be configured to complete a change from a first set of modulation signals M.sub.1, M.sub.2—which achieves, e.g., the region 25 of
(40) The modulation signals M.sub.1, M.sub.2 required to achieve a specific region 25 of increased per-pixel refresh rate RR.sub.xy that hits or covers the ROI 26, i.e., the dependencies of the modulations signals M.sub.1, M.sub.2 on the detected ROI 26, can be programmed into the controller 14 in form of a formula. Alternatively, as shown in
(41) For a specific ROI 26 detected, the controller 14 looks up the region 25 into which the ROI 26 falls (or which falls into that ROI 26) and retrieves from the correspondence between the elements of the matrices 32, 33 the corresponding first and second modulation signals M.sub.1, M.sub.2. The controller 14 then modulates the excitation signals S.sub.1, S.sub.2 with the modulation signals M.sub.1, M.sub.2 retrieved from the look-up table 31.
(42) To perform the modulation, the display apparatus 1 may have discrete modulators 34, 35 receiving the excitation signals S.sub.1, S.sub.2 from the signal generators 23, 24 on the one hand and the modulation signals M.sub.1, M.sub.2 from the controller 14 on the other hand. Alternatively, the signal generators 23, 24 and modulator 34, 35 can be implemented by processing elements within the controller 14.
(43) In general, different types of modulation signals M.sub.1, M.sub.2, can be used which lead to different shapes and sizes of regions 25 of increased per-pixel refresh rate RR.sub.xy. Instead of the saw-tooth signals of
(44) In
(45) In
(46) In an optional variant (not shown), the modulation signal M.sub.1 or M.sub.2 is a repetition of a step function comprised of at least two sections of different respective constant values, wherein the respective values and lengths of the sections depend on the detected ROI 26. It goes with saying that each of the excitations signals S.sub.1, S.sub.2 can be modulated with the same or different modulation signals M.sub.1, M.sub.2, i.e., with modulation signals M.sub.1, M.sub.2 of different frequencies, shapes, amplitudes and offsets.
(47) In the region 25 of increased per-pixel refresh rate RR.sub.xy there may occur a concomitant increase in light intensity perceived by the user. This is not only caused by the increased refresh rate itself but also by the varying local speed of the light beam 12 along its Lissajous trajectory L, as the light beam 12 moves slower in the periphery and faster in the middle of the image area 4. The perceived intensity in a pixel 3 thus depends on the total (i.e., summed) amount of time the light beam 12 spends in that pixel 3 during the time an image frame 2 is displayed. To counter this effect, the controller 14 can optionally decrease the intensity of the light beam 12 via the control input 13 of the light source 11 for a pixel 3 when the amount of time the light beam 12 spends in that pixel 3 during a frame 2 increases, and vice versa.
(48) The disclosed subject matter is not restricted to the specific embodiments described herein, but encompasses all variants, modifications and combinations thereof that fall within the scope of the appended claims.