Systems and methods for rendering a display to compensate for a viewer's visual impairment

Abstract

A system and method is provided for rendering on a display of an electronic device a pre-corrected image of an original image to compensate for a visual impairment of a user of the device, such that the pre-corrected image when displayed on the display would be perceived by the user to be in better optical focus than the user would perceive a display of the original image. The method includes receiving hardware characteristics of the display, receiving at least one parameter describing the visual impairment of the user, receiving at least one parameter describing the conditions under which the display is seen by the user, calculating a pupil function, calculating a Point Spread Function (PSF), calculating a pre-corrected image corresponding to at least a portion of the original image to compensate for the visual impairment of the user, and rendering on the display the pre-corrected image.

Claims

1. A computer-implemented method for rendering on a display of an electronic device a pre-corrected image of an original image to compensate for a second-order visual impairment of a user of the device such that the pre-corrected image when displayed on the display would be perceived by the user to be in better optical focus than the user would perceive a display of the original image, comprising: receiving hardware characteristics of the display; receiving at least one parameter describing the visual impairment of the user, wherein the parameter is at least one from the group consisting of: a sphere parameter, a cylinder and axis parameters, and a near point of accommodation or another mathematically-equivalent set of parameters representing the visual impairment of the user; receiving at least one parameter describing the conditions under which the display is seen by the user; calculating, directly using the at least one parameter describing the visual impairment of the user, a Point Spread Function (PSF), or another mathematically-equivalent representation of an imaging capability of an eye's optical system including an optical transfer function (OTF); calculating a pre-corrected image corresponding to at least a portion of the original image to compensate for the visual impairment of the user, using a constrained optimization technique that maximizes perceived quality of the pre-corrected image according to constraints determined by a set of images capable of being displayed according to the hardware characteristics, and rendering on the display the pre-corrected image, wherein the rendering comprises a separate pre-correction of text elements of the original image implemented as a rendering effect performed by a font rendering module of an operating system, separate from correction to non-text elements of the original image.

2. The computer-implemented method of claim 1, wherein receiving at least one parameter describing the visual impairment of the user comprises at least one of receiving the visual impairment parameter from a remote database, a local database, real-time input, or real-time estimation.

3. The computer-implemented method of claim 1, wherein receiving at least one parameter describing the conditions under which the display is seen by the user comprises at least one of receiving the display condition parameter from real-time estimation, real-time input, or a local database during the rendering.

4. The computer-implemented method of claim 1, wherein receiving hardware characteristics of the display comprises receiving at least one from the group consisting of pixel dimensions and spacing, display width and height, display reflectivity emission spectra of each of a sub-pixel class, and a relationship between pixel value and luminance.

5. The computer-implemented method of claim 1, wherein receiving a parameter describing the conditions under which the display is seen comprises the step of receiving one or more of a measurement of the distance between the eye of the user and the screen, an ambient luminosity, and a measurement of the pupil size of the user.

6. The computer-implemented method of claim 1, wherein calculating the PSF comprises calculating a polychromatic PSF per sub-pixel class of the display using an emission spectrum of the sub-pixel class and an average photopic spectral sensitivity function of the user's retina, and wherein rendering on the display the pre-corrected image comprises rendering the pre-corrected image for each sub-pixel class.

7. The computer-implemented method of claim 1, wherein calculating a pre-corrected image is performed using the constrained optimization technique employing a gradient descent or using a Van-Cittert algorithm.

8. The computer-implemented method of claim 1, wherein the constrained optimization technique is formulated based on a metric comprising a variation of sensitivity of eye receptors due to luminance.

9. The computer-implemented method of claim 1, wherein the constrained optimization technique is formulated based on an objective function representing an overall image quality as perceived by the user.

10. The computer-implemented method of claim 1, wherein at least one from the parameters describing the visual impairment of the user and the parameter describing the conditions under which the display is seen is determined by performing a dichotomy test, the dichotomy test comprising: presenting to the user images pre-corrected using different hypotheses.

11. The computer-implemented method of claim 1, further comprising enabling or disabling rendering by striking a hard key, varying an option, shaking the device, or using an embedded camera to detect eyeglasses being worn by the user.

12. The computer-implemented method of claim 1, wherein the pre-correction is done using a PSF of the right eye only, a PSF of the left eye only, or a PSF resulting from a combination of both PSF's of the left and right eyes, or both PSF's of the left and right eyes and applying two pre-correction results to two elements of an autostereoscopic monitor.

13. A system for rendering on a display of an electronic device a pre-corrected image of an original image to compensate for a second-order visual impairment of a user of the device such that the pre-corrected image when displayed on the display would be perceived by the user to be in better optical focus than the user would perceive a display of the original image, comprising: a computer processor; and a memory connected to said computer processor, wherein the memory is configured with instructions to cause the processor to (1) receive hardware characteristics of the display screen from a local database; (2) receive a parameter describing the visual impairment of the user from at least one of a remote database, a local database, real-time input, or real-time estimation, wherein the parameter is at least one from the group consisting of: a sphere parameter, a cylinder and axis parameters, and a near point of accommodation; (3) receive a parameter describing a condition under which the display is seen by the user from at least one of real-time estimation, real-time input, or local database; (4) calculate, directly using the parameter describing the visual impairment of the user, a Point Spread Function (PSF), or another mathematically-equivalent representation of an imaging capability of an eye's optical system including an optical transfer function (OTF); (5) calculate a pre-corrected image corresponding to at least a portion of the original image to compensate for the visual impairment of the user, using a constrained optimization technique that maximizes perceived quality of the pre-corrected image according to constraints determined by a set of images capable of being displayed according to the hardware characteristics; and (6) render on the display the pre-corrected image, wherein rendering on the display the pre-corrected image comprises a separate pre-correction of text elements of the original image implemented as a rendering effect performed by a font rendering module of an operating system, separate from correction to non-text elements of the original image.

14. A non-transitory computer readable storage medium storing a program configured for execution by a processor, the program when executed being adapted to cause the processor to: receive a hardware characteristic of the display from a database; receive a parameter describing a second order visual impairment of the user from a remote database, a local database, real-time input, or real-time estimation, wherein the parameter is at least one from the group consisting of: a sphere parameter, a cylinder and axis parameters, and a near point of accommodation; receive a parameter describing the conditions under which the display is seen by the user from at least one of real-time estimation, real-time input, or local database; calculate, directly using the parameter describing the visual impairment of the user, a Point Spread Function (PSF), or another mathematically-equivalent representation of an imaging capability of an eye's optical system, including an optical transfer function (OTF); calculate a pre-corrected image corresponding to at least a portion of an original image to compensate for a user's visual impairment, using a constrained optimization technique that maximizes perceived quality of the pre-corrected image according to constraints determined by a set of images capable of being displayed according to the hardware characteristics; and render the pre-corrected image on a display of the electronic device, whereby the display of the pre-corrected image would be perceived by the user to be in better optical focus than the user would perceive a display of the original image, wherein rendering the pre-corrected image on the display comprises a separate pre-correction of text elements of the original image implemented as a rendering effect performed by a font rendering module of an operating system, separate from correction to non-text elements of the original image.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and together with the description serve to explain the principle of the invention. In the drawings:

(2) FIG. 1 shows the formation of a spot on the retina of a presbyopic eye looking at one point source;

(3) FIG. 2 shows the formation of two overlapping spots on the retina of a presbyopic person when he or she looks at two point sources;

(4) FIGS. 3(a) and (b) schematically show the display of a smartphone as seen by a non-visually impaired eye and a presbyopic person, respectively.

(5) FIGS. 4(a) and (b) schematically show a corrected display of a smartphone, according to the present invention, as seen by a non-visually impaired person and by a presbyopic person, respectively;

(6) FIG. 5 is a block diagram illustrating exemplary system architecture of the present invention;

(7) FIG. 6 is a flow diagram showing the succession of steps involved in the vision compensation method of the present invention;

(8) FIG. 7 is a flow diagram showing the succession of steps involved in the calculation of a monochromatic PSF;

(9) FIG. 8 is a diagram schematically illustrating the geometry behind the calculation of a PSF;

(10) FIG. 9 is a flow diagram illustrating the calculation of a polychromatic PSF;

(11) FIG. 10 is a flow diagram detailing how a reference image is received;

(12) FIG. 11 is a flow diagram illustrating the integration of pre-correction at the window manager level; and

(13) FIG. 12 is a flow diagram illustrating the integration of the pre-correction at the font rendering library level.

DETAILED DESCRIPTION

(14) Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous non-limiting specific details are set both in order to assist in understanding the subject matter presented herein. It will be apparent, however, to one of ordinary skill in the art that various alternatives may be used without departing from the scope of the present invention and the subject matter may be practiced without these specific details. For example, it will be apparent to one of ordinary skill in the art that the subject matter presented herein can be implemented on many types of electronic personal devices comprising a digital display.

(15) The solution according to the present invention is software based. Knowing the deformation introduced by the eyes (i.e. the vision impairment condition) the software pre-corrects an image in a manner related to the user's specific vision impairment by introducing an organized blur that will compensate for the impairment and result in the user perceiving an image that is more clear.

(16) In a branch of digital image processing called image restoration, techniques (often called deconvolution) are used to correct for defocus or other optical perturbations and to make it possible to estimate the original object from a blurred image. The use of digital image processing in the context of the present invention is derived from image restoration techniques but in general reverses the order of operations: deconvolution is applied before the image is blurred by the eye. For purposes of the present invention such processing is referred to as “pre-correction.” A defocused eye looking at a pre-corrected image will cancel the effect of pre-correction and see a sharp image. In practice, pre-correction modifies the image displayed to the user so that the overlapping spots generated by individual pixels partially negate each other and produce a sharp retinal image. FIG. 3 (a) is an example of an original or reference image. FIG. 3 (b) is the corresponding simulation of what a presbyopic person sees. FIG. 4 (a) is a simulated pre-corrected image. FIG. 4 (b) is a simulation of what the same person sees when looking at FIG. 4 (a). Although the contrast may be reduced, the global readability of the display, as represented on FIG. 4 (b), is greatly improved compared to FIG. 3 (b).

(17) FIG. 5 is a block diagram illustrating an exemplary system 100 in accordance with the present invention. The exemplary system 100 comprises an electronic device 110 displaying content (e.g., text and/or images) on a monitor 111, after being processed by a pre-correction module 117 so that the eye 101 of the user can see the content sharply even if the user is visually-impaired (e.g., a presbyopic or far-sighted user). The image will not be focused on the retina 102 for an uncorrected eye 101) but will still be perceived as a sharp image.

(18) FIG. 6 is a flowchart illustrating the steps of the processing method according to non-limiting embodiments. Step 201a collects context-dependent data. Context-dependent data represents the conditions under which the device is seen by the user. It is stored in memory 118. It typically includes the pupil size, the viewing distance between the device 110 and the eye 101, and the level of the ambient light. The exact list of parameters depends on the model used for PSF calculation during step 202. In some instances one or more of these parameters can be considered constant and not necessary to update. For example, under normal condition of use the distance between a screen included in the dashboard of an automobile and the eye of the driver can be considered relatively constant. Other parameters may be updated by module 121 when the conditions under which the device is used change. According to an embodiment of the present invention, module 121 detects changes and estimates new values of the parameters using data acquired from embedded sensors 113:

(19) As an example, an image may be received from an embedded front camera and the head of the user may be recognized and measured using conventional techniques in order to estimate the distance between the eye of the user and the display screen. Images may also be received from an embedded front camera in order to recognize and measure the size of the user's pupil using conventional techniques. Also, ambient light measures may be received from an embedded light sensor.

(20) Such sensors are commonly used by mobile devices in order to adjust the brightness of the screen back-light. If direct measurement of the pupil size cannot be made by the device, considering that the pupil size depends on the intensity of light received by the eye, pupil size may be estimated using a model based on a measure of the level of ambient light and the light intensity emitted by the device screen. Radio beam (infrared, WiFi, 3GPP or the like) may be used to estimate the distance between the user and the display screen. This technique, called “a head tracking system,” has been used in devices such as game consoles. Similar to a radio beam, an audio signal can be emitted by a speaker in a device and its echo detected by a microphone in order to estimate the distance between the user and the display screen. Applications for smartphones are known which use the embedded speaker and microphone to estimate the distance to surrounding objects. Also, motion detection may be received from motion sensors (e.g. an accelerometer, a gyroscope or the like) in order to detect position change and estimate the new position a user relative to a device.

(21) Step 201b collects user-dependent data. User-dependent data characterizes the visual impairment of the user. As in the case of context-dependent data, user-dependent data may be stored in memory 120. It may be expressed as a discrete estimate of the wave aberration function of each eye of the user for a given viewing distance and a given wavelength. It may also include the user accommodation expressed, for instance, as the near point of accommodation. The exact type and nature of the parameters depend on the model used for PSF calculation during step 202. Module 122 computes or acquires the user-dependent data from different possible inputs. For example, the user may enter the parameters of en eyeglass prescription, namely the Sphere (S), Cylinder (C) and Axis (A) parameters for each eye. The user may also enter the additional refractive power (ADD or NV) or a self estimate of his or her near point of accommodation. When the user enters new parameters, module 122 determines the user-dependent data and updates memory 120. Mathematical models for computation of the wavefront aberration functions from such parameters are well known to those skilled in optometry, and may involve notably decomposition over Zernike polynomials.

(22) Alternatively, an auto-diagnosis program may be run. Such a program presents the user different patterns on a display, e.g., the user's handheld display, and asks the user to enter whether he or she sees the detail of the pattern in absolute terms or comparatively with respect to other displays. Optionally, the program can be associated with optical elements in order to measure more precisely the refractive errors using known inverse Shack-Hartmann techniques. Alternatively, the auto-diagnosis program presents the user pre-corrected chart images assuming various typical PSF's, the parameters of which can be updated step by step by dichotomy until the user no longer sees improvement. Module 122 derives an estimate of the user-dependent data and updates memory 120 each time the auto-diagnosis program is run.

(23) As another alternative, if the user's device can connect to the internet, the user-dependent data may be retrieved from a remote database through the internet. The data may have been previously stored in the remote database after being collected by one of the above-mentioned alternatives. The data may also have been stored in the remote database after an optometric examination using, e.g., a Shack-Hartmann wavefront sensor.

(24) In yet another alternative, user-dependent data stored in one personal device (e.g., a smartphone, tablet computer, smart card, etc.) may be transferred to another personal device using short-range transmission means (e.g. NFC, Bluetooth, etc.) or through smart card contact. The data may have been previously stored in the personal device after being collected by one of the just-mentioned alternatives. The data may also have been stored in the personal device after an optometric examination using, as mentioned above, a Shack-Hartmann wavefront sensor, for example.

(25) Step 201c collects device-dependent data. Device-dependent data are characteristics of the display screen that affect the computation of the PSF during step 202. The exact list and nature of the parameters depends on the model used for PSF calculation. They typically comprise the pixel (sub-pixel) size, the emission spectra of each pixel or sub-pixel class (e.g. the red sub-pixels, the green sub-pixels and the blue sub-pixels), the luminosity of pixels or sub-pixels as a function of the pixel value (sometimes called the gamma function), and the screen reflectivity. These parameters are stored in memory 119 and in some situations may be considered constant. However, memory 119 may be updated when the screen luminosity or other calibration values change.

(26) Step 202 computes the PSF from the data received during steps 201a, 201b, and 201c (or any other mathematically equivalent representation of the imaging capability of the eye's optical system, like the optical transfer function (OTF)). In the context of a preferred embodiment of the present invention, the PSF is defined as the retinal color image of a pixel, as viewed by the user at a given distance, and comprising a number of sub-pixels, each of them emitting with a dominant color but still having an emission spectrum not necessarily purely monochromatic. As explained herein, various methods can be used during step 202.

(27) FIG. 7 illustrates the sub-steps 301-305 of step 202 for an exemplary embodiment, when the pixels are assumed to emit light in a single frequency. At step 301, the generalized pupil function is calculated and describes how the light wave is modified by the eye's optical system for a different position in the pupil. The pupil function may be expressed according to Equation (1) as:
P(ρ,θ)=α(ρ,θ)exp(ikW(ρ,θ))  (1)
where α is the amplitude transmittance of the pupil, W is the wave aberration function,

(28) $i=\sqrt{-1},k=\frac{2\pi }{\lambda },$
λ is the wavelength, and ρ and θ are the polar coordinates in the pupil.

(29) W is a function of the user-dependent data (e.g., visual impairment and accommodation), the context-dependent data (e.g., view distance), and/or the device-dependent data (e.g., wavelength λ).

(30) In a model, expressed below, a has the value of 1 inside the circle aperture of the pupil, and the value of 0 outside. The model may be expressed according to Equation (2) as:

(31) $\begin{array}{cc}\alpha \left(\rho ,\theta \right)=\{\begin{array}{cc}1& \mathrm{if}\rho <D\\ 0& \mathrm{otherwise}\end{array}& \left(2\right)\end{array}$
where D is the pupil size from the context-dependent data.

(32) In some embodiments the Stiles-Crawford effect, where there is a different sensitivity of cones depending on the light direction, may be considered in order to better approach the true PSF of the eye. In such circumstance, instead of a having a constant value of 1 inside the aperture, the following Equation (3) may be used:

(33) $\begin{array}{cc}\alpha \left(\rho ,\theta \right)=\{\begin{array}{cc}\exp \left(-\beta {\rho }^2/2\right)& \mathrm{if}\rho <D\\ 0& \mathrm{otherwise}\end{array}& \left(3\right)\end{array}$

(34) where β is a constant representing the average amplitude of the Stiles-Crawford effect in the population and may take the value 0.116 mm.sup.−2.

(35) At step 302, the optical field entering the pupil is calculated. Using the assumption that a pixel is a point source with no dimension located at distance z.sub.0 from the pupil on the optical axis (see, e.g., FIG. 8), the field entering the pupil may be expressed in Equation (4) as:

(36) $\begin{array}{cc}S\left(\rho ,\theta \right)=\frac{A_0}{\sqrt{{\rho }^2+z_0^2}}\exp \left(\mathrm{ik}\sqrt{{\rho }^2+z_0^2}\right)& \left(4\right)\end{array}$
where A.sub.0 is the amplitude of the point source. At step 303, the field exiting the pupil is calculated using Equation (5) as:
U(ρ,θ)=S(ρ,θ)P(ρ,θ)  (5)

(37) At step 304, the field on the retinal plane I is calculated. In some embodiments, the Rayleigh-Sommerfeld diffraction formula of the first kind may be used and is expressed in Equation (6) as:

(38) $\begin{array}{cc}I\left(R\right)=\frac{{\mathrm{iA}}_0}{\lambda }\int {\int }_Q^{}U\left(Q\right)\frac{\exp \left(ik.Math.\stackrel{.fwdarw.}{\mathrm{QR}}.Math.\right)}{.Math.\stackrel{.fwdarw.}{\mathrm{QR}}.Math.}\cos \left(\stackrel{.fwdarw.}{n},\stackrel{.fwdarw.}{\mathrm{QR}}\right)\mathrm{dS}& \left(6\right)\end{array}$

(39) where R is a point of the retina, Q is a point of the pupil, dS is a small surface element including Q and {right arrow over (n)}, the normal value to the pupil plane.

(40) Under some assumptions, known as the Fraunhofer approximation, the diffraction formula, I(R), becomes the inverse Fourier transform of U(Q). Some embodiments may use an inverse Fast Fourier Transform instead of direct computation of the Rayleigh-Sommerfeld diffraction formula.

(41) At step 305, the PSF is computed from I(R) as a square modulus and is expressed in Equation (7) as:
PSF=|I(R).sup.2  (7)

(42) The resulting PSF may be called a monochromatic PSF because it is computed under the assumption that the pixel emits light at a single wavelength λ.

(43) FIG. 9 illustrates the sub-steps of step 202 for an exemplary embodiment that calculates a polychromatic PSF for a given sub-pixel class. One polychromatic PSF may be computed per sub-pixel class (e.g., one polychromatic PSF may be computed for each of the Red sub-pixel class the Green sub-pixel class, and the Blue sub-pixel class). Each sub-pixel class may be characterized by an emission spectrum that may be pre-stored in a memory 119 as a part of device-dependent data. The stored emission spectrum may consist of a set of values representing the emitting power of the sub-pixel class at a number of frequencies spanning around the nominal frequency characterizing the color of the sub-pixel class. For each sub-pixel class, the polychromatic PSF may be calculated by computing the weighted average of monochromatic PSF's calculated for each frequency, λ.sub.i which is part of the emission spectrum, and then multiplied by the power w.sub.i from the emission spectrum.

(44) In some embodiments, the terms of the above weighted average may be further weighted by the photopic spectral sensitivity at each frequency. The average photopic spectral sensitivity function of the human retina may be found in academic literature.

(45) The refractive power of an optical system, like an eye, depends on the wavelength. At some point, the eye can only be focused perfectly for a single wavelength. This phenomenon results in so-called chromatic aberrations. In some embodiments there may be a defocus sub-step at step 301, in the wave aberration function depending on the wavelength, to account for such chromatic aberrations. The amount of defocusing to be done may be found in academic literature.

(46) At step 203, the image that will be processed by the pre-correction algorithm is received. This image is called the reference image. How the reference image is received may be based on user preference(s) and criteria stemming from running applications. FIG. 10 illustrates an exemplary embodiment of the image received at step 203. At step 601, a memory containing the user's preference is read. The user may have set his preference to, e.g., “render text,” “render all,” “application-based” rendering, or “render nothing”. Step 602 is performed if the preference is set to “render text”; step 603 is performed if the preference is set to “render all”; step 604 is performed if the preference is set to “application-based”; and, steps 204 and 205 may not be performed if the preference is set to “render nothing.”

(47) In some embodiments, toggling between “render nothing” and the preference stored in memory previously may be performed if the user strike a dedicated hard key on the device; the embedded accelerometer triggers pre-correction on/off; detecting when the user shakes the device in a specified way; or by the camera detecting that the user is wearing his eyeglasses (or not) and transmitting an instruction to, for example, the module(s) that execute any of steps 601-604. The instruction may be to switch the pre-correction process on/off. The foregoing technique may be performed if the user has specified that he does not wear corrective/contact lenses a priori.

(48) In some embodiments, all text displayed to the user like, for example, when he sets an option in the preference menu, may be shown using large-sized fonts until the pre-correction process is turned on.

(49) At step 602, text may received by one or more modules from one or more applications, using a font-rendering library (e.g., of the operating system (OS) or software executing on the OS), when the applications include text. The font-rendering library may be used to output and/or render the appropriate text on small, off-screen images that may become the reference image(s) of step 204.

(50) At step 603, an operating system display module (e.g., a compositing window manager) may ask an application to draw some portion of its display (e.g. by way of a method or function call) or may receive updates from an application for some portion of its display. In some embodiments, a rectangular or other portion of the application screen buffer containing the updated/changed portion may become the reference image of step 204.

(51) At step 604, reference images may be received from the font rendering library or from an operating system display module. Text may be received from the font rendering library if the application calling its services is a text-based application, which may be included in a pre-stored list of such applications (e.g., SMS, e-mail, contact list, caller ID). In some embodiments, an image containing updated parts of an application display may be received from the operating system display module if the application is an image-based application, which may be included in a pre-stored list (e.g., photo manager, navigation application).

(52) At step 204, the reference image received in step 203 is pre-corrected. The reference image, may be composed of a limited number of 1-component images corresponding to each sub-pixel class. In some embodiments, we may assume that, for example, x.sub.ref is such an image, expressed using luminance values, and h.sub.imp is the corresponding component of the PSF of an eye of a visually-impaired user. Assuming a suitable coordinate system and sampling distances are used, a discrete estimate of the image formed on the retina of the eye, f.sub.ref, when the image is displayed, may be expressed in Equation (8) as:
f.sub.ref=x.sub.ref*h.sub.imp  (8)

(53) where * denotes the discrete convolution operator. In some embodiments, pre-correction consists in calculating an image x.sub.cor such that, when such an image is displayed instead of x.sub.ref, it produces on the retina an image f.sub.cor, which may be expressed in Equation (9) as
f.sub.cor=x.sub.cor*h.sub.imp  (9)

(54) The image, f.sub.cor, may be of “better quality” than f.sub.ref. “Better quality” here means that the user may be able to perform a precise task relative to his usage of the device with better performance (for example, like reading the text displayed with less errors).

(55) x.sub.cor may be expressed in Equation (10) as:

(56) $\begin{array}{cc}{x}_{\mathrm{cor}}=\underset{x}{\mathrm{argmin}}d\left(f_{\mathrm{cor}},f_{\mathrm{obj}}\right)& \left(10\right)\end{array}$
where f.sub.obj is the ideal retinal image and d is a metric quantifying the perceived difference between f.sub.cor and f.sub.obj. The minimization may be carried out on the domain, D, containing the images x that can actually be displayed by the device. D is a set of images with pixel values that are not lower than the luminance of black, and not higher than the maximum luminance.

(57) The choices for the values of f.sub.obj, d, and the numerical method used to solve Equation (10) may each lead to various embodiments.

(58) In some embodiments, x.sub.att may be a contrast-attenuated version of x.sub.ref. Each luminance value, x.sub.ref(i), may be rescaled to x.sub.att(i)=R(x.sub.ref(i)), using an increasing function R. Minimum and maximum luminance values of the screen, v.sub.min and v.sub.max, map respectively to v.sub.0=R(v.sub.min) and v.sub.1=R(v.sub.max), where v.sub.0 and v.sub.1 satisfy the inequality expressed in Equation (11) as:
v.sub.min≦v.sub.0<v.sub.1≦v.sub.max  (11)
R may be a linear function of the luminance, expressed in Equation (12) as:

(59) $\begin{array}{cc}R\left(v\right)=v_0+\frac{v_1-v_0}{v_{\mathrm{ma}x}-v_{min}}\left(v-v_{min}\right)& \left(12\right)\end{array}$

(60) Alternatively, R may be a linear function of the pixel value, expressed in Equation (13) as:

(61) $\begin{array}{cc}R\left(v\right)=L\left({\tilde{v}}_0+\frac{{\tilde{v}}_1-{\tilde{v}}_0}{{\tilde{v}}_{m\mathrm{ax}}-{\tilde{v}}_{min}}\left(L^{-1}\left(v\right)-{\tilde{v}}_{min}\right)\right)& \left(13\right)\end{array}$
where L is the display function that converts a pixel value to a luminance value, {tilde over (v)}.sub.0=L.sup.−1(v.sub.0) and {tilde over (v)}.sub.1=L.sup.−1(v.sub.1) are the minimum and maximum pixel values of the contrast-attenuated image, and {tilde over (v)}.sub.min=L.sup.−1(v.sub.min) and {tilde over (v)}.sub.max=L.sup.−1(v.sub.max) are the minimum and maximum pixel values of the screen.

(62) f.sub.obj may be set to x.sub.att, the retinal image produced by a perfectly focused eye, having a PSF limited to a single point, when x.sub.att is displayed.

(63) Alternatively, f.sub.obj may be set to f.sub.obj=x.sub.att*h.sub.ref where h.sub.ref is a realistic PSF of a non-impaired eye estimated by the method described above. h.sub.ref may be the PSF of a diffraction-limited eye, focused on the screen, with a pupil of same size than the user's pupil. In some embodiments, h.sub.ref may also be a polychromatic PSF that accounts for normal chromatic aberrations.

(64) Metric d may be chosen from a variety of metrics, some of them known. The choice of a particular metric does not alter the spirit and scope of the invention.

(65) In some embodiments, d may be the Euclidean metric.

(66) Alternatively, d may be a metric that accounts for the varying sensitivity of the eye's receptors due to luminance, which may be expressed in Equation (14) as:

(67) $\begin{array}{cc}d\left(x,y\right)=\underset{i}{.Math.}{\left(p\left(x\left(i\right)\right)-p\left(y\left(i\right)\right)\right)}^2& \left(14\right)\end{array}$
where p is a non-linear function associating a subjective brightness value to a luminance value. In some embodiments, p may be, for example, the inverse L.sup.−1 of the display function L that converts pixel values to luminance values. In some embodiments, x(i) and y(i) may be the luminance values of a pixel, i, in image(s) x and y, respectively.

(68) Equation 10 may be solved using a variety of known techniques. The choice of a particular technique does not alter the spirit or scope of this invention.

(69) When d is the Euclidean metric, a solution to Equation (10) may be estimated using Wiener deconvolution, which may be expressed in Equation (15) as:

(70) 0$\begin{array}{cc}{X}_{\mathrm{cor}}=\frac{H_{\mathrm{imp}}^{*}}{{.Math.H_{\mathrm{imp}}.Math.}^2+K}{F}_{\mathrm{obj}}& \left(15\right)\end{array}$
where X.sub.cor, H.sub.imp and F.sub.obj are the discrete Fourier transforms of x.sub.cor, h.sub.imp and f.sub.obj respectively, and H*.sub.imp denotes the complex conjugate of H.sub.imp. K, which in image restoration problems is linked to the power density spectrum of the noise, may act here as a regularization parameter that limits the high frequencies in X.sub.cor, and, thus, the odds that x.sub.cor may exhibit values smaller than v.sub.min or larger than v.sub.max. If such values still exist in x.sub.cor, they are clamped in order to obtain an image x.sub.cor, which belongs to domain D containing the images that can be displayed by the screen and may be expressed in Equation (16) as:

(71) $\begin{array}{cc}{x}_{\mathrm{cor}}\leftarrow \{\begin{array}{cc}{x}_{\mathrm{cor}}& \mathrm{if}{v}_{min}\le x_{\mathrm{cor}}\le v_{m\mathrm{ax}}\\ v_{min}& \mathrm{if}{x}_{\mathrm{cor}}<v_{min}\\ v_{m\mathrm{ax}}& \mathrm{if}{x}_{\mathrm{cor}}>v_{m\mathrm{ax}}\end{array}& \left(16\right)\end{array}$

(72) In some embodiments, Equation (10) may be solved using a constrained optimization algorithm. If d is the Euclidean metric, known algorithms for solving quadratic problems may be used. If d is defined by Equation (14), where p is an analytical differentiable function, the optimization may be carried out using a gradient descent method.

(73) In some embodiments, as an alternative to a gradient descent method, the optimization may be carried out using the Van Cittert algorithm with constraints applied using the Jaanson techniques.

(74) In some embodiments, as an alternative to Wiener deconvolution and iterative algorithms like gradient descending techniques or the Van Cittert algorithm, gradient descending techniques or the Van Cittert algorithm may be used after initializing their values with the result(s) of a Wiener deconvolution.

(75) In some embodiments, the pre-corrected images may be determined without solving Equation (10) for each color component. Alternatively, the pre-corrected images for each sub-pixel class x.sub.cor1, x.sub.cor2, etc. may be determined simultaneously and together with function R in order to use the contrast attenuation that yields the viewed image with the “best quality.” If function R is characterized by parameters v.sub.0 and v.sub.1 according to Equations (12) or (13), the pre-corrected images may be expressed in Equation (17) as:

(76) $\begin{array}{cc}\left(x_{\mathrm{cor}1},\mathrm{.Math.},x_{\mathrm{corJ}},v_{\mathrm{opt}0},v_{\mathrm{opt}1}\right)\underset{x_1,\mathrm{.Math.}{x}_J,v_0,v_1}{\arg \min }Q\left(x_1,\mathrm{.Math.}{x}_J,v_0,v_1\right)& \left(17\right)\end{array}$
where J is the number of sub-pixel classes, v.sub.opt0 and v.sub.opt1 are the parameters characterizing the optimal function R, and Q is a function that depends on d(x.sub.corj,f.sub.objj) and on the magnitude of contrast attenuation. The choice for function Q does not alter the spirit or scope of this invention.

(77) In some embodiments, a metric d that accounts for the non-linear luminance sensitivity of the eye, Q, may be expressed in Equation (18) as:

(78) $\begin{array}{cc}Q\left(x,v_0,v_1\right)=\frac{1}{{n\left(v_1-v_0\right)}^2}\underset{j=1}{\overset{J}{.Math.}}\underset{i}{.Math.}{\left(p\left(f_{\mathrm{corj}}\left(i\right)\right)-p\left(f_{\mathrm{objj}}\left(i\right)\right)\right)}^2+\mu \frac{v_{m\mathrm{ax}}-v_{min}}{v_1-v_0}& \left(18\right)\end{array}$
where n is the numbers of sub-pixels and μ is a constant parameter quantifying the relative contribution of contrast attenuation in the perceived image quality.

(79) In some embodiments, the PSF may be computed from data that are only approximations of physical quantities. For example, the user's pupil size may be measured or estimated with a significant confidence interval, and the parameters describing the user's visual impairment cannot characterize perfectly the behavior of the eye's optical system. In some embodiments, in order to account for the resulting uncertainty on the PSF, instead of minimizing the perceived difference between f.sub.obj and f.sub.cor computed from a single PSF, the difference between f.sub.obj and a set of f.sub.cor computed from a plurality of PSFs may be minimized. Other embodiments may define a probabilistic function instead of metric d or function Q depending on the probability distribution of the uncertain parameters and minimize an index computed from the probabilistic function, like the expectation.

(80) For some users, each eye may be affected by significantly different impairments, thus yielding a different PSF and, consequently, a different pre-correction. In some embodiments, in order to address such a scenario properly, a single PSF may be computed by combining the PSF's of both eyes or by computing a single PSF from combined user-dependent data of both eyes. The combination may be, for example, an arithmetic average of the PSF samples or of the user-dependent data, like the S.C.A. vision parameters. Alternatively, another mathematical combination may be used.

(81) In order to further address the situation where each eye of the user may be affected by different impairments, in some embodiments, there may be an embedded autostereoscopic monitor capable of showing a different image to each eye. In such a scenario, the PSF of each eye may be stored and two corresponding pre-corrections may be computed for each graphical/image element.

(82) In order to further address the situation where each eye of the user may be affected by different impairments, in some embodiments, a plurality of options may be presented to the user as preferences. The options may include, for example, pre-correcting using the PSF of the right eye only, pre-correcting using the PSF of the left eye only, pre-correcting using a PSF resulting from a combination of both PSF's of the left and right eyes, or pre-correcting using both PSF's of the left and right eyes as they are applied to the two elements of an autostereoscopic monitor.

(83) In some embodiments, in order to limit the amount of processing performed each time new content is available for display to the user, or each time a part of the content displayed present should be updated/changed, step 204 may be separated into two sub-steps. First, a pre-corrected image may be calculated for a special image where all pixel values may be set to 0 and the value of one central pixel is set at its maximum luminance, for each sub-pixel class. The resulting pre-corrected image may be called the “inverse PSF” and stored in memory. Pre-correction of the actual images may be performed by convolving the reference image with the inverse PSF. In some embodiments, the convolution may be performed in the frequency domain, by: (a) applying a Fourier transform of the inverse PSF that may be stored in memory and a Fast Fourier Transform (FFT) to the image of reference, (b) performing a point-wise multiplication with the Fourier transform of the inverse PSF, and (c) applying an inverse FFT to the result of the multiplication.

(84) FIG. 6 illustrates exemplary steps 201-205 for pre-correcting a reference image. Steps 201-205 do not need to be repeated for each new reference image. In some embodiments, caching techniques may be used to limit the amount of processing. Caching techniques may include, for example, where (a) the PSF is stored in memory 114 and its calculation during step 202 may be performed when there is a received data change, (b) the pre-corrected image resulting from step 204 may be stored in memory 114 if the same reference image is determined to be likely to be received again during, for example, step 203. In some embodiments, when pre-correcting text, the pre-correction may be performed separately for each letter and pre-corrected letters may be stored. When pre-corrected letters are stored, the next time text contains the same letter at the same size, the execution of step 204 for the letter may be limited to retrieving the stored letter from memory 114. In some embodiments, if step 204 uses an inverse PSF as described above, the inverse PSF may be stored with the PSF in memory 114 so that step 204, in the future, is limited to retrieving the inverse PSF and performing a convolution.

(85) In some embodiments, step 205 outputs/displays the pre-corrected image.

(86) FIG. 11 illustrates an exemplary embodiment of the pre-correction process. As shown in FIG. 11, the pre-correction module(s) are called by the compositing window manager 702, which then displays the pre-corrected image by modifying the screen buffer 704. The compositing window manager is a classical component of an operating system that aggregates the off-screen buffers 703 of each application before drawing the entire graphical interface to the screen buffer 704. In this embodiment, pre-correction may be implemented during composition of the application's display into the screen buffer and may be applied to screen buffers, or a part of the screen buffers, that contain the reference image received in step 203. This implementation may suitable when the user's preference is set to “render all” (as described above).

(87) FIG. 12 illustrates an exemplary embodiment of the pre-correction process. As shown in FIG. 12, the pre-correction module(s) are called by a font rendering module 802 of the operating system which then paints the resulting pre-corrected text. This embodiment may be used when the user preference is set to “render text”. When the application 801 needs to display text, pre-correction may be performed by the font rendering module 802, which may implement pre-correction as it would implement a rendering effect like, for example, anti-aliasing.