Method and device for ambient light estimation
10176763 ยท 2019-01-08
Assignee
Inventors
Cpc classification
G09G2370/047
PHYSICS
G09G3/3283
PHYSICS
G09G2370/045
PHYSICS
G09G2310/0272
PHYSICS
G09G2370/10
PHYSICS
G09G2310/0275
PHYSICS
G09G2300/0876
PHYSICS
G09G3/3655
PHYSICS
G09G2370/14
PHYSICS
G09G2360/16
PHYSICS
G09G3/3291
PHYSICS
International classification
G09G5/00
PHYSICS
G09G3/3291
PHYSICS
Abstract
A method of forming a control parameter dependent on ambient light. The method comprises the steps of acquiring light values from an ambient light sensor and acquiring positional status values from a positional status sensor. The control parameter depends on the light values and is filtered in dependence on the positional status values.
Claims
1. A method of forming at least one control parameter dependent on ambient light, the method comprising: acquiring light values from an ambient light sensor; acquiring positional status values from a positional status sensor; determining a variation of the light values over a time period; determining a temporal covariance value between the light values and the positional status values over the time period; applying a first temporal filtering of the light values to form a first control parameter for adjusting a display brightness of a display; and applying a second temporal filtering of the light values to form a second control parameter for adjusting pixel values of the display, the first temporal filtering different from the second temporal filtering, wherein at least one of the first control parameter or the second control parameter are formed in dependence on the temporal covariance value.
2. The method of claim 1, further comprising determining a correlation between the light values and the positional status values.
3. The method of claim 1, wherein the positional status values represent angular positions, respectively, and the method further comprises mapping the light values onto an array as a function of the angular positions.
4. The method of claim 3, wherein the angular positions are with respect to a reference angular position, the method further comprising setting the reference angular position according to an external coordinate system or according to a current angular position of the positional status detector after a fixed time interval after a change in angular position larger than a predetermined value.
5. The method of claim 1, further comprising forming at least one of a temporal, positional or orientational average of the light values.
6. The method of claim 1, wherein the positional status values represent angular positions, respectively, and the method comprises weighting the light values in dependence on the angular positions, respectively.
7. The method of claim 6, wherein the weighting is with a distribution which decreases with angle or a uniform distribution or a uniform distribution with a peak.
8. The method of claim 1, wherein the positional status values include motion.
9. The method of claim 1, wherein the ambient light includes diffuse illumination and spot illumination and at least one of a first value of the first control parameter or a second value of the second control parameter differentiates between the diffuse illumination and the spot illumination.
10. The method of claim 1, wherein at least one of a first value of the first control parameter or a second value of the second control parameter differentiates between changes in the light values due to changes in the ambient light and due to changes in a positional status of a device comprising the display.
11. The method of claim 1, comprising adjusting the display brightness of the display using the first control parameter and adjusting the pixel values of the display using the second control parameter.
12. The method according to claim 11, wherein the adjusting the pixel values comprises applying a gamma correction having an exponent dependent on the second control parameter.
13. The method of claim 1, wherein the display brightness increases with the light values.
14. The method of claim 1, comprising adjusting the pixel values such that a dynamic range of data to be displayed is smaller than or equal to a dynamic range of the display.
15. The method of claim 1, wherein the first control parameter is the display brightness and the second control parameter is a strength of dynamic compression.
16. The method of claim 1, further comprising acquiring light values from two or more ambient light sensors.
17. The method of claim 1, further comprising acquiring positional status values from two or more positional status sensors.
18. The method of claim 1, wherein the positional status relates to at least one of position and motion.
19. The method according to claim 1, comprising: determining a mean light value representative of a mean of the light values over the time period; and determining a mean positional status value representative of a mean of the positional status values over the time period, wherein the temporal covariance value corresponds to a mean of a product of a deviation of the light values from the mean light value and a deviation of the positional status values from the mean positional status value.
20. The method according to claim 1, comprising setting at least one of a first recursive filtering coefficient for the first filtering or a second recursive filtering coefficient for the second filtering in dependence on the positional status values.
21. A system for forming at least one control parameter dependent on ambient light, the system comprising: an ambient light sensor having an output to supply light values; a positional status sensor having an output to supply positional status values; and a combiner, wherein: the output of the ambient light sensor and the output of the positional status sensor are connected to inputs of the combiner, and the combiner is configured to: determine a variation of the light values over a time period; determine a temporal covariance value between the light values and the positional status values over the time period; provide a first value of a first control parameter obtained by applying a first temporal filtering to the light values, for adjusting a display brightness of a display; and provide a second value of a second control parameter obtained by applying a second temporal filtering to the light values, for adjusting pixel values of the display, the first temporal filtering different from the second temporal filtering, wherein at least one of the first control parameter or the second parameter are formed in dependence on the temporal covariance value.
22. The system of claim 21, wherein the combiner is adapted such that at least one of the first value of the first control parameter or the second value of the second control parameter differentiates between diffuse illumination and spot illumination, where the ambient light includes the diffuse illumination and the spot illumination.
23. The system of claim 21, wherein the combiner is configured such that at least one of the first value of the first control parameter or the second value of the second control parameter differentiates between changes in the light values due to changes in the ambient light and due to changes in a positional status of a device comprising the display.
24. The system of claim 21, further comprising the display, wherein the display is arranged to receive the first control parameter and the second control parameter as inputs.
25. A non-transitory computer-readable storage medium comprising computer-executable instructions which, when executed by a processor, cause a computing device to form at least one control parameter dependent on ambient light by: acquiring light values from an ambient light sensor; acquiring positional status values from a positional status sensor; determining a variation of the light values over a time period; determining a temporal covariance value between the light values and the positional status values over the time period; applying a first temporal filtering of the light values to form a first control parameter for adjusting a display brightness of a display; and applying a second temporal filtering of the light values to form a second control parameter for adjusting pixel values of the display, the first temporal filtering different from the second temporal filtering, wherein at least one of the first control parameter or the second control parameter are formed in dependence on the temporal covariance value.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
(1)
(2)
(3)
(4)
(5)
(6)
(7)
DETAILED DESCRIPTION OF CERTAIN INVENTIVE EMBODIMENTS
(8)
(9) The ambient light sensor 3 is a conventional light sensor capable of measuring a light value, i.e. a light level of the incident ambient light. The sensor is may be integrated in the device, such that the incident light captured by the sensor provides a fair estimate of the ambient light incident on the display. Ambient light is defined as the environmental lighting in which the device is being used, which may be composed of different sources including direct or spot illuminators and diffuse illuminators.
(10) The positional status sensor 4 can be a position sensor and/or a motion sensor. The positional status sensor is mechanically connected to the ambient light sensor and measures the positional status of the ambient light sensor. The sensor 4 in the figure is shown by a dashed line to indicate that the sensor is located inside the device 1. The positional status sensor can be a translational position sensor, such as a Global Positioning System (GPS) receiver and/or an angular or rotational position sensor, such as a gyroscope. The positional status sensor can also be a translational motion sensor, such as an accelerometer or a GPS receiver including a differentiator, and/or an angular or rotational motion sensor, such as a gyroscope including a differentiator. In general, a motion sensor may be based on a position sensor or an accelerometer.
(11)
(12) An output of the brightness control unit 7 is input to a display controller 9, where it may be used for improving the viewing by for example controlling the display brightness. The brightness control is dependent on a control parameter input from the combiner 5, which is derived from the ambient light. The brightness control is used, for example, for setting the intensity of a backlight of the display or for setting the maximum brightness of a pixel such as in an OLED display. The control usually sets the display brightness to low in dark ambient light conditions and to high in high ambient light conditions. Adjustment of the display brightness may for example be achieved by implementing a look-up table providing the value of display brightness for a given measured light value.
(13) The dynamic range control unit 8, which may be a video processor, adjusts the pixel values of the display. The adjustment is dependent on a control parameter input from the combiner 5, derived from the ambient light. The adjustment is used to alter the appearance of imagery displayed in order to improve the viewing experience. Adjustment of the pixel values may for example be achieved by applying a tone curve having a shape dependent on the measured light value to the input pixel values, by applying a gamma correction having an exponent defined by the measured light value to the input image or video, by adjusting the histogram of pixel values, or by applying a spatially-varying transform which acts to reduce the dynamic range of the pixel values. The dynamic range of the data to be displayed may be controlled to be smaller than or equal to a dynamic range of the display. The output of the dynamic range compression unit, for example a video stream, is input to the display controller 9.
(14) The output of the display control system, including elements 5, 7, 8 and 9, is connected to the display 2. The display, e.g. an LCD, OLED or electrowetting display, forms an image of the content for viewing by a user. Whereas the embodiment in
(15) Displays are known in the prior art where the display brightness and the video content adjustment is controlled only by an ambient light sensor. The control may produce rapid and noticeable changes in the display appearance to the viewer under conditions where the estimation of ambient light is either rapidly changing or unreliable. The problem is caused by the measured ambient light value being different from the true value of the ambient light incident on the display. The difference may be caused by inaccuracy in the ambient light measurement, e.g. due to limitation in field of view, which may be caused by a bezel in which the ambient light sensor is placed, or shading of the sensor by the viewer. The difference may also be caused by the inability of the ambient light sensor to distinguish spot illumination from diffuse illumination. Hence, a rapid change in measured light value may not indicate a commensurate change in real ambient light conditions of the display. The known control of the display by low-pass temporal filtering of the output of the ambient light sensor does, however, not differentiate between variations in the light value due to changing ambient light conditions in the environment and variations due to the relative position of the ambient light sensor itself or fluctuations in the measured light value due to errors in the measurement while the actual ambient light is unchanging.
(16) The above disadvantages in the control of the prior art displays are mitigated or removed by a control in which light values and positional status values are combined and filtered as shown in
(17) By selecting a specific way of combining the values, the control parameter may distinguish between diffuse illumination and spot illumination. The control parameter can be made dependent mainly or only on the diffuse illumination or mainly or only on the spot illumination. Dependence on a specific combination of diffuse and spot illumination is also possible. The diffuse illumination level is usually a better measure for controlling the brightness of a display than a combined spot and diffuse illumination level as used in the prior art. The control parameter may differentiate between changes in light value due to changes in the ambient light and due to changes in the positional status of the device. The control of the display brightness and the control of the pixel values may require a different dependence of the respective control parameters on the light values for a suitable adaptation of the display to the lighting conditions. For example, when the ambient light is changing rapidly, the display brightness may respond slower than the pixel values to avoid visible flicker. This can be achieved by averaging the light values over a longer period for controlling the display brightness than for controlling the pixel values.
(18) The combiner 5 outputs parameter values to the brightness controller 7 for setting the display brightness. The control parameter may be the display brightness B depending on the light values x, defined by
B.sub.i=LUT(x.sub.i);<B>.sub.i=(1a)<B>.sub.i-1+aB.sub.i(1)
(19) The display brightness B.sub.i is related via a look-up table (LUT) to the current light value x.sub.i obtained from the ambient light sensor 3. The average display brightness <B>.sub.i output to the brightness controller 7, is obtained by recursive filtering of the values of B.sub.i. Parameter a is a recursive filtering coefficient and the average is taken over successive values in time.
(20) The combiner 5 may also output parameter values to the dynamic range controller 8 for setting for example the gamma correction. The control parameter may be the strength S of dynamic range adjustment of pixel values, defined by
S.sub.i=LUT(x.sub.i);<S>.sub.i=(1b)<S>.sub.i-1+bS.sub.i(2)
(21) The parameter S.sub.i, output to the dynamic range controller 8, may be proportional to the exponent of gamma correction. Parameter b is a recursive filtering coefficient and the average is taken over successive values in time.
(22) The larger the filtering parameters a and b, the more rapidly the display brightness and pixel values are adjusted to changes in the measured light value. The value of the parameters a and b, determining the speed of adaptation of the display to changes in the ambient light, may depend on the viewing conditions or the type of lighting environment. The type of lighting environment may be determined by the variation of the light value, the variation of the positional status value and/or the relation between the light value and the positional status value. Other linear and non-linear averaging methods may be used for determining an average value of B and S.
(23) The variation of the measured light value or the positional status value may be determined by combiner 5 as the following exemplary scaled variance measure
X.sub.t=(N<x>.sup.2).sup.1.sub.i(x.sub.i<x>).sup.2
(24) where N light values or positional status values x.sub.i are taken over a defined time period between times t and t+t and <x> is the average of the N levels. The averaging period t depends on the sampling rate of the ambient light sensor or positional status sensor and is typically between 2 and 30 seconds. A usual sampling rate is 10 samples per second. If t is 5 seconds, N is 50 for the usual sampling rate.
(25) The variation of the light values is a suitable parameter for comparison with the positional status values, allowing to make a clear distinction between different illuminations. The variation may be the variance of the light values.
(26) The relation between the light value x.sub.i and the positional status value y.sub.i can, for example, be determined by the combiner 5 as the temporal covariance C.sub.t of pairs of sensor readings
C.sub.t(X,Y)=(N<x><y>).sup.1.sub.i|(x.sub.i<x>)(y.sub.i<y>)|
(27) The positional status values may for example consist of a gyroscope reading over which the average or maximal absolute value is taken over the three axes. This example does not take account of motion direction; hence y.sub.i=|y.sub.i| for all i.
(28) The embodiment of
(29) The control parameter can be an average value of the ambient light obtained by filtering of the light values x.sub.i. The average value, filtered in dependence on the positional status values, can be a more accurate estimate of the ambient light. The average value may be tuned to a specific purpose, such as the control of a display. When the light values are averaged using angular position values, a wide-angle measurement of the ambient light can be made using a narrow-angle light sensor. As the device orientation changes naturally in the hands of a user, the light values recorded will have different angular positions. The measurement at different angular positions allows simulation of a light sensor having different angular characteristics than the actual light sensor.
(30)
(31) The reference angular position may be set with respect to an external coordinate system, such as a system fixed to the surroundings or environment in which the light sensor captures the ambient light. The external coordinate system may be fixed to the earth. The position of the device with respect to the reference angular position may be determined using a gyro or a GPS as positional status sensor. Alternatively, the reference angular position may be set by the angular position of the positional status sensor, i.e. usually the angular position of the device, at a certain moment in time. In the embodiment of
(32) The reference angular position can be updated after fixed time intervals, e.g. by setting the reference angular position equal to the current direction perpendicular to the display 2, which may be determined using a gyro or a GPS. The reference angular position may also be a moving average over the angular positions of the positional status sensor over time. Alternatively, the reference angular position may be updated after a large change in orientation is detected, i.e. a change in angular position larger than a predetermined value, e.g. 10 degrees.
(33) An angular filtering of the time-varying ambient light sensor readings can be defined as follows:
X=N.sup.1x(.sub.1,.sub.2)(.sub.1,.sub.2)d.sub.1.sub.2(3)
(34) where x is an ambient light value recorded when the device is at a given orientation .sub.1, .sub.2; is an angular weighting function which is typically radially symmetric, N is a normalization factor, and X is the orientational average ambient light value. A single ambient light value X is thereby obtained from a sequence of ambient light samples recorded at different times while the device changes its orientation.
(35) The value X.sub.i representing the value of X calculated at time i then replaces the time-sampled x in Eqs 1 and 2
B.sub.i=LUT(X.sub.i);<B>.sub.i=(1a)<B>.sub.i-1+aB.sub.i(5)
S.sub.i=LUT(X.sub.i);<S>.sub.i=(1b)<S>.sub.i-1+bS.sub.i(6)
(36) A light sensor having a wide-angle lens can be simulated by choosing a function such that it weights the ambient light values acquired by a narrow-angle light sensor with a distribution which decreases with angle between the current angular position v at which the sensor measures the ambient light and the reference angular direction, such as a cosine function. Alternatively, different cosine functions in two perpendicular angular directions, e.g. as shown by .sub.1, .sub.2 in
(37) When the cosine-function is replaced with a function that is uniformly distributed except for a narrow peak near the reference angular position, a combination of a narrow-angle ambient light sensor and a very wide-angle ambient light sensor is simulated. This allows for direct determination of the specular, haze and diffuse components of the ambient light.
(38) It may be desirable to reset the filtering provided in Eq (4) whenever the device experiences a large or rapid displacement or orientational change, indicating that the device has moved into a different ambient light environment.
(39) For computational purposes, it is convenient to map the measured light values onto a two-dimensional array of nodes at discrete angular positions 1 and 2, as shown in
X=N.sup.1.sub..sub.
(40) where now the sums run over discrete angular positions and the value x represents the ambient light sensor value sampled at the coordinate closest to a given (.sub.1, .sub.2) node in the array, weighted by a value dependent on the node position. In practice, only a subset of nodes will be occupied with ambient light values at a given time, as indicated by the black dots in
(41) The array of light values can be used to generate a map of the ambient light sources in the environment of the device. For example, a spot source will generate a characteristic pattern of high values at array positions corresponding to direct illumination, while a diffuse source will contribute uniformly to array positions. Such an ambient light map may be used to record for example from which direction sunlight is incident, e.g. through a window, and to adjust selectively device parameters such as screen brightness when the device is held facing to such a source. For example, array node values in excess of 1000 lux may typically indicate directions from which natural light is incident in the environment, while values less than 100 lux indicated directions in which the environment is in shadow.
(42)
(43) The following four exemplary scenarios show how the control parameters can distinguish between different viewing conditions. In the examples, the term high is associated with an average sensor value which exceeds a noise threshold, set to exclude values below the accuracy of the measurement system, and low with values below such a threshold. The four scenarios use an accelerometer as positional status sensor. Similar results can be obtained with any translational motion sensor.
(44) In the first scenario the variance of the light value ll, X.sub.ll, t, is low. Hence, the device is likely in diffuse lighting conditions. The display can be set to adapt smoothly to changes in light value. Since the device is stationary or moving slowly because of the small accelerometer value, a cautious adaptation to the light value is preferred; for example a=b=0.5. The control policy is shown in
(45) In the second scenario the average accelerometer value is high and the variance of the light value is low. Since the changes in device position and/or orientation imply a sampling of the lighting environment and these samples show a low variance, it is probable that the device is in diffuse lighting. In such an environment the rate of adaptation can be increased, for example a=b=0.75.
(46) In the third scenario the average accelerometer value is high and the variance of the light value is also high. This is suggestive of a spot lighting environment, where the measured light values are highly dependent on device position and/or orientation. Under such conditions, it is unsafe to adapt the display quickly to light value changes, implying or example a=b=0.05. The control policy is shown in
(47) In the fourth scenario the average accelerometer value is low and the variance of the light value is high. This is suggestive of a moving vehicle, where ambient light may vary rapidly while the device is held with relative stability. Here, the changes in ambient light may be due to passing under trees, in and out of shadows cast by buildings, or in and out of tunnels. To minimize the effect of the rapidly changing ambient light on the user's impression of the display, it is desirable to adapt the screen as rapidly as possible. To avoid the risk of visible flicker, this may be achieved by pixel brightness adjustment, which can be performed uniquely for each video frame, as opposed to screen brightness changes, which may yield observable flicker. For example, a=0.5, b=1. The control policy is shown in
(48) The above four scenarios can be more accurately differentiated if alternative or additional positional status sensor information is available, e.g. by arranging an alternative or additional positional status sensors in the device. For example, an integrated gyroscope provides relative device orientation along each axis. If the covariance between light value and device orientation along an axis C.sub.t(X.sub.ll, Y.sub.x-gyro) is high, this implies that the light value is a function of device orientation, which strongly indicates that the device is viewed in spot lighting conditions. The light values may be acquired by one or more further ambient light sensors, which can improve the differentiation between scenarios.
(49) The graph in
(50) A GPS receiver may be used to differentiate more accurately the fourth scenario above, in which the device is being viewed within a moving vehicle. If accelerometer and gyroscope sensors indicate that the device is exhibiting a low degree of relative motion, while the GPS indicates the device is in uniform motion, changes in light value readings can be confidently associated with changes in ambient light.
(51) The positional status sensor may be used to determine whether readings from the ambient light sensor are accurate or not. For example, when the display is in a preferred position, such as with the ambient light sensor on top, the readings will be accurate. If the positional status sensor detects that the ambient light sensor is not on top, for example because the display is held upside down, the measured light values may be inaccurate and the value of the control parameter may be kept at a constant value until the display is brought back to a preferred orientation.
(52) An alternative application is a display of a mobile device, such as a mobile phone, on which a graphical application is shown, such as a video game, or an augmented reality application where computer graphics are superimposed onto a video display. The graphics may be adjusted depending on the direction the device is pointing in relation to the sun to make the graphics blend well into the surroundings.
(53) It will be understood that the processors or processing systems referred to herein may in practice be provided by a single chip or integrated circuit or plural chips or integrated circuits, optionally provided as a chipset, an application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), etc. The chip or chips may comprise circuitry (as well as possibly firmware) for embodying at least one or more of a data processor or processors, a digital signal processor or processors, which are configurable so as to operate in accordance with the exemplary embodiments. In this regard, the exemplary embodiments of the method may be implemented at least in part by one or more computer programs stored in memory and executable by the processor, or by hardware, or by a combination of tangibly stored software and hard-ware (and tangibly stored firmware). The one or more computer programs may be stored on a record carrier.
(54) The above embodiments are to be understood as illustrative examples. Further embodiments are envisaged. Alternative statistics will be evident to a person skilled in the art. For example, it may be desirable to correlate the rate of change of the light value measurements with the rate of change of device orientation. The values and formulae above are intended for illustration only and other alternatives will be evident to a person skilled in the art. It is to be understood that any feature described in relation to any one embodiment may be used alone, or in combination with other features described, and may also be used in combination with one or more features of any other of the embodiments, or any combination of any other of the embodiments. Furthermore, equivalents and modifications not described above may also be employed without departing from the scope of the accompanying claims.