Method for outputting a signal from an event-based sensor, and event-based sensor using such method

Abstract

An event-based sensor having a sensing part that produces events asynchronously. The output signal includes information relating to at least some of the produced events. The method comprises: estimating a rate of production of events by the sensing part; while the estimated rate is less than a threshold, transmitting the signal having a variable rate; and when the estimated rate is above the threshold, transmitting the signal including information relating to only some of the produced events, such that a rate of events for which information is included in the signal remains within the threshold.

Claims

1. A method for outputting a signal from an event-based sensor, wherein the event-based sensor has a sensing part that produces events asynchronously and the signal includes information relating to at least some of the produced events, the method comprising: estimating a rate of production of events by the sensing part; while the estimated rate is less than an event rate threshold, transmitting the signal having a variable rate; and when the estimated rate is above the event rate threshold, transmitting the signal including information relating to only some of the produced events, such that a rate of events for which information is included in the signal remains within the event rate threshold.

2. The method of claim 1, further comprising receiving a value of the event rate threshold as a configuration parameter.

3. The method of claim 1, further comprising receiving a value of a period for estimating the rate of production of events as a configuration parameter.

4. The method of claim 1, wherein at least one filter is applied to the produced events in response to the estimated rate exceeding the event rate threshold, so as to discard selected events in the signal to be outputted.

5. The method of claim 4, wherein the at least one filter comprises a temporal filter in which the selected events occur in accordance with a temporal pattern.

6. The method of claim 4, wherein the sensing part comprises a plurality of spatially distributed sensing elements each having a respective address and producing a respective sequence of events, and wherein the at least one filter comprises a spatial filter in which the selected events are events produced by sensing elements at addresses corresponding to a spatial pattern.

7. The method of claim 6, wherein the sensing elements of the sensing part are arranged in a two-dimensional matrix, and wherein the at least one filter includes a first spatial filter in which the selected events are events produced by sensing elements at addresses corresponding to rows of the matrix and a second spatial filter in which the selected events are events produced by sensing elements at addresses corresponding to columns of the matrix.

8. An event-based sensor, comprising: a sensing part for producing events asynchronously; an output port; and a controller for providing a signal to the output port, including information relating to at least some of the events produced by the sensing part, wherein the controller is configured to: estimate a rate of production of events by the sensing part; while the estimated rate is less than the event rate threshold, provide the signal having a variable rate to the output port; and when the estimated rate is above the event rate threshold, provide the signal including information relating to only some of the produced events to the output port, such that a rate of events for which information is included in the signal remains within the event rate threshold.

9. The device of claim 8, wherein the controller is configured to receive a value of the event rate threshold as a configuration parameter.

10. The device of claim 8, wherein the controller is configured to receive a value of a period for estimating the rate of production of events as a configuration parameter.

11. The device of claim 8, wherein the controller comprises at least one filter which is applied to the produced events in response to the estimated rate exceeding the event rate threshold, so as to discard selected events in the signal provided to the output port.

12. The device of claim 11, wherein the at least one filter comprises a temporal filter in which the selected events occur in accordance with a temporal pattern.

13. The device of claim 11, wherein the sensing part comprises a plurality of spatially distributed sensing elements each having a respective address and producing a respective sequence of events, and wherein the at least one filter comprises a spatial filter in which the selected events are events produced by sensing elements at addresses corresponding to a spatial pattern.

14. The device of claim 13, wherein the sensing elements of the sensing part are arranged in a two-dimensional matrix, and wherein the at least one filter includes a first spatial filter in which the selected events are events produced by sensing elements at addresses corresponding to rows of the matrix and a second spatial filter in which the selected events are events produced by sensing elements at addresses corresponding to columns of the matrix.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) FIG. 1 is a block diagram of a system using an event-based sensor according to an embodiment.

(2) FIG. 2 is a block diagram of an example of event rate controller usable in a system as shown in FIG. 1.

DETAILED DESCRIPTION

(3) The system shown in FIG. 1 comprises an event-based asynchronous vision sensor 10 placed in front of a scene and receiving light from the scene through acquisition optics 11 comprising one or more lenses. The sensor 10 includes a sensing part 12 placed in the image plane of the acquisition optics 11. The sensing part 12 may comprise an array of photosensitive elements organized as a matrix of pixels. Each pixel corresponding to a photosensitive element p produces successive events ev(p, t) at times t as a function of variations of light in the scene.

(4) A processor 20 processes the asynchronous information originating from the sensor 10, including events ev(p, t) received asynchronously from the photosensitive elements p of the sensing part 12. The processor 20 operates on digital signals using any suitable programming technique, or hardware arrangement. The type of processing executed by the processor 20 is application-dependent and is not a feature of the present disclosure.

(5) The sensing part 12 of the sensor 10 generates a sequence of events for each pixel p of the matrix, using the variations of light detected by the pixel in the scene that appears in the field of vision of the sensor. Those event sequences from the pixels are aggregated into a signal by a controller 15 associated with the output interface of the sensor 10. The signal uses an address-event representation (AER) and is supplied to the processor 20 through an output port 16 of the sensor 10 and a communication link 18. Each event in the output signal includes at least one spike whose temporal position indicates a change in luminance seen by a pixel, associated with an address of that pixel in the matrix. One or more luminance attribute(s), such polarity information and/or a luminance value, can also be associated with the event. The communication link 18 may use any suitable communication medium and any suitable communication protocol. As an illustration, it may be based on a universal serial bus (USB) port 16.

(6) By way of example, the sensing part 12 that carries out the acquisition of events can be embodied as a dynamic vision sensor (DVS) of the type described in “A 128×128 120 dB 15 μs Latency Asynchronous Temporal Contrast Vision Sensor”, P. Lichtsteiner, et al., IEEE Journal of Solid-State Circuits, Vol. 43, No. 2, February 2008, pp. 566-576, or in patent application US 2008/0135731 A1.

(7) Another example of an asynchronous sensor that can take advantage of an embodiment is the asynchronous time-based image sensor (ATIS) of which a description is given in the article “A QVGA 143 dB Dynamic Range Frame-Free PWM Image Sensor With Lossless Pixel-Level Video Compression and Time-Domain CDS”, C. Posch, et al., IEEE Journal of Solid-State Circuits, Vol. 46, No. 1, January 2011, pp. 259-275.

(8) The controller 15 is referred to as an event rate controller (ERC) in the following. A possible embodiment of the ERC is illustrated in FIG. 2.

(9) In that embodiment, the ERC 15 processes data in series as provided by the readout circuitry of the sensing part 12. The ERC 15 has a rate measurement component 30 which receives the events generated by the array 12 of sensing elements. The event rate R.sub.E is estimated by the component 30, for example by simply counting the events detected by the array 12 in a given time period T. The event count C.sub.E (=R.sub.E×T) is provided to a comparator 31 that compares it to a threshold S.

(10) While C.sub.E≤S, namely while the event rate R.sub.E is within a threshold S/T, the events produced by the array 12 of sensing elements are directly fed to the output port 16 for transmission to the processor 20. This is illustrated diagrammatically in FIG. 2 with switches 32. In this case, the signal transmitted through the output port 16 retains the variable rate of the events produced by the array 12.

(11) When the comparator 31 reveals that C.sub.E>S, namely that the event rate R.sub.E exceeds the threshold S/T, the stream of events from the array 12 is subjected to a filter bank of the ERC 15 which, in the illustrated embodiment, includes two spatial filters 33, 34 and a temporal filter 35 (the illustrative switches 32 are then in the positions shown in FIG. 2).

(12) The filters 33-35 apply sub-sampling to the events from the array 12 such that the rate of events in the signal transmitted through the output port 16 remains within the threshold S/T. In other words, each filter 33-35 discards some events in the signal, the events discarded by one of the filters being selected according to a respective pattern H_P, V_P, T_P indicated by a configuration module 40 of the ERC 15.

(13) In the embodiment shown in FIG. 2, the filter bank includes two spatial filters 33-34 followed by a temporal filter 35.

(14) In the spatial filters 33-34, the patterns H_P, V_P for discarding events correspond to regions of the array of sensing elements, i.e. to specific addresses of sensing elements in the matrix forming the array 12. There is a horizontal filter 33 in which the events to be discarded originate from sensing elements at addresses corresponding to rows of the matrix, followed by a vertical filter 34 in which the selected events originate from sensing elements at addresses corresponding to columns of the matrix. The rows of the matrix where the horizontal filter 33 discards events may be regularly spaced, with a density D.sub.H. The columns of the matrix where the vertical filter 34 discards events may be regularly spaced, with a density D.sub.V.

(15) In the temporal filter 35, the pattern T_P for discarding events corresponds to certain time intervals in which events are discarded, with a drop duty ratio D.sub.T.

(16) When the filters 33-35 are cascaded, as shown in FIG. 2, the overall drop rate D′ applied by the filter bank may be calculated as D′=[1−(1−D.sub.H)×(1−D.sub.V)×(1−D.sub.T)]. The respective drop rates D.sub.H, D.sub.V, D.sub.T of the filters 33, 34, 35 are set by the configuration module 40 as a function of a target drop rate D given by the comparator 31, such that D′=D or D′≈D.

(17) The order in which the three filters process the stream of events may be different from the one illustrated in FIG. 2. In addition, there can be less than three filters, or more than three filters, to drop events in a controlled manner when the estimated event rate R.sub.E exceeds the threshold S/T. In addition, the space/time filtering of the events can have forms different from those presented above. For example, a spatial filter may apply a drop pattern in which the selected events are located both in specified columns and in specified rows

(18) In an embodiment of the ERC, the comparator 31 provides, at each period T, a target drop rate D that depends on how the estimated event rate R.sub.E compares with the threshold S/T. By way of example, the target drop rate D may be quantized using a number N of bits representing an integer A=0, 1, 2, . . . , 2N.sup.−1, such that D=A/2.sup.N, where A=└2.sup.N.Math.(1−S/C.sub.E)┘. When C.sub.E≤S, A is zero, i.e. no events are discarded by the filters 33-35. Otherwise, the comparator 31 puts the value of the event count C.sub.E in one of 2.sup.N−1 bins determined based on the threshold S. Which bin receives the event count C.sub.E determines the N bits quantizing the integer A that gives the drop rate D. Again, no complex computation is needed for such processing.

(19) The configuration module 40 receives the drop rate D from the comparator 31 and determines patterns H_P, V_P, T_P to be used by the filters 33-35 when R.sub.E>S/T, by setting their respective event drop rates D.sub.H, D.sub.V, D.sub.T as mentioned above. The patterns H_P, V_P, T_P may be determined by the configuration module 40 using a predefined lookup table accessed using addresses formed by the bits representing the target drop rate D.

(20) How the target drop rate D is split into three event drop rates D.sub.H, D.sub.V, D.sub.T may depend on a configuration parameter provided by the designer of the configuration module, or received from the processor 20 through the communication link 18. Depending on the application run by the processor 20, filtering strategies with more or less importance on the temporal, vertical or horizontal drop patterns may be more suitable. This can be specified by setting the configuration parameter. There can be a default strategy, for example to split the target drop rate D equally between the three rates D.sub.H, D.sub.V, D.sub.T, when no configuration parameter is provided.

(21) The reference period T in which the events are counted is another configuration parameter which can be set by the user interface provided on the communication link 18. The processor 20 may select the reference period T according to its own dynamic behavior given that, when a longer period T is selected there can remain a risk that bursts of events cause temporary overflow at the output of the sensor 10 or at the input of the processor 20. Typically, when the abovementioned examples of event-based sensors 10 are used, the reference period T may be in a range of 10 to 500 microseconds.

(22) A main configuration parameter is the value of the threshold S to which the event count C.sub.E is compared. The ratio S/T is directly linked to the throughput of the signal sent by the sensor 10 to the processor 20. The value of S can also be set by the user interface provided on the communication link 18.

(23) The ERC 15 illustrated diagrammatically in FIG. 2 does not include a buffer for temporarily storing events prior to filtering. In practice, there can be an event buffer, whose dimension is larger when the reference period T increases, such that the target drop rate D determined for a period T is applied to the set of events generated by the array 12 during the same period T. However, presence of such a buffer is not a requirement in all applications. It is also possible to buffer events over a time period shorter than the reference period T.

(24) It is noted that the process run by the ERC 15 is fairly simple to achieve the function of removing excess events in a controlled manner. This removes the need to have a filter using CPU capacity to sort events.

(25) It will be appreciated that the embodiments described above are illustrative and that various modifications can be made without departing from the scope as defined in the appended claims.

(26) For example, some of the events produced by the array 12 may be discarded prior to transmission to the processor 20 due to other functions carried out at the level of the sensor 10, for example functions that remove events carrying little information (e.g. noise, flickering effects, trail filter, etc.). The ERC processing described above is usable at the output interface to make sure that the output signal of the event-based sensor will meet the desired data rate.

(27) In other embodiments, the event drop rate may be set to vary across the field of view of the sensing part 12. For example, spatial filters can be used to provide a grid of areas for preferentially dropping events. Which filters are applied by the ERC 15, and with which parameters, may be configured as a function of the application run by the processor 20.

(28) An embodiment may be useful in a variety of devices including, without limitation, autonomous vehicles, prosthetic devices, autonomous and robotic apparatus such as manufacturing robots, military or medical robotic devices.

(29) It may further be applicable in a wide variety of stationary and portable devices, such as, for example, smart phones, portable communication devices, notebook, netbook and tablet computers, surveillance systems, and practically any other computerized device configured to process vision data.

(30) Implementations may be used in many applications including computer human interaction (e.g., recognition of gestures, voice, posture, face, and/or other applications), controlling processes (e.g., an industrial robot, autonomous and other vehicles), following movements of a set of interest points or objects (e.g., vehicles or humans) in the image sequence and with respect to the image plane, augmented reality applications, virtual reality applications, access control (e.g., opening a door based on a gesture, opening an access way based on detection of an authorized person), detecting events (e.g., for visual surveillance or people or animal), counting, tracking, etc. Myriads of other applications exist that will be recognized by those of ordinary skilled in the art given the present disclosure.