Data rate control for event-based vision sensor

Abstract

In dynamic vision sensor (DVS) or change detection sensors, the chip or sensor is configured to control or modulate the event rate. For example, this control can be used to keep the event rate close to a desired rate or within desired bounds. Adapting the configuration of the sensor to the scene by changing the ON-event and/or the OFF-event thresholds, allows having necessary amount of data, but not much more than necessary, such that the overall system gets as much information about its state as possible.

Claims

1. A change detection sensor, comprising: a pixel array comprising pixels that detect light, wherein the pixels comprise photosensors and photoreceptor circuits that output first signals based on the response of the photosensors; event detectors for detecting events associated with light received by the pixels by comparing the first signals and a first reference voltage; an event rate detector for counting the events with a counter; and a controller for changing how the pixels detect the events based on the event rate detector by changing the first reference voltage; wherein the controller compares an event rate provided by the event rate detector to a maximum event rate, if the event rate is larger than the maximum event rate the controller changes a time resolution of the sensor.

2. The sensor as claimed in claim 1, further comprising a threshold generation circuit for setting thresholds applied by the event detectors by providing the first reference voltage.

3. The sensor as claimed in claim 1, wherein the controller changes the first reference voltage in response to the assessment provided by the event rate detector.

4. The sensor as claimed in claim 1, wherein the controller sets thresholds for ON events based on ON events in the pixel array based on the counter.

5. The sensor as claimed in claim 1, wherein each of the pixels comprises a memory capacitor of one of the event detectors for storing a charge corresponding to received light when the pixel was reset.

6. The sensor as claimed in claim 1, wherein each of the pixels comprises one or more comparators of one of the event detectors for comparing the changes in received light to the first reference voltage and a second reference voltage to detect the events.

7. The sensor as claimed in claim 1, further comprising comparators of the event detectors for comparing changes in received light to the first reference voltage to detect the events, the comparators being located in a readout circuit.

8. The sensor as claimed in claim 1, wherein the counter is incremented based on the events read out by a readout circuit.

9. The sensor as claimed in claim 1, wherein the counter is reset by the controller when a new count is required.

10. The sensor as claimed in claim 1, wherein the controller reduces the number of events during a sleep cycle and then increases the number of events during normal operating mode.

11. The sensor as claimed in claim 1, wherein the controller sets thresholds for ON events and OFF events based on ON events and OFF events in the pixel array based on the counter.

12. A method of operation of a change detection sensor, the method comprising: detecting light with pixels of a pixel array, wherein the pixels comprise circuits that output first signals based on the response of photosensors of the pixels; detecting events associated with light received by the pixels by comparing the first signals and a first reference voltage; counting the events with a counter with an event rate detector; changing how the pixels detect the events based on the event rate detector by changing the first reference voltage; comparing an event rate provided by the event rate detector to a maximum event rate; and if the event rate is larger than the maximum event rate, changing a time resolution of the sensor.

13. The method as claimed in claim 12, further comprising changing the first reference voltage in response to the assessment provided by the event rate detector.

14. The method as claimed in claim 12, further comprising setting thresholds for ON events based on ON events in the pixel array based.

15. The method as claimed in claim 12, wherein each of the pixels comprises a memory capacitor for storing a charge corresponding to received light when the pixel was reset.

16. The method as claimed in claim 12, wherein each of the pixels comprises one or more comparators for comparing the changes in received light to the first reference voltage and a second reference voltage to detect the events.

17. The method as claimed in claim 12, further comprising comparing changes in received light to the first reference voltage to detect the events using comparators located in a readout circuit.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) In the accompanying drawings, reference characters refer to the same parts throughout the different views. The drawings are not necessarily to scale; emphasis has instead been placed upon illustrating the principles of the invention. Of the drawings:

(2) FIG. 1A is a block diagram of an event based change detection sensor including a event rate control.

(3) FIG. 1B is a circuit diagram showing one example of a change detection pixel that could be used in the pixel array 110.

(4) FIG. 2 is a block diagram showing one embodiment of the event rate detector 200 implemented as an event counter that is reset periodically.

(5) FIGS. 3A and 3B are flow diagrams illustrating the operation of the controller 120 for controlling the threshold generation circuit based on the event rate detector.

(6) FIG. 4 is a schematic diagram showing an event based change detection sensor that generates an event rate estimate according to another embodiment.

(7) FIG. 5 is a schematic diagram showing an event based change detection sensor that generates an event rate estimate according to still another embodiment.

(8) FIG. 6 is a circuit diagram showing a possible implementation of the current ADC 212 of the previous embodiments.

(9) FIG. 7 is a state diagram showing a possible implementation of the controller 120.

(10) FIG. 8 is a block diagram showing a basic threshold generation circuit.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

(11) FIG. 1A is a block diagram of an event based change detection sensor 100 having event rate control.

(12) In general, the change detection sensor 100 include five elements that are specifically relevant to the present system:

(13) array of change detection pixels 110;

(14) an event rate detector 200;

(15) a controller 120;

(16) a threshold generation circuit 130; and

(17) a readout circuit 140.

(18) The pixel array 110 generates a variable amount of data: change events. The event rate detector 200 periodically assesses the events such as by counting or estimating the change events. Then the controller 120 adapts the sensor configuration so that in the next time step the pixel array 110 generates an amount of change events closer to the desired rate.

(19) The pixel array can be any array of change detection pixels such as described in U.S. Pat. Appl. No. US 2008/00135731, by Lichtsteiner, et al., entitled “Photoarray for Detecting Time-Dependent Image Data”, which is incorporated herein by this reference; or such as described in U.S. patent application Ser. No. 15/858,427, entitled Dynamic Vision Sensor Architecture, filed on Dec. 29, 2017, (hereinafter the Berner Patent Document, which document is incorporated herein by this reference in its entirety); or such as those disclosed in J. Kramer, “An on/off transient imager with event-driven, asynchronous read-out,” IEEE International Symposium on Circuits and Systems, 2002, vol. 2, pp. 165-168, 2002) or alternatively spatial contrast detection pixels such as in Ruedit et al., “A 128×128 pixel 120-dB dynamic-range vision-sensor chip for image contrast and orientation extraction,” IEEE Journal of Solid-State Circuits, vol 38, no 12, pp 2325-2333, December 2003.

(20) FIG. 1B is a circuit diagram of one possible change detection pixel 112.

(21) This exemplary pixel 112 of the two dimensional array 110 includes a photosensor such as a photodiode PD that detects or measures impinging light and converts the light intensity into a signal. Here the signal is a current, Iphoto. The pixel 112 also has an event detector 114 that then determines whether there has been a sufficient change in the light detected by the photosensor such that an event has occurred. There is this accomplished by monitoring the current Iphoto.

(22) In more detail, the exemplary event detector 114 includes a photoreceptor circuit PRC that then generates a photoreceptor signal Vpr dependent on the light intensity. A memory capacitor C1 then remembers past photoreceptor signals.

(23) The comparator A1 compares the difference between current photoreceptor signal Vpr and past photoreceptor signal to a threshold Vb. This threshold Vb is supplied by the threshold generation circuit 130.

(24) Preferably, the comparator A1, or a pair of comparators, detect ON events, based on an ON event threshold, and OFF events based on an OFF event threshold. ON events are characteristic of an increase in the light received by the photosensor, and OFF events are characteristic of a decrease in the light received by the photosensor. If a single comparator is used, then the ON event threshold and the OFF event threshold are provided serially in time as threshold Vb.

(25) This comparator A1 can be in each pixel, or shared between a subset (for example a column) of pixels. In one case, the comparators are shared and located in the readout circuit 140. Different examples of such a configuration are disclosed in the Berner Patent Document.

(26) In the preferred embodiment, however, the comparator A1 will be integral to the pixel 112, with each pixel having one or more dedicated comparators A1.

(27) Memory 50 stores the event or events. The events are taken from the comparator output Vcomp, based on a sample signal from the controller 120. Memory can be a sampling circuit (for example a switch and a parasitic or explicit capacitor) or a digital memory circuit (a latch or a flip-flop). In one embodiment the memory will be a sampling circuit and each pixel will have two memories, one for storing any ON event, and one for storing any OFF event. The memory supplies the output on one or more lines Evt.

(28) A conditional reset circuit R1 provides a conditional reset based on a combination of the state of the memorized comparator output and a reset signal GlobalReset applied by the controller 120.

(29) The pixel circuit 112 and controller 120 operate as follows.

(30) A change in light intensity received by the photosensor PD will translate to a change in photoreceptor signal Vpr. When the reset circuit R1 is not conducting, the changes in Vpr will be reflected also in the voltage Vdiff at the inverting input (−) to the comparator A1. This occurs because the voltage across the memory capacitor C1 stays constant.

(31) At times selected by the controller 120, the comparator A1 of the event detector 114 compares the voltage at the second terminal of the memory capacitor C1 (Vdiff) to a threshold voltage Vb (from the threshold generation circuit 130) applied to the non-inverting input (+) of the comparator A1.

(32) The controller 120 operates the memory 50 to store the comparator output Vcomp. The memory 50 is typically implemented as part of the pixel circuit 112 as shown.

(33) In other embodiments, however, the memory 50 is implemented as part of readout circuit 140 (peripheral circuit, one per each column of the pixel array 110).

(34) In the illustrated embodiment, the sensor's readout circuit 140 reads out the memories 50 for each of the pixels 112 in the array 110. In one example, the readout circuit 140 stores the coordinates of the pixels in the array that have detected change events.

(35) If the state of the stored comparator output held in the memory 50 indicates sufficient change in light intensity (i.e., an event) AND the global reset signal GlobalReset signal from the controller 120 is active, the conditional reset circuit R1 is conducting. Here “AND” indicates the logical AND operator. With the conditional reset circuit R1 in a conductive state, the voltage at the inverting input of the comparator A1 (Vdiff) is reset to a known level. Thus, it stores the current photoreceptor signal Vpr on the memory capacitor C1.

(36) Event Counting

(37) FIG. 2 shows one embodiment of the event rate detector 200.

(38) Here, the event rate detector 200 is implemented as a counter 210 that is periodically reset by the controller 120. This counter is incremented based on the events read out by the readout circuit 140. The event count is provided to the controller 120. The counter 210 is in turn reset by the controller 120 when a new count is required.

(39) In one example, the event rate detector 200 counts the number of events output by the sensor array 110 during a given time window. Here, this is accomplished such that the readout circuit sends an Increment signal to the counter 210 every time it reads out an event from the pixel array 110.

(40) In one specific example, the event rate detector 200 counts the number of both ON events output by the sensor array 110 and OFF events output by the sensor array 110, during a given time window. This is preferably accomplished using a separate ON event counter and OFF event counter within counter 210.

(41) FIG. 3A is a flow diagram illustrating the operation of the controller 120 for controlling the threshold generation circuit 130 based on the event rate detector 200 according to one embodiment.

(42) Specifically, in step 310, the counter 210 of the event rate detector 200 is reset by the controller 120 when a new count is required.

(43) In step 312, the controller 120 reads out the count of the events after a predefined time.

(44) Then, in step 314 the controller calculates a change in the event threshold that is to be applied to the pixel array 110. This new threshold is sent to the threshold generation circuit 130 in step 316. Then the threshold is applied to the event detectors 114 of the sensor array 110 as threshold Vb in determining the next set of events.

(45) FIG. 3B is a flow diagram illustrating the operation of the controller 120 for controlling the event rate detector 200 according to another embodiment.

(46) In this embodiment, the controller 120 waits for an external trigger in step 318 before resetting the counter in step 310. This embodiment would be most useful in the situation in which the sensor should be synchronized to an external timing source.

(47) Event Count Estimator

(48) A second approach relies on generating an event rate estimate. This approach employs an event count estimator within the event rate detector 200. It avoids having to read all the events from the array to determine the number of events. Instead, this approach allows roughly estimating the number of events, before the events are read out or as part of the read out process.

(49) FIG. 4 is a schematic diagram showing an event based change detection sensor 100 that generates an event rate estimate according to another embodiment.

(50) In more detail, an event detector 114 is located in each pixel 112 of the array 110 and is part of each pixel circuit, along with a photoreceptor 116 for detecting incident light. A current source 118 of each pixel circuit 112 is enabled only if an event is present in this pixel.

(51) Specifically, if the event detector 114 registers a change in the light received by the photosensor 116, such as due to an ON event and/or an OFF event, then the event signal Evt becomes active, which closes the switch 117 associated with that pixel 112.

(52) The current of all pixels 112 of the array 110 or a sampling of pixels distributed through the extent of the array 110 of the sensor 100 is summed up and fed to a current analog to digital converter (ADC) 212 of the event rate detector 200. This ADC 212 takes the current and converts the magnitude of the current into a digital representation. This digital representation of the sum of the currents is then provided to the controller 120 as an estimate of the number of detected events, and if performed over a known period of time, it is indicative of the event rate.

(53) In general, the current sources 118 will not be very well matched between pixels 112. This means that the sum of the currents will not allow an exact estimation of how many pixels have an event pending, but it will allow a rough estimate.

(54) FIG. 5 is a schematic diagram showing a change detection sensor that generates an event rate estimate according to another embodiment.

(55) In more detail, a photosensor 116 of each pixel 112 generates a signal that is responsive to and indicative of the instantaneous amount of light detected by that photoreceptor 116. This information is sent to the event detector 114.

(56) Thus, the output of the comparator in the event detector 114 changes when the light received by the photoreceptor 116 has changed by an amount greater than the threshold with respect to the last time it was reset. Often, an ON event threshold is supplied to the pixel as the Pixel Threshold to discriminate ON events, which are associated with an increase in the amount of light detected by the photoreceptor 116. Then an OFF event threshold is supplied to the pixel as the Pixel Threshold to discriminate OFF events, which are associated with a decrease in the amount of light detected by the photoreceptor.

(57) In other cases, the event detector includes two comparators, a first comparator that receives the ON event threshold to discriminate ON events, and a second comparator that receives the OFF event threshold to discriminate OFF events.

(58) If the signal Enable evt counter is active (and thus Disable evt counter inactive), all the readout lines are shorted together by the closing of switches 188 and the readout lines are disconnected from the column readout circuits 180 by the opening of switches 190. With all the readout lines connected to the current ADC 212, a current will flow from the current ADC 212 to all pixels in which the comparator output is high and Evt is active to thereby close the switches 117 in each pixel 112. Further, the OR gates 186 for each row will close the switches 178 in each pixel 112 since Enable Evt counter is active. So the total current will depend on the number of pixels in which the output of the comparator 174 is high or signal line(s) Evt is/are active.

(59) If the signal Enable evt counter is inactive (and thus Disable evt counter active), the pixel readout lines are connected to their corresponding column readout circuits 180 if the corresponding Readout row select X is active.

(60) FIG. 6 shows a possible implementation of the current ADC 212 of the previous embodiments.

(61) The summed current is fed to an operational amplifier A 214 with a resistor R 216 in a negative feedback configuration. The voltage difference between the reference voltage and the voltage at the output of the operational amplifier is proportional to the magnitude of the input current and represents thus the number of pending events.

(62) The voltage at the output of the operational amplifier is then fed to a conventional voltage ADC 218, which can be implemented as a flash ADC, a SAR ADC, a sigma-delta ADC, etc.

(63) After the current to voltage conversion, a voltage sampling step (between the out of the OpAmp 214 and the voltage ADC 218) can be added, so that the ADC conversion, which can be long, can be done while the pixels are read out. Once the ADC conversion is finished, the readout of the pixels can be stopped if needed.

(64) Controller 120

(65) The controller 120 is most likely a digital block that has as input the event rate detector 200 and a desired event rate (or bounds for the desired event rate) from outside the sensor 100, and optionally a trigger input. The controller 120 outputs control signals for the chip configuration (i.e., time resolution and threshold to the threshold generation circuit 130).

(66) The controller 120 can be implemented as a traditional P, PI or PID controller, or more simply as a state machine.

(67) However, when using an event rate estimator, the controller 120 could also be implemented as an analog circuit. The input to this analog controller then would not be a digital representation; instead the output voltage of the operation amplifier 214 in FIG. 6 would typically be directly fed to the controller 120.

(68) FIG. 7 is a state diagram for the controller 120.

(69) The event threshold is modified to hold the event rate within desired bounds based on the signaling from the event rate estimator. Event rate: input from the event rate detector 200. MinEventRate: constant, minimal desired event rate. MaxEventRate: constant, max desired event rate. MinEventThreshold: parameter, minimal threshold setting. MaxEventThreshold: parameter, max threshold setting. Step: parameter, step by which the event threshold is changed during each pass of the loop. EventThreshold: output fed to event generation circuit. Trigger: periodic input signal that starts the loop.

(70) In more detail, the controller 120 waits for the trigger in state 710. When it is received, the controller 120 compares the event rate provided by the event rate detector 200 to a minimum event rate in state 712. If the event rate is smaller than the minimum event rate then the controller proceeds to state 716. In state 716, the controller 120 compares the event threshold to a minimum event threshold; if it is smaller, then it returns to the wait state in 710. On the other hand, if the event threshold is larger than the minimum event threshold then the event threshold is reduced in state 718 and it returns to the wait state 710.

(71) On the other hand, if in state 712, the event rate is larger than the minimum event rate then the event rate is compared to the maximum event rate in state 714, if it is smaller, then the controller 120 returns to the wait state 710. On the other hand, if the event rate is larger than the maximum event rate then the event threshold is compared to the maximum event threshold in state 720. If the event threshold is larger than the maximum event threshold in state 720, then the controller 120 returns to the wait state 710. On the other hand, if the event threshold is smaller than the maximum event threshold then the event threshold is increased by a step in state 722 and returns to the wait state 710.

(72) Threshold Generation Circuit

(73) For change detection pixels such as those disclosed in U.S. Pat. Appl No. US 2008/00135731 by Lichtsteiner, et al., entitled “Photoarray for Detecting Time-Dependent Image Data”, and incorporated herein by this reference, the event thresholds are the difference between a threshold voltage and an amplifier bias voltage. The On Threshold voltage is usually higher than the bias voltage, while the Off Threshold voltage is lower.

(74) Basic Threshold Generation

(75) FIG. 8 shows a basic threshold generation circuit which is a set of voltage digital to analog converters (DACs) or current DACs 810, 812, such as those disclosed in P. Lichtsteiner, C. Posch, and T. Delbruck, “A 128×128 120 dB 15 μs Latency Asynchronous Temporal Contrast Vision Sensor,” IEEE J. Solid-State Circuits, vol. 43, no. 2, pp. 566-576, February 2008; and T. Delbruck, R. Berner, P. Lichtsteiner, and C. Dualibe, “32-bit Configurable bias current generator with sub-off-current capability,” in Proceedings of 2010 IEEE International Symposium on Circuits and Systems, 2010, pp. 1647-1650. Specifically, for the set of DAC's 810, 812, 814, there is one for generating the comparator bias voltage 814, one for the OnThres voltage 810 and one for the OffThres voltage 812.

(76) Event Rate and Balance Control

(77) In a further embodiment, the sensor 100 counts ON (increasing light intensity) and OFF (decreasing light intensity) events, separately. Then the ratio between ON and OFF events is used for controlling the ON and OFF thresholds separately. In general, it is desired a similar amount of ON events and OFF events over a large (larger than seconds) time window.

(78) No-Motion Threshold

(79) A property of event-based change detection sensors is that they do not generate data (or very little data) if there is no motion in the scene. In such a case of no motion, it may not be desirable to change the threshold, so the controller may be extended to include a minimal event rate threshold, below which the control loop to change the EventThreshold output is not activated.

(80) Time Resolution Control

(81) In some pixel designs, the time bin rate (rate of comparison) can be changed. In a pixel such as proposed by described in U.S. Pat. Appl No. US 2008/00135731, the refractory bias can be controlled, which allows limiting the rate at which a pixel can produce events.

(82) While this invention has been particularly shown and described with references to preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the invention encompassed by the appended claims.