Image sensor and image recognition apparatus using the same

Abstract

Disclosed is an image sensor including pixels, and an analog digital converter configured to select output values of the pixels using an analog multiplexer and to simultaneously perform a convolution operation and conversion into a digital signal using a delta-sigma ADC. The size of processed data is reduced, and power consumption is reduced, and thus the image sensor is advantageously applied to a portable device.

Claims

1. An image sensor comprising: pixels; an analog multiplexer configured to select an analog output signal of the pixels; and an analog digital converter configured to convert the analog signal output from the analog multiplexer into a digital signal and to output the digital signal; wherein the analog multiplexer includes a switch, the analog multiplexer is configured in a matrix form, and a pixel for performing a convolution operation is selected; wherein the analog digital converter is a delta-sigma ADC; wherein the delta-sigma ADC includes: a first stage; a second stage configured to additionally process an output signal of the first stage; a decimation filter provided in an output end and configured to apply a weight; and a mode switch configured to determine whether to operate the second stage; and wherein the mode switch switches: a normal photograph mode in which the second stage is operated; and a binarized neural network (BNN) mode in which the second stage is not operated.

2. The image sensor of claim 1, wherein the analog multiplexer includes a weight matrix including a plurality of switches that are arranged in series and in parallel.

3. The image sensor of claim 2, wherein a weight of the weight matrix is determined based on a signal for controlling the analog multiplexer.

4. The image sensor of claim 1, wherein the delta-sigma ADC outputs a value obtained by summing values obtained by multiplying a weight of a weight matrix and values of the pixels; and wherein the weight of the weight matrix is configured with 1 or −1.

5. The image sensor of claim 1, wherein the analog multiplexer, the delta-sigma ADC, and the decimation filter are combined and perform a convolution operation.

6. An image recognition apparatus including the image sensor of claim 1, further including a binarized neural network for processing a digital signal output from the image sensor.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) FIG. 1 is a diagram illustrating a concept of an image recognition apparatus using an image sensor.

(2) FIG. 2 is a diagram illustrating a concept of multiplication of a binary weight and a pixel value and a convolution operation using a delta-sigma analog digital converter.

(3) FIG. 3 is a diagram for explaining a detailed operation of an analog multiplexer for applying a binary weight.

(4) FIG. 4 is a block diagram of a delta-sigma ADC.

(5) FIG. 5 is a block diagram of a decimation filter for calculating a weight of −1.

(6) FIG. 6 is a timing diagram for operating an analog digital converter and a decimation filter.

(7) FIG. 7 is a table showing comparison of computation efficiency according to an embodiment.

DETAILED DESCRIPTION

(8) Hereinafter, embodiments of the present disclosure will be explained in detail with reference to the attached drawings. However, the present disclosure should not be construed as being limited to the embodiments set forth herein and one of ordinary skilled in the art would be obvious that other embodiments included in the scope of the present disclosure are easily proposed via addition, changes, deletion, and supplement of components and are included in the scope of the spirit of the present disclosure.

(9) FIG. 1 is a diagram illustrating a concept of an image recognition apparatus using an image sensor. FIG. 1A is a diagram illustrating a concept of an image recognition apparatus configured by coupling a conventional image sensor and a binarized neural network (BNN). FIG. 1B is a diagram illustrating a concept of an image recognition apparatus using an image sensor according to an embodiment of the present disclosure.

(10) As seen from FIG. 1A, an output data amount (N bit) from an image sensor is larger, and thus, computational load of the binarized neural network (BNN) is increased.

(11) In contrast, as seen from FIG. 1B, data (M bit, N>M) that is output through a convolution operation inside an image sensor 100 may be output and the computational load of the binarized neural network (BNN) may also be reduced.

(12) In more detail, the image recognition apparatus using an image sensor according to an embodiment may include the image sensor 100, and a binarized neural network (BNN) 200. In addition, the image recognition apparatus may further include other devices such as a discriminator, a notifier, or a memory, and these components may be other devices that are well known.

(13) The image sensor 100 may include an analog multiplexer 20 for performing a convolution operation on analog information acquired from pixels 10 provided to an imaging device, and a delta-sigma ADC 30 for converting an analog signal output from the analog multiplexer 20 into a digital signal.

(14) A concept of an operation of the image sensor 100 will be described with reference to FIG. 2.

(15) FIG. 2 is a diagram illustrating a concept of convolution performed by a hardware device inside an image sensor. The drawing illustrating convolution using an analog multiplexer and a delta-sigma ADC with respect to 3×3 pixels and a weight of 3×3.

(16) As seen from FIG. 2, output of the delta-sigma ADC is output as a value obtained by summing values obtained by multiplying a weight of a weight matrix 90 and values of the pixels 10. The weight may be given by a value of 1 and −1. Thus, the weight may be embodied by an analog circuit.

(17) Each weight of the weight matrix may be embodied as a plurality of analog switches that are arranged in series and in parallel to each other, that is, a switch matrix, and may be provided as a structure based on a barrel shifter. In other words, the analog multiplexer 20 may be configured with the switch matrix.

(18) The weight matrix may be predetermined depending on a type of the image recognition apparatus. For example, any switch matrix for a weight matrix appropriate for vehicle recognition may be provided as long as the switch matrix is used in vehicle recognition in autonomous driving.

(19) FIG. 3 is a diagram for explaining a concept of FIG. 2 in more detail and illustrates a procedure of performing a convolution operation of pixels and a weight of 3×3 using a switch and a delta-sigma analog digital converter. Here, the small letter x is a value of each pixel, the capital letter X is a value sampled by the delta-sigma ADC 30 as output of the analog multiplexer 20, Y is a value output from the delta-sigma ADC 30, and Φ.sub.1 and Φ.sub.2 are signals for controlling the analog multiplexer 20.

(20) In FIG. 3, (a), (b), and (c) are listed in a time sequence, and (d) is a voltage of each node according to time. A weight may be configured by an analog multiplexer, a left column may be selected when the weight is 0, and a right column may be selected when the weight is 1.

(21) First, (a) of FIG. 3 illustrates a situation in which signals of Φ.sub.1 and Φ.sub.2 are 0 and 0, respectively, in which case a weight of 1 is applied, and a value of each point is an analog multiplexer output value X.sub.1=x.sub.1 and X.sub.2=x.sub.2 and a digital output value Y.sub.1=x.sub.1 and Y.sub.2=x.sub.2.

(22) Then, (b) of FIG. 3 illustrates a situation in which signals of Φ.sub.1 and Φ.sub.2 are 1 and 0, respectively, in which case a weight of −1 is applied, and a value of each point is an analog multiplexer output value X.sub.1=x.sub.2 and X.sub.2=x.sub.3 and a digital output value Y.sub.1=x.sub.1−x.sub.2 and Y.sub.2=x.sub.2−x.sub.3.

(23) Then, (c) of FIG. 3 illustrates a situation in which signals of Φ.sub.1 and Φ.sub.2 are 0 and 1, respectively, in which case a weight of 1 is applied, and a value of each point is an analog multiplexer output value X.sub.1=x.sub.3 and X.sub.2=x.sub.4 and a digital output value Y.sub.1=x.sub.1−x.sub.2+x.sub.3 and Y.sub.2=x.sub.2−x.sub.3+x.sub.4.

(24) The aforementioned method may be applied in the same way to the remaining weights of FIG. 3 and may also be applied to weights of other combinations. A computational operation of a negative weight (−1) will be described with reference to FIG. 5.

(25) FIG. 4 is a block diagram of a delta-sigma ADC according to an embodiment.

(26) Referring to FIG. 4, the delta-sigma ADC according to an embodiment may be a delta-sigma ADC of an amp sharing method using self-bias for a low-power operation and may include a first stage 31, a second stage 32, a mode switch 33 for determining an operation of the second stage 32, and a decimation filter 40.

(27) The delta-sigma ADC 30 may be operated at first or second order depending on a control state of the mode switch 33. The mode switch 33 may be manipulated according to an external control signal (NN_MODE signal). According to the present control signal, a binarized neural network mode (BNN mode) and a normal photograph mode may be switched with each other. To this end, the NN_MODE signal may perform a function of inputting output of the first stage 31 to the decimation filter 40 and may enable and disable the second stage 32.

(28) The delta-sigma ADC 30 may be operated as a primary delta-sigma ADC for a convolution operation in the BNN mode. This is because it is not appropriate to apply a secondary delta-sigma ADC to a convolution operation since a unique weight is generated during a switching procedure.

(29) The delta-sigma ADC 30 may be operated as the secondary delta-sigma ADC in order to acquire a high-resolution image in the normal photograph mode. To this end, an over sampling rate (OSR) of the ADC may be adjusted.

(30) FIG. 5 is a block diagram of a decimation filter for an operation of an analog digital converter, in which an order is changed depending on an operating mode of an image sensor, and for applying a weight of −1.

(31) Referring to FIG. 5, the delta-sigma ADC 30 may be primarily operated in a binarized neural network mode (BNN mode), and thus, an adder 43 of a second stage 42 of the decimation filter 40 may add 0 according to a mode signal and may output a result of a first stage 41 without change. Here, a negative weight (−1) may be embodied using the first stage 41 of the decimation filter 40 as an ascending/descending adder.

(32) FIG. 6 is a diagram illustrating a control signal according to a photograph mode of a delta-sigma ADC.

(33) As described above, an image sensor according to an embodiment may be operated in two modes that are classified into a normal photograph mode (NN_MODE=“0”) and a binarized neural network mode (NN_MODE=“1”) according to the NN_MODE signal.

(34) In the normal photograph mode, the image sensor may be secondarily operated to finely digital-convert an output voltage of a pixel like in a normal image sensor.

(35) The binarized neural network mode requires characteristics of simply summing sampled signals of the ADC, and thus, it may be required to disable the second stage 32. Thus, signals S2_PH1 and S2_PH2 are fixed to a constant value.

(36) In the binarized neural network mode, initialization of an analog digital converter may be controlled by EXT_RST.

(37) FIG. 7 is a table showing computation efficiency according to an embodiment and shows comparison of CPU, GPU, FPGA, and embodiments with respect to convolution operations of 2 million times in consideration of power consumption. As seen from FIG. 7, the embodiment of the present disclosure has excellent computation performance based on power consumption compared with CPU, GPU, and FPGA.

(38) As described in detail above, with regard to a conventional image sensor, since image sensor and neural network functional chips are each developed and are integrated into each other, a weight is adjusted in a digital manner, and thus, power consumption in an analog area may not be reduced.

(39) According to the present disclosure, a neural network image sensor that is easily variable to 1 to N bits may be embodied using a delta-sigma ADC and a kernel convolution matrix.

(40) According to the present disclosure, an image sensor for a BNN may be applied to enable low power consumption and a rapid operation. In addition, an NN mode and a normal mode may be easily switched with each other to acquire a normal high-resolution image as well as an NN application.

(41) The present disclosure may be used to various application fields such as pattern classification), IoT, and a vision sensor for autonomous driving, and may be advantageously embodied by slightly changing a conventional structure.

(42) The image sensor and the image recognition apparatus according to the present disclosure may be applied to an image classification function like in a mobile, vehicle, or image monitoring field, and more particularly, may be applied to a mobile device that requires a small amount of energy consumption.

(43) According to the present disclosure, a convolution function of a binarized neural network may be added to an image sensor to reduce the amount of data output from the image sensor. Thus, computational load processed in the binarized neural network and capacity of a memory may be reduced.

(44) According to the present disclosure, a weight of a binarized neural network may be configured with 1 and −1, and thus a convolution operation in an image sensor may be simply embodied almost without power consumption using an analog multiplexer (MUX) and a delta-sigma ADC.

(45) A conventional image sensor and a neural network may cause power consumption in each of image production and an convolution operation, but according to the present disclosure, a combination of the analog multiplexer (MUX) and the delta-sigma ADC may perform the convolution operation in an analog region and may perform multiplication and sum of weights simultaneously with analog digital conversion, and thus, it may be possible to perform the convolution operation using only energy for producing an image by the conventional image sensor, thereby achieving excellent hardware than hardware using the conventional image sensor in terms of power efficiency.

(46) According to the present disclosure, the analog mulitiplexer (MUX) may be controlled to switch between a binarized neural network mode (BNN mode) and a normal photograph mode. Thus, although a single device is used, an effect may be achieved as if two image sensors are used.