Compressive sensing image sensor for enhancing frame rate and operating method thereof

Abstract

A compressive sensing image sensor includes: a pixel array; and a readout circuit configured to receive pixel data on a shot image in an analogue form, and to process the pixel data, wherein the pixel array includes a plurality of blocks each having a plurality of pixels and arranged in an array form, wherein the circuit includes: a compressive sensing multiplexer to which a plurality of pixel data outputted from a corresponding block from among the plurality of blocks are inputted; an LFSR configured to arbitrarily select at least one pixel data from the plurality of pixel data inputted to the compressive sensing multiplexer; and a delta-sigma ADC configured to receive the at least one pixel data selected by the LFSR, to delta-sigma modulate the received at least one pixel data, and to generate compressive sensing data for restoring an image of the corresponding block from among the shot images.

Claims

1. A compressive sensing image sensor comprising: a pixel array comprising a plurality of pixels; and a readout circuit configured to receive pixel data on a shot image in an analogue form from the plurality of pixels, and to process the pixel data, wherein the pixel array comprises a plurality of blocks each having a set of pixels arranged in an array form, wherein the readout circuit comprises: a compressive sensing multiplexer to which a plurality of pixel data outputted from a corresponding block from among the plurality of blocks are inputted; a linear feedback shift register (LFSR) configured to arbitrarily select at least one pixel data from the plurality of pixel data inputted to the compressive sensing multiplexer; and a delta-sigma analog-to-digital converter (ADC) configured to receive the at least one pixel data selected by the LFSR, to delta-sigma modulate the received at least one pixel data, and to generate compressive sensing data for restoring an image of the corresponding block from among the shot image, wherein the compressive sensing data comprises a plurality of compressive sensing data, wherein a number of the plurality of compressive sensing data is changed based on a set compression ratio, and wherein, when the compressive sensing image sensor makes a video, the readout circuit is configured to output, as data for restoring a current frame of the video, at least one compressive sensing data obtained from the current frame of the video, and at least one compressive sensing data of an already-obtained previous frame.

2. The compressive sensing image sensor of claim 1, wherein the number of the plurality of compressive sensing data increases as the set compression ratio increases.

3. The compressive sensing image sensor of claim 1, wherein the number of the plurality of compressive sensing data is changed based on a number of pixels included in the corresponding block and the set compression ratio.

4. The compressive sensing image sensor of claim 1, wherein the plurality of pixel data outputted from the corresponding block from among the plurality of blocks is included in a corresponding block column from among the plurality of blocks and are inputted to the compressive sensing multiplexer.

5. The compressive sensing image sensor of claim 1, wherein the LFSR is configured to arbitrarily select the at least one pixel data from the plurality of pixel data by applying a pseudo-random number generation method.

6. The compressive sensing image sensor of claim 1, wherein the delta-sigma ADC comprises a delta-sigma modulator and a decimation filter, and wherein the delta-sigma modulator is a second-order delta-sigma modulator.

7. An operating method of a compressive sensing image sensor for making a video, the method comprising the steps of: (a) obtaining a plurality of pixel data on a first frame of the video from a block having a plurality of pixels; (b) selecting pixel data of one group comprising at least one pixel data from among the obtained plurality of pixel data; and (c) obtaining first compressive sensing data for restoring the first frame of the video from the block by performing delta-sigma modulation with respect to the selected pixel data of the one group by using an ADC corresponding to the block, wherein the first compressive sensing data comprises an M number of first compressive sensing data, further comprising a step of obtaining, from the block, an N number of second compressive sensing data on a second frame which is a next frame of the first frame, wherein compressive sensing data for restoring the second frame comprises an M-N number of the first compressive sensing data and the N number of the second compressive sensing data, and wherein the N number has a value less than or equal to the M number.

8. The method of claim 7, wherein the step of (a) and the step of (b) are performed at least a single time, and wherein the M number is determined based on a number of pixels of the block and a set compression ratio, and the M number has a value greater than or equal to 1.

9. The method of claim 8, wherein the M number increases as the set compression ratio increases.

10. The method of claim 7, wherein the N is a number of ADCs corresponding to the block, wherein the step of obtaining the N number of the second compressive sensing data is performed on each of the N number of ADCs a single time.

11. The method of claim 7, wherein the step of selecting the pixel data of the one group comprises arbitrarily selecting the pixel data of the one group from the plurality of pixel data by applying a pseudo-random number generation method.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) FIGS. 1A and 1B is a schematic block diagram of a pixel array and an ADC for column-parallel processing of a related-art image sensor;

(2) FIG. 2 is a table showing the number of times of sampling required by a single-slope ADC and a delta-sigma ADC according to a resolution of the ADC;

(3) FIG. 3 is a schematic block diagram of an image sensor according to an embodiment of the present disclosure;

(4) FIG. 4 is a view provided to explain a pixel array, a compressive sensing multiplexer, and a delta-sigma ADC of the image sensor in detail according to an embodiment of the present disclosure;

(5) FIGS. 5 and 6 are views provided to explain a difference between a related-art rolling shutter video coding method and a block-based compressive sensing method according to an embodiment of the present disclosure;

(6) FIG. 7 is a view illustrating comparison of images displayed according to a compression ratio when the block-based compressive sensing method is applied according to an embodiment of the present disclosure;

(7) FIG. 8 is a flowchart provided to explain an operating method of an image sensor applying the block-based compressive sensing method according to an embodiment of the present disclosure;

(8) FIG. 9 is a view illustrating an example of a measurement matrix when a compression ratio is 1/4, in relation to another embodiment of the present disclosure;

(9) FIG. 10 is a graph illustrating an example of a weight set of a third-order delta-sigma modulator when a solution of a 4-partition problem can be derived;

(10) FIG. 11 is a view illustrating an example of splitting into four subsets having the equal sum with respect to FIG. 10 by using the solution of the 4-partition problem;

(11) FIG. 12 is a graph illustrating an example of a weight set of a third-order delta-sigma modulator that cannot derive the solution of the 4-partition problem;

(12) FIG. 13 is a view illustrating an example of splitting into subsets having the equal sum by using the solution of the 4-partition problem if a specific weight is excluded in FIG. 12;

(13) FIG. 14 is a view illustrating an operation of a compressive sensing unit;

(14) FIG. 15 is a flowchart illustrating a compressive sensing method of an image sensor according to an embodiment of the present disclosure;

(15) FIG. 16 is a view illustrating comparison of image qualities according to the number of cycles when second-order and fourth-order delta-sigma modulators are used; and

(16) FIG. 17 is a graph illustrating comparison of the number of cycles required by first-order to fourth-order delta-sigma modulators, which are designed based on the delta-sigma toolbox provided by Mathworks, to obtain respective resolutions.

BEST MODE

(17) Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. The terms “module” and “unit” used for elements in the following description are assigned or interchangeably used only in consideration of easy writing of the specification, and do not have a meaning or a role of being distinguished from each other. In the description of exemplary embodiments of the present disclosure, certain detailed explanations of related art are omitted when it is deemed that they may unnecessarily obscure the essence of the embodiments of the present disclosure. In addition, the accompanying drawings are just for easy understanding of the embodiments of the present disclosure, and the technical concept of the present disclosure is not limited by the accompanying drawings. It should be appreciated that various embodiments of the present disclosure include various changes, equivalents, or replacements included in the technical concept and the technical scope of the present disclosure.

(18) FIG. 3 is a schematic block diagram of an image sensor according to an embodiment of the present disclosure.

(19) Referring to FIG. 3, the image sensor 10 may be implemented by using a complementary metal-oxide semiconductor (CMOS) image sensor, but is not limited thereto. In addition, the image sensor 10 may be implemented by using a compressive sensing image sensor applying a compressive sensing method as will be described below.

(20) The image sensor 10 may include a pixel array 20, a reference and bias circuit 30, a timing control logic 40, a row pixel selector 50, and a readout circuit 60. The image sensor 10 shown in FIG. 3 is merely for convenience of explanation, and may include more or fewer components than the image sensor 10 according to an embodiment.

(21) The pixel array 20 may have an array structure having a plurality of pixels extended in the horizontal direction and the vertical direction. Each of the plurality of pixels may include a photoelectric transformation element such as a photo diode or a photo transistor, and a plurality of transistors to transmit an analogue voltage (or an analogue signal) outputted from the photoelectric transformation element to the readout circuit 60 or the outside of the pixel array 20.

(22) The image sensor 10 according to an embodiment of the present disclosure may apply a block-based compressive sensing method other than a normally used rolling shutter method. Accordingly, the pixel array 20 may be divided into a plurality of blocks, and each of the blocks may be configured with an array having at least one pixel. In this case, the plurality of blocks may be arranged within the pixel array 20 as a kind of array (a plurality of block rows and a plurality of block columns).

(23) In the present specification, it is assumed that one block 21 is configured with an array of a 4×4 pixel structure, and accordingly, 16 pixels are included in one block 21. However, the configuration of the block 21 may be changed freely according to an embodiment.

(24) The reference and bis circuit 30 may provide a bias voltage necessary for operations of the pixels of the pixel array 20, and the readout circuit 60. The timing control logic 40 may generate various timing signals necessary for controlling operations of the respective components 20, 50, 60, and may output the generated timing signals to the respective components.

(25) The row pixel selector 50 may activate the plurality of pixel rows included in the pixel array 20 in sequence. When a specific pixel row is activated, pixel data may be outputted from the respective pixels included in the corresponding pixel row.

(26) The readout circuit 60 may process an analogue voltage or a signal (pixel data) outputted from the pixels of the pixel array 20, and may convert the same into a digital code. In particular, the readout circuit 60 according to an embodiment of the present disclosure may include a compressive sensing multiplexing block 61, a delta-sigma modulation block 63, a decimation filter block 64, a column shift register 65, and a plurality of linear feedback shift registers (LFSRs) 66. Herein, the delta-sigma modulation block 63 and the decimation filter block 64 may be integrated and may be defined as a delta-sigma ADC block 62.

(27) The compressive sensing multiplexing block 61 may include a plurality of compressive sensing multiplexers (MUXs). A plurality of pixel data on a corresponding block from among the plurality of blocks of the pixel array 20 may be inputted to each of the plurality of compressive sensing multiplexers, and at least one pixel data arbitrarily selected from the inputted plurality of pixel data by the LFSR 66 may be outputted to the delta-sigma ADC 62. According to an embodiment, a plurality of blocks included in the same block column may correspond to one compressive sensing multiplexer. In addition, according to an embodiment, the plurality of compressive sensing multiplexers may correspond to one block.

(28) Each of the plurality of LFSRs 66 may cause at least one pixel data arbitrarily selected from the plurality of pixels data inputted to the corresponding compressive sensing multiplexer to be outputted to the delta-sigma ADC 62, thereby enabling compressive sensing for the block 21. To restore compressive sensing data afterward, the LFSR 66 may arbitrarily select at least one pixel data from the plurality of pixel data by applying a pseudo-random number generation method.

(29) The delta-sigma modulation block 63 may include a plurality of delta-sigma modulators, and the decimation filter block 64 may include a plurality of decimation filters. Each of the plurality of delta-sigma modulators and each of the plurality of decimation filters may receive the at least one pixel data (analogue voltage) selected and outputted by the corresponding compressive sensing multiplexer and the LFSR 66, and may delta-sigma modulate the received pixel data and may generate a digital code. The generated digital code may correspond to compressive sensing data. The delta-sigma modulator may be implemented by using a second-order delta-sigma modulator, but is not limited hereto. In addition, the delta-sigma ADC may be configured with the delta-sigma modulator and the decimation filter.

(30) The column shift register 65 may transmit the compressive sensing data generated at the delta-sigma ADC 62 to the outside of the image sensor 10. For example, if the image sensor 10 is implemented in a mobile terminal such as a smartphone, a tablet PC, or the like, the compressive sensing data may be transmitted to a memory of the mobile terminal or may be transmitted to a display of the mobile terminal. A controller of the mobile terminal may restore the compressive sensing data based on information on the pseudo-random number generation method of the LFSR 66. According to the result of restoring, the controller may obtain a compressive sensing image and may display the obtained compressive sensing image through the display. Quality of the compressive sensing image may vary according to a compression ratio, and the number or quantity of compressive sensing data per block may vary according to the compression ratio. As the compression ratio increases, the number or quantity of the compressive sensing data may increase and the quality of the compressive sensing image may be enhanced. On the other hand, as the compression ratio decreases, the number or quantity of the compressive sensing data may decrease and the quality of the compressive sensing image may be reduced.

(31) FIG. 4 is a view provided to explain the pixel array, the compressive sensing multiplexer, and the delta-sigma ADC of the image sensor in detail according to an embodiment of the present disclosure.

(32) Referring to FIG. 4, the image sensor 10 according to an embodiment of the present disclosure may use a block-based compressive sensing method, and accordingly, one ADC may not be connected to every pixel column of the pixel array 20 as in the related-art technology. That is, according to the present disclosure, at least one ADC may be connected to each of the block columns, and may perform the block-based compressive sensing method. Therefore, the number of ADCs included in the image sensor 10 can be reduced. Accordingly, in the image sensor 10 of high pixels requiring a high-resolution ADC, a delta-sigma ADC having a fast operating speed compared to a single-slope ADC may be used.

(33) As described above, the plurality of blocks included in the pixel array 20 may be arranged in an array form. In this case, each of the plurality of block columns may correspond to at least one of the plurality of compressive sensing multiplexers included in the compressive sensing multiplexing block 61, and may correspond to at least one of the plurality of delta-sigma ADCs of the delta-sigma ADC block 62.

(34) For example, as shown in FIG. 4, a first block column BC1 including a first block 21 may correspond to a first compressive sensing multiplexer 611 of the compressive sensing multiplexing block 61, and may correspond to a first delta-sigma ADC 621 of the delta-sigma ACD block 62.

(35) A process of obtaining compressive sensing data from pixel data of the first block 21 will be described briefly as follows.

(36) A 1.sup.st row ROW1 to a 4.sup.th row ROW4 of the first block 21 may be activated in sequence by the row pixel selector 50. As the 1.sup.st row ROW1 to the 4.sup.th row ROW4 are activated in sequence, respective pixel data of a 1.sup.st pixel P1 to a 16.sup.th pixel P16 of the first block 21 may be outputted to the first delta-sigma ADC 621 through the first compressive sensing multiplexer 611. In this case, the LFSR 66 may arbitrarily select pixel data of one group from the respective pixel data of the 1.sup.st pixel P1 to the 16.sup.th pixel P16 based on the pseudo-random number generation method, and may output the pixel data of the one group to the first delta-sigma ADC 621. The pixel data of the one group may refer to at least one pixel data outputted from each of at least one pixel of the 1.sup.st pixel P1 to the 16.sup.th pixel P16.

(37) The first delta-sigma ADC 621 may perform delta-sigma modulation with respect to the pixel data group outputted through the first compressive sensing multiplexer 611 and the LFSR 66. As a result of the delta-sigma modulation, the image sensor 10 may obtain compressive sensing data having a digital code format. The obtained compressive sensing data may be outputted to the outside of the image sensor 10. A controller or an image signal processor (ISP) of various types of electronic devices including the image sensor 10 may obtain a compressive sensing image by restoring the compressive sensing data. When one compressive sensing data is obtained regarding the 16 pixels P1-P16 included in the first block 21 as shown in the above-described embodiment, the compression ratio may correspond to 1/16.

(38) When there are a plurality of compressive sensing multiplexers and a plurality of delta-sigma ADCs corresponding to a specific block as will be described in FIG. 6, a plurality of compressive sensing data may be obtained from the corresponding block. For example, when there are four compressive sensing multiplexers and four delta-sigma ADCs corresponding to the first block 21, the ADCs may operate in parallel with one another. When each ADC operates a single time, four compressing sensing data in total may be obtained, and, in this case, the compression ratio may correspond to 1/4. When each ADC operates four times, 16 compressive sensing data in total may be obtained, and the compression ratio may be 1. Therefore, a compressive sensing image having the same quality as an original image may be obtained. In this case, an operating time is the same as the operating time of a rolling shutter method using four ADCs.

(39) That is, since the compressive sensing image having the compression ratio of 1/4 is obtained simply by operating each ADC a single time, the quality of the image may be slightly reduced, but a more enhanced frame rate can be provided. Since the above-described compressive sensing data obtaining process uses only the pixel data of some pixels selected from the pixels included in the first block 21 by the LFSR 66, the quality of the compressive sensing image obtained by restoring at least one compressive sensing data may be lower than the quality of the original image. In this case, the quality of the compressive sensing image obtained from the plurality of compressive sensing data may be enhanced by repeating the above-described compressive sensing data obtaining process with respect to the first block 21, but it may be difficult to ensure enhancement of the frame rate in comparison to the rolling-shutter method.

(40) Regarding this, the image sensor 10 according to an embodiment of the present disclosure may generate a compressive sensing image on a current frame by using already obtained compressive sensing data and compressive sensing data obtained regarding the current frame when making a video, and in particular, may perform the compressive sensing data obtaining process with respect to the current frame a single time, such that a frame rate can be enhanced while maintaining a high compression ratio. This will be described hereinbelow with reference to FIGS. 6 to 8.

(41) FIGS. 5 and 6 are views provided to explain a difference between a related-art rolling shutter video coding method and a compressive sensing video coding method according to an embodiment of the present disclosure.

(42) In FIGS. 5 and 6, a first image frame F1 is defined as a start frame of a video image made by the image sensor 10, and a second image frame F2 is defined as the next frame of the first image frame F1.

(43) In FIGS. 5 and 6, it is assumed that four ADCs are connected to each image frame F1, F2. In the case of FIG. 5, one ADC may be connected to each of the four pixel columns, and in the case of FIG. 6, four ADCs may be connected to the block of the image frame F1, F2.

(44) Referring to FIG. 5, in the case of the rolling shutter video coding method, all pixel data from the 1.sup.st Row1 to the 4.sup.th row Row4 may be processed in configuring one image frame (for example, the first image frame F1), and then one image frame may be generated. Since most of the time taken to process one row is a readout time, a frame rate may be restricted by the readout time in the case of the rolling shutter video coding method. In the embodiment of FIG. 5, 16 pixels data in total for each image frame may be processed by the four ADCs. Each ADC may operate four times with respect to the 1.sup.st row Row1 to the 4.sup.th row Row4. In particular, in the case of the related-art rolling shutter video coding method, the above-described process should be performed for every image frame, and thus, each ADC may operate four times with respect to the second image frame F2 as shown in FIG. 5.

(45) On the other hand, referring to FIG. 6, in the case of the compressive sensing video coding method according to an embodiment of the present disclosure, one image frame may be configured by obtaining a plurality of compressive sensing data with respect to a frame of a specific block. In this method, data enough to configure one image frame can be obtained simply by operating each ADC a single time, although the compression ratio is low. Accordingly, the time taken to obtain one image frame can be reduced in comparison to the rolling shutter video coding method.

(46) For example, in the embodiment of FIG. 6, it is assumed that the compression ratio is 1, that is, 16 compressive sensing data are obtained from the original image frame to obtain a compressive sensing image of the same quality as the original image frame by using the compressive sensing video coding method. In this case, the image sensor 10 may obtain 1.sup.st compressive sensing data to 16.sup.th compressive sensing data CSF1-CSF16 with respect to the first image frame F1, and may output the 16 compressive sensing data to the outside of the image sensor 10 as data for restoring the first image frame F1. To achieve this, each of the ADC operates four times, and thus the operating time is substantially the same as that of the rolling shutter video coding method of FIG. 5.

(47) Thereafter, the image sensor 10 may obtain 17.sup.th compressive sensing data to 20.sup.th compressive sensing data CSF17-CSF20 from the second image frame F2, and may obtain a second compressive sensing image frame by using the already obtained 5.sup.th compressive sensing data and 16.sup.th compressive sensing data CSF5-CSF16, and the 17.sup.th compressive sensing data to the 20.sup.th compressive sensing data CSF17-CSF20. That is, regarding the second image frame F2, each ADC operates only a single time to obtain the 17.sup.th compressive sensing data to the 20.sup.th compressive sensing data CSF17-CSF20. Therefore, compared to that in the embodiment of FIG. 5, a processing speed of the second image frame F2 can increase four times, and the compression ratio is maintained as 1. Accordingly, the second compressive sensing image frame having substantially the same quality as that of the second image frame F2 can be obtained.

(48) In addition, a frame rate of the image sensor can be additionally enhanced by using the delta-sigma ADC instead of a related-art method using a single-slope ADC of a high resolution.

(49) The number or quantity of compressive sensing data required to obtain one compressive sensing image frame may be changed according to a set compression ratio. As described above, on the assumption that 16 compressive sensing data are obtained if the compression ratio is 1, 8 compressive sensing data may be obtained if the compression ratio is 1/2, and a compressive sensing image frame may be obtained.

(50) FIG. 7 is a view illustrating comparison of images displayed according to compression ratios when the compressive sensing video coding method is applied according to an embodiment of the present disclosure.

(51) Referring to FIG. 7, it can be seen that, when the block-based compressive sensing method is applied according to an embodiment of the present disclosure, quality of an image increases as a compression ratio (CR) increases. For example, a PSNR of the image may be about 88 dB if the compression ratio CR is 1, but a PSNR of the image may be about 29 dB if the compression ratio (CR) is 1/4. That is, in the case of a still image, quality of the compressive sensing image may be slightly reduced according to the compression ratio (CR).

(52) However, when an image of a video is shot, the image sensor 10 according to an embodiment of the present disclosure may obtain a plurality of compressive sensing data and may generate compressive sensing image frames, and in particular, may perform a compressive sensing data obtaining process only a single time with respect to the other frames except for a start frame. Accordingly, since the operating time of the readout circuit 60 for one frame is reduced, a frame rate can be enhanced, and also, a high compression ratio can be maintained. Therefore, quality of a compressive sensing image frame restored afterward can also be maximized.

(53) FIG. 8 is a flowchart provided to explain an operating method of an image sensor according to an embodiment of the present disclosure.

(54) Although the embodiment of FIG. 8 illustrates a process of obtaining compressive sensing data from a certain block 21 from among the blocks of the pixel array 20, the corresponding embodiment may be performed in the same way with respect to all of the blocks of the pixel array 20.

(55) Referring to FIG. 8, the image sensor 10 may obtain m number of first compressive sensing data on a first frame from a first block of the pixel array 20, based on a set compression ratio (S100).

(56) Herein, the first frame is defined as a start frame when the image sensor 10 makes a video.

(57) The number m of the first compressive sensing data may be related to the number of pixels of the first block and the compression ratio. For example, the m may correspond to multiplication of the number of pixels of the first block and the compression ratio. For example, if the number of pixels is 16 and the compression ratio is 1, m may be 16. If the number of pixels is 16 and the compression ratio is 1/2, m may be 8.

(58) The image sensor 10 may output the m number of the first compressive sensing data obtained as data for restoring the first frame (S110).

(59) The m number of the first compressive sensing data obtained may be outputted to the outside of the image sensor 10. For example, if the image sensor 10 is implemented in a mobile terminal such as a smartphone, a tablet PC, or the like, the first compressive sensing data may be transmitted to a memory of the mobile terminal or may be transmitted to a display of the mobile terminal. A controller or an ISP of the mobile terminal may obtain a first compressive sensing image frame by restoring the first compressive sensing data by using information on the pseudo-random number generation method of the LFSR 66.

(60) The image sensor 10 may obtain n number of second compressive sensing data on a second frame from the first block (S120). The second frame may correspond to the next frame of the first frame.

(61) The number n of the second compressive sensing data may be less than or equal to m, and may be the same as the number of ADCs corresponding to the first block. That is, each of the n number of ADCs corresponding to the first block may operate a single time with respect to the second frame, and as a result, the n number of the second compressive sensing data may be obtained.

(62) The image sensor 10 may output, as data for restoring the second frame, m number of compressive sensing data including at least one of the m number of the first compressive sensing data and the n number of the second compressive sensing data (S130).

(63) That is, the data for restoring the second frame may include the m-n number of the first compressive sensing data and the n number of the second compressive sensing data. Since the n number of ADCs operate only a single time with respect to the second frame as described above, the frame rate of the image sensor 10 can be enhanced.

(64) According to an embodiment, the image sensor 10 may output only the n number of the second compressive sensing data as the data for restoring the second frame. In this case, the controller or the ISP of the electronic device may restore a second compressive sensing image frame corresponding to the second frame by using the n number of the second compressive sensing data obtained and the m-n number of the first compressive sensing data already obtained.

(65) FIG. 9 illustrates an example of a simple deterministic measurement matrix used in the present disclosure if a compression ratio for determining a sensing method is 1/4.

(66) Referring to FIG. 9, the number of ‘1’ forming one row of the measurement matrix is the same as the reciprocal of the compression ratio. In addition, this is the same as the number of data included in one measurement value derived through a linear combination of the data.

(67) FIG. 10 illustrates an example of a weight function of a third-order delta-sigma modulator regarding cycles 1 to 10.

(68) As shown in FIG. 10, a high-order delta-sigma modulator may have a different weight according to each input cycle.

(69) FIG. 11 illustrates an example of a case where, on the assumption that weights obtained from the weight function of the third-order delta-sigma modulator are a universal set, the universal set can be split into four subsets having the equal sum by being applied to a 4-partition problem.

(70) FIG. 11 illustrates a result of splitting the weights of the third-order delta-sigma modulator suggested in FIG. 10 into four subsets having the equal sum.

(71) FIG. 12 illustrates an example of the weight function of the third-order delta-sigma modulator regarding cycles 1 to 14.

(72) FIG. 13 illustrates an example of a case where, on the assumption that weights obtained from the weight function of the third-order delta-sigma modulator are a universal set, the universal set cannot be split into four subsets having the equal sum by being applied to the 4-partition problem.

(73) FIG. 13 illustrates that, since the weights of the third-order delta-sigma modulator suggested in FIG. 12 are not split into four subsets having the equal sum, arbitrarily selected cycles (11.sup.th, 12.sup.th cycles) are excluded and the weights of the other cycles are split into four sets having the equal sum, by being applied to the 4-partition problem In this case, 0 or a specific direct current voltage may be inputted to the excluded cycles.

(74) FIG. 14 is a view illustrating an operation of a sensing unit.

(75) As shown in FIG. 14, the sensing unit 400 of the image sensor may include a compressed sensing multiplexer (CS-MUX) 410 and a high-order delta-sigma modulator 420.

(76) Referring to FIG. 10, the weight gradually decreases with reference to an initial value in a non-linear pattern in the case of the third-order delta-sigma modulator. If an input signal is inputted to the delta-sigma modulator in sequence, the weight is applied according to the weight function. Accordingly, in the case of a second or higher-order delta-sigma modulator, different weights are applied to signals inputted in sequence, and thus there is a problem in deriving a linear combination.

(77) Referring to FIG. 9, all pixels to constitute a linear combination should have the same weight at the output of the modulator at the time conversion ends in order to derive the linear combination. The sensing unit 400 may operate as follows to derive a linear combination from the second or higher order according to the present disclosure.

(78) Pixels corresponding to the number of pixels to constitute a linear combination are inputted to the compressed sensing multiplexer 410. In this case, the number of pixels to be inputted may be determined by a form of a measurement matrix. The embodiment shown in FIG. 14 illustrates a case where the compression rate is 4 as shown in FIG. 9. Accordingly, it can be seen from FIG. 9 that the linear combination is formed of four pixels. According to FIG. 14, the compressed sensing multiplexer 410 receiving a specific number of pixels simultaneously may input the pixels to the high-order delta-sigma modulator 420 according to the derived inputting order by utilizing a solution of a subset problem as suggested in the present disclosure.

(79) In other words, FIG. 14 illustrates an example if the compression rate is 1/4, and it can be seen that the order of pixels inputted to the high-order delta-sigma modulator is the same as the inputting order derived from the solution of the partition problem as shown in FIG. 10.

(80) Accordingly, when the pixels are inputted to the delta-sigma modulator in the above-described order, all of the pixels have the same weight sum at the time conversion ends. Accordingly, the linear combination of all pixels having the same weight can be derived.

(81) FIG. 15 is a flowchart illustrating a compressive sensing method of the image sensor according to an embodiment of the present disclosure.

(82) The compressive sensing method according to an embodiment of the present disclosure (specifically, a method for compressive sensing a natural image signal by using a second or higher-order delta-sigma modulator) includes a step of selecting a certain pixel group according to a measurement matrix in the arrangement of the image sensor (S501), a step of deriving a sampling order by applying a solution of a partition problem to determine an order in which individual pixels in the selected group are inputted to the delta-sigma modulator (S503), a step of performing oversampling according to the sampling order derived by applying the solution of the partition problem (S505), and a step of adding voltages of pixels corresponding to respective sampling orders in sequence after performing the oversampling, and outputting a digital code (S507).

(83) In this case, the second or higher-order sigma-delta modulator may have a different weight function according to the order of the modulator used.

(84) In addition, at the step of selecting the group of the pixels according to the measurement matrix, the measurement matrix and the number of pixels to be selected may be determined according to a compression ratio.

(85) Herein, the step of determining the order of inputting the pixels to the delta-sigma modulator may further include a step of deriving a universal set of weights of respective cycles during the total cycles in consideration of a given sampling cycle and a weight function of the used delta-sigma modulator, a step of splitting the derived universal set of the weights into a plurality of subsets having the equal sum by using a solution of a partition problem, and a step of deriving a sampling order to cause the weights of the individual pixels to be the same at the output of the delta-sigma modulator based on the split set.

(86) Herein, the step of performing the oversampling may include a step of inputting the individual pixels of the pixel group derived according to the measurement matrix into the delta-sigma modulator according to the derived sampling order.

(87) FIG. 16 is a view illustrating comparison of qualities of restored images according to the number of cycles by applying the sampling method of the embodiment when second-order and fourth-order delta-sigma modulators are used. Compared to the second-order method, the fourth-order method can increase a sampling speed.

(88) FIG. 17 is a graph illustrating comparison of the number of cycles for first-order to fourth-order delta sigma modulators designed based on the delta-sigma toolbox provided by Mathworks to have their respective resolutions. It is illustrated that the number of necessary cycles is abruptly reduced as the order increases.

(89) According to an embodiment of the present disclosure, the above-described method can be embodied as processor-readable codes on a medium having a program recorded thereon. Examples of the processor-readable medium include a read only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, or the like, and may be implemented in the form of a carrier wave (for example, transmission through the Internet).

(90) The display device described above is not limited to the configurations and the method of the embodiments described above, and an entirety or a part of the respective embodiments can be selectively combined and configured to make various changes to the above-described embodiments.