Systems and Methods for Improving Resolution in Lensless Imaging
20170153599 ยท 2017-06-01
Inventors
- Patrick R. Gill (Sunnyvale, CA)
- David G. Stork (Portola Valley, CA)
- John Eric Linstadt (Palo Alto, CA)
Cpc classification
H04N25/702
ELECTRICITY
G03H1/0443
PHYSICS
H10F39/806
ELECTRICITY
G03H1/041
PHYSICS
G03H2001/0428
PHYSICS
G03H2001/0088
PHYSICS
G03H2001/0441
PHYSICS
International classification
Abstract
An infrared imaging system includes a phase grating overlying a two-dimensional array of thermally sensitive pixels. The phase grating comprises a two-dimensional array of identical subgratings that define a system of Cartesian coordinates. The subgrating and pixel arrays are sized and oriented such that the pixels are evenly distributed with respect to the row and column intersections of the subgratings. The location of each pixel thus maps to a unique location beneath a virtual archetypical subgrating.
Claims
1. An imaging device comprising: an optical phase grating layer including a repeating pattern of subgratings, with a first subgrating number gx of the subgratings along a first axis X and a second subgrating number gy of the subgratings along a second axis Y; and a pixel array having pixels, including: a first pixel number px, coprime with the first subgrating number gx, of the pixels along the first axis X; and a second pixel number py, coprime with the second subgrating number gy, of the pixels along the second axis Y.
2. The imaging device of claim 1, wherein the repeating patterns of the subgratings comprise rows and columns of the subgratings.
3. The imaging device of claim 2, wherein the pixel array comprises rows and columns of the pixels.
4. The imaging device of claim 3, wherein the columns of the subgratings extend at an angle with respect to the columns of the pixels.
5. The imaging device of claim 4, wherein the angle is of a Pythagorean triple.
6. The imaging device of claim 1, wherein the subgratings are identical. The imaging device of claim 1, the phase grating to cast an interference pattern on the pixel array, each subgrating including boundaries of odd symmetry separating stepped features on opposite sides of each boundary, the stepped features on the opposite sides of each boundary offset from the pixel array to induce a phase difference of half of a wavelength within the infrared spectrum, plus an integer multiple of the wavelength, to produce curtains of destructive interference at the pixel array.
8. The imaging device of claim 1, further comprising superfluous pixels.
9. The imaging device of claim 8, wherein the superfluous pixels comprise defective pixels.
10. The imaging device of claim 1, wherein each of the pixels is located beneath a respective subgrating at a location relative to the respective subgrating, the pixel array further comprising redundant pixels at the locations relative to others of the subgratings.
11. An imaging device comprising: an array of px by py pixels arranged in px columns and py rows; a phase grating over the array and including gx by px subgratings, including gx columns of subgratings and gy rows of subgratings defining a subgrating grid, each subgrating having a grid location; wherein each pixel includes a pixel center beneath a corresponding one of the subgratings at a pixel location that is modulo the grid location of the corresponding one of the subgratings; and wherein the modulos are evenly spaced with respect to the subgrating grid.
12. The imaging device of claim 11, wherein the modulos are evenly spaced along the columns of subgratings and the rows of subgratings.
13. The imaging device of claim 11, wherein each pixel location has a different modulo along at least one of the columns of subgratings and the rows of subgratings.
14. The imaging device of claim 13, wherein each pixel location has a different modulo along both the columns of subgratings and the rows of subgratings.
15. The imaging device of claim 11, wherein each modulo includes an x dimension and a y dimension.
16. The imaging device of claim 15, wherein at least one of the x dimension and the y dimension is unique for each pixel location.
17. The imaging device of claim 11, wherein there are px times py unique modulos.
18. The imaging device of claim 11, wherein columns of pixels diverge from the columns of subgratings.
19. The imaging device of claim 18, wherein the columns of pixels diverge from the columns of subgratings by an angle of a Pythagorean triple.
20. The imaging device of claim 11, wherein the subgratings are identical.
21. The imaging device of claim 11, further comprising superfluous pixels.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
[0002] The detailed description is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:
[0003]
[0004]
[0005]
[0006]
[0007]
[0008]
[0009]
[0010]
[0011]
[0012]
[0013]
[0014]
[0015]
[0016]
[0017]
[0018]
DETAILED DESCRIPTION
[0019]
[0020] Phase gratings of the type used for subgratings g.sub.i,j are detailed in U.S. patent application Ser. No. 14/677,878 to Stork et al, which is incorporated herein by this reference. Briefly, and in connection with subgrating g.sub.3,3, subgratings g.sub.i,j are of a material that is transparent to IR light. The surface of subgratings g.sub.i,j includes transparent features 110 (black) and 115 (white) that define between them boundaries of odd symmetry. Features 110 are raised in the Z dimension (normal to the view) relative to features 115, and are shown in black to elucidate this topography. As detailed below, the boundaries between features 110 and 115 produce an interference pattern on the underlying pixel array that contains rich spatial information about an imaged scene.
[0021] Subgratings g.sub.i,j are arranged in a two-dimensional array with a number gx along the X axis and a number gy along the Y axis. Both gx and gy are four in this simple example, but either or both numbers can be the same or different. The array of pixels p.sub.i,j includes a number px along the X axis and a number py along the Y axis. Numbers px and py are nine and eleven, respectively, but either or both can be different.
[0022] Features 110 and 115 within each subgrating g.sub.i,j form asymmetrically shaped elements with curved boundaries. Such shapes can be described mathematically by starting with identical, evenly spaced elements that approximate the desired shapes. The x/y locations of the points making up these elements are then translated by an amount that depends on x and y, where this translation is smooth. In this example, these distortions are constrained to the spaces between the edges of the subgratings such that the edges match between subgratings. The boundaries of odd symmetry are thus contiguous between subgratings (see e.g.
[0023] Pixel count px is coprime with the subgrating count gx along the X axis; that is, numbers px and gx share no common integer factor other than one. Likewise, the pixel count py is coprime with the subgrating count gy along the Y axis. With this arrangement, imaging device 100 obtains px times py independent samples of the interference pattern created by a single instance of a subgrating g.sub.i,j. The effective pixel pitch is 1/gy times pixel pitch Py in the Y dimension and 1/px times the pixel pitch Px in the X dimension. Unless otherwise specified, the X and Y dimensions refer to the Cartesian coordinate system defined by the array of subgratings.
[0024] Pixel arrays for IR cameras, also called thermographic or thermal-imaging cameras, can be cooled to support certain types of IR sensitive semiconductors. Cooling adds complexity, cost, and requires power. The pixels in uncooled detectors are mostly based on pyroelectric and ferroelectric materials or microbolometer technology. Uncooled sensors offer inferior image quality, but are substantially simpler, smaller, and less expensive.
[0025] Though not shown, pixel arrays can include superfluous pixel structures that are e.g. defective or redundant and not used for image capture. Such superfluous structures are not pixels as that term is used herein, as that term refers to elements that provide a measurement of illumination that is used for image acquisition. Redundant pixels can be used to take multiple measurements of pixels in equivalent positions, reducing noise.
[0026]
[0027]
[0028] Adjacent features 110 and 115 form six illustrative odds-symmetry boundaries 300, each indicated using a vertical, dashed line. The lower features 115 induce phase retardations of half a wavelength ( radians) relative to upper features 110. Features 305 and 310 on either side of each boundary exhibit odd symmetry. With this arrangement, paired features induce respective phase delays that differ by approximately half a wavelength over the wavelength band of interest, approximately 5 m (half of 10 m) in this example for imaging within the IR spectrum. The different phase delays produce curtains of destructive interference separated by relatively bright foci to produce an interference pattern on pixel array 107. Features 305 and 310 are of uniform width in this simple illustration, but vary across each subgrating g.sub.i,j and collection of subgratings. Curved and divergent boundaries of odd symmetry, as illustrated in
[0029] IC 315 includes a processor 320, random-access memory (RAM) 325, and read-only memory (ROM) 330. ROM 330 can store a digital representation of the point-spread function (PSF) of grating layer 105 from which a noise-dependent deconvolution kernel may be computed. ROM 330 can also store the deconvolution along with other parameters or lookup tables in support of image processing. Processor 320 captures digital image data from the pixel array and uses that data with the stored PSF to compute e.g. images and other image data. Processor 320 uses RAM 325 to read and write data in support of image processing. Processor 320 may support specialized processing elements that aid fast, power-efficient Fourier-or spatial-domain deconvolution, for example.
[0030]
[0031] The center of pixel p.sub.7,6, with reference to the array of subgratings, is located at gx3+mod(px7,gx) in the X dimension and gy2+mod(py6,gy) in the Y dimension. The displacements mod(px7,gx) and mod(py6,gy) place the center of pixel p.sub.7,6 within overlaying subgrating g.sub.3,2. The combination of mod(px7,gx) and mod(py6,gy) for pixel p.sub.7,6 is unique among pixels p.sub.i,j with reference to their corresponding subgratings. The same is true of the remaining pixels. That is, no two pixels p.sub.i,j are located at the same position relative to their respective subgrating. Moreover, the modulo coordinates are evenly spaced along the X and Y axes.
[0032]
[0033] The number of unique modulo coordinates preserves the resolution provided by the pixel array. The even spacings of the modulo coordinates in the X and Y dimensions simplifies the mathematics required to extract image information using e.g. Fourier deconvolution, and thus limits the time and processing power required for image processing.
[0034] Due to the finite size of each pixel, there will be some spatial frequencies whose period in either of the X or Y dimensions fits an integral number of times into the pixel extent in that dimension, and thus may not be observed by device 100. Such frequencies can be in the null space for imaging tasks. However, the precise spatial frequencies causing such nulls may not be part of the discrete Fourier basis of frequencies describing a finite-sized sensor array. In other words, while the frequency response of the pixels incurs a sinc penalty from the square-wave profile of the pixels, this sinc (which has zeros) will not in general be sampled at its zeros if the total lateral dimensions of the array are not integer multiples of the extent of the pixel. In imaging device 100, for example, there is no spatial frequency in the discrete-Fourier-transform basis that cannot be sampled, so the system has full rank, and, with a sufficiently high signal-to-noise ratio (SNR), can produce images with as many free parameters as there are pixels.
[0035]
[0036]
[0037] The displacements mod(px7,gx) and mod(py7,gy) place the center of pixel p.sub.7, 7 within overlaying subgrating g.sub.2,1. The combination of mod(px7,gx) and mod(py7,gy) for pixel p.sub.7,7 is unique among pixels p.sub.i,j with reference to their corresponding subgratings, and the same is true of the remaining pixels. The centers of the shaded pixels are evenly spaced along the X and Y axes. The combination of the array of subgratings g.sub.i,j and the shaded pixels p.sub.i,j can thus be modeled as a single virtual subgrating 650 with three rows and three columns of evenly spaced virtual pixels.
[0038]
and the angle
The center of each pixel is marked with a small circle, each shaded or filled to identify them as members of groups of pixels associated with corresponding subgratings. For example, pixel p.sub.0,0 is shaded like the other pixels beneath subgrating g.sub.0,1.
[0039]
[0040]
[0041]
[0042] The five leftmost pixels 705 in the lowermost row include a member from each class. The leftmost corner of each of these pixels is marked with a circle to provide a reference point relative to the respective overlaying subgrating 710. Each of these reference points is mapped to an archetypal subgrating 715, with the resulting constellation of circles 720 representing the positional diversity of each class of pixel relative to its respective and identical subgratings. The five circles 720 are distributed evenly in both the X and Y dimensions relative to the row and column intersections of the subgrating gird. The incremental spacing in the X dimension is Px/5, where Px is the pixel and subgrating pitch in the X dimension; the incremental spacing in the Y dimension is Py/5, where Py is the pixel and subgrating pitch in the Y dimension. In this particular arrangement, each pixel 705 in the array remaps to one of these five locations. The array of subgratings g.sub.i,j and pixels p.sub.i,j can thus be modeled as a single virtual subgrating 715 with five virtual pixels evenly spaced in the X and Y dimensions. Imaging device 700 thus provides five-times oversampling.
[0043]
[0044]
[0045]
[0046]
[0047] Subgratings g.sub.i,j are identical, and each casts essentially the same interference pattern on the underlying pixel array. The identical patterns are not sampled identically, however, as each subgrating is offset by a different amount relative to the Cartesian coordinates defined by the underlying pixels. In this example, grating columns g.sub.i,0 and g.sub.i,1 are separated by a spacing Ssx of eighteen and one-half times the pixel pitch px in the X dimension, and grating rows g.sub.0,j , and g.sub.1,j are separated by a spacing Ssy of eighteen and one-half times the pixel pitch py in the Y dimension. In other embodiments, not every subgrating will have a unique alignment with the pixel array; multiple subgratings can still be useful for improving the signal to noise ratio and for lightfield sampling used for e.g. depth estimation.
[0048]
[0049] Pixel pitches px and py are relatively large in this example for ease of illustration. Pitches px and py are no bigger than about one-third the widest dimension of one of the spiral arms formed by the features of subgratings g.sub.#,#, and can be much smaller. In one embodiment, for example, the pixel array of
[0050] While the subject matter has been described in connection with specific embodiments, other embodiments are also envisioned. For example, the wavelength band of interest can be broader or narrower than those of the foregoing examples, and may be discontinuous. A linear array of pixels can be used alone or in combination with other linear arrays to sense one-dimensional aspects of a scene from one or more orientations. Moreover, if a given subgrating exhibits some Fourier nulls, then two or more general regions that potentially have different aspect ratios, grating designs or orientations, or any combination of the above, could provide independent measurements of the scene. Other variations will be evident to those of skill in the art. Therefore, the spirit and scope of the appended claims should not be limited to the foregoing description. Only those claims specifically reciting means for or step for should be construed in the manner required under the sixth paragraph of 35 U.S.C. 112.