Abstract
Continuing a sequence of lensless light-field imaging camera patents beginning 1999, the present invention adds light-use efficiency, predictive-model design, distance-parameterized interpolation, computational efficiency, arbitrary shaped surface-of-focus, angular diversity/redundancy, distributed image sensing, plasmon surface propagation, and other fundamentally enabling features. Embodiments can be fabricated entirely by printing, transparent/semi-transparent, layered, of arbitrary size/curvature, flexible/bendable, emit light, focus and self-illuminate at zero-separation distance between (planar or curved) sensing and observed surfaces, robust against damage/occultation, implement color sensing without use of filters or diffraction, overlay on provided surfaces, provided color and enhanced multi-wavelength color sensing, wavelength-selective imaging of near-infrared/near-ultraviolet, and comprise many other fundamentally enabling features. Embodiments can be thinner, larger/smaller, more light-use efficient, and higher-performance than recently-popularized coded aperture imaging cameras. Vast ranges of diverse previously-impossible applications are enabled: credit-card cameras/phones, in-body monitoring of healing/disease, advanced biomarker analysis systems, perfect eye-contact video conferencing, seeing fabrics/skin/housings, and manufacturing-monitoring, wear-monitoring, and machine vision capabilities.
Claims
1. A lensless light-field imaging system, comprising: an array of light sensing elements, each light-sensing element comprising a light-sensing area and each light-sensing element configured to generate an electrical photocurrent responsive to an amplitude of incoming light striking a light-sensing surface, each light-sensing surface arranged to experience angularly-varying sensitivity responsive to a direction of each path of the incoming light striking the light-sensing surface; electronics configured to interface the array of light sensing elements and further configured to provide a plurality of electronically-represented digital numbers, each digital number responsive to light received by at least one light-sensing element in the array of light sensing elements, producing a result comprising a plurality of electronically-represented digital numbers; an algorithm configured to execute on a computational processor, the algorithm for computing a two-dimensional image representation from the plurality of electronically-represented digital numbers, the two-dimensional image representation corresponding to portion of a focused image at a separation distance value measured perpendicular to the light-sensing surface of the one of the light sensing elements in the array of light sensing elements, there being a plurality of separation distance values, wherein each of the electronically-represented digital numbers are responsive to the amplitude of incoming light striking the light-sensing surface of an associated light sensing element in the array of light sensing elements and a plurality of focused image portions, and wherein the plurality of separation distance values are not a substantially same numeric value.
2. The lensless light-field imaging system of claim 1, wherein the light sensing elements of the array of light sensing elements are oriented in space to form a curved surface.
3. The lensless light-field imaging system of claim 1, wherein spatial positions of the plurality of focused image portions form a planar surface.
4. The lensless light-field imaging system of claim 1, wherein the light sensing elements of the array of light sensing elements are oriented in space to form a planar surface.
5. The lensless light-field imaging system of claim 1, wherein spatial positions of the plurality of focused image portions form a curved surface.
6. The lensless light-field imaging system of claim 1, wherein the light sensing elements of the array of light sensing elements are oriented in space to form a first curved surface and spatial positions of the plurality of focused image portions form a second curved surface.
7. The lensless light-field imaging system of claim 1, wherein the algorithm is controlled by at least one separation distance parameter.
8. The lensless light-field imaging system of claim 1, wherein the light sensing elements comprise organic semiconductors.
9. The lensless light-field imaging system of claim 1, wherein the light sensing elements comprise semiconductors that are co-optimized for both light emission and light sensing.
10. The lensless light-field imaging system of claim 1, wherein the light sensing elements are arranged to emit light for an interval of time.
11. The lensless light-field imaging system of claim 1, wherein the angularly-varying sensitivity of the light sensing elements results at least in part from a structure of the light sensing elements.
12. The lensless light-field imaging system of claim 1, wherein the angularly-varying sensitivity of the light sensing elements results at least in part from a structure attached to the array of light sensing elements.
13. The lensless light-field imaging system of claim 12, wherein the structure attached to the array of light sensing elements comprises segregated optical paths.
14. The lensless light-field imaging system of claim 13, wherein the segregated optical paths are created by separating surfaces.
15. The lensless light-field imaging system of claim 13, wherein the separating surfaces are at least partially-reflective.
16. The lensless light-field imaging system of claim 13, wherein at least one of the light sensing elements is color selective.
17. The lensless light-field imaging system of claim 13, wherein a color selective property results from a band gap property of a semiconductor device element comprised by the at least one of the light sensing elements.
18. The lensless light-field imaging system of claim 1, wherein the algorithm comprises array multiplication of numerical values obtained from calculation of a generalized inverse matrix.
19. The lensless light-field imaging system of claim 1, wherein the algorithm comprises array multiplication of numerical values obtained from an interpolation.
20. The lensless light-field imaging system of claim 1, wherein the algorithm comprises array multiplication of numerical values obtained from a predictive analytical model.
21. The lensless light-field imaging system of claim 1, wherein the algorithm comprises array multiplication of numerical values derived from a predictive analytical model.
22. The lensless light-field imaging system of claim 1, wherein the algorithm comprises array multiplication of numerical values derived from empirical measurements.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
[0233] The above and other aspects, features and advantages of the present invention will become more apparent upon consideration of the following description of preferred embodiments taken in conjunction with the accompanying drawing figures, wherein:
[0234] FIG. 1 depicts an example conceptual view of the underlying principles of the invention, facilitating a wide range of implementation methods and architectures.
[0235] FIG. 2 depicts an illustrative representational view of the confluence of the expanded features and capabilities taught in the inventor's 1999 patent family.
[0236] FIG. 3 depicts an illustrative representational view of the confluence of the expanded features and capabilities taught in the inventor's 2008 patent family. The 2009 addition of optical tomography capabilities is also noted.
[0237] FIG. 4 depicts an illustrative representational view of the confluence of the expanded features and capabilities associated with the present invention. This depiction and the elements therein are intended as only illustrative and representative and does not provide or suggest a comprehensive or exhaustive listing, structure, or characterization.
[0238] FIG. 5 depicts a more detailed view of the inventor's comprehensive lensless light-field imaging program (beginning with the inventor's 1999 patent family) is and recently-popularized coded-aperture lensless imaging.
[0239] FIG. 6 depicts a functional timeline view of non-pinhole lensless imaging, including both the inventor's comprehensive lensless light-field imaging program (beginning with the inventor's 1999 patent family) is and recently-popularized (2011-2016) coded-aperture lensless imaging (recently termed a lensless Computational Renaissance[P6]) stemming from radiation-imaging work dating from 1968 [P62].
[0240] FIG. 7 depicts a timeline of representative technology-related patents and literature with respect to the inventor's comprehensive lensless light-field imaging program.
[0241] FIG. 8a, adapted from FIG. 42 of the present inventor's U.S. Pat. No. 8,830,375 and related cases, depicts a vector space of trade-offs for semiconducting diode-junction devices.
[0242] FIG. 8b, adapted from FIG. 42 of the present inventor's U.S. Pat. No. 8,830,375 and related cases, depicts an adaptation of FIG. 8a wherein different optimizations are used for implementing single function diode-junction devices such as (but not limited to) switching diodes versus light-emitting diodes (LEDs) versus photodiodes.
[0243] FIG. 9a, adapted from the figure available on the internet at https://en.wikibooks.org/wiki/Introduction_to_Inorganic_Chemistry/Electronic_Properties_of Materials:_Superconductors_and_Semiconductors#/media/File:PnJunction-E.PNG as retrieved Jul. 3, 2017 (top portion), depicts a representation of the active carrier flow of a forward biased switching diode wherein, by design, current-flow directional switching functions are optimized and light-emission and light-detection capabilities of PN junctions are suppressed.
[0244] FIG. 9b, adapted from the figure available on the internet at https://en.wikibooks.org/wiki/Introduction_to_Inorganic_Chemistry/Electronic_Properties_of Materials:_Superconductors_and_Semiconductors#/media/File:PnJunction-E.PNG as retrieved Jul. 3, 2017 (middle portion), depicts the blocked carrier flow of a reversed biased situation for the switching diode depicted in FIG. 9a.
[0245] FIG. 9c, adapted from the figure available on the internet at https://en.wikibooks.org/wiki/Introduction_to_Inorganic_Chemistry/Electronic_Properties_of Materials:_Superconductors_and_Semiconductors#/media/File:PnJunction-E.PNG as retrieved Jul. 3, 2017 (bottom portion), depicts an energy-band representation of a switching diode wherein, by design, current-flow directional switching functions are optimized and light-emission and light-detection capabilities of PN junctions are suppressed.
[0246] FIG. 9d, adapted from the image available at http://www.leamabout-electronics.org/Semiconductors/diodes 23.php as visited on Jun. 20, 2017, depicts a representation of the physical construction of a switching diode wherein, by design, current-flow directional switching functions are optimized and light-emission and light-detection capabilities of PN junctions are suppressed.
[0247] FIG. 10a, adapted from the top portion of a figure available on the internet at https://en.wikipedia.org/wiki/Light-emitting_diode#/media/File:PnJunction-LED-E.svg as retrieved Jul. 3, 2017, depicts a carrier-process representation of an operating (inorganic or organic) semiconducting PN junction light-emitting diode (LED).
[0248] FIG. 10b, adapted from the bottom portion of a figure available on the internet at https://en.wikipedia.org/wiki/Light-emitting_diode#/media/File:PnJunction-LED-E.svg as retrieved Jul. 3, 2017, depicts an energy-transition representation of an operating (inorganic or organic) semiconducting PN junction light-emitting diode (LED).
[0249] FIG. 11a, adapted from figure 4.7.1 of the on-line notes Principles of Semiconductor Devices by B. Van Zeghbroeck, 2011, available at https://ecee.colorado.edu/-bart/book/book/chapter4/ch4_7. htm as retrieved Jul. 3, 2017, depicts an abstracted structural representation of an example (inorganic or organic) simple (simple-heterostructure) semiconducting PN junction light-emitting diode (LED).
[0250] FIG. 11b, adapted from Figure 7.1 of the on-line table of figures available on the internet at https://www.ecsarpi.edu/-schubert/Light-Emitting-Diodes-dot-org/chap07/chap07.htm as retrieved Jul. 3, 2017, depicts an abstracted structural representation of an example (inorganic or organic) more complex double-heterostructure semiconducting PN junction light-emitting diode (LED), here effectively configured as a two-PN junction sandwich. FIG. 12a (adapted from G. Gu, G. Parthasarathy, P. Burrows, T. Tian, I. Hill, A. Kahn, S. Forrest, Transparent stacked organic light emitting devices. I. Design principles and transparent compound electrodes, Journal of Applied Physics, October 1999, vol. 86 no. 8, pp. 4067-4075) depicts an example high-level structure of a three-color transparent Stacked OLED (SOLED) element.
[0251] FIG. 12b (also adapted from G. Gu, G. Parthasarathy, P. Burrows, T. Tian, I. Hill, A. Kahn, S. Forrest, Transparent stacked organic light emitting devices. I. Design principles and transparent compound electrodes, Journal of Applied Physics, October 1999, vol. 86 no. 8, pp. 4067-4075) depicts a more detailed structure of a three-color transparent SOLED element.
[0252] FIG. 13a, depicts an example energy-transition representation of an operating (inorganic or organic) simple semiconducting PN junction photodiode.
[0253] FIG. 13b, simplified and adapted from the first two figures in Comparison of waveguide avalanche photodiodes with InP and InAlAs multiplication layer for 25 Gb/s operation by J. Xiang and Y. Zhao, Optical Engineering, 53(4), published Apr. 28, 2014, available at http://opticalengineering.spiedigitallibrary.org/article.aspx?articleid=1867195 as retrieved Jul. 3, 2017, depicts an example structural representation of an example simple layered-structure PIN (inorganic or organic) simple semiconducting PN junction photodiode.
[0254] FIG. 14a, adapted from FIG. 2 of U.S. Pat. No. 7,202,102 Doped Absorption for Enhanced Responsivity for High Speed Photodiodes to J. Yao, depicts a combined energy/structure representation of a more specialized example layered-structure avalanche semiconducting PN junction photodiode.
[0255] FIG. 14b, adapted from the first two figures in Comparison of waveguide avalanche photodiodes with InP and InAlAs multiplication layer for 25 Gb/s operation by J. Xiang and Y. Zhao, Optical Engineering, 53(4), published Apr. 28, 2014, available at http://opticalengineering.spiedigitallibrary.org/article.aspx?articleid=1867195 as retrieved Jul. 3, 2017, depicts an example structural representation of an example layered-structure avalanche semiconducting PN junction photodiode.
[0256] FIG. 15a depicts material science and fabrication relationships among (1) transparent/non-transparent electronics and optoelectronics, (2) flexible/non-flexible electronics and optoelectronics, (3) printed/non-printed electronics and optoelectronics, and (4) organic/non-organic electronics and optoelectronics.
[0257] FIG. 15b provides a version of FIG. 15a where certain types of the electronics and optoelectronics are marked with asterisks (*) to signify functional contributions to various aspects of the present invention.
[0258] FIG. 16a, adapted from [P5], depicts a schematic representation of the arrangements and intended operational light paths for a pinhole camera.
[0259] FIG. 16b, adapted from [P5], depicts a schematic representation of the arrangements and intended operational light paths for a (simplified or single-lens) lens-based camera.
[0260] FIG. 16c, adapted from [P5], depicts a schematic representation of the arrangements and intended operational light paths for a mask-based camera, such as those discussed in [P62]-[P67].
[0261] FIG. 16d depicts to a schematic representation of some aspects of the present invention and the inventor's more comprehensive lensless light-field imaging program.
[0262] FIG. 17 depicts an array of parallel-oriented vignetting cavities; the bottom of each cavity can comprise or direct isolated light to light-sensing structure.
[0263] FIG. 18, adapted from FIG. 12 of the present inventor's U.S. Pat. No. 8,816,263 and related cases, illustrates a simplified view of how a vignette structure can limit the range of incident angles at which rays of light within a light field are able to reach the surface of the light-sensing element within a vignetting structure covering a light-sensing element. (Importantly, reflective effects within the vignette and diffraction effects are not illustrated.)
[0264] FIG. 19, composited and adapted from FIGS. 8 and 9a through 9b of the present inventor's U.S. Pat. No. 8,830,375 and related cases, illustrates a simplified view of the process by which the degree of vignette overlap increases as separation between the object in the scene and its distance from the micro-optic structure and light sensor array increases and how the degree of vignette overlap increases from 0% to values approaching 100% as the separation distance between a scene object and the micro-optic structure and light sensor array increases. (Importantly, reflective effects within the vignette and diffraction effects are not illustrated.)
[0265] FIGS. 20a through 20c depict illustrative representations of reflection and scattering effects within a vignette. (Importantly, diffraction effects are not illustrated.)
[0266] FIG. 21, adapted from FIG. 11b of the present inventor's U.S. Pat. No. 8,816,263 and related cases, depicts an array of parallel-oriented instances of alternating short light-sensing structures and tall, each parallel-oriented instance alternately staggered to create vignetting cavities surrounded by the sides of neighboring tall structures (which in some implementations can be light-emitting), the bottom of each cavity comprising a short light-sensing structure. In some implementations, the tall structures can be light-emitting.
[0267] FIGS. 22a through 22c depict differing illustrative 3-dimensional views of a plane the containing the sensing surface of a planar image sensor arrangement, and extending spatially in front of the planar image sensor a coordinate grid defining numerically quantizing regions on an incoming light field that can be observed by the planar image sensor arrangement. Depending on the directional capabilities of the planar image sensor arrangement, the shape of the observable light field can have a different shape than the illustrative rectangular parallelepiped.
[0268] FIG. 23a depicts an example spatial quantization of a light field extending spatially in front of the planar image sensor into a lattice of distinct indexable volume elements (voxels).
[0269] FIG. 23b depicts an example spatial quantization of the light field voxel lattice of FIG. 23a by representing the aggregate of light-emission, light reflection, and/or light propagation within the voxel as (1) having a composite quantitative value of light representing the combined aggregate of light-emission, light reflection, and/or light propagation within the volume of voxel which is (2) concentrated at a point in the interior of the voxel.
[0270] FIG. 24 depicts a pair of illustrative 3-dimensional views of an example arrangement comprising a planar array of (emitted and/or reflected) light source elements and a parallel planar array of light-sensing elements, and a spatially-quantized light-field representation between the planes. The roles of planar array of light source elements and a parallel array of light-sensing elements can be interchanged.
[0271] FIGS. 25a and 25b depict a pair of illustrative 3-dimensional views of an example variation of the arrangement depicted in FIG. 24 wherein a planar array of (emitted and/or reflected) light source elements and a planar array of light-sensing elements, are not parallel planes. The roles of planar array of light source elements and a parallel array of light-sensing elements can be interchanged.
[0272] FIG. 26a depicts another illustrative 3-dimensional view of an example variation of the arrangement depicted in FIGS. 25a and 25b wherein the dihedral angle between the planes is farther from parallel. The roles of planar array of light source elements and a parallel array of light-sensing elements can be interchanged.
[0273] FIG. 26b depicts an illustrative 3-dimensional view of another example of the arrangements depicted in FIGS. 25a, 25b, and 26a wherein the dihedral angle between the planes is sloped in two dimensions. The roles of planar array of light source elements and a parallel array of light-sensing elements can be interchanged.
[0274] FIGS. 27a and 27b depict a pair of illustrative 3-dimensional views of an example of a non-planar curved surface and a planar surface with a spatially-quantized light-field representation between the two surfaces.
[0275] FIGS. 28a and 28b depict a pair of illustrative 3-dimensional views of a variation on FIGS. 27a and 27b featuring different example non-planar curved surface.
[0276] FIG. 29a depicts an illustrative example of non-planar curved surface sensor and non-planar curved surface object with a spatially-quantized light-field representation between the two surfaces. Either or both of the curved surfaces can be configured to be a camera.
[0277] FIG. 29b depicts an example cross-section of a (rigid or flexible) curved imaging surface, for example as may be fabricated by printing or other deposition fabrication processes.
[0278] FIGS. 30a through 30c depict a variety of illustrative 3-dimensional views of an example variation of the arrangement depicted in FIGS. 24, 25a, 25b, 26a, and 25b wherein the array of (emitted and/or reflected) light source elements are split among a plurality of smaller parallel planes at different separation distances from the planar array of light-sensing elements. Depending on the spatial arrangement, some portions of some of the smaller parallel planes can be observationally occulted from some regions of the planar surface. The roles of the plurality of planar arrays of light source elements and a parallel array of light-sensing elements can be interchanged.
[0279] FIGS. 31a and 31b depict a variety of illustrative 3-dimensional views of an example variation of the arrangement depicted in FIGS. 24, 25a, 25b, 26a, and 25b wherein the array of (emitted and/or reflected) light source elements are distributed over a connected group of planes, at least one parallel to the planar array of light-sensing elements.
[0280] FIGS. 32a through 32c depict a variety of illustrative 3-dimensional views of an example wherein the array of (emitted and/or reflected) light source elements are distributed over a complex collection of connected and disconnected planes, some of which are parallel to the planar array of light-sensing elements, and some of which observationally occulted others from some regions of the planar surface by being situated directly in front of others.
[0281] FIGS. 33a and 33b depict example inward-directed or outward-directed sensor-pixel lattice locations distributed on a rigid or elastic curved convex-shaped surface.
[0282] FIGS. 34a through 34d depicts examples of pairs of curved and sharply-angled surfaces, one of the pair inside the other of that pair. In any of the arrangements depicted, at least one of the inner surface and the outer surface can be a camera arranged to view the other surface. As considered elsewhere, the camera can be configured to provide self-illumination.
[0283] FIGS. 35a through 35c depict illustrative examples of bumpy and/or pitted sensor surfaces that can provide angular diversity. Such arrangements can also be used to provide sensor robustness via spatial diversity, to provide directed angle-orientation viewing, and to provide other types of functions. These can be combined with the generous recovery capabilities described in the mathematical treatment to follow, and enhanced further by the statistical corrections obtainable using the Moore-Penrose pseudo-inverse, to provide immense imaging robustness to a wide range of degradation and partial occultation effects.
[0284] FIG. 36 depicts example correspondences between a physical optical arrangement comprising an optical process (including for example vignetting optics and free-space separation should the image sensor not be in contact with the actual source image) and a mathematical model of that physical optical arrangement (transforming an actual image array data to measured image array data by a numerical model of the optical process.
[0285] FIG. 37 depicts an illustrative example of Mathematical Recovery of an approximate representation of the actual Image from Measured Image Array Data obtained by operating on the Measured Image Array Data by a Numerical Inverse of the Model of the Optical Process as depicted in FIG. 36.
[0286] FIG. 38, adapted from FIG. 2b of the present inventor's U.S. Pat. No. 8,830,375 and related cases, depicts an exemplary embodiment comprising a micro-optic structure, a light sensor array, an image formation signal processing operation and an optional additional subsequent image processing operations, herein the micro-optic structure and light sensor array are grouped into a first subsystem, and the image formation signal processing operation and subsequent image processing operations are grouped into a second subsystem. As discussed in the present inventor's U.S. Pat. No. 8,830,375 and related cases, various other arrangements are possible and provided for by aspects of the invention.
[0287] FIG. 39a depicts an example scheme wherein manufacturing, physical, optical, and mathematical considerations are used to create a reproducible manufacturing design such that is adequate manufacturing tolerances are obtained an analytical predictive model can be used to produce numerical models of the optical situations to be recovered without the use of empirical measurements.
[0288] FIG. 39b depicts an example variation on the scheme presented in FIG. 39a wherein post-manufacturing empirical measurements are used to further fine-calibrate the system performance of each particular manufactured article.
[0289] FIG. 40 depicts an example representation of example serialization processes and de-serialization processes for image recovery from an inverse or pseudo-inverse model as provided for by the invention.
[0290] FIG. 41 depicts a representation of example serialization processes transforming a measured image produced by a light-field travelling through an optical structure (here a vignette array) at being measured by a sensor array.
[0291] FIG. 42 depicts a representation of example empirical image-basis training sequence, or alternatively a collection of predictive-model-generated image-bases that directly populate a JKNM matrix providing a numerical model of the optical environment from which a future image will later be recovered as provided for by aspects of the invention.
[0292] FIG. 43a depicts a representation of example image recovery process using an inverse square-matrix representing an approximate inverse model as provided for by the invention.
[0293] FIG. 43b depicts a representation of example image recovery process using a generalized-inverse or pseudo-inverse matrix representing an approximate pseudo-inverse underspecified model as provided for by the invention.
[0294] FIG. 43c depicts a representation of example image recovery process using a generalized-inverse or pseudo-inverse matrix representing an approximate pseudo-inverse overspecified model as provided for by the invention.
[0295] FIG. 44a depicts a simple computational approach for image recovery as provided for by the invention.
[0296] FIG. 44b depicts an example representation of a far more numerically-complex spectral or transform computation (which numerically amounts to additional basis rotation transformation steps) as would be used in spectral or transform methods.
[0297] FIG. 44c depicts a comparison of the more direct computational approach depicted in FIG. 44a and the approach depicted in FIG. 44b, demonstrating comparative reasons to reject the far more numerically-complex spectral or transform computational approached represented in FIG. 44b.
[0298] FIG. 45 depicts the use of classical ill-posed inverse-problem regularization methods as employed in the Rambus [P4] and Rice University Flat Cam [P5] implementations as well as other coded aperture imaging implementations. These classical ill-posed inverse-problem regularization methods are widely used in many areas but also have an established role in optical systems design and analysis; see for example [B5] and [B6].
[0299] FIG. 46 depicts an abstract representation of an Identity structure within a (necessarily-sparse) Identity 4-tensor.
[0300] FIG. 47 depicts an abstract representation of a dilation around the Identity structure within a (necessarily-sparse) Identity 4-tensor.
[0301] FIG. 48 depicts a ((77)(77)) matrix-of-matrices representation of a (7777) Identity 4-tensor as abstracted in FIG. 46.
[0302] FIG. 49 depicts an example ((77)(77)) matrix-of-matrices representation of the dilation around the Identity structure of a (7777) 4-tensor as abstracted in FIG. 46.
[0303] FIG. 50 depicts an abstract representation of the trade-off between Space/Inverse-Space/Spatial Frequency Localization Methods and the sparcity of numerical model tensors, matrices, and their inverses. This can be used, for example, in designing vignetting and aperture arrays.
[0304] FIGS. 51a through 51e depict how the sparcity of an example numerical model matrix serializing a numerical model 4-tensor degrades as the image source moves farther and farther from the image sensor (using numerical models at each distances predicted by an analytical geometric predictive model provided for by the invention).
[0305] FIG. 52 depicts an example ((77)(77)) matrix-of-matrices representation of the numerical model 4-tensor and example measured image (right) at zero separation distance from a single-illuminated-pixel source image (left). As described elsewhere, the source image need not be on a parallel plane and can be distributed arbitrarily in space.
[0306] FIG. 53 depicts an example ((77)(77)) matrix-of-matrices representation of the numerical model 4-tensor and example measured image (right) at a small non-zero separation distance from the same single-illuminated-pixel source image (left) as in FIG. 52.
[0307] FIG. 54 depicts an example ((77)(77)) matrix-of-matrices representation of the numerical model 4-tensor and example measured image (right) at zero separation distance from a two-illuminated-pixel source image (left). As described elsewhere, the source image need not be on a parallel plane and can be distributed arbitrarily in space.
[0308] FIG. 55 depicts an example ((77)(77)) matrix-of-matrices representation of the numerical model 4-tensor and example measured image (right) at a small non-zero separation distance from the same two-illuminated-pixel source image (left) as in FIG. 54.
[0309] FIG. 56 depicts an example ((77)(77)) matrix-of-matrices representation of the numerical model 4-tensor and example measured image (right) at zero separation distance from a more closely-spaced two-illuminated-pixel source image (left). As described elsewhere, the source image need not be on a parallel plane and can be distributed arbitrarily in space.
[0310] FIG. 57 depicts an example ((77)(77)) matrix-of-matrices representation of the numerical model 4-tensor and example measured image (right) at a small non-zero separation distance from the same more closely-spaced two-illuminated-pixel source image (left) as in FIG. 56.
[0311] FIGS. 58a through 58c depict three of forty-nine steps of an example empirical training sequence as provided for by the invention. In an implementation of the invention, such a procedure could be performed for a selected collection of one or more separation distances between the image sensor and the source image. As described elsewhere, the source image need not be on a parallel plane and can be distributed arbitrarily in the observable space.
[0312] FIG. 59 depicts an example polynomial fitting function interpolation method for interpolating the numerical model, its inverse/pseudo-inverse, and/or other functions for separation distances values lying between k empirically-trained separation distances. As described elsewhere, the source image need not be on a parallel plane and can be distributed arbitrarily in the observable space
[0313] FIG. 60 depicts an example polynomial fitting function interpolation method for interpolating the numerical model, its inverse/pseudo-inverse, and/or other functions for separation distances values lying between k separation distances used by a predictive-model. As described elsewhere, the source image need not be on a parallel plane and can be distributed arbitrarily in the observable space.
[0314] FIG. 61 depicts an example of very poor curve-fit interpolation of the matrix elements of the numerical model between measure data distances resulting from not having correct or sufficient terms in a model polynomial.
[0315] FIG. 62 depicts an example piecewise-linear interpolation of the matrix elements of the numerical model between k empirically-trained separation distances.
[0316] FIG. 63a depicts how with very small separation distances and certain non-alignment of source-pixel locations and sensor pixel locations some sensor pixel locations cannot receive light from proximate source-pixel locations, while FIG. 63b depicts how at slightly greater separation distances this condition does not occur; such processes can give rise to the rising and falling of selected curves in the example empirical training data shown in the example of FIG. 61.
[0317] FIG. 64 depicts an example piecewise-linear interpolation method for interpolating the numerical model, its inverse/pseudo-inverse, and/or other functions for separation distances values lying between k empirically-trained separation distances. As described elsewhere, the source image need not be on a parallel plane and can be distributed arbitrarily in the observable space.
[0318] FIG. 65 depicts an example piecewise-linear interpolation method for interpolating the numerical model, its inverse/pseudo-inverse, and/or other functions for separation distances values lying between k separation distances used by a predictive-model. As described elsewhere, the source image need not be on a parallel plane and can be distributed arbitrarily in the observable space.
[0319] FIG. 66 depicts an abstract representation of the mathematical recovery of image from measured image array data.
[0320] FIG. 67 depicts a variation on the representation of the mathematical recovery of image from measured image array data shown in FIG. 66 wherein more measurements are made than needed, resulting in an over-specified collection of measurements that can be expected to lead to inconsistent calculation outcomes when different subgroups of measurements are used to solve for the recovered image. As provided for by the invention, a Moore-Penrose pseudo-inverse operation gives a least-squares fit to the expected inconsistent outcomes; this is depicted in the bottom portion of the figure.
[0321] FIG. 68a through 68d depict example numerical model outcomes responsive to a single-illuminated pixel as generated for various separation distances by an example predictive analytical geometric model. The particular example predictive analytical geometric model used only accounts for vignette occultation as calculated by simple ray-trancing and does not include the effects of vignette-internal reflections, vignette-internal scattering, vignette-aperture diffraction, surface plasmoid propagation, etc.
[0322] FIG. 69a depicts three example interactive quantization-effect modeling outcomes wherein controllable quantization is artificially imposed on measurement data so as to predictively model and characterize the effects of quantizing nonlinearities imposed by electronic digital-to-analog converter processes on empirical training, predictive-model generated, and real-time measurements as they influence the quality of image recovery.
[0323] FIG. 69b depicts three example interactive offset-effect modeling outcomes wherein controllable measurement offset is artificially imposed on measurement data so as to predictively model and characterize the effects of measurement offsets imposed by electronic digital-to-analog converter processes on empirical training, predictive-model generated, and real-time measurements as they influence the quality of image recovery.
[0324] FIG. 69c depicts an example interactive noise-modeling control used to introduce synthetically-generated noise and noise processes so as to predictively model and characterize its effects. The selection shown controls additive Gaussian noise, but other noise processes associated with photodiodes can similarly be introduced.
[0325] FIGS. 70a through 70d depict example interactive modeling outcomes showing the effect of noise, offset, and both these for a 32-step (5-bit) quantized (DN [B2]) measurement dynamic range.
[0326] FIGS. 71a through 71d depict example interactive modeling outcomes showing the effect of noise, offset, and both these for a 64-step (6-bit) quantized (DN) measurement dynamic range.
[0327] FIG. 72a through 72d depict example interactive modeling outcomes showing the effect of noise, offset, and both these for a 128-step (7-bit) quantized (DN) measurement dynamic range.
[0328] FIGS. 73a through 73d depict example interactive modeling outcomes showing the effect of noise, offset, and both these for a 140-step (slightly more than 7-bit) quantized (DN) measurement dynamic range.
[0329] FIG. 74a through 74d depict example interactive modeling outcomes showing the effect of noise, offset, and both these for a 150-step (yet more than 7-bit) quantized (DN) measurement dynamic range.
[0330] FIG. 75a, adapted from [B7] Figure C2.5.16(a), depicts a first example pixel-cell multiplexed-addressing circuit for an individual OLED within an OLED array that includes a dedicated light-coupled monitoring photodiode for use in regulating the light output of the individual OLED so as to prevent user-observed fading or other brightness-variation processes.
[0331] FIG. 75b, adapted from [B7] Figure C2.5.16(b), depicts a second example pixel-cell multiplexed-addressing circuit for an individual OLED within an OLED array that includes a dedicated light-coupled monitoring photodiode for use in regulating the light output of the individual OLED so as to prevent user-observed fading or other brightness-variation processes.
[0332] FIG. 76a, adapted from [B18], depicts an example pixel-cell multiplexed-addressing circuit for an individual OLED within an OLED array with a monitoring feature.
[0333] FIG. 76b, also adapted from [B18], depicts an example pixel-cell multiplexed-addressing circuit for an isolated high-performance photodiode or phototransistor within a high-performance photodiode or phototransistor array with a forced-measurement provision.
[0334] FIG. 77 depicts a simplified view of FIG. 1 showing only the signal and computational portion of FIG. 1 as will be useful in a subsequent discussion.
[0335] FIG. 78 depicts a variation of the arrangement represented in FIG. 77 wherein the Inverse Model is rendered as a parameterized Inverse Model which can be altered responsive to one or more provided parameters.
[0336] FIG. 79 depicts a variation of the arrangement represented in FIG. 78 wherein the electrical signals and/or computational data produced by the Optical Sensor are in parallel provided in whole or selected (or selectable) part to a plurality of Inverse Models, each producing one or more computationally-produced images responsive to the sensor data.
[0337] FIG. 80 depicts a variation of the arrangement represented in FIG. 78 wherein the electrical signals and/or computational data produced by the Optical Sensor is handled by a computer or computational element (such as a microprocessor, GPU, DSP chip, ALU, FPLA, combination of two or more these, pluralities of these, etc.) in some fashion that at least permits the electrical signals and/or computational data produced by the Optical Sensor to be stored as a file.
[0338] FIG. 81 depicts a variation of the arrangement represented in FIG. 78 wherein the aforementioned handling by a computer or computational element is controlled in some manner by a control parameter.
[0339] FIG. 82a depicts a variation of the arrangement represented in FIG. 81 wherein a plurality of stored files is created, for example with different parameter values associated with each stored file.
[0340] FIG. 82b depicts an example arrangement wherein a stored file is used by a fixed Inverse Model to create a computationally-produced image.
[0341] FIG. 82c depicts an example arrangement wherein a stored file is used by a parameterized Inverse Model to create a computationally-produced image, and further wherein parameter value(s) associated the parameterized Inverse Model are externally associated with each stored file.
[0342] FIG. 82d depicts an example arrangement wherein a stored file is used by a parameterized Inverse Model to create a computationally-produced image, and further wherein parameter value(s) associated the parameterized Inverse Model are derived from or obtained from the stored file.
[0343] FIG. 83 depicts four example conformations of a particular bendable, flexible, and/or or pliable optical-sensor/optical-structure sheet, each conformation giving rise to an associated model for how a light field is sensed in the context of reconstructing an image, and each model giving rise to its own associated inverse model. It is noted that, as dependent on the properties and limitations of the optical-structure, there can be small blind spots in regions of sufficiently-high curvature.
[0344] FIG. 84a through FIG. 84d depict how various conformations can be used to render a computationally derived image
[0345] FIG. 85a depicts an example arrangement for an optical-sensor/optical-structure of fixed conformation.
[0346] FIG. 85b depicts an example arrangement for an optical-sensor/optical-structure of variable conformation, for example should the surface bend, deform, hinge, expand, contract, etc.
[0347] FIG. 85c depicts a variation of FIG. 85b wherein the model and the Inverse Model are parameterized.
[0348] FIG. 86a depicts an arrangement useful for using optical training to sense the present conformation of an optical-sensor/optical-structure of variable conformation, for example should the surface bend, deform, hinge, expand, contract, etc.
[0349] FIG. 86b depicts an arrangement useful for using internal sensing means to sense the present conformation of an optical-sensor/optical-structure of variable conformation, for example should the surface bend, deform, hinge, expand, contract, etc.
[0350] FIG. 86c depicts an arrangement useful for using information about a movable or changing support structure or contact arrangement to identify the present conformation of an optical-sensor/optical-structure of variable conformation, for example should the surface bend, deform, hinge, expand, contract, etc.
[0351] FIG. 86d depicts an arrangement useful for using external observation mean, for example such as one or more observing video camera(s) means to sense the present conformation of an optical-sensor/optical-structure of variable conformation, for example should the surface bend, deform, hinge, expand, contract, etc.
[0352] FIG. 87a depicts an optical-sensor/optical-structure of variable conformation, for example should the surface bend, deform, hinge, expand, contract, etc. producing an output signal and/or output data.
[0353] FIG. 87b depicts a controllable variable-conformation material whose shape/conformation can be controlled by externally-provide control stimulus.
[0354] FIG. 87c depicts an optical-sensor/optical-structure of variable conformation, for example should the surface bend, deform, hinge, expand, contract, etc. producing an output signal and/or output data fabricated on, with, or co-integrated with controllable variable-conformation material whose shape/conformation can be controlled by externally-provide control stimulus.
[0355] FIG. 88a depicts use of conformational sensing information to derive or compute parameter values for a parameterized Inverse Model.
[0356] FIG. 88b depicts use of conformational control parameter information to derive or compute parameter values for a parameterized Inverse Model.
[0357] FIG. 88c depicts use of both conformational sensing information and conformational control parameter information to derive or compute parameter values for a parameterized Inverse Model.
DETAILED DESCRIPTION
[0358] In the following description, reference is made to the accompanying drawing figures which form a part hereof, and which show by way of illustration specific embodiments of the invention. It is to be understood by those of ordinary skill in this technological field that other embodiments may be utilized, and structural, electrical, as well as procedural changes may be made without departing from the scope of the present invention.
[0359] In the following description, numerous specific details are set forth to provide a thorough description of various embodiments. Certain embodiments may be practiced without these specific details or with some variations in detail. In some instances, certain features are described in less detail so as not to obscure other aspects. The level of detail associated with each of the elements or features should not be construed to qualify the novelty or importance of one feature over the others.
[0360] FIG. 7 depicts a timeline of representative patents and literature and patents in lensless imaging and light-field imaging with respect to the inventor's comprehensive lensless light-field imaging program.
[0361] FIG. 8a, adapted from FIG. 42 of the present inventor's U.S. Pat. No. 8,830,375 and related cases, depicts a vector space of trade-offs for semiconducting diode-junction devices. As taught in the inventor's earlier patents, there are opportunities to, rather than exclusively optimize for light-sensing or light-emission, to instead co-optimize for both light-sensing and light-emission.
[0362] The present invention provides for co-optimizing of doping, electrode configurations, structure, and other attributes for both light-sensing and light-emission, giving rise to entirely new kinds of semiconductor optoelectronic elements and devices. Rapidly evolving organic semiconductor material science methods, including polymer properties and meta-material properties, can be used to improve quantum efficiency, noise performance, transparency, size requirements, electrical characteristics, etc. as well as facilitate useful manufacturing techniques such as high-resolution printing. Additional features, such as angular-selectivity and wavelength selectivity, can also be included. Additional structures, such as vignetting or aperturing arrays, reflective optical path walls to reduce incident light-loss, angular diversity, curvature, flexibility, etc. can be co-integrated, and can for example be designed to produce predictable reproducible optical sensing behaviors. Exotic features, such as predictable or and/or reproducible surface plasmon propagation to selected light sensors to further reduce incoming light loss and use of quantum dots, can be included.
[0363] FIG. 8b, adapted from FIG. 42 of the present inventor's U.S. Pat. No. 8,830,375 and related cases, depicts an adaptation of FIG. 8a wherein different optimizations are used for implementing single function diode-junction devices such as (but not limited to) switching diodes versus light-emitting diodes (LEDs) versus photodiodes.
[0364] FIG. 9a, adapted from the figure available on the internet at https://en.wikibooks.org/wiki/Introduction_to_Inorganic_Chemistry/Electronic_Properties_of Materials:_Superconductors_and_Semiconductors#/media/File:PnJunction-E.PNG as retrieved Jul. 3, 2017 (top portion), depicts a representation of the active carrier flow of a forward biased switching diode wherein, by design, current-flow directional switching functions are optimized and light-emission and light-detection capabilities of PN junctions are suppressed.
[0365] FIG. 9b, adapted from the figure available on the internet at https://en.wikibooks.org/wiki/Introduction_to_Inorganic_Chemistry/Electronic_Properties_of Materials:_Superconductors_and_Semiconductors#/media/File:PnJunction-E.PNG as retrieved Jul. 3, 2017 (middle portion), depicts the blocked carrier flow of a reversed biased situation for the switching diode depicted in FIG. 9a.
[0366] FIG. 9c, adapted from the figure available on the internet at https://en.wikibooks.org/wiki/Introduction_to_Inorganic_Chemistry/Electronic_Properties_of Materials:_Superconductors_and_Semiconductors#/media/File:PnJunction-E.PNG as retrieved Jul. 3, 2017 (bottom portion), depicts an energy-band representation of a switching diode wherein, by design, current-flow directional switching functions are optimized and light-emission and light-detection capabilities of PN junctions are suppressed.
[0367] FIG. 9d, adapted from the image available at http://www.leamabout-electronics.org/Semiconductors/diodes 23.php as visited on Jun. 20, 2017, depicts a representation of the physical construction of a switching diode wherein, by design, current-flow directional switching functions are optimized and light-emission and light-detection capabilities of PN junctions are suppressed.
[0368] FIG. 10a, adapted from the top portion of a figure available on the internet at https://en.wikipedia.org/wiki/Light-emitting_diode#/media/File:PnJunction-LED-E.svg as retrieved Jul. 3, 2017, depicts a carrier-process representation of an operating (inorganic or organic) semiconducting PN junction light-emitting diode (LED).
[0369] FIG. 10b, adapted from the bottom portion of a figure available on the internet at https://en.wikipedia.org/wiki/Light-emitting_diode#/media/File:PnJunction-LED-E.svg as retrieved Jul. 3, 2017, depicts an energy-transition representation of an operating (inorganic or organic) semiconducting PN junction light-emitting diode (LED).
[0370] FIG. 11a, adapted from Figure 4.7.1 of the on-line notes Principles of Semiconductor Devices by B. Van Zeghbroeck, 2011, available at https://ecee.colorado.edu/-bart/book/book/chapter4/ch4_7. htm as retrieved Jul. 3, 2017, depicts an abstracted structural representation of an example (inorganic or organic) simple (simple-heterostructure) semiconducting PN junction light-emitting diode (LED).
[0371] FIG. 11b, adapted from Figure 7.1 of the on-line table of figures available on the internet at https://www.ecsarpi.edu/-schubert/Light-Emitting-Diodes-dot-org/chap07/chap07.htm as retrieved Jul. 3, 2017, depicts an abstracted structural representation of an example (inorganic or organic) more complex double-heterostructure semiconducting PN junction light-emitting diode (LED), here effectively configured as a two-PN junction sandwich. When component layers are properly doped, a P-I-N (P-type/Intrinsic/N-type) structure is formed, confining charge carriers into a small energy gap surrounded by abrupt energy discontinuities that can be used to create a quantum well; the charge carriers recombine in the Intrinsic region and emit photons with wavelengths defined by corresponding discrete permissible energy transitions.
[0372] FIG. 12a (adapted from G. Gu, G. Parthasarathy, P. Burrows, T. Tian, I. Hill, A. Kahn, S. Forrest, Transparent stacked organic light emitting devices. I. Design principles and transparent compound electrodes, Journal of Applied Physics, October 1999, vol. 86 no. 8, pp. 4067-4075) depicts an example high-level structure of a three-color transparent Stacked OLED (SOLED) element.
[0373] FIG. 12b (also adapted from G. Gu, G. Parthasarathy, P. Burrows, T. Tian, I. Hill, A. Kahn, S. Forrest, Transparent stacked organic light emitting devices. I. Design principles and transparent compound electrodes, Journal of Applied Physics, October 1999, vol. 86 no. 8, pp. 4067-4075) depicts a more detailed structure of a three-color transparent SOLED element.
[0374] FIG. 13a, depicts an example energy-transition representation of an operating (inorganic or organic) simple semiconducting PN junction photodiode.
[0375] FIG. 13b, simplified and adapted from the first two figures in Comparison of waveguide avalanche photodiodes with InP and InAlAs multiplication layer for 25 Gb/s operation by J. Xiang and Y. Zhao, Optical Engineering, 53(4), published Apr. 28, 2014, available at http://opticalengineering.spiedigitallibrary.org/article.aspx?articleid=1867195 as retrieved Jul. 3, 2017, depicts an example structural representation of an example simple layered-structure PIN (inorganic or organic) simple semiconducting PN junction photodiode.
[0376] FIG. 14a, adapted from FIG. 2 of U.S. Pat. No. 7,202,102 Doped Absorption for Enhanced Responsivity for High Speed Photodiodes to J. Yao, depicts a combined energy/structure representation of a more specialized example layered-structure avalanche semiconducting PN junction photodiode.
[0377] FIG. 14b, adapted from the first two figures in Comparison of waveguide avalanche photodiodes with InP and InAlAs multiplication layer for 25 Gb/s operation by J. Xiang and Y. Zhao, Optical Engineering, 53(4), published Apr. 28, 2014, available at http://opticalengineering.spiedigitallibrary.org/article.aspx?articleid=1867195 as retrieved Jul. 3, 2017, depicts an example structural representation of an example layered-structure avalanche semiconducting PN junction photodiode.
[0378] FIG. 15a depicts material science and fabrication relationships among (1) transparent/non-transparent electronics and optoelectronics, (2) flexible/non-flexible electronics and optoelectronics, (3) printed/non-printed electronics and optoelectronics, and (4) organic/non-organic electronics and optoelectronics.
[0379] FIG. 15b provides a version of FIG. 15a where certain types of the electronics and optoelectronics are marked with asterisks (*) to signify functional contributions to various aspects of the present invention.
[0380] FIG. 16a, adapted from [P5], depicts a schematic representation of the arrangements and intended operational light paths for a pinhole camera. The box represents a light-tight enclosure with a pinhole opening on the left side that blocks much of the incoming light field (depicted as approaching from the right) but permits transmission of narrow-diameter incoming light rays to enter the enclosure and travel through a region of free-space so as to widen the light area to match that of a (typically rectangular) image sensor, film emulsion, display surface, etc.
[0381] FIG. 16b, adapted from [P5], depicts a schematic representation of the arrangements and intended operational light paths for a (simplified or single-lens) lens-based camera. The box represents a light-tight enclosure with a lens and supporting opening for the lens on the left side that bends most rays of the incoming light field (depicted as approaching from the right) for transmission and travel through a region of free-space defined by the lens focal length and the lens law equation so as to create focused image of a selected depth-of-field onto a (typically rectangular) image sensor, film emulsion, display surface, etc.
[0382] FIG. 16c, adapted from [P5], depicts a schematic representation of the arrangements and intended operational light paths for a mask-based camera, such as those discussed in [P62]-[P67]. The relatively flatter box represents a light-tight enclosure with a masked opening on the left side that blocks some of the incoming light field (depicted as approaching from the right) and permits transmission the remaining incoming light rays to enter the enclosure and travel through a shorter region of free-space so as to widen the light area to match that of a (typically rectangular) image sensor.
[0383] FIG. 16d depicts to a schematic representation of some aspects of the present invention and the inventor's more comprehensive lensless light-field imaging program. No free space is needed and any vignetting optical structure can directly contact and/or be co-integrated or layered upon (by deposition, printing, etc.) the image sensor surface. The optical width of such a vignetting optical structure can be as small as one light-sensing pixel in a light-sensing array, and such a vignetting optical structure can (unlike a mask or the Rambus [P4] diffraction element) have a very simple structure.
[0384] The invention further provides for vignetting arrays, aperturing arrays, or other optical structures attached to, co-fabricated on, or co-fabricated with an array of light sensors to include for example, reflective optical path walls to reduce incident light-loss, angular diversity, curvature, flexibility, etc.
[0385] The invention further provides for vignetting arrays, aperturing arrays, or other optical structures attached to, co-fabricated on, or co-fabricated with an array of light sensors to be designed to produce predictable reproducible optical sensing behaviors. The invention further provides for vignetting arrays, aperturing arrays, or other optical structures attached to, co-fabricated on, or co-fabricated with an array of light sensors to include or facilitate advance light-processing features such as predictable or and/or reproducible surface plasmon propagation to selected light sensors to further reduce incoming light loss, use of quantum dots, etc.
[0386] Additionally, the invention provides for each light-sensing pixel element in a light-sensing array to comprise one or separate wavelength-selective light-sensing sub-elements, for example as taught in the inventor's 1999 and 2008 patent families. In some implementations these sub-elements can be spatially adjacent and share the same vignetting or other light-structuring pathway. In other implementations it is advantageous to stack two or more wavelength-selective light-sensing sub-elements in layers, analogous to Stacked Organic Light Emitting Diodes (SOLEDs) as discussed in the inventor's 1999 and 2008 patent families. It is further noted that structures stacking layers of two or more wavelength-selective light-sensing sub-elements can be designed to limit or advantageously structure different vignetting effects each wavelength-selective light-sensing sub-element will experience at each particular depth in the layered stack. It is noted that recent (2016) developments in this area implement light-field imaging (without the use of microlenses) employing layers of optical sensors [P43].
[0387] FIG. 17 depicts an array of parallel-oriented vignetting cavities; the bottom of each cavity can comprise or direct isolated light to light-sensing structure.
[0388] FIG. 18, adapted from FIG. 12 of the present inventor's U.S. Pat. No. 8,816,263 and related cases, illustrates a simplified view of how a vignette structure can limit the range of incident angles at which rays of light within a light field are able to reach the surface of the light-sensing element within a vignetting structure covering a light-sensing element. (Importantly, reflective effects within the vignette and diffraction effects are not illustrated.)
[0389] FIG. 19, composited and adapted from FIGS. 8 and 9a through 9b of the present inventor's U.S. Pat. No. 8,830,375 and related cases, illustrates a simplified view of the process by which the degree of vignette overlap increases as separation between the object in the scene and its distance from the micro-optic structure and light sensor array increases and how the degree of vignette overlap increases from 0% to values approaching 100% as the separation distance between a scene object and the micro-optic structure and light sensor array increases. (Importantly, reflective effects within the vignette and diffraction effects are not illustrated.)
[0390] FIGS. 20a through 20c depict illustrative representations of reflection and scattering effects within a vignette. (Importantly, diffraction effects are not illustrated.)
[0391] FIG. 21, adapted from FIG. 11 b of the present inventor's U.S. Pat. No. 8,816,263 and related cases, depicts an array of parallel-oriented instances of alternating short light-sensing structures and tall, each parallel-oriented instance alternately staggered to create vignetting cavities surrounded by the sides of neighboring tall structures (which in some implementations can be light-emitting), the bottom of each cavity comprising a short light-sensing structure. In some implementations, the tall structures can be light-emitting.
Light-Field Origins, Propagation, and Lensless-Light-Field Sensing
[0392] Returning to the depiction illustrated in FIG. 1, an Optical Scene creates a Light-Field that is directed to an Optical Sensor which is preceded by one or more Lensless Optical Structure(s) that in some manner alters the light field in a predictable spatial manner. The Optical Sensor produces (typically time-varying) electrical signals and/or computational data responsive (instantly and/or within some time-delay) to light incident to the surface or other substructure(s) within the Optical Sensor at any given moment.
[0393] In terms of the mathematical development above, objects or situations producing reflected, refracted, and/or light-emitted contributions to the Light-Field can be represented in a spatially-quantized manner as a light-field source array.
[0394] FIGS. 22a through 22c depict differing illustrative 3-dimensional views of a plane the containing the sensing surface of a planar image sensor arrangement, and extending spatially in front of the planar image sensor a coordinate grid defining numerically quantizing regions on an incoming light field that can be observed by the planar image sensor arrangement. Depending on the directional capabilities of the planar image sensor arrangement, the shape of the observable light field can have a different shape than the illustrative rectangular parallelepiped.
[0395] FIG. 23a depicts an example spatial quantization of a light field extending spatially in front of the planar image sensor into a lattice of distinct indexable volume elements (voxels).
[0396] FIG. 23b depicts an example spatial quantization of the light field voxel lattice of FIG. 23a by representing the aggregate of light-emission, light reflection, and/or light propagation within the voxel as (1) having a composite quantitative value of light representing the combined aggregate of light-emission, light reflection, and/or light propagation within the volume of voxel which is (2) concentrated at a point in the interior of the voxel. If the light-field is indexed by wavelength or wavelength range, the composite quantitative value can be represented as a function of an associated wavelength index or quantized-wavelength index. The point within each voxel in the light field can be used to define a spatially-quantized vector field representing a physical spatially-continuous vector field (and/or numerical representation thereof). Accordingly, composite quantitative value of light representing the combined aggregate of light-emission, light reflection, and/or light propagation within the volume of voxel can further be represented as a function with a directional argument. In such a manner, the spatial and spectral (wavelength) aspects of a spatially (and if relevant, spectrally) quantized representation of a physical light field can be computationally represented as a multiple-index array.
Case A: Fixed Separation Distance:
[0397] Although it will be shown that the constraints on this arrangement can be extremely relaxed, in can be initially convenient to regard the objects or situations producing contributions to the light-field as lying in a plane parallel to an image sensor plane, and the contributions to the light-field comprising a planar (for example rectangular, other shapes explicitly admissible) array of light-providing light-source spatially-quantized pixels, each light-source pixel emitting light that in various manners make their way to a parallel spatially-separated image sensor plane. The image sensor plane comprises a planar (for example rectangular, other shapes explicitly admissible) array of light-providing spatially-quantized light-sensing pixels, these light-sensing pixels producing an electrical signal that can be further processed. The discussion and capabilities of this development explicitly include cases with zero separation distance between at least a planar array of light-providing light-source pixels and at least a planar array of light-sensing pixels.
[0398] As an illustration of Case A, FIG. 24 depicts a pair of illustrative 3-dimensional views of an example arrangement comprising a planar array of (emitted and/or reflected) light source elements and a parallel planar array of light-sensing elements, and a spatially-quantized light-field representation between the planes. Note the roles of planar array of light source elements and a parallel array of light-sensing elements can be interchanged.
Case B: Continuous Spatially-Varying Separation DistancesNon-Parallel Planes:
[0399] Relaxing the constraints in Case A, the above-described planes of (a) the objects or situations producing contributions to the light-field and (b) the image sensor are not parallel but rather oriented at some non-parallel and non-perpendicular dihedral angle. The resulting light-field has separation mixed distances. The discussion and capabilities of this development explicitly include cases with zero separation distance between at least a subset (even a 1-dimensional edge or even a single point) of a planar array of light-providing light-source pixels and a subset (even a 1-dimensional edge or even a single point) of at least a planar array of light-sensing pixels.
[0400] As an illustration of Case B, FIGS. 25a and 25b depict a pair of illustrative 3-dimensional views of an example variation of the arrangement depicted in FIG. 24 wherein a planar array of (emitted and/or reflected) light source elements and a planar array of light-sensing elements, are not parallel planes. Note the roles of planar array of light source elements and a parallel array of light-sensing elements can be interchanged.
[0401] As another illustration of Case B, FIG. 26a depicts another illustrative 3-dimensional view of an example variation of the arrangement depicted in FIGS. 25a and 25b wherein the dihedral angle between the planes is farther from parallel. Note the roles of planar array of light source elements and a parallel array of light-sensing elements can be interchanged.
[0402] As yet another illustration of Case B, FIG. 26b depicts an illustrative 3-dimensional view of another example of the arrangements depicted in FIGS. 25a, 25b, and 26a wherein the dihedral angle between the planes is sloped in two dimensions. Note the roles of planar array of light source elements and a parallel array of light-sensing elements can be interchanged.
Example C: Continuous Spatially-Varying Separation DistancesCurved Surfaces
[0403] Relaxing the constraints in Case A in yet another way, one or both of (a) the objects or situations producing contributions to the light-field and/or (b) the image sensor reside on smoothly-curved non-planar surface. The resulting light-field has separation mixed distances and in some cases possible occultation depending on variations in curvature. The discussion and capabilities of this development explicitly include cases with zero separation distance between at least a subset (even a 1-dimensional edge or even a single point) of a planar array of light-providing light-sourcepixels and a subset (even a 1-dimensional edge or even a single point) of at least a planar array of light-sensing pixels.
[0404] As an illustration of Case C, FIGS. 27a and 27b depict a pair of illustrative 3-dimensional views of an example of a non-planar curved surface and a planar surface with a spatially-quantized light-field representation between the two surfaces. Note that conceptually that (1) the planar surface could comprise light-sensing elements that observe the non-planar curved surface which in this role comprises (reflective or emitting) light source elements, or (2) if the imaging surface can be rendered as the depicted illustrative non-planar curved surface, the non-planar curved surface could comprise light-sensing elements that observe the planar surface which in this role comprises (reflective or emitting) light source elements. It is noted that regions of the non-planar curved surface that are convex or concave with respect to the planar surface can be observationally occulted from some regions of the planar surface.
[0405] As another illustration of Case C, FIGS. 28a and 28b depict a pair of illustrative 3-dimensional views of a variation on FIGS. 27a and 27b featuring different example non-planar curved surface. Note here, too, that conceptually that (1) the planar surface could comprise light-sensing elements that observe the non-planar curved surface which in this role comprises (reflective or emitting) light source elements, or (2) if the imaging surface can be rendered as the depicted illustrative non-planar curved surface, the non-planar curved surface could comprise light-sensing elements that observe the planar surface which in this role comprises (reflective or emitting) light source elements. It is also noted that regions of the non-planar curved surface that are convex or concave with respect to the planar surface can be observationally occulted from some regions of the planar surface.
[0406] As a variation of Case C, FIG. 29 depicts an illustrative example cross-section of a non-planar (rigid or flexible) curved surface sensor and non-planar curved surface object with a spatially-quantized light-field representation between the two surfaces. Either or both of the curved surfaces can be configured to be a camera, and such arrangements can be fabricated by printing or other deposition fabrication processes. Depending on the spatial arrangement, some portions of some of one of the curved surfaces can be observationally occulted from some regions of the other curved surface. Note the roles of the plurality of planar arrays of light source elements and a parallel array of light-sensing elements can be interchanged.
Case D: Multiple Parallel Planes of Mixed Discrete Separation Distances:
[0407] Relaxing the constraints in Case A in still another way, either one or both of (a) the objects or situations producing contributions to the light-field and/or (b) the image sensor resides on more than one planar surfaces but with various separation distances, typically a mix of separation distances. A resulting light-field comprises mixed separation distances with abrupt changes in the separation distance.
[0408] As an illustration of Case D, FIGS. 30a through 30c depict a variety of illustrative 3-dimensional views of an example variation of the arrangement depicted in FIGS. 24, 25a, 25b, 26a, and 25b wherein the array of (emitted and/or reflected) light source elements are split among a plurality of smaller parallel planes at different separation distances from the planar array of light-sensing elements. Depending on the spatial arrangement, some portions of some of the smaller parallel planes can be observationally occulted from some regions of the planar surface. Note the roles of the plurality of planar arrays of light source elements and a parallel array of light-sensing elements can be interchanged.
Case E: Combinations of at Least One Parallel Planes and at Least One Non-Multiple Parallel Plane:
[0409] Generalizing in Case D further, one or more instances of the situations of Case A and Case B are combined, resulting in a more complex light-field. In many situations occultation of portions of light fields can occur, for example in cases where one non-transparent source array blocks at least a portion of the light emitted by another source array as viewed by (one or more of the) sensor array(s).
[0410] As an illustration of Case E, FIGS. 31a and 31b depict a variety of illustrative 3-dimensional views of an example variation of the arrangement depicted in FIGS. 24, 25a, 25b, 26a, and 25b wherein the array of (emitted and/or reflected) light source elements are distributed over a connected group of planes, at least one parallel to the planar array of light-sensing elements. Depending on the spatial arrangement, some portions of some of the non-parallel planes can be observationally occulted from some regions of the planar surface. Note the roles of the plurality of planar arrays of light source elements and a parallel array of light-sensing elements can be interchanged.
Case F: More Complex Combinations of Mixed Discrete and Continuous Spatially-Varying Separation Distances:
[0411] Generalizing in Case E further, one or more instances of the situations of at least one of Case A and Case B are combined with at least one instance of the situation of Case C, resulting in yet a more complex light-field. In many situations occultation of portions of light fields can occur, for example in cases where one non-transparent source array blocks at least a portion of the light emitted by another source array as viewed by (one or more of the) sensor array(s).
[0412] As an illustration of Case F, FIGS. 32a through 32c depict a variety of illustrative 3-dimensional views of an example wherein the array of (emitted and/or reflected) light source elements are distributed over a complex collection of connected and disconnected planes, some of which are parallel to the planar array of light-sensing elements, and some of which observationally occulted others from some regions of the planar surface by being situated directly in front of others. Note the roles of the plurality of planar arrays of light source elements and a parallel array of light-sensing elements can be interchanged.
[0413] FIGS. 33a and 33b depict example inward-directed or outward-directed sensor-pixel lattice locations distributed on a rigid or elastic curved convex-shaped surface. As considered elsewhere, leveraging the co-integration of light-emitting and light-sensing elements as taught in the inventor's 1999 patent family, the camera can be configured to provide self-illumination, for example when used as an elastically-fitted cap that serves as a zero-separation-distance contact-imaging camera monitoring the surface of an enclosed object. As explained elsewhere, since an arbitrary-shaped focus-surface can be computationally defined, should the elastic cap only make contact with some portions of an object enshrouded by such a (self-illuminating if needed) elastic cap, focused images of the entire encapsulated region of the non-occulted surface of the enshrouded object can be produced.
[0414] FIGS. 34a through 34d depicts examples of pairs of curved and sharply-angled surfaces, one of the pair inside the other of that pair. In any of the arrangements depicted, at least one of the inner surface and the outer surface can be a camera arranged to view the other surface. As considered elsewhere, the camera can be configured to provide self-illumination.
[0415] FIGS. 35a through 35c depict illustrative examples of bumpy and/or pitted sensor surfaces that can provide angular diversity. Such arrangements can also be used to provide sensor robustness via spatial diversity, to provide directed angle-orientation viewing, and to provide other types of functions.
[0416] FIG. 36 depicts example correspondences between a physical optical arrangement comprising an optical process (including for example vignetting optics and free-space separation should the image sensor not be in contact with the actual source image) and a mathematical model of that physical optical arrangement (transforming an actual image array data to measured image array data by a numerical model of the optical process.
[0417] FIG. 37 depicts an illustrative example of Mathematical Recovery of an approximate representation of the actual Image from Measured Image Array Data obtained by operating on the Measured Image Array Data by a Numerical Inverse of the Model of the Optical Process as depicted in FIG. 36.
[0418] FIG. 38, adapted from FIG. 2b of the present inventor's U.S. Pat. No. 8,830,375 and related cases, depicts an exemplary embodiment comprising a micro-optic structure, a light sensor array, an image formation signal processing operation and an optional additional subsequent image processing operations, herein the micro-optic structure and light sensor array are grouped into a first subsystem, and the image formation signal processing operation and subsequent image processing operations are grouped into a second subsystem. As discussed in the present inventor's U.S. Pat. No. 8,830,375 and related cases, various other arrangements are possible and provided for by aspects of the invention.
[0419] FIG. 39a depicts an example scheme wherein manufacturing, physical, optical, and mathematical considerations are used to create a reproducible manufacturing design such that is adequate manufacturing tolerances are obtained an analytical predictive model can be used to produce numerical models of the optical situations to be recovered without the use of empirical measurements.
[0420] FIG. 39b depicts an example variation on the scheme presented in FIG. 39a wherein post-manufacturing empirical measurements are used to further fine-calibrate the system performance of each particular manufactured article.
[0421] FIG. 40 depicts an example representation of example serialization processes and de-serialization processes for image recovery from an inverse or pseudo-inverse model as provided for by the invention.
[0422] FIG. 41 depicts a representation of example serialization processes transforming a measured image produced by a light-field travelling through an optical structure (here a vignette array) at being measured by a sensor array.
[0423] FIG. 42 depicts a representation of example empirical image-basis training sequence, or alternatively a collection of predictive-model-generated image-bases that directly populate a JKNM matrix providing a numerical model of the optical environment from which a future image will later be recovered as provided for by aspects of the invention.
[0424] FIG. 43a depicts a representation of example image recovery process using an inverse square-matrix representing an approximate inverse model as provided for by the invention.
Generalized Inverse/Pseudo-Inverse Remarks
[0425] There are a number of types of generalized inverses that have been developed and surveyed in the literature; for example see [B9] Section 3.3, [B10] pp. 110-111, and the tables in [B11] pp. 14-17. Some types of generalized inverses are uniquely-defined while others are non-uniquely defined in terms of infinite families. The notion of a generalized inverse applies to not only to finite-dimensional matrices but more broadly to (infinite-dimensional) linear operators; see for example [B12].
[0426] The Moore-Penrose generalized inverse, a special case of the Bjerhammar intrinsic inverses (see [B10] p.105 and [P1], is uniquely-defined ([B13] p.180), exists for any rectangular (or square) matrix regardless of matrix rank ([B13] p.179, [B14] p.196, [B15] p.19) and provides many properties found in matrix inverse ([B14], p.196) and beyond.
[0427] In particular, the Moore-Penrose generalized inverse inherently provides a unique solution providing a Least-Squares statistical fit in cases where solvable subsets of the larger number of equations give different inconsistent solutions; see for example [B15] pp. 17-19.
[0428] There are other types of generalized inverses that also provide least-squares properties; for example see entries annotated (3.1) and (3.2) in the table pp. 14 as well as sections 3.1-3.2 of [B11] as well as section 4.4.1 of [B13].
[0429] Further, the Moore-Penrose generalized inverse can be used to determine whether a solution to a set of linear equations exists ([B13] pp. 190-191).
[0430] Various extended definitions and generalized forms of the Moore-Penrose generalized inverse exist; see for example section 4.4.3 of [B13].
[0431] Some of the other types of generalized inverses are not useful for solving over-specified systems of linear equations (more equations than variables), for example the Drazin inverse which is restricted to square matrices and has more abstract applications; see for example [B13] Section 5.5.
[0432] FIG. 43b depicts a representation of example image recovery process using a generalized-inverse or pseudo-inverse matrix representing an approximate pseudo-inverse underspecified model as provided for by the invention. Underspecifed arrangements have been of interest in sub-Nyquist rate compressive sampling which can also be applied more broadly than optical sensing (see for example [P60]), and naturally fit into the classical regularized ill-posed inverse problem paradigm employed endemically in the coded aperture lensless imaging systems and approaches reviewed in review section A at the opening of the present patent application. The regularized ill-posed inverse problem paradigm gives rise to Moore-Penrose pseudo-inverse matrices or matrices similar to those (as presented in [P4] for example). Additionally, it is noted that the Moore-Penrose pseudo-inverse matrices and some of the other types of generalized inverse matrices can provide best-fit (Least square error) solutions to underspecified (fewer numbers of measurements than needed to uniquely solve for an image), fully-specified (exactly the number of measurements needed to uniquely solve for an image), and overspecified (fewer numbers of measurements than needed to uniquely solve for an image) situations and arrangements. However, the use of the Moore-Penrose pseudo-inverse matrices and other types of generalized inverses as taught in the inventor's 2008 patent family are directed to deliver additional advantages for pre-designed overspecified measurement situations and not the result of the regularized ill-posed inverse problem paradigm now widely used in coded aperture imaging and many other types of optical systems design and analysis see for example [B5], [B6]).
[0433] FIG. 43c depicts a representation of example image recovery process using a generalized-inverse or pseudo-inverse matrix representing an approximate generalized-inverse or pseudo-inverse overspecified model as provided for by the invention.
[0434] Use of a generalized-inverse or pseudo-inverse (and use of the Moore-Penrose pseudo-inverse in particular) in solving for a best-fit image from overspecified (and likely inconsistent) measurement data was introduced in a 2008 inventor's patent family. It is noted that slightly-related work in the area of improving digital Image resolution by use of oversampling can be found in the far earlier publication by Wiman [P3], but that is a different idea and goal. Rather, the inventor's use of use of a generalized-inverse or pseudo-inverse (and Moore-Penrose pseudo-inverse in particular) in solving for a best-fit image from overspecified (and likely inconsistent) measurement data provides robustness of image recovery with respect to damage or occultation of portions of the image sensor, etc.
[0435] FIG. 44a depicts a simple computational approach for image recovery as provided for by the invention, for example as taught in the inventor's 1999 and 2008 patent families. Such a depicts a simple computational approach compares favorably with far more numerically-complex spectral or transform computation (which numerically amounts to additional basis rotation transformation steps) as would be used in spectral or transform methods. For comparison, FIG. 44b depicts an example representation of a far more numerically-complex spectral or transform computation (which numerically amounts to additional basis rotation transformation steps) as would be used in spectral or transform methods.
[0436] The inventor's comprehensive lensless light-field imaging program (beginning with the inventor's 1999 patent family) includes a framework covering the approaches depicted in FIGS. 44a and 44b and various situations leading to these such as deconvolution methods. It is the inventor's view that the inventor's comprehensive lensless light-field imaging program (beginning with the inventor's 1999 patent family) includes a framework admitting most of visual-light coded aperture and angular-selective light-sensor approaches to various degrees.
[0437] FIG. 44c depicts a comparison of the approach depicted in FIG. 44a and the approach depicted in FIG. 44b, demonstrating comparative reasons to reject the far more numerically-complex spectral or transform computational approached represented in FIG. 44b.
[0438] FIG. 45 depicts the use of classical ill-posed inverse-problem regularization methods as employed in the Rambus [P4] and Rice University Flat Cam [P5] implementations as well as other coded aperture imaging implementations. These classical ill-posed inverse-problem regularization methods are widely used in many areas but also have an established role in optical systems design and analysis; see for example [B5] and [B6].
[0439] FIG. 46 depicts an abstract representation of an Identity structure within a (necessarily-sparse) Identity 4-tensor.
[0440] FIG. 47 depicts an abstract representation of a dilation around the Identity structure within a (necessarily-sparse) Identity 4-tensor.
[0441] FIG. 48 depicts a ((77)(77)) matrix-of-matrices representation of a (7777) Identity 4-tensor as abstracted in FIG. 46.
[0442] FIG. 49 depicts an example ((77)(77)) matrix-of-matrices representation of the dilation around the Identity structure of a (7777) 4-tensor as abstracted in FIG. 46.
[0443] FIG. 50 depicts an abstract representation of the trade-off between Space/Inverse-Space/Spatial Frequency Localization Methods and the sparcity of numerical model tensors, matrices, and their inverses.
[0444] FIGS. 51a through 51e depict how the sparcity of an example numerical model matrix serializing a numerical model 4-tensor degrades as the image source moves farther and farther from the image sensor (using numerical models at each distances predicted by an analytical geometric predictive model provided for by the invention).
[0445] FIG. 52 depicts an example ((77)(77)) matrix-of-matrices representation of the numerical model 4-tensor and example measured image (right) at zero separation distance from a single-illuminated-pixel source image (left). As described elsewhere, the source image need not be on a parallel plane and can be distributed arbitrarily in space.
[0446] FIG. 53 depicts an example ((77)(77)) matrix-of-matrices representation of the numerical model 4-tensor and example measured image (right) at a small non-zero separation distance from the same single-illuminated-pixel source image (left) as in FIG. 52.
[0447] FIG. 54 depicts an example ((77)(77)) matrix-of-matrices representation of the numerical model 4-tensor and example measured image (right) at zero separation distance from a two-illuminated-pixel source image (left). As described elsewhere, the source image need not be on a parallel plane and can be distributed arbitrarily in space.
[0448] FIG. 55 depicts an example ((77)(77)) matrix-of-matrices representation of the numerical model 4-tensor and example measured image (right) at a small non-zero separation distance from the same two-illuminated-pixel source image (left) as in FIG. 54.
[0449] FIG. 56 depicts an example ((77)(77)) matrix-of-matrices representation of the numerical model 4-tensor and example measured image (right) at zero separation distance from a more closely-spaced two-illuminated-pixel source image (left). As described elsewhere, the source image need not be on a parallel plane and can be distributed arbitrarily in space.
[0450] FIG. 57 depicts an example ((77)(77)) matrix-of-matrices representation of the numerical model 4-tensor and example measured image (right) at a small non-zero separation distance from the same more closely-spaced two-illuminated-pixel source image (left) as in FIG. 56.
[0451] FIGS. 58a through 58c depict three of forty-nine steps of an example empirical training sequence as provided for by the invention. In an implementation of the invention, such a procedure could be performed for a selected collection of one or more separation distances between the image sensor and the source image. As described elsewhere, the source image need not be on a parallel plane and can be distributed arbitrarily in the observable space.
Lensless Light-Field Imaging as an Associated Inverse Problem
[0452] The Inverse Model depiction illustrated in FIG. 1 can be configured to, in some appropriate manner, undo the effects of the incoming light's optical travel first within the Light-Field preceding the optical structure(s) and then through the Lensless Optical Structure(s) to where it reaches the Optical Sensor where the incident light converted to an electrical signal that can be further processed. If a mathematical model is a close match to the composite effects of these optical and optoelectrical processes, and if the model is mathematically invertible, applying the inverse of the model to the measured data can create a computationally-produced image which, for example, can be further arranged to be useful for human or machine use.
[0453] The Inverse Model can, for example, be implemented as a matrix, a 4-tensor, or other mathematical and/or data and/or logical operation.
[0454] The Inverse Model can be fixed or adjustable, can be implemented in a lumped or distributed manner, and can be unique or variationally-replicated in various manners. The optical structure can be fixed or reconfigurable, and can be arranged to be in a fixed position with respect to the Optical Sensor or can be configured to be movable in some manner with respect to the Optical Sensor. Additionally, at this level of abstraction, one or both of the Optical Sensor and Lensless Optical Structure(s) themselves can be variable in their electrical, physical, optical, mechanical, and other characteristics. For example, one or both of the Optical Sensor and Lensless Optical Structure(s) themselves can be any one or more of flat, curved, bendable, elastic, elastically-deformable, plastically-deformable, etc.
[0455] The Inverse Model can be derived from analytical optical models, empirical measurements, or combinations of these. In some embodiments the Inverse Model can be parametrized using interpolation.
[0456] Interpolation-based parameterization can be particularly useful if the Inverse Model is based on a collection of selected empirical measurements, or if the analytical optical model involves complex numerical computations.
[0457] FIG. 59 depicts an example polynomial fitting function interpolation method for interpolating the numerical model, its inverse/pseudo-inverse, and/or other functions for separation distances values lying between k empirically-trained separation distances. As described elsewhere, the source image need not be on a parallel plane and can be distributed arbitrarily in the observable space. In general the polynomial fitting function can be expected to include terms with negative exponential powers expected due to the overall 1/r.sup.2 enveloping attention as the serration distance r increases.
[0458] FIG. 60 depicts an example polynomial fitting function interpolation method for interpolating the numerical model, its inverse/pseudo-inverse, and/or other functions for separation distances values lying between k separation distances used by a predictive-model. As described elsewhere, the source image need not be on a parallel plane and can be distributed arbitrarily in the observable space. Hereto in general the polynomial fitting function can be expected to include terms with negative exponential powers expected due to the overall 1/r.sup.2 enveloping attention as the serration distance r increases.
[0459] FIG. 61 depicts an example of very poor curve-fit interpolation of the matrix elements of the numerical model between measure data distances resulting from not having sufficient terms in a model polynomial expansion and/or inclusion of terms with negative exponential powers expected due to the overall 1/r.sup.2 enveloping attention as the serration distance r increases.
[0460] FIG. 62 depicts an example piecewise-linear interpolation of the matrix elements of the numerical model between k empirically-trained separation distances. (Measurement data and piecewise-linear plot made by Michael Hring.)
[0461] Using an empirically-trained numerical model for representing the linear transformation invoked by the optical arrangement, it is clearly possible to train the system to focus on an arbitrarily-shaped surface, including one that is curved, bend, or irregularly-shaped; the inversion math does not care as long as the resulting numerical model matrix is non-singular, and the recovered image will be obtained in the same manner as if the focus-surface was a parallel plane. Accordingly, in principle a predictive analytical model can be used to generate the numerical model matrix, and by either means (empirically-trained or predictively-modeled) the system and methods can be arranged to focus on an arbitrarily-shaped surface, including one that is curved, bend, or irregularly-shaped.
[0462] FIG. 63a depicts how with very small separation distances and certain non-alignment of source-pixel locations and sensor pixel locations some sensor pixel locations cannot receive light from proximate source-pixel locations, while FIG. 63b depicts how at slightly greater separation distances this condition does not occur; such processes can give rise to the rising and falling of selected curves in the example empirical training data shown in the example of FIG. 61. (This analysis performed by Michael Hiking.)
[0463] FIG. 64 depicts an example piecewise-linear interpolation method for interpolating the numerical model, its inverse/pseudo-inverse, and/or other functions for separation distances values lying between k empirically-trained separation distances. As described elsewhere, the source image need not be on a parallel plane and can be distributed arbitrarily in the observable space.
[0464] FIG. 65 depicts an example piecewise-linear interpolation method for interpolating the numerical model, its inverse/pseudo-inverse, and/or other functions for separation distances values lying between k separation distances used by a predictive-model. As described elsewhere, the source image need not be on a parallel plane and can be distributed arbitrarily in the observable space.
[0465] FIG. 66 depicts an abstract representation of the mathematical recovery of image from measured image array data.
[0466] FIG. 67 depicts a variation on the representation of the mathematical recovery of image from measured image array data shown in FIG. 66 wherein more measurements are made than needed, resulting in an over-specified collection of measurements that can be expected to lead to inconsistent calculation outcomes when different subgroups of measurements are used to solve for the recovered image. As provided for by the invention, a Moore-Penrose pseudo-inverse operation gives a least-squares fit to the expected inconsistent outcomes; this is depicted in the bottom portion of the figure.
[0467] FIG. 68a through 68d depict example numerical model outcomes responsive to a single-illuminated pixel as generated for various separation distances by an example predictive analytical geometric model. The particular example predictive analytical geometric model used only accounts for vignette occultation as calculated by simple ray-trancing and does not include the effects of vignette-internal reflections, vignette-internal scattering, vignette-aperture diffraction, surface plasmoid propagation, etc.
[0468] FIG. 69a depicts three example interactive quantization-effect modeling outcomes wherein controllable quantization is artificially imposed on measurement data so as to predictively model and characterize the effects of quantizing nonlinearities imposed by electronic digital-to-analog converter processes on empirical training, predictive-model generated, and real-time measurements as they influence the quality of image recovery.
[0469] FIG. 69b depicts three example interactive offset-effect modeling outcomes wherein controllable measurement offset is artificially imposed on measurement data so as to predictively model and characterize the effects of measurement offsets imposed by electronic digital-to-analog converter processes on empirical training, predictive-model generated, and real-time measurements as they influence the quality of image recovery.
[0470] There are many noise processes inherent to light sensing and associated electronics and various resulting performance limitations and tradeoffs; see for example [P44], [B2]. A very general performance perspective is provided in the book by Janesick [B2]. In the limit, highest performance will be obtained by single-electron sensors and amplifies; as to steps towards array sensors of this type see the paper by Richardson [P33]. The invention provides for inclusion of these considerations. FIG. 69c depicts an example interactive noise-modeling control used to introduce synthetically-generated noise and noise processes so as to predictively model and characterize its effects. The selection shown controls additive Gaussian noise, but other noise processes associated with photodiodes (1/f noise, dark-current shot (Poissonian) noise, photon shot (Poissonian) noise, Johnson and other circuit noise, dark-current thermal noise, spectral noise, detector amplifier noise, etc.) can similarly be introduced.
[0471] FIGS. 70a through 70d depict example interactive modeling outcomes showing the effect of noise, offset, and both these for a 32-step (5-bit) quantized (DN [B2]) measurement dynamic range.
[0472] FIGS. 71a through 71d depict example interactive modeling outcomes showing the effect of noise, offset, and both these for a 64-step (6-bit) quantized (DN) measurement dynamic range.
[0473] FIGS. 72a through 72d depict example interactive modeling outcomes showing the effect of noise, offset, and both these for a 128-step (7-bit) quantized (DN) measurement dynamic range.
[0474] FIGS. 73a through 73d depict example interactive modeling outcomes showing the effect of noise, offset, and both these for a 140-step (slightly more than 7-bit) quantized (DN) measurement dynamic range.
[0475] FIGS. 74a through 74d depict example interactive modeling outcomes showing the effect of noise, offset, and both these for a 150-step (yet more than 7-bit) quantized (DN) measurement dynamic range.
[0476] FIG. 75a, adapted from [B7] Figure C2.5.16(a), depicts a first example pixel-cell multiplexed-addressing circuit for an individual OLED within an OLED array that includes a dedicated light-coupled monitoring photodiode for use in regulating the light output of the individual OLED so as to prevent user-observed fading or other brightness-variation processes. The adjacent photodiodes used for pixel-by-pixel closed-loop feedback of OLED brightness.
[0477] FIG. 75b, adapted from [B7] Figure C2.5.16(b), depicts a second example pixel-cell multiplexed-addressing circuit for an individual OLED within an OLED array that includes a dedicated light-coupled monitoring photodiode for use in regulating the light output of the individual OLED so as to prevent user-observed fading or other brightness-variation processes. There are many other subsequent developments since the publishing of this books tentatively-toned remarks; for example recently-announced OLED phones are said to be using this technique.
[0478] FIG. 76a, adapted from [B18], depicts an example pixel-cell multiplexed-addressing circuit for an individual OLED within an OLED array with a monitoring feature.
[0479] FIG. 76b, also adapted from [B18], depicts an example pixel-cell multiplexed-addressing circuit for an isolated high-performance photodiode or phototransistor within a high-performance photodiode or phototransistor array with a forced-measurement provision.
Additional Functional Architectures
[0480] As descried earlier, FIG. 1 depicts an example conceptual view of the underlying principles of the invention, facilitating a wide range of implementation methods and architectures. In this depiction, an Optical Scene creates a Light-Field that is directed to an Optical Sensor which is preceded by one or more Lensless Optical Structure(s) that in some manner alters the light field in a predictable spatial manner.
[0481] The Optical Sensor produces (typically time-varying) electrical signals and/or computational data responsive (instantly and/or within some time-delay) to light incident to the surface or other substructure(s) within the Optical Sensor at any given moment. The depicted Inverse Model can be configured to, in some appropriate manner, undo the effects of the incoming light's optical travel first within the Light-Field preceding the optical structure(s) and then through the Lensless Optical Structure(s) to where it reaches the Optical Sensor, resulting in a computationally-produced image which, for example, can be arranged to be useful for human or machine use. The Inverse Model can, for example, be implemented as a matrix, a 4-tensor, or other mathematical and/or data and/or logical operation. The Inverse Model can be fixed or adjustable, can be implemented in a lumped or distributed manner, and can be unique or variationally-replicated in various manners. The optical structure can be fixed or reconfigurable, and can be arranged to be in a fixed position with respect to the Optical Sensor or can be configured to be movable in some manner with respect to the Optical Sensor. Additionally, at this level of abstraction, one or both of the Optical Sensor and Lensless Optical Structure(s) themselves can be variable in their electrical, physical, optical, mechanical, and other characteristics. For example, one or both of the Optical Sensor and Lensless Optical Structure(s) themselves can be any one or more of flat, curved, bendable, elastic, elastically-deformable, plastically-deformable, etc.
[0482] FIG. 77 depicts a simplified view of FIG. 1 showing only the signal and computational portion of FIG. 1 as will be useful in a subsequent discussion. It is understood that in FIG. 77 and related figures that the Optical Scene, Light-Field, and Lensless Optical Structure(s) can have the arrangements with respect to the Optical Sensor like that, similar to, extensible from, or in appropriate manners alternative to that depicted in FIG. 1.
[0483] FIG. 78 depicts a variation of the arrangement represented in FIG. 77 wherein the Inverse Model is rendered as a parameterized Inverse Model which can be altered responsive to one or more provided parameters. For example, the parameters provided to the parameterized Inverse Model could control a surrogate viewpoint, specify one or more focus planes (or more generally focus surfaces), etc. rating to what the Inverse Model undoes. Accordingly, this arrangement allows a plurality of imaging capabilities and functions to be selectably and/or adjustably be supported by the more general arrangement of FIG. 77.
[0484] FIG. 79 depicts a variation of the arrangement represented in FIG. 78 wherein the electrical signals and/or computational data produced by the Optical Sensor are in parallel provided in whole or selected (or selectable) part to a plurality of Inverse Models, each producing one or more computationally-produced images responsive to the sensor data. Although the Inverse Models shown in FIG. 79 are depicted as parameterized Inverse Models, the invention provides for some or all of the plurality of Inverse Models to be non-parameterized Inverse Models.
[0485] FIG. 80 depicts a variation of the arrangement represented in FIG. 78 wherein the electrical signals and/or computational data produced by the Optical Sensor is handled by a computer or computational element (such as a microprocessor, GPU, DSP chip, ALU, FPLA, combination of two or more these, pluralities of these, etc.) in some fashion that at least permits the electrical signals and/or computational data produced by the Optical Sensor to be stored as a file.
[0486] FIG. 81 depicts a variation of the arrangement represented in FIG. 78 wherein the aforementioned handling by a computer or computational element is controlled in some manner by a control parameter. The control parameter for example can specify an aspect of the name of the Stored File, specify an aspect of the format of the Stored File, specify a selection of specific portions of the electrical signals and/or computational data produced by the Optical Sensor, specify or control data operations on the electrical signals and/or computational data produced by the Optical Sensor, specify or mathematical operations on the electrical signals and/or computational data produced by the Optical Sensor, specify or control logical operations on the electrical signals and/or computational data produced by the Optical Sensor, etc.
[0487] FIG. 82a depicts a variation of the arrangement represented in FIG. 81 wherein a plurality of stored files is created, for example with different parameter values associated with each stored file. Although a separate computer function is depicted for each of the stored file, the invention provides for these computer functions to be implemented with or executed on any of a single computer or computational element (such as a microprocessor, GPU, DSP chip, ALU, FPLA, combination of two or more these, pluralities of these, etc.), a plurality of computers or computational elements, an individually-dedicated computer or computational element, etc. The parameter values used in creating each Stored File can be either externally associated with each stored file, or can be stored as part of the stored file in a direct or encoded form.
[0488] FIG. 82b depicts an example arrangement wherein a stored file is used by a fixed Inverse Model to create a computationally-produced image.
[0489] FIG. 82c depicts an example arrangement wherein a stored file is used by a parameterized Inverse Model to create a computationally-produced image, and further wherein parameter value(s) associated the parameterized Inverse Model are externally associated with each stored file.
[0490] FIG. 82d depicts an example arrangement wherein a stored file is used by a parameterized Inverse Model to create a computationally-produced image, and further wherein parameter value(s) associated the parameterized Inverse Model are derived from or obtained from the stored file.
[0491] FIG. 83 depicts four example conformations of a particular bendable, flexible, and/or or pliable optical-sensor/optical-structure sheet, each conformation giving rise to an associated model for how a light field is sensed in the context of reconstructing an image, and each model giving rise to its own associated inverse model. It is noted that, as dependent on the properties and limitations of the optical-structure, there can be small blind spots in regions of sufficiently-high curvature.
[0492] FIG. 84a through FIG. 84d depict how various conformations can be used to render a computationally derived image
[0493] FIG. 85a depicts an example arrangement for an optical-sensor/optical-structure of fixed conformation.
[0494] FIG. 85b depicts an example arrangement for an optical-sensor/optical-structure of variable conformation, for example should the surface bend, deform, hinge, expand, contract, etc.
[0495] FIG. 85c depicts a variation of FIG. 85b wherein the model and the Inverse Model are parameterized.
[0496] FIG. 86a depicts an arrangement useful for using optical training to sense the present conformation of an optical-sensor/optical-structure of variable conformation, for example should the surface bend, deform, hinge, expand, contract, etc.
[0497] FIG. 86b depicts an arrangement useful for using internal sensing means to sense the present conformation of an optical-sensor/optical-structure of variable conformation, for example should the surface bend, deform, hinge, expand, contract, etc. It is noted that a prototype for a commercial (panoramic camera) product employing a flexible OLED display, (c) conformation deformation sensing has been presented and is described in [P36].
[0498] FIG. 86c depicts an arrangement useful for using information about a movable or changing support structure or contact arrangement to identify the present conformation of an optical-sensor/optical-structure of variable conformation, for example should the surface bend, deform, hinge, expand, contract, etc.
[0499] FIG. 86d depicts an arrangement useful for using external observation mean, for example such as one or more observing video camera(s) means to sense the present conformation of an optical-sensor/optical-structure of variable conformation, for example should the surface bend, deform, hinge, expand, contract, etc.
[0500] FIG. 87a depicts an optical-sensor/optical-structure of variable conformation, for example should the surface bend, deform, hinge, expand, contract, etc. producing an output signal and/or output data.
[0501] FIG. 87b depicts a controllable variable-conformation material whose shape/conformation can be controlled by externally-provide control stimulus.
[0502] FIG. 87c depicts an optical-sensor/optical-structure of variable conformation, for example should the surface bend, deform, hinge, expand, contract, etc. producing an output signal and/or output data fabricated on, with, or co-integrated with controllable variable-conformation material whose shape/conformation can be controlled by externally-provide control stimulus.
[0503] FIG. 88a depicts use of conformational sensing information to derive or compute parameter values for a parameterized Inverse Model.
[0504] FIG. 88b depicts use of conformational control parameter information to derive or compute parameter values for a parameterized Inverse Model.
[0505] FIG. 88c depicts use of both conformational sensing information and conformational control parameter information to derive or compute parameter values for a parameterized Inverse Model.
Imaging Algorithms
[0506] The development to follow is broad enough to cover a wide variety of sensor types and imaging frameworks, and can be expanded further. Although the development to follow readily supports the advanced features made possible by curved, flexible/bendable, transparent, light-emitting, and other types of advanced sensors taught in the present patent application and earlier inventor patent families, many of the techniques can be readily applied to appreciate optical arrangements, devices, and situations employing traditional image sensors such as CMOS, CCD, vidicon, etc. Accordingly, the present invention provides for the use of a wide range of image sensor types including CMOS, CCD, vidicon, flat, curved, flexible/bendable, transparent, light-emitting, and other types of advanced sensors taught in the present patent application and earlier inventor patent families, as well as other know and future types of image sensors.
[0507] Traditional Notation Conventions for Vectors, Matrices, and Matrix-Vector (Left) Multiplication
[0508] Let x be an M-dimensional column vector (i.e., an array of dimension 1M) comprising elements {x.sub.m} 1mM
[00001]
and y be a J-dimensional column vector (i.e., an array of dimension 1M) comprising elements {y.sub.i}1jJ
[00002]
[0509] Let A be a JM matrix (an array of dimension JM) comprising elements {a.sub.jm},1jJ, 1mM:
[00003]
[0510] The matrix product of a JM matrix A with an M-dimensional column vector x can produce a J-dimensional column vector y; by left multiplication convention this is denoted as
y=Ax
where each element y.sub.i of resulting a J-dimensional column vector y is calculated as
[00004]
for each j an integer such that 1jJ. Then the entire a J-dimensional column vector y is given by
[00005]
Use to Represent Spatial Line-Array (1-Dimensional) Imaging (as Used in Fax and Spectrometry Sensors)
[0511] In the above, one is using the matrix A as a transformational mapping from column vector x to column vector y
[00006]
analogous 4-Tensor Representation of Spatial Grid-Array (2-dimensional) Imaging.
[0512] For example, if column vector x represents a line-array of data (such as light source values directed to a line-array light-measurement sensor used in a fax scanner or optical spectrometer), the matrix A can represent a composite chain of linear optical processes, electro-optical processes, and interface electronics (transconductance, transimpedance, amplification, etc.) processes that result in measured data represented by a column vector y. Here the indices of the vectors and matrix signify unique well-defined discrete (step-wise) spatial positions in an underlying 1-dimensional spatial structure.
[0513] A visual image as humans experience it through vision and conventional photography, as well as other analogous phenomena, has an underlying 2-dimensional spatial structure. In digital imaging, unique well-defined discrete (step-wise) spatial positions within an underlying 2-dimensional spatial structure are identified and/or employedin the context of this discussion these may be called pixels.
[0514] Mathematically, a 2-dimensional array of mathematically-valued elements, such as a matrix, can provide a mathematical representation of an image wherein the indices of an element identify an individual pixel's spatial location and the value of that mathematical element represents the brightness of that pixel. For example, an JK array of measured 2-dimensional image data arranged as row and columns can be represented as a matrix of brightness values:
[00007]
[0515] A convenient shorthand for this can be denoted as
Q={q.sub.jk}
where it is understood each of the two indices span a range of consecutive non-zero integer values
1jJ, 1kK.
[0516] Similarly, a source image (2-dimensional array) can be represented as a matrix:
[00008]
[0517] A similar convenient shorthand for this is denoted
S={s.sub.mn}
where it is understood each of the two indices span a range of consecutive non-zero integer values
1mM, 1nN.
[0518] A source image S can be transformed by linear optical processes, linear sensor processes, and linear electronics processes into a measured image Q. This can be represented mathematically as a linear matrix-to-matrix transformation ![custom-character]()
mapping the matrix S to the matrix Q
[00009]
akin to employing the earlier matrix A as a linear vector-to-vector transformational (for example, mapping column vector x to column vector y):
[00010]
[0519] Most generally this linear transformation ![custom-character]()
can be represented by a 4-dimensional array 4-tensor:
![custom-character]()
={t.sub.jkmn}
1jJ
1kK
1mM
1nN
with the understanding that the following multiplication rule is represented by the tensor ![custom-character]()
multiplying the matrix S, namely each element q.sub.jk of resulting matrix Q is given by
[00011]
[0520] Note this convention corresponds in form to the matrix case presented earlier:
[00012]
[0521] The corresponding multiplicative product of the 4-tensor ![custom-character]()
with matrix S to give matrix Q can be represented as
Q=![custom-character]()
S
which compares analogously to the multiplicative product of the matrix A with the vector x to give the vector y
y=Ax
[0522] With its four tensor-element indices and tensor-matrix product organized in this way, the 4-tensor ![custom-character]()
with elements t.sub.jkmn can be readily and conveniently represented as a matrix-of-matrices where interior matrix blocks are indexed by row j and column k and the elements with each of the interior matrices are indexed by row m and column n. More specifically, the 4-tensor element t.sub.jkmn resides in an inner matrix residing in row m and column n of an inner matrix that resides in row j and column k of the outer matrix. The resulting matrix-of-a-matrices representation for the 4-tensor ![custom-character]()
={t.sub.jkmn} with
1jJ, 1kK, 1mM, 1nN
would be:
[00013]
[0523] This matrix-of-a-matrices structure where the mapping
[00014]
is defined by
[00015]
provides several important opportune outcomes, among these being:
[0524] Property: [0525] The individual entries t.sub.jkmn of each interior matrix having block-position index {j, k} scale the contribution of each element s.sub.mn which sum together to the quantity q.sub.jk comprised within the matrix Q; this can be seen directly by just considering fixed values for {j, k} in the defining relation
[00016]
[0526] Utility: [0527] this is also the way MatrixForm[*] displays the 4-array ![custom-character]()
in Mathematica
[0528] Remark 1: It is noted that the 4-tensor ![custom-character]()
as defined thus far is represented in this matrix of matrices structure, the interior matrices are organized so that each interior matrix having block-position index {j,k} is associated with the corresponding outcome quantity q.sub.jk. An attractive property of this representation, as called out above, is that the value of the output quantity q.sub.jk is the sum of all the pointwise products of values of source image pixels s.sub.mn scaled by the corresponding elements in the interior matrix that has block-position index {j, k}. Thus, in an optical imaging context, the values of elements in the interior matrix that has block-position index {j, k} graphically show the multiplicative gain (or sensitivity) attributed to each of the same-positioned source image pixels s.sub.mn in the image source matrix S. In more pedestrian but intuitively useful terms, the values of elements in the interior matrix that has block-position index {j,k} display the heat map of responsiveness of an observed or measured sensor pixel q.sub.jk in the observed or measured image matrix Q to source image pixels s.sub.mn in the image source matrix S.
[0529] Remark 2: From this it is further noted that other kinds of 4-tensors could be reorganized in other ways that have other attractive merits. For example, a 4-tensor comprising elements .sub.mnjk can be defined by the simple index reordering
.sub.mnjk=t.sub.jkmn;
each interior matrix in the matrix of matrices structure for the 4-tensor having block-position index {m, n} represents a discrete point-spread function for a source pixel at position {m, n} into individual outcome pixels at position {j, k} as can be seen from the resulting relation
[00017]
[0530] Although point-spread function representation imposed by the matrix of matrices structure for 4-tensor has obvious customary attraction, the discourse will continue in terms of the 4-tensor ![custom-character]()
comprising elements t.sub.jkmn as defined by
[00018]
because of its organizational similarity with the conventional matrix definition
[00019]
and with the understanding that all the subsequent development can be transformed from the definitions used for 4-tensor ![custom-character]()
to the point-spread oriented for 4-tensor by the index re-mapping
.sub.mnjk=t.sub.jkmn.
[0531] Remark 3: It is noted that for a (variables-separable) separable two-dimensional transform, such as the two-dimensional DFT, DCT, DST, etc., commonly used in traditional spectral image processing affairs of the j and m indices are handled entirely separate from affairs of the k and n indices, so q.sub.ik takes the restricted variables-separable form, for example when J=M and K=N
[00020]
in which case
t.sub.jkmn=d.sub.jmd.sub.kn
[0532] For example, for the normalized DFT matrices operating on an image of M rows and N columns, these d.sub.jm and d.sub.kn element are:
[00021]
where i={square root over (1)}.
[0533] As another example of other kinds of 4-tensors be reorganized in other ways with attractive merits, a variables-separable 4-tensor comprising elements .sub.mnjk can be defined by the simple index reordering
.sub.jmkn=t.sub.jkmn
which separately associates rows (indexed by j) in matrix Q with rows (indexed by m) in matrix S and separately associates columns (indexed by k) in matrix Q with columns (indexed by n) in matrix S.
[0534] Remark 4: To further illustrate details and develop intuition of the matrix of matrices structure, or at least the aforedescribed organization of indices and multiplication rule, one could employ the dimension signifier (JJ)(MN). As some examples:
[00022]
Examples of Transpose Operations for 4-Tensors
[0535] Various types of Transpose operations involving self-inverting index-exchange operations for one or two pairs of indices can be defined from (4.Math.3.Math.2.Math.1)1=23 index re-organizations overall.
[0536] Perhaps some of the most useful of these would include: [0537] 2134-Transpose: Exchanging rows and columns within the outer matrix structure (i.e., exchanging order of first two indices) .sub.kjmn=t.sub.jkmn; t.sub.kjmn=.sub.jkmn; [0538] 1243-Transpose: Exchanging rows and columns within the inner matrix structure (i.e., exchanging order of last two indices) .sub.jknm=t.sub.jkmn; t.sub.jknm=.sub.jkmn; [0539] 2143-Transpose: Exchanging rows and columns within the outer matrix structure (exchanging order of first two indices) and exchanging rows and columns within the inner matrix structure (exchanging order of last two indices) together .sub.kjnm=t.sub.jkmn; t.sub.kjnm=.sub.jkmn; [0540] 3412-Transpose: Exchanging row-column pair of outer matrix structure (first two indices) with the row-column pair of inner matrix structure (last two indices) .sub.mnjk=t.sub.mnjk; t.sub.mnjk=.sub.jkmn; [0541] 1324-Transpose: Grouping the row indices of both the inner and outer matrix structures (first and third indices) followed by grouping the column indices of both the inner and outer matrix structures (second and fourth indices) .sub.jmkn=t.sub.jkmn; t.sub.jmkn=.sub.jkmn.
[0542] Incidentally it is noted, for example that:
[00023] [0543] (matrix analogy to point spread function re-organization from above)
[00024] [0544] (point spread function to matrix analogy re-organization)
[00025] [0545] (matrix analogy to variables-separable re-organization from above)
[00026] [0546] (variables-separable to matrix analogy re-organization)
The Identity 4-Tensor
[0547] As with an NN Identity matrix employing the mapping
[00027]
to map an N-dimensional vector to a copy of itself, using
a.sub.mn=.sub.mn
where .sub.Pq is the Kronecker delta
[00028]
an Identity 4-tensor (for example J=M and K=N) mapping a MN matrix to an MN copy of itself results from employing the mapping:
[00029]
[0548] Note that with this (variables-separable) structure gives q.sub.jk=s.sub.jk for each 1jM, 1kN.
[0549] Using the matrix-of-matrices representation, a (33)(33) Identity 4-tensor ![custom-character]()
.sub.(33)(33) would have the form:
[00030]
[0550] Such a (33)(33) Identity 4-tensor would map a 33 pixel source image S to a 33 pixel result image Q with Q=S.
[0551] More generally Identity 4-Tensors map an MN matrix to an MN matrix, but the matrices need not individually (row-column) symmetricthat is one does not require M=N.
[0552] For example, using the matrix-of-matrices representation, a (32)(32) Identify 4-Tensor ![custom-character]()
.sub.(32)(32) that maps a 32 matrix to a 32 matrix would have the form:
[00031]
[0553] Such a (32)(32) Identity 4-tensor would map a 32 pixel source image S to a 32 pixel result image Q with Q=S.
[0554] For each of these Identity 4-tensor examples, regarding Remark 1 above (as to interpreting the values of elements in the interior matrix with block-position index {j, k} in an (mN)(MN) matrix of matrices as representing a heat map of responsiveness of an observed or measured sensor pixel q.sub.jk in the observed or measured image matrix Q to source image pixels s.sub.mn in the image source matrix S), the structure of an MNMN Identity 4-tensor is crystal clear as to it rendering q.sub.jk=s.sub.jk for each 1jM, 1kN.
Re-Indexing and Reorganization a 4-Tensor-Operator Matrix-to-Matrix (Image-to-Image) Equation as a Matrix-Operator Vector-to-Vector Equation
[0555] Although in an image the row and column ordering, two-dimensional neighboring arrangement of pixels, and other such two-dimensional indexing details are essential, some linear transformations act entirely independently of the two-dimensional index structure. An example, are situations where one can regard the relationships defined by a tensor mapping between matrices such as
Q=![custom-character]()
S
as simply representing a set of simultaneous equations
[00032]
[0556] In such circumstances one could without consequence uniquely re-index the variables with an indexing scheme that serializes the index sequence in an invertible way. For example, one can define two serializing indices p and q to serialize a JKMN dimensional 4-tensor ![custom-character]()
comprising elements t.sub.jkmn into a JKMN dimensional matrix T comprising elements t.sub.pq using the index-mapping relations
p=(j1)K+k
r=(m1)N+n
those relations can be inverted via
j=Mod(p1,K)+1
k=Floor(p.sup.1/K)+1=Ceiling[p/K]
m=Mod(r1,N)+1
n=Floor(r.sup.1/N)+1=Ceiling[r/N]
[0557] Using these, one can define the serialized-index vectors q, comprising elements
q.sub.p 1pJK,
and s, comprising elements
s.sub.r 1rMN,
which are simply scanned or flattened versions of matrix Q, comprising elements
q.sub.jk 1jJ, kK
and matrix S, comprising elements
s.sub.mn 1mM, 1nN
[0558] An example scanning or flattening index correspondence is
q.sub.(j1)K+k.Math.q.sub.jk 1jJ, 1kK
s.sub.(m1)N+n.Math.s.sub.mn 1mM, 1nN
and its corresponding inverse correspondence is
q.sub.p.Math.q.sub.Mod(p1,N)+1,Ceiling(p/K), 1pJK
s.sub.r.Math.s.sub.Mod(r1,N)+1,Ceiling(r/N), 1rMN.
[0559] The last pair of these index correspondences can be used to formally define index-serializing mappings
q.sub.p=q.sub.Mod(p1,N)+1,Ceiling(p/K), 1pJK
s.sub.r=s.sub.Mod(r1,N)+1,Ceiling(r/N), 1rMN
that provide a flattening reorganization of the elements q.sub.jk comprised by the JK-dimensional matrix Q into a vector q comprising elements q.sub.p, and a flattening reorganization of the elements s.sub.mn comprised the MN-dimensional matrix S into a vector s comprising elements s.sub.r. These result in flattening transformations Q.fwdarw.q and .fwdarw.s.
[0560] The first pair of the index correspondences can be used to formally define index-vectorizing mappings
q.sub.jk=q.sub.(j1)K+k 1jJ, 1kK
s.sub.mn=s.sub.(m1)N+n 1mM, 1nN
that provide a partitioning reorganization of the elements q.sub.p of vector q into the elements q.sub.jk comprised by the JK-dimensional matrix Q, and a partitioning reorganization of the elements s.sub.r of vector q into the elements s.sub.mn comprised the MN-dimensional matrix S. These result in partitioning transformations q.fwdarw.Q and s.fwdarw.S which reconstruct the matrices Q and S from the serialized vectors q and s.
[0561] In a corresponding way, one can use these same serialized-indices to correspondingly re-label and reorganize the values of the (JK)(MN)-dimensional tensor ![custom-character]()
to the JKMN-dimensional matrix T. The mapping ![custom-character]()
.fwdarw.T is given by
t.sub.(j1)K+k,(m1)N+n=t.sub.jkmn
1jJ, 1kK, 1mM, 1nN
and the reverse mapping T.fwdarw.![custom-character]()
is given by
t.sub.Mod(p1,K)+1,Ceiling(p/K),Mod(r1,K)+1,Ceiling(r/N)=t.sub.pr
1pJK, 1rMN
[0562] Thus, because of the transformational equivalence between
[00033]
for the same (but re-indexed) variables, this allows one to exactly represent the matrix-tensor equation
Q=![custom-character]()
S
as an equivalent vector-matrix equation
q=Ts
[0563] More generally, the index serialization functions can be arbitrary as long as they are one-to-one and onto over the full range and domain of the respective indices, and invertably map pairs of integers to single integers. For example they could be organized as a scan in other ways, or even follow fixed randomly-assigned mapping. In general one can write:
[00034]
or more compactly
[00035]
or more abstractly
[00036]
[0564] This is extremely valuable as it allows for matrix methods to solve inverse problems or implement transformations on images in terms of matrices. Of course matrix methods have been used in variables-separable image processing for decades employing various ad hoc constructions. Those ad hoc constructions could be formalized with the aforedescribed 4-tensor representation should one be interested in the exercises, but more importantly the computation of the aforedescribed 4-tensor representation and the formal isomorphic equivalence between 4-tensor linear transformations mapping matrices (representing images) to matrices (representing images) and matrix transformations mapping vectors to vectors allows clarity and methodology to complicated non-variables-separable linear imaging transformations, inverses, pseudo-inverses, etc. Also importantly the aforedescribed 4-tensor representation readily extends to mappings among tensors as may be useful in color, multiple-wavelength, tomographic, spatial-data, and many other settings and applications.
[0565] Additionally, as an aside: the aforedescribed 4-tensor representation naturally defines eigenvalue/eigenmatrix and eigenvalue/eigentensor problems; for example the eigenvalue/eigenmatrix problem
![custom-character]()
Z.sub.i=.sub.iZ.sub.i 1iJK
for ![custom-character]()
a JKJK 4-tensor, the collection of indexed scalars {.sub.i} 1iJK the scalar eigenvalues, and the collection of indexed matrices {Z.sub.i} 1iJK the eigenmatrices is equivalent to the eigenvalue/eigenvector problem
[00037]
for calculation and analysis and transformed back via
[00038]
[0566] These general process can be order-extended and further generalized to similarly transform eigenvalue/eigentensor problems into equivalent eigenvalue/eigenvector problems, and extended further in various ways to replace the eigenvalue scalars with an eigenvalue array.
[0567] As and additional aside, these same and similar approaches employing
[00039]
and other combined or more generalized reorganization methods
[00040]
can be order-extended and further generalized to similarly transform the vast understanding, rules, bases, transformations, vector spaces, spaces of matrices, properties of matrices, and matrix-vector equations into a wide range of tensor understandings, tensor rules, tensor bases, tensor transformations, and properties of tensors, spaces of tensors, and tensor-matrix and tensor-tensor equations.
[0568] Attention is next directed to inversion and then to image formation, and then after first developing and using extensions of the aforedescribed 4-tensor representation to mappings among tensors) expanding these to color/multiple-wavelength imaging applications.
Inverse of a 4-Tensor
[0569] Accordingly, for
Q=![custom-character]()
S with M=J,N=K,
if all the represented individual equations are linearly independent and of full rank, then the matrix T defined by
[00041]
is invertible and the pixel values of the source image S can be obtained from the pixel values of the measurement Q by simply inverting the matrix T:
s=T.sup.1q
where the corresponding flattening and partitioning index transformations are employed among the matrices and vectors
[00042]
[0570] Further, the pixel values of the source image S can be obtained from the pixel values of the measurement Q by simply inverting the matrix T to obtain T.sup.1, multiplying the flattened measurement data q with T.sup.1 to obtain the vector s, and partitioning the result into the source (image) matrix S:
[00043]
[0571] It is noted that effectively the column vectors of the matrix T serve as the natural linearly-independent spanning basis of the composite sensor and optical arrangement corresponding to a particular positioning situation. The natural linearly-independent spanning basis is not necessarily orthogonal, although it can of course be orthogonalized if useful using Gram-Schmitt of other methods. Additionally, the natural linearly-independent spanning basis can be transformed into other coordinate systems defined by other basis functions should that be useful. Such transformations can include the effects of discrete Fourier transforms, wavelet transforms, Walsh/Hadamard transforms, geometric rotations and scaling transforms, etc.
[0572] The simple approach employing T.sup.1 reconstructs the image by simply reproducing individual columns of an identity matrix, more precisely a diagonal matrix whose non-zero diagonal elements represent the light amplitude at a particular pixel. The invention provides for the replacement of this simple approach with other methods fitting into the same structure or delivering the same effect; for example projection techniques, matched filters, generalized inverses, SVD operations, sparse matrix operations, etc. These can be formatted in Tensor or matrix paradigms in view of the formal transformational tensor/matrix isomorphism established above. An example of this, namely the pseudo inverse case of a generalized inverse operation.
[0573] It is noted that the matrix T can become quite large, making inversion and subsequent operations described above numerically and computationally challenging. The invention provides for separating matrix T operations into smaller blocks (for example JPEG and MPEG regularly employ 88 and 1616 blocks). The invention provides for these blocks to be non-overlapping, to overlap, and to be interleaved. The invention further provides for blocked inversion results involving overlapping blocks or interleaving blocks to be combined by linear or other operations to suppress block-boundary artifacts.
Pseudo-Inverse of a 4-Tensor
[0574] Further, because in image capture a system usually spatially quantizes natural source image without a pixel structure, it is additional possible to measure a larger number of pixels than will be used in the final delivered image, that is M<J and N<K.
[0575] In traditional image processing such an excess-measurement scheme can be used in various oversampling methods, or could be decimated via resampling. Instead of these, the excess measurements can be used to create an over-specified system of equations that provides other opportunities. For example, the resulting over-specified matrix T can be used to generate a generalized inverse T.sup.+.
[0576] For example, if the 4-tensor ![custom-character]()
represents a transformation of a 2-dimensional (monochromatic) source image represented as an MN matrix of brightness values:
[00044]
to a JK array of measured 2-dimensional (monochromatic) image data represented as a JK matrix of brightness values:
[00045]
represented as
Q=![custom-character]()
S
then a pseudo-inverse tensor ![custom-character]()
.sup.+ can be defined via:
[00046]
and represented as
S=![custom-character]()
.sup.+Q
Further, the pixel values of the source image S can be obtained from the pixel values of the measurement Q by forming the pseudo-inverse of the matrix T, multiplying the flattened measurement data q with T.sup.+ to obtain the vector s, and partitioning the result into the source (image) matrix S:
[00047]
[0577] There are a number of pseudo-inverses and related singular-value decomposition operators, but of these it can be advantageous for the optical imaging methods to be described for the generalized inverse T.sup.+ to be specifically the Moore-Penrose generalized (left) inverse defined (when a matrix T has all linearly-independent columns) using the matrix transpose T.sup.T or conjugate transpose T.sup. of T and matrix inverse operations as:
T.sup.+=(T.sup.T T).sup.1T.sup.T for real-valued T
T.sup.+=(T.sup.T).sup.1 T.sup. for complex-valued T
[0578] (There is also Moore-Penrose generalized right inverse defined when a matrix T has all linearly-independent rows.) The Moore-Penrose generalized inverse inherently provides a Least-Squares statistical fit where solvable subsets of the larger number of equations give different inconsistent solutions. This Least-Squares statistical fit can provide robustness to the imaging system, for example in the case where one or more sensor elements degrade, are damaged, are occulted by dirt, are occulted by objects, are altered by transparent or translucent droplets or deposits, etc.
[0579] Using the Moore-Penrose generalized inverse for real-valued pixel quantities, the pixel values of the source image S can be obtained from the pixel values of the measurement Q by forming the pseudo-inverse of the matrix T, multiplying the flattened measurement data q with T.sup.+ to obtain the vector s, and partitioning the result into the source (image) matrix S:
[00048]
Configurations for Applications
[0580] Drawing on the functionality described above and taught in the Inventor's related lensless imaging patent filings listed at the beginning of this application, a wide range of additional provisions and configurations can be provided so as to support of vast number of valuable and perhaps slightly revolutionary imaging applications.
[0581] In an example generalizing assessment, the invention provides for a rigid or flexible surface to be configured to implement a lensless light-field sensor, producing electrical signals that can be used in real time, or stored and later retrieved, and provided to a computational inverse model algorithm executing on computational hardware comprising one or more computing elements so as to implement a lensless light-field camera.
[0582] In another aspect of the invention, a rigid surface is configured to additionally function as a housing and thus operate as a seeing housing.
[0583] In another aspect of the invention, a rigid surface is configured to additionally function as a protective plate and thus operate as a seeing armor.
[0584] In another aspect of the invention, a rigid surface is configured to additionally function as an attachable tile and thus operate as a seeing tile.
[0585] In another aspect of the invention, a rigid surface is configured to additionally function as an attachable film and thus operate as a seeing film.
[0586] In another aspect of the invention, a flexible surface is configured to additionally function as an attachable film and thus operate as a seeing film.
[0587] In another aspect of the invention, a flexible surface is configured to additionally function as a garment and thus operate as a seeing garment.
[0588] In another aspect of the invention, a flexible surface is configured to additionally function as a shroud and thus operate as a seeing shroud.
[0589] In another aspect of the invention, a flexible surface is configured to additionally function as an enveloping skin and thus operate as a seeing skin.
[0590] In another aspect of the invention, the rigid or flexible surface is small in size.
[0591] In another aspect of the invention, the rigid or flexible surface is large in size.
[0592] In another aspect of the invention, the rigid or flexible surface is flat.
[0593] In another aspect of the invention, the rigid or flexible surface is curved.
[0594] In another aspect of the invention, the rigid or flexible surface is rendered as a polytope.
[0595] In another aspect of the invention, the rigid or flexible surface is rendered as a dome.
[0596] In another aspect of the invention, the rigid or flexible surface is rendered as a part of a sphere.
[0597] In another aspect of the invention, the rigid or flexible surface is rendered as a part of a spheroid.
[0598] In another aspect of the invention, the rigid or flexible surface is rendered as a sphere.
[0599] In another aspect of the invention, the rigid or flexible surface is rendered as a spheroid.
[0600] In another aspect of the invention, the rigid or flexible surface is transparent.
[0601] In another aspect of the invention, the rigid or flexible surface is translucent.
[0602] In another aspect of the invention, the rigid or flexible surface is opaque.
[0603] In another aspect of the invention, the rigid or flexible surface performs contact sensing.
[0604] In another aspect of the invention, the rigid or flexible surface is configured to perform contact image sensing with near-zero separation distance.
[0605] In another aspect of the invention, the rigid or flexible surface is configured to perform contact image sensing with zero separation distance.
[0606] In another aspect of the invention, the rigid or flexible surface performs distributed optical imaging.
[0607] In another aspect of the invention, the rigid or flexible surface performs distributed optical sensing.
[0608] In another aspect of the invention, the rigid or flexible surface performs image sensing of ultraviolet light.
[0609] In another aspect of the invention, the rigid or flexible surface performs image sensing of infrared light.
[0610] In another aspect of the invention, the rigid or flexible surface performs image sensing of selected ranges of visible color light.
[0611] In another aspect of the invention, the rigid or flexible surface performs imaging.
[0612] In another aspect of the invention, the rigid or flexible surface performs distributed chemical sensing employing optical chemical sensing properties of at least one material.
[0613] In another aspect of the invention, the rigid or flexible surface performs distributed radiation sensing employing optical radiation sensing properties of at least one material.
[0614] In another aspect of the invention, the rigid or flexible surface performs distributed magnetic field sensing employing optical magnetic field sensing properties of at least one material.
[0615] In another aspect of the invention, the rigid or flexible surface is configured to emit light.
[0616] In another aspect of the invention, the rigid or flexible surface is configured to operate as a light-emitting display.
[0617] In another aspect of the invention, the rigid or flexible surface is configured to operate as a selectively self-illuminating contact imaging sensor.
[0618] In another aspect of the invention, the computational inverse model algorithm is configured to provide variable focusing.
[0619] In another aspect of the invention, the computational inverse model algorithm is configured to mixed depth-of-field focusing.
[0620] In another aspect of the invention, the computational inverse model algorithm is configured to implement a viewpoint with a controllable location.
[0621] In another aspect of the invention, the computational inverse model algorithm is configured to implement a plurality of viewpoints, each viewpoint having a separately controllable location.
[0622] In another aspect of the invention, the computational inverse model algorithm is configured to provide pairs of outputs so as to function as a stereoscopic camera.
[0623] In another aspect of the invention, the computational inverse model algorithm is configured to capture a panoramic view.
[0624] In another aspect of the invention, the computational inverse model algorithm is configured to capture a 360-degree view.
[0625] In another aspect of the invention, the computational inverse model algorithm is configured to capture a partial spherical view.
[0626] In another aspect of the invention, the computational inverse model algorithm is configured to capture a full spherical view.
[0627] In another aspect of the invention, the rigid or flexible surface is configured to perform enveloping image sensing with near-zero separation distance.
[0628] In another aspect of the invention, the rigid or flexible surface is configured to perform contact enveloping sensing with zero separation distance.
[0629] In another aspect of the invention, the rigid or flexible surface is configured to operate as a selectively self-illuminating enveloping imaging sensor.
[0630] In another aspect of the invention, the computational inverse model algorithm is configured to operate at slow-frame video rates.
[0631] In another aspect of the invention, the computational inverse model algorithm is configured to operate at conventional video rates.
[0632] In another aspect of the invention, the computational inverse model algorithm and computational hardware is configured to operate at high-speed video rates.
CLOSING
[0633] The terms certain embodiments, an embodiment, embodiment, embodiments, the embodiment, the embodiments, one or more embodiments, some embodiments, and one embodiment mean one or more (but not all) embodiments unless expressly specified otherwise. The terms including, comprising, having and variations thereof mean including but not limited to, unless expressly specified otherwise. The enumerated listing of items does not imply that any or all of the items are mutually exclusive, unless expressly specified otherwise. The terms a, an and the mean one or more, unless expressly specified otherwise.
[0634] The foregoing description, for purpose of explanation, has been described with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The embodiments were chosen and described in order to best explain the principles of the invention and its practical applications, to thereby enable others skilled in the art to best utilize the invention and various embodiments with various modifications as are suited to the particular use contemplated.
[0635] While the invention has been described in detail with reference to disclosed embodiments, various modifications within the scope of the invention will be apparent to those of ordinary skill in this technological field. It is to be appreciated that features described with respect to one embodiment typically can be applied to other embodiments.
[0636] The invention can be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein.
[0637] Although exemplary embodiments have been provided in detail, various changes, substitutions and alternations could be made thereto without departing from spirit and scope of the disclosed subject matter as defined by the appended claims. Variations described for the embodiments may be realized in any combination desirable for each particular application. Thus particular limitations and embodiment enhancements described herein, which may have particular advantages to a particular application, need not be used for all applications. Also, not all limitations need be implemented in methods, systems, and apparatuses including one or more concepts described with relation to the provided embodiments. Therefore, the invention properly is to be construed with reference to the claims.
CITED REFERENCES
Cited Books
[0638] [B1] D. J. Brady, Optical Imaging and Spectroscopy, Wiley, 2009, ISBN 978-0-470-04823-8. [0639] [B2] J. R. Janesick, Photon Transfer DN.fwdarw., SPIE Press, 2007, ISBN 978-0-819-6722-5. [0640] [B3] A. J. Devaney, Mathematical Foundations of Imaging, Tomography and Wavefield Inversion, Cambridge University Press, 2012, ISBN 978-0-521-11974-0. [0641] [B4] C. Zhang and T. Chen, Light Field Sampling, Morgan and Claypool, 2006, ISBN 978-1-598-29076-9. [0642] [B5] O. Scherzer, M. Grasmair, H. Grossauer, M. Haltmeier, F. Lenzen, Variational Methods in Imaging, Springer, 2009, ISBN 978-0-387-30931-6. [0643] [B6] M. Elad, Sparse and Redundant Representations, Springer, 2010, ISBN 978-1-4419-7010-4. [0644] [B7] J. P. Dakin, R. G. W. Brown, eds. Handbook of Optoelectronics Vol 1, Taylor and Francis, 2012, ISBN 978-0-7503-0646-1. [0645] [B8] H. Caulfied, Holography Shadow Casting, in E. Camatini (ed.), Progress in Electro-Optics: Reviews of Recent Developments, NATO Advanced Study Institute, Series B (Physics), 1973/1975. [0646] [B9] H. Yanai, K. Takeuchi, Y. Takane, Projection Matrices, Generalized Inverse Matrices, and Singular Value Decomposition, Springer, 2011, ISBN 978-1-4419-9886-6. [0647] [B10] A. Bjerhammar, Theory of Errors and Generalized Matrix Inverses, Elsevier, 1973. [0648] [B11] R. R. Roa, S. K. Mitra, Generalized inverses of Matrices and its Applications, Wiley, 1971, ISBN 0-471-70821-6. [0649] [B12] C. W. Groetsch, Generalized Inverses of Linear OperatorsRepresentation and Approximation, Marcel Dekker, Inc., 1977, ISBN 0824766156. [0650] [B13] R. Piziak, P. L. Odell, Matrix TheoryFrom Generalized Inverses to Jordan Form, Chapman & Hall/CRC, 2007, ISBN 978-1-58488-625-0. [0651] [B14] E. Chang, The Generalized Inverse and Interpolation Theory, in Recent Applications of Generalized Inverses, S. L. Campbell (ed.), Pitman, 1982, ISBN 0-273-08550-6. [0652] [B15] A. Albert, Regression and the Moore-Penrose Pseudoinverse, Academic Press, 1972, Library of Congress No. 72-77337 (subsequently ISBN 0-120-48450-1). [0653] [B16] J. Wagner D. Keszler, R. Presley, Transparent Electronics, Springer 2008, ISBN 978-0-378-72341-9. [0654] [B17] A. Facchetti, T. Marks, eds., Transparent ElectronicsFrom Synthesis to Applications, 2010, ISBN 978-0-470-99077-3. [0655] [B18] R. Chaji, A. Nathan, Thin Film Transistor Circuits and Systems, Cambridge, 2013, ISBN 978-1-107-01233-2. [0656] [B19] T. Tsujimura, OLED DisplaysFundamentals and Applications, Wiley, 2012, ISBN 978-1-118-14051-2. [0657] [B20] G. Held, Introduction to Light Emitting Diode Technology and Applications, C R C, 2009, ISBN 978-1-4200-7662-2. [0658] [B21] R. Shinar, J. Shinar (eds.), Organic Electronics in Sensors and Biotechnology, McGraw Hill, 2009, ISBN 978-0-07-159675-6, Chapter 6 and Section 5.5. [0659] [B22] D. R. Gamota, P. Brazis, K. Kalyanasundaram, J. Zhang, Printed Organic and Molecular Electronics, Kluwer, 2004, ISBN 1-4020-7707-6. [0660] [B23] E. Cantatore (ed.), Applications of Organic and Printed ElectronicsA Technology-Enabled Revolution, Springer, 2013, ISBN 978-1-4614-3160-2. [0661] [B24] A. E. Javier, Solution-Processable Materials for Printable Electronics, UMI/Proquest, 2010, ISBN: 978-1-1240-0854-7 [0662] [B25] Z. Cui, Printed Electronics: Materials, Technologies and Applications, Wiley, 2016, ISBN 978-1-1189-2092-3. [0663] [B26] A. Facchetti, Materials and Process Engineering for Printed and Flexible Optoelectronic Devices, in Frontiers of Engineering: National Academy of Engineering, Reports on Leading-Edge Engineering from the 2013 Symposium, National Academies Press, 2014, ISBN 978-0-309-29603-8, pp 113-125. [0664] [B27] B. Gnade, N. Fruehauf, B. Chalamala, J. Jang, Jin (eds), Flexible ElectronicsMaterials and Device Technology, (Proceedings 769), Materials Research Society, 2003, ISBN 1558997067. [0665] [B28] W. Wong, Flexible Electronics: Materials and Applications, Springer, 2009, ISBN 978-0-3877-4362-2. [0666] [B29] S. Logothetidis, Handbook of Flexible Organic Electronics, Woodhead Publishing, 2014, ISBN 978-1-7824-2035-4. [0667] [B30] M. Caironi, Large Area and Flexible Electronics, Wiley-VCH, 2015, ISBN 978-3-5273-3639-5. [0668] [B31] G. Shen, Flexible Electronics: From Materials to Devices, World Scientific Publishing Co, 2015, ISBN 978-9-8146-5198-1. [0669] [B32] F. Gardiner, Polymer ElectronicsA Flexible Technology, Smithers Rapra Technology, 2009, ISBN 978-1-8473-5421-1.
Cited Articles, Presentations, and Technical Papers
[0670] [P1] A. Bjerhammar, A Generalized Matrix Algebra, N.R.C. Can. Div. Appl. Phys., Ottawa., 1957. [0671] [P2] R. D. Jansen-van Vuuren, A. Armin, A. K. Pandey, P. L. Burn, P. Meredith, Organic Photodiodes: The Future of Full Color Detection and Image Sensing, Advanced Materials, Vol 28, Issue 24, Jun. 22, 2016, pp. 4766-4802. [0672] [P3] H. Wiman, Improvement of Digital Image Resolution by Oversampling, XVIIth ISPRS CongressTechnical Commission II: Systems for Data Processing and Analysis, Aug. 2-14, 1992, Washington, D.C., USA, L. W. Fritz, J. R. Lucus (eds), in ISPRS ArchivesVolume XXIX Part B2, 1992, International Society for Photogrammetry and Remote Sensing, 1993, pp. 323-327. [0673] [P4] D. G. Stork, P. R. Gill. Lensless Ultra-Miniature CMOS Computational Imagers and Sensors. Proc. SENSORCOMM (2013): 186-190. [0674] [P5] M. S. Asif, et al. Flatcam: Thin, Lensless Cameras Using Coded Aperture and Computation. IEEE Transactions on Computational Imaging (2017). [0675] [P6] V. Boominathan, et al., Lensless Imaging: A Computational Renaissance, IEEE Signal Processing Magazine 33(5), September 2016, pp. 23-35. Available at http://www.ece.rice.edu/vb10/documents/2016/Lensless_Imaging_Computation al_Renaissance.pdf [0676] [P7] G. Williams, C. Backhouse, H. Aziz, Integration of Organic Light Emitting Diodes and Organic Photodetectors for Lab-on-a-Chip-Detection Systems, Electronics, Vol 3, Feb. 13, 2014, pp. 43-75. [0677] [P8] F. Krujatz, O. R. Hild, K. Fehse, et al. Exploiting the Potential of OLED-Based Photo-Organic Sensors for Biotechnological Applications Chemical Sciences Journal, Vol 7, Issue 3, Jul. 18, 2016. [0678] [P9] M. Punke, S. Mozer, M. Stroisch, Organic Semiconductor Devices for Micro-Optical Applications, Proc of Spie, Vol 6185, Apr. 3, 2006. [0679] [P10] R. Bhattacharya, et al. Organic LED Pixel Array on a Dome, Proceedings of the IEEE, Vol 93, Issue 7, Jul. 5, 2005, pp. 1273-1280. [0680] [P11] D. L. Cade, Hitachi's Lensless Camera Uses Moire and Math, Not Glass, to Take Photos, Hitachi Press Release, Nov. 16, 2016. Available at https://petapixel.com/2016/11/16/hitachis-lensless-camera-uses-moire-math-not-glass-take-photos/ as retrieved Jun. 20, 2017. [0681] [P12] Hitachi, Lensless-Camera Technology for Easily Adjusting Focus on Video Images after Image Capture, Hitachi Press Release, Nov. 15, 2016. Retrieved Jun. 20, 2017 from http://www.hitachi.com/New/cnews/month/2016/11/161115.html [0682] [P13] Hitachi, Technical Explanation of Hitachi Lensless Camera, Hitachi, Dec. 5, 2016 Retrieved Jun. 20, 2017 from https://physics.stackexchange.com/questions/296640/technical-explanation-of-hitachi-lensless-camera. [0683] [P14] Hitachi, Lensless Camera Technology for Adjusting Video Focus After Image Capture, Hitachi Press Release, Nov. 21, 2016. Retrieved Jun. 20, 2017 from https://phys.org/news/2016-11-lensless-camera-technology-adjusting-video.html [0684] [P15] Sumito, Organic Photodiodes, Sumito Chemical Printed Electronics, visited on Jun. 20, 2017. Retrieved Jun. 20, 2017 from https://www.sumitomo-chem.co.jp/printedelectronics/en/application/photodiodes.html. [0685] [P16] Fraunhofer Research Institution for Organics, Materials And Electronic Devices, Smart Optoelectronic Micro-Sensors By OLED-On-Silicon, Fraunhofer Comedd. Available at https://www.comedd.fraunhofer.de/content/dam/comedd/common/products/COM EDD/oledcmos-e.pdf retrieved Jun. 20, 2017.) [0686] [P17] M. G. Han, K. B. Park, et al. Narrow-Band Organic Photodiodes for High-Resolution Imaging, Applied Materials & Interfaces, Vol 8, Issue 39, Sep. 13, 2016, pp. 26143-26151. [0687] [P18] W. Wang, F. Zhang, et al. Highly Narrowband Photomultiplication Type Organic Photodetectors, Nano Letters, Vol 17, Issue 3, pp. 1995-2002, Feb. 6, 2017. [0688] [P19] D. H. Kim, K. S. Kim, et al. A High Performance Semitransparent Organic Photodetector with Green Color Selectivity Applied Physics Letter, Vol 105, 2014. [0689] [P20] R. Hany, Transparent Organic Photodiodes, EMPA, Retrieved Jun. 20, 2017 from https://www.empa.ch/web/s209/organic-photodiodes. [0690] [P21] F. Arca, Organic Photodiodes for Industrial Sensing and Medical Imaging, Dissertation, Technische Universitt Mnchen Lehrstuhl fr Nanoelektronik, Mar. 3, 2013. Retrieved Jun. 20, 2017 from https://mediatum.ub.tum.de/doc/1197763/458492.pdf [0691] [P22] Organic Photodiodes for Sensor Applications, Fraunhofer Comedd, May 14, 2014. Retrieved Jun. 20, 2017 from https://phys.org/news/2014-05-photodiodes-sensor-applications.html. [0692] [P23] B. A. Katchman, J. T. Smith, et al. Application of Flat Panel OLED Display Technology for the Point-of-Care Detection of Circulating Cancer Biomarkers, Scientific Reports, Vol 6, Article number: 29057, Jul. 4, 2016. [0693] [P24] Y. Y. Noh, D. Y. Kim, Organic Phototransistor Based on Pentacene as an Efficient Red Light Sensor Solid-State Electronics, Vol 51, Issue 7, July 2007, pp. 1052-1055. [0694] [P25] X. Liu, E. K. Lee, et al. Flexible Organic Phototransistor Array with Enhanced Responsivity via Metal-Ligand Charge Transfer, Applied Materials & Interfaces Vol 8, Issue 11, Mar. 1, 2016, pp. 7291-7299. [0695] [P26] H. W. Zan, S. C. Kao, S. R. Ouyang, Pentacene-Based Organic Phototransistor with High Sensitivity to Weak Light and Wide Dynamic Range IEEE, Vol 31, Issue 2, Jan. 19, 2010, pp. 135-137. [0696] [P27] K. J. Baeg, M. Binda, Organic Light Detectors: Photodiodes and Phototransistors, Advanced Materials, Volume 25, Issue 31, Aug. 21, 2013, Pages 4267-4295. [0697] [P28] A. Koppelhuber, O. Bimber, Towards a Transparent, Flexible, Scalable and Disposable Image Sensor Using Thin Film Luminescent Concentrators Optics Express, Vol 21, Issue 4, 2013, pp. 4796-4810. [0698] [P29] A. Pierre, A. Gaikwad, A. C. Arias, Charge-integrating organic heterojunction phototransistors for wide-dynamic-range image sensors, Nature Photonics 11, 2017, pp. 193-199. [0699] [P30] M. Ramuz, L. BUrgi, P. Seitz, High sensitivity organic photodiodes with low dark currents and increased lifetimes, Organic Electronics, Vol. 9, Issue 3, June 2008, pp. 369-376. [0700] [P31] A. Busboom, H. Elders-Boll, H. Schotten, Uniformly Redundant Arrays, Experimental Astronomy, June 1998, Volume 8, Issue 2, pp. 97-123. [0701] [P32] R. Marcia, Z. Harmany, R. Willett, Compressive Coded Aperture Imaging, Computational Imaging VII, SPIE Proceedings Vol. 7246 (72460G), Feb. 3, 2009. [0702] [P33] W. H. Richardson, Design of an Ultrasensitive Active Pixel Sensor That Is Based on Silicon Nanostructures, SPIE DSS Conference, Micro- and Nanotechnology Sensors, Systems, and Applications III, Paper 8031-92, (May 2011). [0703] [P34] M. Hirsch, BiDi Screen: Depth and Lighting Aware Interaction and Display, MS Thesis, MIT, Aug. 13, 2009 (Available at https://dam-prod.media.mit.edu/x/files/thesis/2009/mhirsch-ms.pdf as retrieved Jul. 2, 2017.) [0704] [P35] M. Grosse, G. Wetzstein, A. Grundhoefer, O. Bimber, Coded Aperture Projection, ACM Transactions on Graphics 29(3), Volume 29 Issue 2, March 2010; also ACM SIGGRAPH, Jul. 25-29, 2010. (Available at http://web.media.mit.edu/gordonw/CodedApertureProjection/ as retrieved Jul. 2, 2017.) [0705] [P36] T. Barribeau, Shooting Full Panoramas Is Easy with Bendable Flexcam Camera, Imaging Resource, Aug. 19, 2013. (Available at http://www.imaging-resource.com/news/2013/08/19/Shooting-full-panoramas-is-easy-with-bendable-Flexcam-camera as retrieved Jul. 1, 2017.) [0706] [P37] H. Evarts, A Flexible Camera: A Radically Different Approach to Imaging Columbia Engineering, Apr. 13, 2016 (Available at http://engineering.columbia.edu/flexible-camera-radically-different-approach-imaging as retrieved Jul. 1, 2017) [0707] [P38] D. Sims, Y. Yue, S. Nayar, Towards Flexible Sheet Cameras: Deformable Lens Arrays with Intrinsic Optical Adaptation, IEEE International Conference on Computational Photography (ICCP), May 2016. (Available at http://www.cs.columbia.edu/CAVE/projects/flexible_sheet_cameras/Sims_ICCP1 6.pdf as retrieved Jul. 1, 2017.)) [0708] [P39] R. Perkins, Ultra-Thin Camera Creates Images Without Lenses, CalTech, Jun. 21, 2017. (Available at http://www.caltech.edu/news/ultra-thin-camera-creates-images-without-lenses-78731 as retrieved Jul. 2, 2017.) [0709] [P40] R. Ng, M. Levoy, M. Bredif, G. Duval, M. Horowitz, P. Hanrahan, Light Field Photography with a Hand-Held Plenoptic Camera, Stanford University Computer Science Tech Report CSTR 2005-02, 2005. (Available at https://graphics.stanford.edu/papers/Ifcamera/Ifcamera-150 dpi.pdf visited Jul. 2, 2017.) [0710] [P41] R. Butler, Lytro Light Field Camera first look with Ren Ng, Digital Photography Review, Oct. 19, 2011. (Available at https://www.dpreview.com/articles/7237351494/lytro-light-field-camera-first-look-with-ren-ng visited Jul. 2, 2017.) [0711] [P42] V. Koifman, Toshiba Announces Light Field Camera Module, Image Sensors World, Dec. 27, 2012. (Available http://image-sensors-world.blogspot.com/2012/12/toshiba-announces-light-field-camera.html as retrieved Jul. 2, 2017.) [0712] [P43] LightField Forum, New Light Field Tech to use Sensor Layers instead of Microlenses, LightField Forum, May 15, 2016. (Available at http://lightfield-forum.com/2016/03/new-light-field-tech-to-use-transparent-sensor-layers-instead-of-microlenses/ as retrieved Jul. 2, 2017.) [P44] W. Wang, Optical Detectors, slide presentation (date unknown). Available at http://depts.washington.edu/mictech/optics/sensors/detector.pdf as retrieved Jul. 2, 2017. [0713] [P45] E. Fenimore, Coded Aperture Imaging: The Modulation Transfer Function for Uniformly Redundant Arrays, Appl Opt 19 (14), Jul. 15, 1980, pp. 2465-2471. [0714] [P46] M. Levoy, Light fields and computational imaging, Computer, vol. 39, no. 8, August 2006, pp. 46-55. [0715] [P47] A. Zomet and S. K. Nayar, Lensless imaging with a controllable aperture, in IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, 2006, pp. 339-346. [0716] [P48] M. Duarte, M. Davenport, D. Takhar, J. Laska, T. Sun, K. Kelly, R. Baraniuk, Single-Pixel Imaging via Compressive Sampling, IEEE Signal Processing Magazine, 2008, 25, pp. 83-91. [0717] [P49] A. Nathan, A. Ahnood, M. Cole, Flexible Electronics: The Next Ubiquitous Platform, Proceedings of the IEEE, Volume: 100, Issue: Special Centennial Issue, May 13, 2012, pp. 1486-1517. [0718] [P50] Phys Org, Researchers Achieve Major Breakthrough in Flexible Electronics, Phys Org News, Jan. 13, 2017, available at https://phys.org/news/2017-01-major-breakthrough-flexible-electronics.html retrieved Jul. 8, 2017. [0719] [P51] Phys Org, New Hybrid Inks for Printed, Flexible Electronics Without Sintering, Phys Org News, Apr. 10, 2017, available at https://phys.org/news/2017-04-hybrid-inks-flexible-electronics-sintering.html as retrieved Jul. 8, 2017. [0720] [P52] Interuniversitair Micro-Electronica Centrum (IMEC), Thin-film flexible electronics, Available at https://www.imec-int.com/en/thin-film-flexible-electronics retrieved Jul. 8, 2017. [0721] [P53] A. Abouraddy, O. Shapira, M. Bayindir, J. Arnold, F. Sorin, D. Hinczewski, J. Joannopoulos, Y. Fink, Large-scale optical-field measurements with geometric fibre constructs, Nature Mat. 5(7), 532-536 (2006). [0722] [P54] T. Ng, W. Wong, M. Chabinyc, S. Sambandan, and R. Street, Flexible image sensor array with bulk heterojunction organic photodiode, Appl. Phys. Lett. [0723] 92(21), 213303 (2008). [0724] [P55] T. Hayes, Flexible image sensors move nearer to market, Optics.org, Jul. 16, 2013, available at http://optics.org/news/4/7/22 as retrieved Jul. 9, 2017. [0725] [P56] K. Chopra, S. Major, D. Pandya, Transparent conductorsA Status Review, Thin Solid Films, Volume 102, Issue 1, Apr. 8, 1983, pp. 1-46. [0726] [P57] F. Yu, S. Wu, X. Wang, G. Zhang, H. Luaband, L. Qiu, Flexible and low-voltage organic phototransistors, RSC Advances 7(19), February 2017, pp. 11572-11577. [0727] [P58] S. Garlapati, T. Baby, S. Dehm, M. Hammad, V. Chakravadhanula, R. Kruk, H. Hahn, S. Dasgupta, Ink-Jet Printed CMOS Electronics from Oxide Semiconductors, Nano Micro Small, Volume 11, Issue 29, Aug. 5, 2015, pp. 3591-3596. [0728] [P59] G. Huang, H. Jiang, K. Matthews, P. Wilford, Lensless Imaging by Compressive Sensing, IEEE International Conference on Image Processing, ICIP 2013, Paper #2393, (Available at https://arxiv.org/abs/1305.7181 as retrieved Jul. 10, 2017.) [0729] [P60] D. Thapaa, K. Raahemifarb, V. Lakshminarayanan, Less is more: compressive sensing in optics and image science, Journal of Modern Optics, Vol. 62, No. 3, 2015, pp. 169-183. [0730] [P61] Erickson, Evan L., et al. Miniature lensless computational infrared imager. Electronic Imaging 2016.12 (2016): 1-4. [0731] [P62] R. H. Dicke, Scatter-Hole Cameras for X-Rays and Gamma Rays, Astrophys. J. 153, L101 (1968). [0732] [P63] E. Fenimore and T. Cannon, Coded Aperture Imaging with Uniformly Redundant Arrays, Appl. Opt. 17, 337 (1978). [0733] [P64] E. Fenimore, Coded aperture imaging: predicted performance of uniformly redundant arrays. Applied Optics 17.22 (1978): 3562-3570. [0734] [P65] S. Gottesman, Coded Apertures: Past, Present, and Future Application and Design. Optical Engineering Applications. International Society for Optics and Photonics, 2007. [0735] [P66] C. Zhou, S. Nayar, Computational cameras: Convergence of Optics and Processing. IEEE Transactions on Image Processing 20.12 (2011): 3322-3340. [0736] [P67] A. Veeraraghavan, et al., Dappled photography: Mask enhanced cameras for heterodyned light fields and coded aperture refocusing. ACM Trans. Graph. 26.3 (2007): 69. [0737] [P68] A. Levin, et al., Image and depth from a conventional camera with a coded aperture. ACM transactions on graphics (TOG) 26.3 (2007): 70 [0738] [P69] H. Jiang, G. Huang, P. Wilford, Multi-view in lensless compressive imaging, APSIPA Transactions on Signal and Information Processing, 3. (2014) doi:10.1017/ATSIP.2014.16 [0739] [P70] G. Kim, et al. Lensless Photography with only an image sensor, arXiv preprint arXiv:1702.06619 (2017). [0740] [P71] X. Yuan, et al., Lensless compressive imaging, arXiv preprint arXiv:1508.03498 (2015). [0741] [P72] Yuan, Xin, et al. Block-wise Lensless Compressive Camera, arXiv preprint arXiv:1701.05412 (2017). [0742] [P73] O. Cossairt, M. Gupta, S. Nayar, When Does Computational Imaging Improve Performance? IEEE Transactions on Image Processing 22.2 (2013): 447-458. [0743] [P74] Pierre, Adrien, et al., High Detectivity All-Printed Organic Photodiodes. Advanced Materials 27.41 2015, pp. 6411-6417. [0744] [P75] Someya, Takao, et al. Integration of Organic FETs with Organic Photodiodes for a Large Area, Flexible, and Lightweight Sheet Image Scanners. IEEE Transactions on Electron Devices, 52.11 2005, pp. 2502-2511.
