Image capture and processing

Abstract

A method of image processing includes receiving a first image of human skin. The first image corresponds to a first, uniform broadband illumination condition. The method also includes receiving a second image which has the same field of view and contents as the first image. The second image corresponds to a second illumination condition which comprises a uniform narrowband illumination condition. The method also includes processing the first and second images to fit parameter maps for a spectral bidirectional scattering surface reflectance distribution function skin model. The parameter maps include a modelled melanin concentration, a modelled haemoglobin concentration, a modelled melanin blend-type fraction and a modelled epidermal haemoglobin fraction. At least three of the parameter maps are independent.

Claims

1. A method of image processing, comprising: receiving a first image of human skin, the first image corresponding to a first, uniform broadband illumination condition; receiving a second image which has the same field of view and contents as the first image, the second image corresponding to a second illumination condition which comprises a uniform narrowband illumination condition; receiving or determining first spectral data corresponding to the uniform broadband illumination condition and second spectral data corresponding to the second illumination condition; processing the first and second images to fit parameter maps for a spectral bidirectional scattering surface reflectance distribution function skin model, the parameter maps comprising a modelled melanin concentration, a modelled haemoglobin concentration, a modelled melanin blend-type fraction and a modelled epidermal haemoglobin fraction; wherein at least three of the parameter maps are independent; wherein processing the first and second images to fit parameter maps comprises applying a neural network model to inputs comprising: spectral information comprising the first and second spectral data; the first image; and the second image; wherein a final stage of the neural network outputs an output RGB albedo image, and wherein the neural network is configured to determine the parameter maps which minimise differences between the output RGB albedo image and the first image.

2. A method according to claim 1, wherein the uniform narrowband illumination condition corresponds to a blue narrowband illumination condition.

3. A method according to claim 1, wherein: the second illumination condition consists of the uniform narrowband illumination condition; or the second illumination condition comprises the uniform narrowband illumination condition superposed with the uniform broadband illumination condition.

4. A method according to claim 1, wherein the second illumination condition comprises the uniform narrowband illumination condition superposed with the uniform broadband illumination condition, and wherein applying a neural network model to inputs comprising the spectral information, the first image and the second image comprises: generating a narrowband illumination image based on the first and second images; and passing the first image, the narrowband illumination image and the spectral information as inputs to the neural network model.

5. A method according to claim 1, wherein determining first spectral data and second spectral data comprises: receiving a third image of a colour test card, the third image corresponding to the uniform broadband illumination condition; determining first spectral data corresponding to the broadband illumination condition based on the third image; receiving a fourth image of the colour test card, the fourth image corresponding to the second illumination condition; determining second spectral data corresponding to the second illumination condition based on the fourth image.

6. A method comprising: receiving an albedo image showing human skin; receiving or determining spectral data corresponding to an illumination condition used to obtain the albedo image; processing the albedo image to fit parameter maps for a spectral bidirectional scattering surface reflectance distribution function skin model, the parameter maps comprising a modelled melanin concentration, a modelled haemoglobin concentration, a modelled melanin blend-type fraction and a modelled epidermal haemoglobin fraction; wherein at least three of the parameter maps are independent; wherein processing the albedo image to fit parameter maps comprises applying a neural network model to inputs comprising: the albedo image; and the spectral data; wherein a final stage of the neural network outputs an output RGB albedo image, and wherein the neural network is configured to determine the parameter maps which minimise differences between the output RGB albedo image and the albedo image.

7. A method according to claim 6, wherein determining spectral data comprises: receiving a calibration image of a colour test card, the calibration image corresponding to the illumination condition; determining spectral data corresponding to the illumination condition based on the calibration image.

8. A method according to claim 6, further comprising: receiving a first input parameter map related to melanin concentration and corresponding to the albedo image; receiving a second input parameter map related to haemoglobin concentration and corresponding to the albedo image; wherein inputs to the neural network model further comprise: the first input parameter map; and the second input parameter map.

9. A method of image processing, comprising: receiving a first image of human skin, the first image corresponding to a first, uniform broadband illumination condition; receiving a second image which has the same field of view and contents as the first image, the second image corresponding to a second illumination condition which comprises a uniform narrowband illumination condition; receiving or determining first spectral data corresponding to the uniform broadband illumination condition and second spectral data corresponding to the second illumination condition; processing the first and second images to fit parameter maps for a spectral bidirectional scattering surface reflectance distribution function skin model, the parameter maps comprising a modelled melanin concentration, a modelled haemoglobin concentration, a modelled melanin blend-type fraction and a modelled epidermal haemoglobin fraction; wherein at least three of the parameter maps are independent; wherein processing first and second images to fit parameter maps comprises using a three-dimensional spectral look-up table or using a four-dimensional spectral look-up table; wherein the spectral look-up table is constructed based on spectral information comprising the first spectral data and the second spectral data, and further based on the spectral skin reflectance model.

10. A method according to claim 9, wherein the uniform narrowband illumination condition corresponds to a blue narrowband illumination condition.

11. A method according to claim 9, wherein: the second illumination condition consists of the uniform narrowband illumination condition; or the second illumination condition comprises the uniform narrowband illumination condition superposed with the uniform broadband illumination condition.

12. A method according to claim 9, wherein determining first spectral data and second spectral data comprises: receiving a third image of a colour test card, the third image corresponding to the uniform broadband illumination condition; determining first spectral data corresponding to the broadband illumination condition based on the third image; receiving a fourth image of the colour test card, the fourth image corresponding to the second illumination condition; determining second spectral data corresponding to the second illumination condition based on the fourth image.

13. A method comprising: receiving an albedo image showing human skin; receiving or determining spectral data corresponding to an illumination condition used to obtain the albedo image; processing the albedo image to fit parameter maps for a spectral bidirectional scattering surface reflectance distribution function skin model, the parameter maps comprising a modelled melanin concentration, a modelled haemoglobin concentration, a modelled melanin blend-type fraction and a modelled epidermal haemoglobin fraction; wherein at least three of the parameter maps are independent; wherein processing the albedo image to fit parameter maps comprises using a three-dimensional spectral look-up table or using a four-dimensional spectral look-up table; wherein the spectral look-up table is constructed based on the spectral data, and further based on the spectral skin reflectance model.

14. A method according to claim 13, wherein determining spectral data comprises: receiving a calibration image of a colour test card, the calibration image corresponding to the illumination condition; determining spectral data corresponding to the illumination condition based on the calibration image.

15. A method comprising: receiving an albedo image showing human skin; receiving or determining spectral data corresponding to an illumination condition used to obtain the albedo image; receiving a first input parameter map related to melanin concentration and corresponding to the albedo image; receiving a second input parameter map related to haemoglobin concentration and corresponding to the albedo image; processing the albedo image to fit parameter maps for a spectral bidirectional scattering surface reflectance distribution function skin model, the parameter maps comprising a modelled melanin concentration, a modelled haemoglobin concentration, a modelled melanin blend-type fraction and a modelled epidermal haemoglobin fraction; wherein at least three of the parameter maps are independent; wherein processing the albedo image to fit parameter maps comprises: processing the albedo image to fit first and second intermediate parameter maps for a two parameter spectral bidirectional scattering surface reflectance distribution function skin model, wherein the first intermediate parameter map comprises an intermediate melanin concentration and the second intermediate parameter map comprises an intermediate haemoglobin concentration, wherein the intermediate parameter maps are determined using a first two-dimensional spectral look up table constructed based on the spectral data; re-scaling the first and second input parameter maps based on the intermediate parameter maps; setting the parameter map corresponding to the modelled melanin concentration equal to the re-scaled first input parameter map; setting the parameter map corresponding to the modelled haemoglobin concentration equal to the re-scaled second input parameter map; processing the albedo image to fit the parameter maps corresponding to the modelled melanin blend-type fraction and the modelled epidermal haemoglobin fraction, wherein the parameter maps are determined using a second two-dimensional spectral look up table constructed based on the spectral data.

16. A method according to claim 15, wherein determining spectral data comprises: receiving a calibration image of a colour test card, the calibration image corresponding to the illumination condition; determining spectral data corresponding to the illumination condition based on the calibration image.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

(2) Certain embodiments of the present invention will now be described, by way of example, with reference to the accompanying drawings, in which:

(3) FIG. 1A to FIG. 1C: Renderings of acquired faces (FIG. 1A, FIG. 1B) and a patch of skin (back of hand) using our proposed practical spectral measurement and reconstruction approaches. Spatially-varying spectral skin parameters for subject 1 (a, Mediterranean skin type), and subject 2 (FIG. 1B, Asian skin type) acquired using an LED sphere, while the back of the hand (FIG. 1C, South Asian skin type) has been acquired using a hand-held dermatological skin measurement device (Antera3D). Top-row: Renderings (using PBRT) with a spectral BSSRDF under a point source illumination with broadband D65 spectrum. Bottom-row: Diffuse-only renderings under six different LED spectras (W57, W40, W27, Red, Green, Blue).

(4) FIG. 2A to 2D: Spectral skin reflectance model variation due to β.sub.m (FIG. 2A, FIG. 2B), and C.sub.he (FIG. 2C, FIG. 2D) parameters, respectively. Individual 2D LUTs represent variation due to the dominant parameters C.sub.m (x-axis) and C.sub.h (y-axis).

(5) FIG. 3A to FIG. 3C: Comparison of a facial photograph under uniform broad-band (W57 cool white LED) illumination (FIG. 3B) to reconstructions using the reduced 2D spectral model of Jimenez et al. [JSB*10] (FIG. 3A), and using the complete 4D model (FIG. 3C) which enables a much closer match to the photograph.

(6) FIG. 4A and FIG. 4B: (FIG. 4A) Spectral power distributions of the 6 different LEDs installed on the multispectral LED sphere. (FIG. 4B) Spectra of D65′ and D65′+blue illumination, jointly used to isolate the blue response (Section 4.2).

(7) FIG. 5A to FIG. 5F: Proposed practical spectral measurements of skin reflectance (FIG. 5A, FIG. 5B). (FIG. 5A) Broadband D65 metamer (D65′) illumination. (FIG. 5B) Mix of D65′+blue LED illumination. (FIG. 5C) Synthesized response to pure blue LED illumination, and its isolated blue channel data (FIG. 5F) employed for parameter estimation. (FIG. 5D) Direct measurement of skin response to blue illumination, and its isolated blue channel data (FIG. 5E).

(8) FIG. 6A and FIG. 6B: (FIG. 6A) The chromaticity of the narrow band blue LED lies outside both the sRGB ITU-R BT.709 and Adobe RGB colour spaces. Mixing the narrow band blue LED with a broadband illumination (D65′ or W57) reports the chromaticity of the illumination within both gamuts. (FIG. 6B) Additional advantage is that the spectrum of the synthesized blue is narrower than the actual blue LED, contributing to sharp details correlated with melanin concentration.

(9) FIG. 7A to FIG.7H: Estimated spectral parameters for subject's face (FIG. 7A to FIG. 7D), and comparison of photographs with reconstructions under D65′ metamer broadband illumination (FIG. 7E, FIG. 7F), and warmer W27 LED illumination not employed for parameter estimation (FIG. 7G, FIG. 7H).

(10) FIG. 8A to FIG. 8H: Estimated spectral parameters (FIG. 8A to FIG. 8D) for faces of two subjects with different skin types, and comparison (FIG. 8E TO FIG. 8H) of photographs with reconstructions under different types of broadband illumination. Measurement using D65′ for broadband illumination.

(11) FIG. 9A to FIG. 9E: Measurement of a subject's cheek with the Antera 3D® (FIG. 9D) and example of acquired chromophore maps (FIG. 9A to 9C). Photo-graph of the device's multi-LED illumination setup surrounding the camera lens (FIG. 9E).

(12) FIG. 10A to FIG. 10L: Estimating skin patch parameters using Antera data. Parameters obtained by adapting and augmenting Antera's maps (FIG. 10E to FIG. 10H) exhibit higher physiological correlation and lower noise than parameters estimated using a full 4D search based on the albedo (FIG. 10I to 10L). However, reconstruction with the latter approach is a closer match to the albedo map.

(13) FIG. 11A to FIG. 11B: (FIG. 11A) Spectrum of Antera's LEDs estimated using proposed genetic programming based optimization. (FIG. 11B) Validation of estimation procedure on D65′ spectrum achieved with the LED sphere.

(14) FIG. 12: Pipeline for neural parameter estimation and reconstruction from our measurements.

(15) FIG. 13A to FIG. 13H: Spectral parameters for subject's face estimated by the cascaded feed-forward MLP (FIG. 13A to FIG. 13D). The bottom row shows comparison of photographs with reconstructions under D65′ metamer broadband illumination (FIG. 13E, FIG. 13F), and warmer W27 LED illumination not employed for parameter estimation (FIG. 13G, FIG. 13H).

(16) FIG. 14A to FIG. 14K: Estimating skin patch parameters using the cascaded MLP network on Antera data. Parameters obtained by adapting and augmenting Antera's maps (FIG 14D to FIG. 14G) and from albedo only input (FIG. 14H to FIG. 14K). As with the look-up based search, reconstruction with the latter approach is a slightly closer match to the input albedo.

(17) FIG. 15A to FIG. 15F: Comparison of reconstruction with D65′ vs W57 for broadband measurement. Top-row: Reconstructions (FIG. 15A and FIG. 15C) under D65′ illumination compared to photograph (FIG. 15B). Bottom-row: Reconstructions (FIG. 15D and FIG.15F) under W57 illumination compared to photograph (FIG. 15E). Left column: Reconstructions (FIG. 15A and FIG. 15D) with maps estimated with our preferred D65′ for broadband illumination. Right column: Reconstructions (FIG. 15C and FIG. 15F) with maps estimated with W57 (cool white LED) for broadband illumination.

(18) FIG. 16A to FIG. 16D: Examples of physiologically based edits to Antera's adapted 4D parameters (FIG. 16A) producing realistic tanning (FIG. 16B), drained (FIG. 16C), and flushed (FIG. 16D) appearance.

(19) FIG. 17A to FIG. 17H: D65′ and synthesized blue facial textures (FIG. 17A, FIG. 17B), and the estimated chromophore maps (FIG. 17C, FIG. 17D, FIG. 17E, FIG. 17F) mapped onto UV texture-space of a face scan. Reconstructed facial geometry (FIG. 17G), and diffuse-only rendering with heterogeneous subsurface scattering (FIG. 17H).

(20) FIG. 18A to FIG. 18F: Comparison of photographs of subject lit with cross-polarized point light source with 6 different LED spectras (top-row), against renderings driven by spectral parameters estimated using two complementary measurements (bottom-row).

(21) FIG. 19A to FIG. 19D: Quality of D65 metamers created by various combinations of LEDs on the LED sphere. (FIG. 19A) Ideal metamer created when combining all 6 LEDs. (FIG. 19B) Employing only cool white LED (W57) in combination with the RGB LEDs also achieves a decent quality metamer. The quality drops off when combining RGB LEDs with one of the warmer white LEDs (FIG. 19C, FIG. 19D).

(22) FIG. 20A to FIG. 20C: Comparison of facial photographs under uniform broad-band (W57 cool white LED) illumination (FIG. 20B) to reconstructions using the reduced 2D spectral model of Jimenez et al. [JSB*10] (FIG. 20A), and using the complete 4D model (FIG. 20C).

(23) FIG. 21A to FIG. 21H: Estimated spectral parameters for a face of a Caucasian male subject, and comparison of photographs with reconstructions under different types of broadband illumination. Measurement using W 57 cool white LED for broadband illumination.

(24) FIG. 22: Spectra of W57 and W57+blue illumination, jointly used to isolate the blue response.

(25) FIG. 23A to FIG. 23F: Proposed practical spectral measurements of skin reflectance (FIG. 23A, FIG. 23B). (FIG. 23A) Broadband W 57 illumination. (FIG. 23B) Mix of W57+blue LED illumination. (FIG. 23C) Synthesized response to pure blue LED illumination, and its isolated blue channel data (FIG. 23F) employed for parameter estimation. (FIG. 23D) Direct measurement of skin response to blue illumination, and its isolated blue channel data (FIG. 23E).

(26) FIG. 24A to FIG. 24C: Skin reflectance spectrum (from [DJ06]) and its response under: (FIG. 24A) Flat illumination spectrum. (FIG. 24B) D50 spectrum (employed by [JSB*10]). (FIG. 24C) D65 spectrum.

(27) FIG. 25A to FIG. 25D: Estimated spectral parameters for a palm. Top-row: Using just broad band W57 LED illumination. Center-row: Estimation using D65 metamer (D65′) broadband illumination. Bottom-row: Joint-estimation using D65′ broadband+narrow band blue LED illumination.

(28) FIG. 26A to FIG. 26D: Photographs (top-row) vs reconstructions (bottom-row) of the palm under uniform illumination with four different spectra. The reconstructions use the estimated parameters in FIG. 6 (bottom-row).

(29) FIG. 27A to FIG. 27D: Comparison of estimated spectral parameters for the palm. Top-row: Using broad band W57+narrow band blue LED illumination. Bottom-row: Joint-estimation using D65′ broadband+narrow band blue LED illumination.

(30) FIG. 28A to FIG. 28C: Additional examples of skin patch measurements with Antera. Center-left: Initial 2D mapping of Antera's parameters. Center-right: Adapting and augmenting Antera's parameters for the skin model. Right: Parameter estimation using full 4D search.

(31) FIG. 29A to FIG. 29C: Photographs of a female subject acquired under normal (FIG. 29A), and flushed (FIG. 29B) skin conditions. Reconstruction of simulated flushing (FIG. 29C) using spectral parameters estimated under nor-mal condition and applying scaling to the C.sub.h and C.sub.he parameters.

(32) FIG. 30A to FIG. 30F: Renderings of acquired faces (top and center row) and a patch of skin (bottom-row) under a point light source with six different LED spectrums (W57, W40, W27, Red, Green and Blue).

(33) FIG. 31A to FIG. 31C: Comparison of our proposed renderings of subsurface scattering with spectral parameters estimated using two complementary measurements (FIG. 31A), against photographs of two different subjects lit with a cross-polarized point light source (W40 LED) (FIG. 31B). (FIG. 31C) Comparison renderings with spectral parameters estimated using a single broadband measurement under D65′.

(34) FIG. 32A to FIG. 32B: Failure cases for our proposed 4D model and measurement approach. Top-row: Veins in the inner forearm. Bottom-row: Tattoo on the back of neck.

DETAILED DESCRIPTION

(35) In this work, we propose a novel practical spectral skin measurement approach (using an LED sphere) that, in conjunction with driving the spectral skin BSSRDF of [JSB*10] with appropriate model complexity, is suitable for facial capture of live subjects with realistic spectral appearance reproduction (Section 4). The acronym BSSRDF stands for Bidirectional Scattering Surface Reflectance Distribution Function We demonstrate that the proposed model complexity involving four parameters (melanin C.sub.m and hemoglobin C.sub.h concentration, melanin blend-type fraction β.sub.m, and epidermal hemoglobin C.sub.he fraction) is required to match subject appearance in photographs, which may not be possible using a reduced model with just two free parameters (melanin and hemoglobin), and our proposed measurement protocol combining two complementary broad and narrow-band spectral illumination conditions provides higher quality estimates of spectral parameters than those obtained using just broadband illumination. Additionally, we demonstrate how to acquire ideal broadband and narrowband illumination measurements for parameter estimation within the practical constraints of LED illuminants and regular color cameras. Thus, as our primary contribution, we propose a minimal measurement and modeling complexity for data-driven reproduction of spatially varying spectral appearance of skin, including human faces.

(36) We also demonstrate how practical measurements with a hand-held off-the-shelf skin measurement device designed for dermatological applications (a Miravex Antera3D camera) can be adapted for realistic skin appearance reproduction and rendering (Section 5). Here, we demonstrate how to appropriately transform the output pigmentation and redness maps produced by the device into melanin and hemoglobin concentrations respectively, and augment them with additional model parameters (β.sub.m and C.sub.he) which are not provided by the device. Additionally, we demonstrate how neural networks can be employed for faster, improved parameter estimation given our measurements (Section 6). Finally, we demonstrate realistic rendering of subsurface scattering with our estimated parameters (in PBRT) using spatially-varying diffusion profiles (Section 7), achieving renderings of human faces using a biophysically based spectral BSSRDF that are, for the first time, comparable to photographs (see FIG. 1A to FIG. 1C).

(37) To summarize, our central high-level contributions in this work are as follows: Practical measurement of spectral skin reflectance suitable for facial capture in conjunction with appropriate model complexity of spectral BSSRDF for matching skin appearance in photographs. Adaption and augmentation of chromophore maps obtained from a hand-held dermatological skin measurement device for realistic rendering. Novel estimation of parameters from our measurements using neural networks, which is significantly faster than a look-up table search along with reduced quantization.

(38) We additionally propose the following practical contributions for spectral measurements with a color camera: An optimization procedure for combining a set of broad- and narrow-band LED illuminants to construct a metamer for desired D65 illumination. Novel indirect measurement of narrow-band LED response that enables higher quality measurement with sharper spectral isolation than direct measurement with a color camera. A genetic programming algorithm for estimation of unknown illumination spectra from a single color chart observation.

3. SPECTRAL BSSRDF MODEL COMPLEXITY

(39) We aim to drive a spectral skin BSSRDF model with practical measurements and hence prefer a model with an appropriately minimal complexity to simplify measurements while simultaneously having sufficient complexity to match the observed spatial variation in skin, particularly facial appearance. In this respect, we aim for a data-driven modeling of skin and facial appearance rather than striving for strict bio-physical accuracy of the estimated parameters. We choose the model of Jimenez et al. [JSB*10], originally proposed for facial measurements, as the starting point for our work. The model includes the following four parameters: melanin concentration (C.sub.m) in epidermis, melanin blend-type fraction β.sub.m (blend between eumelanin and pheumelanin), and hemoglobin concentration (C.sub.h) in dermis, and epidermis (C.sub.he), respectively (please see Supplemental material for details). However, unlike Jimenez et al., we have empirically found that all four parameters in the model need to be varied over the skin surface in order to closely match the appearance of real skin. Variation in β.sub.m is particularly useful for reconstructing facial appearance variation due to facial hair and around eyelids, while a higher fraction of epidermal hemoglobin C.sub.he is necessary to match the very reddish areas of a face such as the lips and cheeks. Hence, we allow β.sub.m to vary between 0.0 and 1.0, and C.sub.he to vary between 0.0 and 0.6. We also set epidermal thickness d to 0.33 mm instead of 0.25 mm suggested in previous work in order to better match the appearance of subjects with the above spectral model.

(40) This leads to a 4D spectral skin appearance model. In practice, in order to restrict the search space of the various parameters for model-fitting, we employ very coarse discretization for β.sub.m (10 bins) and C.sub.he (4 bins) parameters which have a more subtle effect on the overall appearance, while employing a large number of bins to model the dominant variation in C.sub.m and C.sub.h. FIG. 2A to FIG. 2D shows the variation due to the two dominant parameters as a 2D LUT, while showing more subtle variation due to the β.sub.m parameter across (FIG. 2A, FIG. 2B) (for fixed C.sub.he=0.25), and variation due to the C.sub.he fraction parameter across (FIG. 2C, FIG. 2D) (for fixed β.sub.m=0.5).

(41) FIG. 3A to FIG. 3C shows comparison of a facial photograph under uniform broadband illumination, and its reconstruction using the reduced 2D model employed by Jimenez et al. [JSB*10] with only two free parameters (C.sub.m and C.sub.h) (FIG. 3A), and reconstruction using the complete 4D model with two additional free parameter (β.sub.m, and C.sub.he) (FIG. 3C). As can be seen, the 4D model complexity allows a closer match to the spatially varying appearance of skin in the photograph. We provide additional comparisons in the Supplemental material.

4. PRACTICAL SPECTRAL ACQUISITION FOR FACES

(42) We now present our practical measurement protocol for robustly estimating the four parameters of the BSSRDF model with a minimal set of measurements suitable for facial capture. We employ a multispectral LED sphere equipped with a combination of narrow band Red, Green, and Blue LEDs, and three types of broad band LEDs (warm 2700K, neutral 4000K, and cool 5700K which we refer to as W27, W40, and W57 respectively), and 9 color DSLR cameras (Canon 800D) for multiview acquisition of a subject. The LEDs on the sphere are all cross-polarized w.r.t. the cameras, allowing specular cancellation according to the method of [GFT*11]. We also measured the individual spectral distributions of our illuminants using a spectrometer (Sekonic SpectroMaster C700) placed at the center of the LED sphere (see FIG. 4A). With this setup, we simultaneously record the diffuse reflectance response of a subject from multiple viewpoints under specific uniform spectral illumination conditions as described next.

(43) 4.1. Measurement Protocol

(44) With this setup, when we restricted ourselves to a single observation (as a baseline), we found the best individual LED illumination on our LED sphere for estimating model parameters to be uniform W57 (cool white LED) illumination (see FIG. 3B). Here, W57 illumination provided the highest color contrast in the skin reflectance for parameter estimation using CIELAB space color matching given a pre-computed 4D look-up table for the skin model under W57 illumination (more details in Supplemental material). Our choice of W57 broadband illumination is also supported by the study of Preece & Claridge [PC04] who found a peak around 560 nm to be useful for measurement of hemoglobin concentration, and a peak around 485 nm to be useful for measurement of melanin. Consistent with this study, we found improved parameter estimation when using two complementary spectral illumination conditions: a combination of broad band+narrow band blue LED illumination (480 nm peak response). The premise here is that the blue illumination primarily only excites epidermal reflectance [PC04], while broadband illumination excites both epidermal and dermal reflectance. The response to narrow-band blue illumination exhibits sharper skin texture (melanin response) due to reduced scattering of the wavelength compared to broadband illumination. This enables us to estimate sharper descattered parameter maps when jointly employing the two complementary illumination conditions for LUT search. Furthermore, instead of making direct measurements under W57 and blue LED illumination respectively, we instead employ the following procedure:

(45) Ideal broadband measurement: Given the 6 types of LEDs in our LED sphere, we instead create a more ideal broadband illumination by computing a weighted combination of all 6 LEDs to create a D65 metamer spectra (D65′) which we instead employ for our broadband measurements. We notice an even higher contrast in skin color, particularly coloration due to skin pigmentation and redness, under the D65 metamer illumination and we make the observation that D65 spectrum (blue dominant) balances the somewhat skewed red-dominant reflectance spectrum of human skin (see Supplemental material). This is consistent with studies on the human visual system [CXW19] where D65 spectrum has been reported to be most desirable for discerning differences in skin color. Note that true D65 illumination is not possible with the LEDs on our LED sphere. Instead, the D65 metamer is created to be an approximation of the ideal D65 spectra achieved by the available LEDs such that it minimizes the color difference between the 24 color patches measured on an Xrite color chart vs the reference 24 colors on an ideal color chart in sRGB color space (which assumes ideal D65 spectrum). Our computed metamer spectrum can be seen in comparison to the ideal D65 spectrum in FIG. 4B. We provide more details on how to compute the D65 metamer in Appendix A.

(46) Narrow-band measurement: Directly recording the reflectance response of a subject under blue LED illumination unfortunately results in some colors being outside the gamut of most off-the-shelf color cameras, which typically work in the sRGB and Adobe RGB colour spaces, with the chromaticity of the narrow band being noticeably outside both colour spaces (FIG. 6A). This results in suboptimal narrow-band measurements. Instead, we propose a novel procedure to overcome this gamut limitation of regular color cameras by instead recording a mix of broadband and blue LED illumination during measurement, and given another measurement under uniform broadband illumination, we can then computationally isolate the narrow band response to just blue LED illumination afterwards as described next.

(47) 4.2. Isolation of Blue Response

(48) We propose an indirect scheme in order to measure skin response under blue LED illumination. We capture two photographs of a subject, the first one under broadband lighting (I.sub.W) and the second one under broadband plus blue LED (I.sub.W+nb). We then separately apply to both the images a Chromatic Adaptation Transform (CAT) (defined in Appendix B), to predict colours appearance under D65 lighting and within the sRGB gamut, and computationally recover the desired narrow band response I.sub.nb as follows:
I.sub.nb=δ×(CAT(I.sub.W+nb)−CAT(I.sub.W)/δ),  (1)

(49) where the factor δ accounts for the difference in intensities of the LEDs in the two conditions.

(50) In our case, we preferably employ D65 metamer illumination for the broadband measurement (I.sub.W=I.sub.D65′) which greatly simplifies its CAT transform to identity matrix. However, the transform is general and can be employed with any broadband illumination (e.g., W=W57 cool white LED) for computing the spectral isolation as per Equation 1. The above scheme offers an additional advantage of actually measuring the skin response under a slightly narrower band than the one offered directly by blue LED (FIG. 6B), further contributing to sharper details in the measurement.

(51) The capture process can be visually seen in FIG. 5A to FIG. 5F where the response of a subject's face to our proposed two complimentary uniform spectral illumination conditions (FIG. 5A, FIG. 5B) are shown in the top row. Afterwards, we can computationally synthesize the response to pure blue illumination as shown in (FIG. 5C). Note that only the blue channel of this synthesized image is relevant for our analysis, while the red and green channels of (FIG. 5C) are residuals of CAT transform after white-balancing, which is why they are ignored. The isolated blue measurement from the synthesized image can be seen in grayscale in (FIG. 5F) showing skin texture and blemishes in sharp detail which is very correlated with skin melanin concentration. FIG. 5D instead shows the subject's direct response to pure blue LED illumination as recorded by the color camera, where some spatial details of the skin texture are lost in the isolated blue component FIG. 5E of image FIG. 5D. We include similar example images for a subject acquired using W57 LED for the broadband condition in the Supplemental material. Note that our preferred choice of broadband illumination (D65′) and indirect measurement of the narrow-band illumination also has a practical advantage of employing all types of LED illuminants in our measurement setup, thereby creating much brighter lighting conditions for faster higher quality measurements compared to when employing a single type of LED.

(52) 4.3. Results

(53) With the measurement protocol described in Section 4.1, we do a joint look-up table search for best matching color values (in CIELAB space) under simulated D65 metamer illumination (FIG. 5A), and best matching synthesized blue channel response under simulated blue LED illumination (FIG. 5F). FIG. 7A to FIG. 7H (top-row) presents the various parameter maps (C.sub.m, β.sub.m, C.sub.h, C.sub.he) estimated for a subject's face using the above acquisition and fitting procedure. Here, we are visualizing the total C.sub.he fraction and not as a fraction of C.sub.h. FIG. 7A to FIG. 7H (bottom-row) presents comparisons of photographs of the subject and our reconstruction with the estimated parameters under D65 metamer illumination and a warmer spectral illumination condition (W27) which was not employed in parameter estimation. A similar comparison under W57 illumination can be seen in FIG. 3A to FIG. 3C.

(54) FIG. 8A to 8H presents the estimated parameter maps using our pro-posed two complementary spectral measurements, and comparison of spectral reconstructions to photographs for two subjects with different skin types. These include a male subject with Asian skin type, and a Caucasian female subject with a pale skin type whose measurements we acquired using the D65 metamer (D65′) as the broadband illumination. As can be seen, the reconstructions are a close match to the photographs under different types of illumination spectrum. The estimated parameters maps not only well reconstruct the appearance of skin well under cooler illumination spectrum, they also appropriately predict the softening and blurring of the skin texture seen under the warmer W27 illumination. The Supplemental material includes an additional example of a Caucasian male subject whose measurement was acquired using the cool white LED (W57) as the broadband illumination, as well as results for a palm of a hand. We provide additional analysis in Section 7.

5. PRACTICAL SKIN MEASUREMENTS WITH ANTERA3D

(55) The focus of the previous section was on practical spectral measurements of skin in a controlled setup suitable for facial capture. For more free-form measurement of skin, we employ a hand-held off-the-shelf device—Antera 3D© (Miravex Limited, Ireland), a camera for image acquisition and corresponding software for analysis of single skin patches (56×56 mm.sup.2). This is an instrument employed in dermatology: it has been compared with most commonly used devices in dermatological research and is reported to be robust, sensitive and precise for skin colour analysis [MFCN15, LWA*18]. For a single measurement, the camera is placed onto a skin patch without applying excessive pressure. The typical measurement procedure along with sample results for a cheek patch is shown in FIG. 9A to FIG. 9E. Any area of the human body can be captured in less than 2 seconds at a resolution of 0.1 mm. The acquisition procedure with the device is as follows: The device consists of a camera and various LEDs covering seven different spectral bands (narrow- and broad band) in a single self-contained unit. During acquisition, the device illuminates the skin patch sequentially with its spectral LEDs from different angles. The obtained reflectance data are transformed by the vendor's proprietary software into skin absorption coefficients and used to quantify chromophore concentrations using mathematical correlation with known spectral absorption data of hemoglobin and melanin. The software provides spatially varying maps of these two chromophore concentrations as well as corresponding 3D surface geometry of the skin patch which is estimated using photometric stereo. The device also provides a color (albedo) map of the skin surface estimated using the acquisition process.

(56) This is a good starting point for employing the data for spectral rendering of skin. However, the device does not capture all of the parameters we have identified as necessary for reproduction of skin appearance. Importantly, for the two parameters that are provided, the device provides chromophore concentrations in terms redness and pigmentation which do not directly map as C.sub.m and C.sub.h for the BSSRDF model. Hence, we have to undertake a number of steps in order to adapt the Antera measurements for driving the skin appearance model.

(57) 5.1. Parameter Remapping and Estimation

(58) Given that Antera estimates only the two primary parameters (pig-mentation and redness) related to melanin and hemoglobin concentration, in a first step we adapt these maps based on Jiminez et al.'s reduced 2D model of skin appearance. In order to do this, we employ the albedo map provided by Antera and estimate corresponding C.sub.m and C.sub.h for the Jimenez model using the same look-up table search procedure (in CIELAB space) described in the previous section. Given our best fit to the 2D model, we then scale antera's maps for pigmentation and redness in an appropriate manner to match the mean and variance of our estimated C.sub.m and C.sub.h parameters (using color space matching), and set these scaled pigmentation and redness maps as our final estimate of C.sub.m and C.sub.h for reconstruction. FIG. 10B shows reconstruction for the measured cheek patch using this 2D remapping procedure using the reduced Jimenez model.

(59) Note that this above remapping step requires us to simulate a 2D lookup table (fixed β.sub.m and C.sub.he) under the same illumination spectra employed for measuring Antera's albedo map. However, this information of the illumination spectrum for the Antera albedo is not provided by the device or the vendor and, being a proprietary device, we have no control over its LED illumination system in order to make a direct measurement of the appropriate LEDs: the device cycles through all the LEDs very rapidly, making direct measurement of spectrum (e.g., with a spectrometer) difficult. Hence, we estimate the unknown illumination spectrum for the albedo measurement using corresponding measurements of color squares on an Xrite color chart and then solve for the illumination spectrum (see FIG. 11A) using a genetic algorithm based optimization detailed in Appendix C. We note that the recovered spectrum for Antera's albedo seems to be an approximation of D65 illumination obtained with the LED illuminants on the device. In this respect, it is conceptually very similar to the D65 metamer illumination we create using the LEDs on our LED sphere. Given that we lack the ground truth for Antera's illumination profile, we indirectly validated the spectrum estimation process using a color chart observation under D65 metamer illumination using the LED sphere (see FIG. 11B).

(60) After recovering the illumination spectrum for Antera's albedo, we can remap Antera's chromophore maps to the appropriate scale using a 2D look-up table based on the reduced Jimenez model. We then fix the C.sub.m and C.sub.h parameters and then re-fit the albedo data to the complete 4D model, this time searching for appropriate values of β.sub.m and C.sub.he parameters in a 4D look-up table for fixed values of C.sub.m and C.sub.h. FIG. 10C shows reconstruction for the measured cheek patch using this procedure which augments Antera's maps with the two additional parameters required by the 4D model. As can be seen, the reconstruction result of this remapping and augmentation step is a closer match to the albedo map than just remapping the data to the 2D model. We also experimented with directly estimating all four parameters for the model using the albedo map which results in an even closer match for the reconstruction to the measured albedo map (see FIG. 10D). However, the parameter maps estimated using just the albedo map are more noisy and show less physiologically correlated structure than the parameters obtained by remapping and augmenting Antera's maps. This is consistent with our findings in the previous section on practical skin measurements with two complementary spectral illumination conditions. Thus, we believe that parameter maps directly estimated using only the albedo data may be somewhat overfitting to the input data (hence lower reconstruction error) and suffer from some baked-in subsurface scattering visible in the albedo, while the maps estimated by adapting and augmenting Antera's measurements seem to have a higher physiological correlation and hence likely to better generalize to other spectral conditions or for predicting changes in physiological state of skin. We provide additional results for acquired skin patches in Section 7 and Supplemental material.

6. NEURAL PARAMETER ESTIMATION

(61) Thus far, we described how to estimate the spectral parameters of skin from our measurements using a look-up table search (for best matching color values in CIELAB space). This process is slow and the results can be prone to image noise and quantization due to discrete values in the look-up table. Hence, instead we also explored a neural prediction approach for obtaining the spectral parameters from our measurements using a cascaded feed-forward multilayer perceptron (MLP) architecture (see FIG. 12). We note some conceptual similarity in our approach with that of [YSJR17] who employed MLPs for parameter conversion from one rendering model to another (volumetric scattering to dipole diffusion) for global scattering in hair/fur fibers.

(62) Each of the 4 parameters of our model is estimated by a different MLP (see FIG. 12. From left to right C.sub.mNNET, β.sub.mNNET, C.sub.heNNET and C.sub.hNNET), which all share the input from an RGB image, either from our acquisition setup under broadband illumination (D65′ or W57) or the albedo map from Antera. In the cascaded order, C.sub.mNNET estimates C.sub.m from the RGB input, and provides the prediction in input to all the subsequent MLPs; similarly, β.sub.mNNET provides its output to C.sub.heNNET and C.sub.hNNET, and so on. The last MLP in the cascade sequence, RGBAlbedoNNET, takes in input the estimated C.sub.m, β.sub.m, C.sub.he and C.sub.h to predict the RGB albedo, ideally matching the input photograph.

(63) To achieve higher accuracy, when the RGB input photograph is provided by our LED sphere setup, C.sub.mNNET and β.sub.mNNET can also take in input the synthesized response to pure blue LED illumination. Additionally, if the RGB input is provided by Antera, its remapped C.sub.m and C.sub.h measurements can be used to bypass respectively C.sub.mNNET and C.sub.hNNET.

(64) 6.1. Design and Training of the MLPs

(65) The first layer of each MLP is the input layer, followed by two hidden layers (L1 and L2) and the output layer. It is well know that two-hidden-layer feedforward networks can approximate complex nonlinear mappings with arbitrary accuracy given enough nodes in the hidden layers and training data. In our pipeline, we train the MLPs on synthetic RGB data, i.e the 4D LUTs provided by the spectral skin reflectance model, augmented by zero-mean white Gaussian noise to simulate photon shot noise. The noise variance is directly estimated from homogeneous areas in a photograph of a color chart. We then analyze the effect of the noise on the estimated parameters in order to derive a suitable noise model to inject in the skin model parameters used to train the MLPs.

(66) To determine the number of nodes N.sub.L.sub.1 and N.sub.L.sub.2 to use in each hidden layer, we observe that the upper bound of the number of nodes per layer can be computed through the following equations [Gua03], where m is the number of output nodes (m=1 for C.sub.mNNET, β.sub.mNNET, C.sub.heNNET and C.sub.hNNET, while m=3 for RGBAlbedoNNET) and Nis the number of distinct training samples (i.e. the size of the LUTs):

(67) $\begin{array}{cc}{U}_{N_{L_1}}=\sqrt{\left(m+2\right)N}+2\sqrt{\frac{N}{m+2}}& \left(2\right)\\ U_{N_{L_2}}=m.Math.\sqrt{\frac{N}{m+2}}.& \left(3\right)\end{array}$

(68) Since such upper bounds might overfit the input data [Gua03], to avoid overfitting and to increase the generalization capabilities of our MLPs we set N.sub.L.sub.1=U.sub.N.sub.L1/k and N.sub.L.sub.2=U.sub.N.sub.L/k, with k≈100, and train the networks using Bayesian regularization backpropagation [Mac92]. Finally, we use a hyperbolic tangent function as activation function.

(69) We train the parameter networks C.sub.mNNET, β.sub.mNNET, C.sub.heNNET and C.sub.hNNET once for each input type (e.g., broadband-only or broadband+synth. blue) with D65′ as the broadband spectrum for LED sphere data, and the recovered Antera spectrum as the broadband for Antera data. However, the RGBAlbedoNNET used for albedo reconstructions from the input parameters is specific for a given illumination spectrum, and needs to be trained separately for different target illumination spectras.

(70) 6.2. Results

(71) FIG. 13A to FIG. 13H presents spectral parameter maps and albedo reconstructions under two uniform spectral illumination conditions, achieved with the above described cascaded MLP network for the subject shown earlier in FIG. 7A to FIG. 7H. As can be seen, both the estimated parameters and the RGB albedo reconstructions are very comparable to that achieved using look-up table search, with the added advantage of reduced noise and quantization, as well as significantly faster parameter estimation and reflectance reconstruction (3.5 seconds for MLP network vs 40 minutes with LUT search, i.e., >680× speedup for 2K resolution on a laptop with 2.7 GHz Intel Core i7 processor and 16 GB RAM).

(72) FIG. 14A to FIG. 14K demonstrates the cascaded MLP network on the input provided by Antera on the cheek patch shown earlier in FIG. 10A to FIG. 10L, either using its C.sub.m and C.sub.h remapped measurements to bypass C.sub.mNNET and C.sub.hNNET, thus only estimating β.sub.m, C.sub.he and albedo (FIG. 14A to FIG. 14K, second row), or directly estimating all four parameters of the model using the albedo map (FIG. 14A to FIG. 14K, third row). As can be seen, the parameters estimated with the MLP network are qualitatively very similar to those obtained with look-up table search shown in FIG. 10A to FIG. 10L, with the advantage of much faster estimation. Similar to the look-up table search, estimating the parameters directly from the Antera photograph results in a slightly closer match of the reconstruction to the input albedo map compared to when employing the Antera adapted C.sub.m and C.sub.h parameters.

7. ADDITIONAL RESULTS AND RENDERING

(73) We now present additional set of results with our proposed practical spectral measurements for faces, and skin-patch measurements with the Antera3D device. FIG. 15A to FIG. 15F compares the quality of spectral reconstructions with parameters maps estimated when employing our preferred D65 metamer for the broadband measurement (left-column) vs parameter maps estimated when employing the baseline W57 cool white LED illumination for the broadband measurements (right-column). As can be seen, measurements of the spectral parameters using D65′ illumination is able to well predict the appearance under W57. However, when the parameters are estimated using W57 illumination, the predicted appearance under D65′ is slightly less accurate, resulting in more saturated colors. This supports our preference for the D65′ illumination as the broadband condition for our skin measurements. However, as also shown in FIG. 8A to FIG. 8H, our approach still enables parameter estimation of sufficient quality even when employing just two types of LEDs (W57 and blue) for the broadband and narrow-band measurements.

(74) FIG. 16A to FIG. 16D presents examples of physiologically based edits performed on Antera data (a forehead patch). Here, we employ the parameters maps obtained after adapting and augmenting the Antera data to show the original reconstruction of skin appearance (FIG. 16A), followed by simulated tanning (FIG. 16B) by scaling up the C.sub.m parameter by 1.1 in addition to an offset of 0.08 units in cubic root space. Similarly we simulate pale drained appearance in (FIG. 16C) by scaling down the original C.sub.h parameter by 1.5 in cubic root space. Finally, flushed skin is simulated in (FIG. 16D) by scaling up the original C.sub.h parameter by 1.1. As can be seen, such simple edits to the adapted Antera maps achieve very realistic physiological changes in skin appearance. We provide a similar example of physiologically based edit for a face in the Supplemental material.

(75) We note that parameters estimated using a single broadband measurement of the albedo encodes some amount of subsurface scattering in the parameter maps due to subsurface scattering being baked in the albedo (also observed by Jimenez et al. [JSB*10]). Since our look-up table based reconstruction of the albedo does not do explicit simulations of subsurface scattering, we actually found closer matches (in CIELAB space) to the input photographs when reconstructing the albedo using parameters estimated under a single broadband condition compared to our proposed approach of combining the broadband measurement with narrow-band blue response (also true for Antera data where full 4D search results in better match to input photograph). The reconstructions with our proposed measurements are slightly sharper, with less baked-in subsurface scattering compared to the input photograph which we attribute to the sharp descattered measurement of the narrow-band response. This is actually a desirable outcome for rendering of subsurface scattering with the measured parameter maps. Jimenez et al. employed their measured maps to only reconstruct the albedo which they employed as a modulation texture to a homogeneous subsurface rendering with a fixed diffusion profile. In contrast, our estimated maps allow us to render subsurface scattering with spatially varying albedo and diffusion profiles as explained next.

(76) 7.1. Rendering Subsurface Scattering

(77) Inspired by the work of Donner et al. [DWd*08], we employ our estimated parameters to render heterogeneous subsurface scattering in skin. We employed PBRT v2 [PH10] to generate renderings with pseudo-heterogeneous subsurface scattering driven by our estimated spatially varying parameters of the spectral BSSRDF model. Given a facial scan of a subject obtained with multiview acquisition in the LED sphere (we employ COLMAP [SF16, SZPF16] for the base geometry reconstruction), we first project the input data of broadband D65′ response (which is also the input to COLMAP for geometry reconstruction) and synthesized narrow-band blue response into the UV texture-space of the face scan (see FIG. 17A, FIG. 17B). We then compute the four spectral parameters of the skin BSSRDF in this UV texture-space and project it onto the base geometry within PBRT for rendering with the skin coloration model.

(78) In order to render heterogeneous subsurface scattering, we modified the provided subsurface scattering implementation in PBRT from the default dipole diffusion kernel to our specified spatially varying profiles implementing two-layered diffusion. For each sampled color generated by the skin model under chosen illumination spectrum, we first pre-compute corresponding spectral reflectance and transmittance profiles for epidermis and dermis separately. We use a dipole model for dermis and a multipole model for epidermis as suggested by [DJ06]. We then convolve these spectral profiles according to Kubelka-Munk formula [DJ05] and store the overall radial reflectance profile function in linear RGB color space. At each point on the surface, PBRT framework extracts the corresponding precomputed spectral reflectance profile from the tabulated set and integrates it over a given radial distance and performs the same operation for all other sampled points on the geometry to add contribution from different spectral profiles, thereby rendering heterogeneous subsurface scattering (more details in Supplemental material). While we convert the spectral profile contributions after integration to RGB within PBRT, it is possible to pre-compute the profiles in sRGB space for usage in a standard RGB rendering pipeline.

(79) FIG. 17H shows such a diffuse-only rendering of spatially-varying subsurface scattering for a male subject lit with a point light source (D65′ spectrum). FIG. 1A to FIG. 1C (top-row) presents renderings with additional specular reflectance for the same subject, as well as an additional subject, and a skin patch measured using the Antera camera. Here, we render the specular reflectance as a separate layer in PBRT using a microfacet BRDF with Blinn-Phong distribution, and add it to the diffuse-only rendering as a post-process. We provide additional diffuse-only renderings of the subjects under the illumination spectras of the six types of LEDs in our LED sphere (FIG. 1A to FIG. 1C (bottom-row), also see Supplemental material).

(80) FIG. 18A to FIG. 18F presents comparisons of renderings of Subject 1 to photographs under each of the six types of LEDs on the LED sphere. Here, the subject was acquired on a different day compared to the data used for FIG. 1A, with noticeable difference in facial hair. As can be seen, renderings with the estimated parameters correctly predict the change in appearance across the six spectrums, with characteristic softening of skin texture under the warmer spectrums. We additionally provide qualitative comparisons (against photographs) of renderings with parameters estimated using our proposed two complementary measurements vs using a single broadband measurement for parameter estimation (which results in softer, blurry skin appearance) in the Supplemental material.

(81) Note that the facial geometry and spectral measurements for the face renderings were acquired using a multiview capture setup (9 DSLR cameras) placed around the LED sphere. For rendering subsurface scattering in PBRT v2, we had to significantly downsample the mesh vertices, thereby rendering a smooth base mesh seen in FIG. 17A to FIG. 17H. For rendering the specular layer in FIG. 1A to FIG. 1C, we compute shading based on a photometric normal map acquired using the LED sphere. The geometry for the back of the hand in FIG. 1A to FIG. 1C is provided by the Antera software which computes the shape of a relatively planar skin patch using photometric stereo which we directly employ for both the subsurface and specular rendering.

(82) 7.2. Limitations and Discussion

(83) The employed 4D skin BSSRDF model is well suited to reconstruct the appearance of skin and facial hair but has limitations and cannot well reconstruct the appearance of dominant veins or tattoos in skin (see Supplemental material). This is because veins and tattoos cannot be modeled with melanin and hemoglobin concentrations [DWd*08]. We currently do not model any fluorescence in skin, although our broadband measurements likely include some effects of dermal fluorescence [GZAK00]. Through our analysis, we show that parameter estimation using just a single broadband illumination is possible, although suboptimal. The quality and structural physiological correlation of the estimated parameters increases with multiple measurements. In this respect, our proposed two shot acquisition with the complementary spectral illumination conditions is a practical middle ground between the highly accurate measurements possible with detailed spectral imaging as demonstrated by [DWd*08] (also employed by the Antera camera), and just single broadband illumination previously employed for practical measurements of faces. Our choice of illuminants is also a function of the LEDs available in our facial capture setup, and the optimal choices may vary slightly for other spectral illumination setups. However, we demonstrate through our analysis the general trend of the desired illumination conditions for skin measurements, and a method for approximating desirable D65 spectrum using combination of available LEDs. Note that an LED sphere is not a strict requirement for the measurements since the method only requires uniform illumination, and hence can be adapted for many facial capture setups. When employing the Antera camera, we are rather restricted to only using the processed outputs of its proprietary software which does not allow access to the raw data of the spectral measurements for further analysis. However, we demonstrate how to adapt black-box measurements from such a custom dermatological scanning device for realistic rendering.

(84) For the LED sphere measurements, we currently estimate parameters assuming uniform illumination with no occlusions. However, for faces there is partial ambient occlusion around eye sockets and the nose that is baked into the measurements and by extension in our estimated parameter maps. We also do not currently explicitly account for the effect of exitant Frensel in these measurements which slightly affects the parameters estimated for surfaces seen at a grazing angle (most visible in the β.sub.m map).

(85) This is not really a problem for our 3D renderings which employ data seen from 9 different viewpoints and hence the facial parameter maps in the UV parameterization of the geometry is composed of mostly near normal incidence estimates. Finally, our look-up table based reconstructions reproduce the coloration of albedo texture under uniform illumination, but do not model lateral scattering of light within skin which requires explicit rendering of subsurface scattering.

8. CONCLUSION

(86) In summary, we have presented novel practical spectral measurements of skin reflectance using both a dedicated spectral illumination setup (LED sphere) and an off-the-shelf skin measurement device (Antera3D), and employed them to drive a spectral skin BSSRDF model with appropriate complexity to match the appearance of real skin. In this respect, our main contribution is proposing a sweet spot both for measurement and data-driven modeling complexity for reproducing the appearance of skin, including human faces. Our additional contributions include investigating desirable illumination spectra realizable with common LEDs, practical analysis of the gamut limitations of regular RGB color cameras for measuring response to narrow band LED illumination, and proposing a novel indirect measurement protocol that overcomes the gamut limitation and achieves improved spectral isolation compared to direct measurement with a color camera. We also demonstrate how to adapt practical hand-held physiological measurements from a dermatological skin measurement device to our application of realistic rendering which can have a significant impact for dermatological visualizations. Additionally, we also demonstrate how neural networks can be employed for much more efficient parameter estimation and spectral reconstructions given various types of measurements. We see this as a promising initial step towards efficient machine-learning based spectral skin rendering and diagnostics. We demonstrate highly realistic reconstructions of skin with our approach, including renderings of human faces using a biophysically based skin BSSRDF that are, for the first time, comparable to photographs. Future work in this direction could investigate practical measurements and modeling of changes in skin parameters due to skin dynamics or physiological factors of interest for medical diagnostics, as well as applications of any skin products.

Appendix A: D65 Metamer

(87) We address the problem of reproducing a desired spectral illumination (D65) using the LEDs in our LED sphere relying on the faithful reproduction of the appearance of a color chart. We first acquire a set of images of the color chart with known reflectance, individually under each of the n=6 LEDs. The measurements are taken at 3 different LED intensities, under uniform spherical illumination. Given the knowledge of the camera spectral sensitivity CSS, measured with a monochromator, this data allows us to recover the per-channel (ch) non-linear response of the camera sensor γ.sub.ch, and the LEDs relative intensities α.sub.k.

(88) This is similar to the approach of LeGendre et al. [LYL*16], except that instead of focusing only on the color chart appearance to a given camera, we also aim at maximizing the faithfulness of perceived color appearance to a human observer. The reason for simultaneously accounting for a percetual metric besides camera sensitivity in our optimization is that the Chromatic Adaptation Transform (Appendix B)

(89) $\begin{array}{cc}{w}_c{t}_c+w_h{t}_h& \left(4\right)\\ t_c=\underset{j=1}{\overset{N_p}{.Math.}}\underset{\mathrm{ch}=r,g,b}{\overset{}{.Math.}}.Math.{\mathrm{DG}}_{j,\mathrm{ch}}-{\left(\underset{k=1}{\overset{n}{.Math.}}{a}_k^{\prime }\underset{\lambda =380}{\overset{720}{.Math.}}{\mathrm{CSS}}_{\mathrm{ch},\lambda }{S}_{k,\lambda }{R}_{j,\lambda }\right)}^{{\gamma }_{\mathrm{ch}}}.Math.& \left(5\right)\\ t_h=\frac{1}{N_p}\underset{j=1}{\overset{N_p}{.Math.}}\Delta E_{2k}\left({\mathrm{Lab}}_{D65,j},\mathrm{XYZ}.fwdarw.\mathrm{Lab}\left(\left[X_{j,({\alpha }_k^{\prime }},Y_{j,{\alpha }_k^{\prime }},Z_{j,{\alpha }_k^{\prime }}\right]\right)\right);& \left(6\right)\end{array}$
employed for spectral isolation of narrow-band response, and look-up table search for the spectral parameters are all based on perceptual metrics. In order to find a spectrum which satisfies the above, given the weights w.sub.c and w.sub.h for the camera and perceptual terms respectively, we need to find a set of coefficients α*.sub.k, for the n=6 LEDs which minimizes the following equation:
where DLj, ch are the per-channel digital levels of the patch j of the color chart, Lab.sub.D65,j are the CIELab values of the color chart patches under D65 illumination, XYZ.fwdarw.Lab is a standard conversion using the D65 reference white [WS82], and [X.sub.j,α′.sub.k,Y.sub.j,α′.sub.k,Z.sub.j,α′.sub.k] are the XYZ tristimulus values of the color chart patches under the spectrum resulting from the coefficient α′.sub.k, computed using the {circumflex over (x)},ŷ,{circumflex over (z)} a CIE 2° colour matching functions. The coefficients α*.sub.k are found by means of a convex non-linear optimization, constraining the solution to have all non-negative values.

(90) FIG. 19A shows the quality of the D65 metamer found by means of the above optimization, by comparing the color chart appearance under the ideal D65 spectrum (background of each square), and the appearance of the color chart under the metamer shown in the circles at the center of each square (mostly barely visible or invisible). Given that this solution for the D65 metamer requires our specific set of 6 LEDs which may not be common, we repeated the experiment by using the Red, Green, and Blue LEDs along with only one of the broadband white LEDs at a time since an RGBW lighting system might be more common. The results, reported in FIG. 19B, FIG. 19C, FIG. 19D, show that the overall color rendition quality of the system is somewhat reduced. However, a very reasonable approximation of the D65 can be still found by using the W57 cool white LED in combination with RGB LEDs which is consistent with the findings of [LYL*16].

Appendix B: Chromatic Adaptation Transform

(91) The Chromatic Adaptation Transform (CAT) of an image CAT (I.sub.III) is computed in the CIE XYZ tristimulus values space, where its predicted [X.sup.D65Y.sup.D65Z.sup.D65].sup.T values under D65 illumination are derived as follows:

(92) $\begin{array}{cc}\left[\begin{array}{c}{X}^{D65}\\ Y^{D65}\\ Z^{D65}\end{array}\right]=M_{\mathrm{CAT}}*\left[\begin{array}{ccc}{\alpha }^{D65}/{\alpha }^{\mathrm{Ill}}& & \\ & {\beta }^{D65}/{\beta }^{\mathrm{Ill}}& \\ & & {\gamma }^{D65}/{\gamma }^{\mathrm{Ill}}\end{array}\right]*M_{\mathrm{CAT}}*\left[\begin{array}{c}{X}^{\mathrm{Ill}}\\ Y^{\mathrm{Ill}}\\ Z^{\mathrm{Ill}}\end{array}\right]& \left(7\right)\\ \mathrm{where}& \\ \left[\begin{array}{c}{\alpha }^{D65}\\ {\beta }^{D65}\\ {\gamma }^{D65}\end{array}\right]=M_{\mathrm{CAT}}*\left[\begin{array}{c}{X_w}^{D65}\\ {Y_w}^{D65}\\ {Z_w}^{D65}\end{array}\right]& \left(8\right)\\ \mathrm{and}& \\ \left[\begin{array}{c}{\alpha }^{\mathrm{Ill}}\\ {\beta }^{\mathrm{Ill}}\\ {\gamma }^{\mathrm{Ill}}\end{array}\right]=M_{\mathrm{CAT}}*\left[\begin{array}{c}{X_w}^{\mathrm{Ill}}\\ {Y_w}^{\mathrm{Ill}}\\ {Z_w}^{\mathrm{Ill}}\end{array}\right].& \left(9\right)\end{array}$

(93) In the above, [X.sub.w.sup.IIIY.sub.w.sup.IIIZ.sub.w.sup.III].sup.T and [X.sub.w.sup.D65Y.sub.w.sup.D65Z.sub.w.sup.D65].sup.T respectively represent the tristimulus values of the source III and D65 illuminants. The 3×3 matrix M.sub.CAT models human color perception at the LMS cone response level [BS10b], and is derived by numerical optimization [BS10a]. Please note that, given the typical overlapping design of camera sensors spectral sensitivity, the red and green channels of I.sub.nb might contain non-zero signal, similarly to a direct acquisition of a photograph under narrow band illumination.

Appendix C: Estimating Antera's LED Spectrum

(94) The lack of control over Antera's LEDs, both in terms of switching sequence and speed, makes it difficult to directly measure the lighting spectra using a spectrometer given the typical integration time of over 1 second per measurement. Moreover, the individual LEDs have different orientations, thus requiring an integrating sphere for accurate spectral measurements. Hence, we opted for spectral recovery by means of a Genetic Algorithm (GA) based optimization by providing as input to our algorithm a sequence of photographs of a standard reference colour chart, with Np=24 colour patches of known spectral reflectance R.

(95) Our GA begins with a random guess for the Antera's LEDs spectra S: the optimization is driven by a fitness function ƒ, which measures the differences between the acquired digital levels and the simulated ones according to a standard camera model:

(96) $\begin{array}{cc}f\left(S,\mathrm{CSS}\right)=\underset{n=1}{\overset{N_p}{.Math.}}\underset{\mathrm{ch}=r,g,b}{\overset{}{.Math.}}.Math.{\mathrm{DG}}_{n,\mathrm{ch}}-{\left(\underset{\lambda =380}{\overset{720}{.Math.}}{\mathrm{CSS}}_{\mathrm{ch},\lambda }.Math.S_{\lambda }.Math.R_{n,\lambda }\right)}^{{\gamma }_{\mathrm{ch}}}.Math.,& \left(10\right)\end{array}$
where DG.sub.n,ch is the acquired digital level of the ch channel of the colour patch n and γ.sub.ch models the per-channel non-linear response of the sensor.

(97) The above equation implies the joint estimation of the incident spectra S and the Antera's spectral sensitivity CSS. Hence, a naïve implementation would be extremely underconstrained. However, the specifications of the Antera camera report that it can be used as a colorimeter, thus implying that the spectral sensitivity must fulfill Luther's condition (i.e. it is a linear transformation of the CIE 1931 2-degree Colour Matching Functions), greatly reducing the search space. Furthermore, we enforce the recovery of the LEDs spectra by augmenting ƒ(S,CSS) with a smoothness constraint:

(98) $\begin{array}{cc}{f}^{\prime }\left(S,\mathrm{CSS}\right)=f\left(S,\mathrm{CSS}\right).Math.\left(1+.Math.\frac{\sigma \left(\delta \left(S\right)\right)}{\stackrel{\_}{\delta \left(S\right)}}.Math./k\right),& \left(11\right)\end{array}$
where K is a scale constant related to the desired resolution in nm of S and CSS; a and S respectively indicates standard deviation and derivative. FIG. 11A to FIG. 11B reports the recovered overall spectrum of the illumination employed for Antera's albedo measurement. Note that the various peaks correspond to the various types of narrow band LEDs on the device, and the intensity of the recovered peaks might be affected by the different orientations of the LEDs.
Supplementary Material
S1. Details of the Spectral Skin BSSRDF Model

(99) Various terms, parameters and coefficients involved in the spectral skin BSSRDF model of [JSB*10] employed in this work are tabulated along with their descriptions in Table 1. The BSSRDF employs a multipole model for scattering in the thin epidermis, and a dipole model for scattering in the thicker dermis. To model subsurface scattering using the multipole and dipole formulations for epidermis and dermis layers, we need to first compute their absorption and scattering coefficients.

(100) The wavelength λ dependent spectral absorption coefficient for the epidermal layer is given as:
σ.sub.a.sup.epi=C.sub.m[β.sub.mσ.sub.a.sup.em(λ)+(1−β.sub.m)σ.sub.a.sup.pm(λ)]+C.sub.he[γσ.sub.a.sup.oxy(λ)+(1−γ)σ.sub.a.sup.deoxy(λ)]+(1−C.sub.m−C.sub.he)σ.sub.a.sup.base,   (S1)
where, the absorption coefficients for eumelanin and pheomelanin (two types of melanin in skin) is computed as:
σ.sub.a.sup.em(λ)=6.6×10.sup.10×λ.sup.−3.33 mm.sup.−1,   (S2)
σ.sub.a.sup.pm(λ)=2.9×10.sup.14×λ.sup.−4.75 mm.sup.−1, and   (S3)
the baseline absorption coefficient σ.sup.base for the cellular matrix is defined as:
σ.sub.a.sup.base(λ)=0.0244+8.53e.sup.−(λ−154)/66.2 mm.sup.−1,   (S4)
Note that X in above equations is defined in nanometers. The absorption coefficients σ.sub.a.sup.oxy and σ.sub.a.sup.deoxy for the oxygenated and deoxygenated hemoglobin are borrowed from measurements provided in medical literature [DJ06].

(101) Similar to epidermis, the absorption coefficient for the dermal layer is defined as:
σ.sub.a.sup.derm(λ)=C.sub.h(γσ.sub.a.sup.oxy(λ)+(1−γ)σ.sub.a.sup.deoxy(λ))+(1−C.sub.h)σ.sub.a.sup.base(λ).   (S5)

(102) Next, the reduced scattering coefficient for the dermis is computed as:
σ′.sub.s.sup.derm(λ)=7.37λ.sup.−0.22+1.1×10.sup.11×λ.sup.−4, and   (S6)
the reduced scattering coefficient for the epidermis is given by:
σ′.sub.s.sup.derm(λ)=14.74λ.sup.−0.22+2.2×10.sup.11×λ.sup.−4.   (S7)
Using σ.sub.a and σ.sub.s′ as absorption and reduced scattering coefficients for the dipole formulation for dermis, its reflectance profile can be computed as explained by Donner and Jensen [DJ05].

(103) Similarly, they also explain how transmittance and reflectance profiles for the epidermis can be computed using its absorption and reduced scattering coefficients with a multipole model. These individual profiles are then convolved to compute the net reflectance profile which is then subject to surface integration to compute diffuse albedo observed due to subsurface scattering in skin. We refer the reader to Donner&Jensen [DJ06] for further details.

(104) FIG. 20A to FIG. 20C shows comparisons of facial photographs under uniform broadband illumination, and their reconstruction using the reduced 2D spectral model employed by Jimenez et al. [JSB*10] with only two free parameters (C.sub.m and C.sub.h) (FIG. 20A), and reconstruction using the complete 4D model with two additional free parameter (β.sub.m, and C.sub.he) (FIG. 20C). As can be seen, the 4D model allows a closer match to the spatially varying appearance of skin in the photographs.

(105) S2. Measurements with LED Sphere

(106) FIG. 22 and FIG. 23A to FIG. 23F show the spectra of illuminants, and the acquired photographs respectively, employed to estimate spectral parameters of a subject acquired using W57 cool white LED as the broadband in conjunction with narrow-band blue LED illumination. As can be seen, the process is very similar to the employment of D65 metamer for the broadband measurement and the estimated parameters are of sufficiently high quality to reconstruct skin appearance under different illumination spectra (see FIG. 21A to FIG. 21H).

(107) Given the 6 types of LEDs in our LED sphere, we prefer to create an ideal broadband illumination by computing a weighted combination of all 6 LEDs to create a D65 metamer spectra (D65) which we employ for our broadband measurements. We notice a higher contrast in skin color, particularly coloration due to skin pigmentation and redness, under the D65 metamer illumination compared to any of the individual white LEDs including W57. This is consistent with our observation that D65 spectrum (blue dominant) balances the somewhat skewed red-dominant reflectance spectrum of human skin (see FIG. 24A to FIG. 24C).

(108) FIG. 25A to FIG. 25D shows comparisons of parameter maps for a palm of a hand of a darker skin subject obtained using a single measurement under W57 illumination (top-row), vs our proposed two complementary measurements (bottom-row). As can be seen, our proposed approach of two complementary spectral measurements enables higher quality parameter estimation with less noise and clearer spatial structure of chromophore concentrations. Even for a single observation under broadband illumination, we see an improvement in the quality of estimated parameters when employing the D65 metamer illumination for the broadband measurement (center-row). Photograph-reconstruction comparisons of the palm under various illumination spectra can be seen in FIG. 26A to FIG. 26D.

(109) FIG. 27A to FIG. 27D shows the spectral parameter maps for the palm estimated using W57+blue illumination (top-row), and D65′+blue (bottom-row). While qualitatively very similar, the parameters in the bottom-row estimated using D65′ for the broadband measurement exhibit slightly reduced noise and better generalize to novel spectral illumination conditions.

(110) For baseline measurements in the LED sphere, we also did an analysis of which type of broadband illumination is most suitable for estimating the spectral parameters of skin. Across four different skin types ranging from Caucasian, Mediterranean, Asian, and South Asian, we consistently found the reconstruction accuracy of estimation using the cold spectrum broadband illumination (W57) to be higher for reconstructing the appearance of skin under both colder and warmer broadband spectrums. And we also found a clear ordering in decreasing order of accuracy for generalization to a different illumination spectrum from W57, followed by W40, and then W27. This is why we selected the W57 as the choice for the baseline measurement. Measurements under each of these broadband condition were most accurate for reproducing the appearance under their own spectral conditions, pointing to some overfitting to the measurement spectrum. This issue is mitigated to quite an extent when we employ the D65 metamer illumination for broadband measurements, improving the generalization to a different illumination spectrum.

(111) S3. Measurements with Antera3D

(112) FIG. 28A to FIG. 28C presents a few additional examples of skin patches that we measured using the Antera device. This includes a forehead patch of a Caucasian subject (top-row), and the back of the hand (center row), and the palm (bottom row) for a subject with darker (South Asian) skin type. For all of these cases, initial mapping of the Antera parameters to the reduced 2D model shows noticeable differences in the reconstructed albedo (center-left column). However, we can see very good agreement between the reconstruction and the photograph using our procedure for adapting and augmenting Antera's parameter maps (center-right column), and even better matching to the photograph using a full 4D search over the parameter space driven by just the albedo map (right column), with softer reconstruction of the albedo similar to the photograph (which has baked-in subsurface scattering). The reconstruction errors (DeltaE94 metric in CIELAB) for various skin patches are reported in Table 2. Thus, either approaches could be used for rendering purposes in many cases with the caveat that our proposed approach for adapting and augmenting Antera's measurements may be more suitable for physiologically motivated simulations than the full 4D search method which tends to better explain the input data, but at the cost of reduced physiological correlation. This can be seen in the back of the hand example in FIG. 28A to FIG. 28C, where adapting Antera's maps is unable to model the veins on the hand very well (middle column) as veins are not physiologically explained by melanin and hemoglobin concentration. On the other hand, the full 4D search has enough degrees of freedom to explain the data and better reproduce the appearance of veins in this case but with reduced physiological interpretation of the parameter values.

(113) S4. Additional Results and Rendering

(114) FIG. 29A to FIG. 29C presents an example of a female subject where she was acquired twice (under W57 illumination), once under normal skin condition (FIG. 29A) and again when her cheeks were rather flushed (FIG. 29B). Given the estimated parameters for her skin from the first acquisition (shown in FIG. 8A to FIG. 8H in the main paper), we scale the corresponding C.sub.h and C.sub.he maps to simulate flushing in the cheek area shown in the reconstruction in (FIG. 29C). Here, we employed a manually created mask with edge softening to limit the scaling of the hemoglobin to areas around the cheek. As can be seen, the flushed simulation in (FIG. 29C) produces a reconstruction that is qualitatively quite similar to the photograph of actual flushed skin (FIG. 29B).

(115) S4.1. Rendering Subsurface Scattering

(116) In order to render heterogeneous subsurface scattering, we modified the provided subsurface scattering implementation in PBRT from the default dipole diffusion kernel to our specified spatially varying profiles implementing two-layered diffusion. We precompute and store the overall radial reflectance profile due to two-layered diffusion per surface point in linear RGB color space. PBRT framework identifies chromophore parameters mapped to that location and extracts the corresponding precomputed reflectance profile from the tabulated set of all sampled profiles generated with the coloration model for the illumination spectrum. Thereafter, PBRT integrates the selected reflectance profile over radial distances and performs the same operation for all other sampled points on the geometry to add contribution from different spectral profiles, thereby rendering heterogeneous subsurface scattering. The pipeline was implemented by modifying Diffusion-Reflectance structure from the dipole subsurface integrator, which reads in chromophore fractions and finds the index of spectral reflectance profile within the precomputed tabulated set. Then at runtime, for each generated radial distance from the current point on the surface the distance function will extract the radial profile for the closest sampled point and add it to the overall color contribution, thus integrating the spectral reflectance profile according to geometry and spatial variation in chromophores.

(117) FIG. 30A to FIG. 30F presents corresponding diffuse-only renderings of subjects shown in FIG. 1A to FIG. 1C of the paper, rendered with a point light source with individual spectras of the six types of LEDs on the LED sphere (W57, W40, W27, Red, Green and Blue). Skin appearance change under the various LED spectrums, particularly the softening and reduction of skin texture under the warm white and red illumination respectively, is correctly predicted with our implementation of spatially varying subsurface scattering driven by estimated chromophore maps.

(118) FIG. 31A to FIG. 31C presents a qualitative comparison of our proposed rendering with subsurface scattering (FIG. 31A) against photographs (FIG. 31B) of two male subjects with different skin types (Mediterranean and Asian skin-type respectively) lit with a single point light source (W40 neutral white LED not employed for parameter estimation). Here, the light source in the photographs was cross-polarized with respect to the camera, hence eliminating any specular reflection. We also present a rendering where the spectral parameters have been estimated under a single D65′ broadband measurement (FIG. 31C). As can be seen, the renderings have a lot of qualitative similarity with the photographs and well approximate the appearance under a warmer broadband illumination condition. However, the renderings in (FIG. 31A) with joint-estimation under two complementary measurements better preserve skin texture details compared to the renderings in (FIG. 31C) with parameters estimated under a single broadband measurement which encodes a slight blur due to baked-in subsurface scattering.

(119) S5. Limitation

(120) The employed 4D skin BSSRDF model is well suited to reconstruct the appearance of skin and facial hair but has limitations and cannot well reconstruct the appearance of dominant veins or tattoos in skin. An example of this can be seen in FIG. 32A to FIG. 32B, where skin patches containing veins and a tattoo measured with the Antera device are not well reconstructed even with the full 4D search, and rather poorly reconstructed when using the adapted Antera maps for the reconstruction. This is because veins and tattoos cannot be modeled with melanin and hemoglobin concentrations, which is why Donner et al. [DWd*08] introduced an inter-layer absorption in their model to account for these.

(121) Modifications

(122) It will be appreciated that various modifications may be made to the embodiments hereinbefore described. Such modifications may involve equivalent and other features which are already known in the processing of images and/or in the design, manufacture and use of image illuminating and/or capturing apparatuses and component parts thereof and which may be used instead of, or in addition to, features already described herein. Features of one embodiment may be replaced or supplemented by features of another embodiment.

(123) Although claims have been formulated in this application to particular combinations of features, it should be understood that the scope of the disclosure of the present invention also includes any novel features or any novel combination of features disclosed herein either explicitly or implicitly or any generalization thereof, whether or not it relates to the same invention as presently claimed in any claim and whether or not it mitigates any or all of the same technical problems as does the present invention. The applicants hereby give notice that new claims may be formulated to such features and/or combinations of such features during the prosecution of the present application or of any further application derived therefrom.