COMPUTER IMPLEMENTED METHOD AND SYSTEM FOR RETRIEVAL OF MULTI SPECTRAL BRDF PARAMETERS

Abstract

A method for retrieval of multi spectral bidirectional reflectance distribution function (BRDF) parameters by using red-green-blue-depth (RGB-D) data includes capturing, by an RGB-D camera, at least one image of one or more objects in a scene. The captured at least one image of the one or more objects includes RGB-D data including color and geometry information of the objects. A processing unit reconstructs the captured at least one image of the one or more objects to one or more 3D reconstructions by using the RGB-D data. A deep neural network classifies the BRDF of a surface of the one or more objects based on the 3D reconstructions. The deep neural network includes an input layer, an output layer, and at least one hidden layer between the input layer and the output layer. The multi spectral BRDF parameters are retrieved by approximating the classified BRDF by using an iterative optimization method.

Claims

1. A computer-implemented method for retrieval of multispectral bidirectional reflectance distribution function (BRDF) parameters by using red-green-blue-depth (RGB-D) data, the computer-implemented method comprising: capturing, by an RGB-D camera, at least one image of one or more objects in a scene, wherein the captured at least one image of the one or more objects comprises RGB-D data including color and geometry information of the one or more objects; reconstructing, by a processing unit, the captured at least one image of the one or more objects to one or more three-dimensional (3D) reconstructions by using the RGB-D data; classifying, by a deep neural network, the BRDF of a surface of the one or more objects based on the 3D reconstructions, wherein the deep neural network comprises an input layer, an output layer, and at least one hidden layer between the input layer and the output layer; and retrieving the multi spectral BRDF parameters by approximating the classified BRDF by using an iterative optimization method.

2. The method of claim 1, wherein retrieving the multi spectral BRDF parameter further comprises: estimating the multi spectral parameters of the BRDF.

3. The method of claim 1, wherein retrieving the multi spectral BRDF parameter further comprises: estimating spectral properties of an environment map.

4. The method of claim 1, wherein the 3D reconstruction of the one or more objects comprises RGB values and a surface normal of the one or more objects.

5. The method of claim 1, wherein the classified BRDF comprises the spectral parameter of the BRDF and the spectral properties of an environmental mapping of the one or more objects.

6. The method of claim 1, wherein the deep neural network comprises at least a convolutional neural network, a recurrent neural network, or a feedforward neural network.

7. The method of claim 1, wherein the iterative optimization comprises a L2-norm optimization or a Levenberg-Marquandt-optimization.

8. The method of claim 7, wherein the iterative optimization minimizes an approximating error until an amount of error within the estimation is lower than within a manual measurement.

9. A system for retrieval of multi spectral bidirectional reflectance distribution function (BRDF) parameters by using red-green-blue-depth (RGB-D) data, the system comprising: an RGB-D camera configured to capture at least one image of one or more objects in a scene, wherein the captured at least one image of the one or more objects comprises RGB-D data including color and geometry information of the one or more objects; and a computer unit comprising: a memory unit configured to store a deep neural network comprising an input layer, an output layer, and at least one hidden layer between the input layer and the output layer; and at least one processing unit configured to store program instructions that, when executed on the at least one processing unit, cause the system to: reconstruct the captured at least one image of the one or more objects to one or more three-dimensional (3D) reconstructions by using the RGB-D data; classify the BRDF of a surface of the one or more objects based on the 3D reconstructions, using the deep neural network; and retrieve the multi spectral BRDF parameters by approximation of the classified BRDF by using an iterative optimization.

10. (canceled)

11. (canceled)

12. A handheld comprising: an RGB-D camera configured to capture at least one image of one or more objects in a scene, wherein the captured at least one image of the one or more objects comprises RGB-D data including color and geometry information of the one or more objects; and a computer unit comprising: a memory unit configured to store a deep neural network comprising an input layer, an output layer, and at least one hidden layer between the input layer and the output layer; and at least one processing unit configured to store program instructions that, when executed on the at least one processing unit, cause the handheld to: reconstruct the captured at least one image of the one or more objects to one or more three-dimensional (3D) reconstructions by using the RGB-D data; classify the BRDF of a surface of the one or more objects based on the 3D reconstructions, using the deep neural network; and retrieve the multi spectral BRDF parameters by approximation of the classified BRDF by using an iterative optimization.

13. In a non-transitory computer-readable storage medium that stores instructions executable by one or more processors to retrieve multispectral bidirectional reflectance distribution function (BRDF) parameters by using red-green-blue-depth (RGB-D) data, the instructions comprising: capturing, by an RGB-D camera, at least one image of one or more objects in a scene, wherein the captured at least one image of the one or more objects comprises RGB-D data including color and geometry information of the one or more objects; reconstructing, by a processing unit, the captured at least one image of the one or more objects to one or more three-dimensional (3D) reconstructions by using the RGB-D data; classifying, by a deep neural network, the BRDF of a surface of the one or more objects based on the 3D reconstructions, wherein the deep neural network comprises an input layer, an output layer, and at least one hidden layer between the input layer and the output layer; and retrieving the multi spectral BRDF parameters by approximating the classified BRDF by using an iterative optimization method.

14. The non-transitory computer-readable storage medium of claim 13, wherein retrieving the multi spectral BRDF parameter further comprises: estimating the multi spectral parameters of the BRDF.

15. The non-transitory computer-readable storage of claim 13, wherein retrieving the multi spectral BRDF parameter further comprises: estimating spectral properties of an environment map.

16. The non-transitory computer-readable storage medium of claim 13, wherein the 3D reconstruction of the one or more objects comprises RGB values and a surface normal of the one or more objects.

17. The non-transitory computer-readable storage medium of claim 13, wherein the classified BRDF comprises the spectral parameter of the BRDF and the spectral properties of an environmental mapping of the one or more objects.

18. The non-transitory computer-readable storage medium of claim 13, wherein the deep neural network comprises at least a convolutional neural network, a recurrent neural network, or a feedforward neural network.

19. The non-transitory computer-readable storage medium of claim 13, wherein the iterative optimization comprises a L2-norm optimization or a Levenberg-Marquandt-optimization.

20. The non-transitory computer-readable storage medium claim 19, wherein the iterative optimization minimizes an approximating error until an amount of error within the estimation is lower than within a manual measurement.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0044] FIG. 1 shows a flow chart of a possible embodiment of a method for retrieval of multi spectral bidirectional reflectance distribution function (BRDF parameters; and

[0045] FIG. 2 shows a block diagram of a possible embodiment of a system architecture for retrieval of multi spectral bidirectional reflectance distribution function (BRDF) parameters by using red-green-blue-depth (RGB-D) data.

DETAILED DESCRIPTION

[0046] FIG. 1 schematically shows a flowchart of a possible embodiment of a method 1 for retrieval of multi spectral bidirectional reflectance distribution function (BRDF) parameters. The BRDF is the function for the reflection behavior of surfaces of a material at any angle of incidence. For each light beam incident on the material at a given angle of incidence, the BRDF provides the quotient of radiation density and irradiance for each light beam emitted. BRDFs are used, for example, in realistic 3D computer graphics, where BRDFs represent part of the fundamental render equation and serve to represent surfaces as realistically and physically correct as possible.

[0047] The method includes, in the illustrated exemplary embodiment, a number of main acts. In a first act S1, at least one image of one or more objects 30 in a scene is captured by an RGB-D camera 11 (see FIG. 2 and the corresponding description). The captured at least one image of the one or objects 30 includes RGB-D data including color and geometry information of the objects 30. In act S1, the parameters of the material are deduced. This includes measuring by the RGB-D camera 11 for each incoming angle the outgoing angle to create a model for each individual material. Each material sketches the light on its surface in a different way, which is described by parameters. The parameters may be estimated by processing the images captured by the RGB-D camera 11.

[0048] By using the known spectral curves of the RGB-D camera 11, the multispectral BRDF properties are retrieved and the multi spectral environment maps may be discriminated. A material may be measured just by using an RGB-D camera 11 on one spot. For example, a plate with a surface of a unique material in a specific size is captured. The RGB-D camera 11 captures with its images the points of the surface of the unique material as pixels including the RGB-D data. The RGB-D data include the color information red, green, and blue as well as the depth information. The image captured by the RGB-D camera 11 includes all the light information (reflection) coming from the point of the surface at which the RGB-D camera 11 is pointing. Further, the light coming from the environment is summed up with the reflection of the surface. From this, the present embodiments estimate the light direction and the parameters of the BRDF.

[0049] In a further act S2, the at least one image of the one or more objects 30 is reconstructed by a processing unit to one or more 3D reconstructions by using the RGB-D data. The RGB-D data provided by the RGB-D camera includes the RGB values and the depth values. These values describe 3D points on the surface of the object 30 that are recorded by the images. The RGB values and the depth values may be used to create a surface that conforms to these points. The created surface is used to estimate the normal. This results in digital information of the point on the surface from the object 30 that includes directional RGB-D data. The digital information includes depth information of the point for each direction.

[0050] In a further act S3, the BRDF of a surface of the one or more objects 30 is classified by a deep neural network 20 based on the 3D reconstruction. The deep neural network 20 includes an input layer, an output layer, and at least one hidden layer between the input layer and the output layer. The 3D reconstruction defining the object as a surface normal and the direction of the RGB values are put in different layers of the deep neural network 20. The deep neural network 20 may be trained by several known input and output values. The training includes determining relations between the provided training input and output values. According to the trained data sets, further BRDF classifications may be performed by using unknown or new input data. The training may be performed, for example, for a number of material models, which are required for testing and simulation purposes.

[0051] In a further act S4, the multi spectral BRDF parameters are retrieved by approximating the classified BRDF by using an iterative optimization method. In an embodiment, act S4 may include estimating the spectral parameters of the BRDF and estimating the spectral properties of the environment map. Act S4 is iterated until a certain small error value is achieved. The environment map defines the incoming light from each direction. The environment properties include the amount value of light coming from each direction. The iterative optimization method stops when a criterion (e.g., an estimated amount of errors) is achieved. The amount of errors may be determined by practical studies with the goal of achieving the greatest accuracy and/or performance of the method. The optimization method may include the L2-norm optimization or the Levenberg-Marquandt-optimization. In one embodiment, with the method, the BRDF parameters of a surface from an object 30 may be estimated in each state and independent of the lighting condition.

[0052] In equation (1), the outgoing radiance of a surface from an object 30 may be calculated. The outgoing radiance Lo(x, ω.sub.o, λ) may be calculated by the integral of Ω.sup.+ over the incoming radiance Li(ω.sub.i, λ) on the surface point of object 30 and the bidirectional reflectance distribution function ρ(x, ω.sub.i, ω.sub.o, λ, p).

[00001]$\begin{array}{cc}Lo\left(x,{\omega }_o,\lambda \right)={\int }_{{\Omega }^{+}}Li\left({\omega }_i,\lambda \right)\rho \left(x,{\omega }_i,{\omega }_o,\lambda ,p\right)\left(n.Math.{\omega }_i\right)d{\omega }_i& \left(1\right)\end{array}$

[0053] The bidirectional reflectance distribution function includes x as the location on the surface of the object 30 to be scanned. The bidirectional reflectance distribution function further includes the incoming direction vector ω.sub.i and the outgoing direction vector ω.sub.o. The parameter λ describes the wavelength of the light. The parameter n describes the surface normal.

[0054] In equation (2), the tristimulus representation of the color response (e.g., RGB, XYZ) after the reconstruction step is shown. From each point of the surface of the object 30, the RGB value including the color response C.sub.k(x, ω.sub.o) is estimated. Further, for estimating the color response C.sub.k(x, ω.sub.o), a wavelength-based stimuli function ∫.sub.λCIE.sub.k(λ) is used. CIE.sub.k describes a standard curve (e.g., CIE 1931 RGB color matching function).

[00002]$\begin{array}{cc}{C}_k\left(x,{\omega }_o\right)={\int }_{\lambda }{\mathrm{CIE}}_k\left(\lambda \right){\int }_{{\Omega }^{+}}Li\left({\omega }_i,\lambda \right)\rho \left(x,{\omega }_i,{\omega }_o,\lambda ,p\right)\left(n.Math.{\omega }_i\right)d{\omega }_{i}d\lambda & \left(2\right)\end{array}$

[0055] In equation (3), specifying the classifications act S3, the wavelength λ is to be discretized. Equation (3) describes the output C.sub.k(x, ω.sub.o) of the sensor (RBG-D camera). The parameters x, ω.sub.o are the result of the reconstruction, and C.sub.k is the result of the RGB values. Equation (3) is to be modified to get the bidirectional reflectance distribution function p and the incoming radiance Li.

[00003]$\begin{array}{cc}{C}_k\left(x,{\omega }_o\right)={.Math.}_{j=0}{\mathrm{CIE}}_k\left({\lambda }_j\right){\int }_{{\Omega }^{+}}Li\left({\omega }_i,{\lambda }_j\right)\rho \left(x,{\omega }_i,{\omega }_o,{\lambda }_j,p\right)\left(n.Math.{\omega }_i\right)d{\omega }_i\Delta \lambda & \left(3\right)\end{array}$

[0056] In equation (5), the incoming light is represented with spherical harmonics. The equation (5) is manipulated by inserting equation (4) to achieve a better performance while computing the method and to optimize the processing of the method.

[00004]$\begin{array}{cc}{\Sigma }_{m=0}{c}_{m,k,j}{Y}_m\left({\omega }_i\right)& \left(4\right)\end{array}$

[0057] Equation (4) describes the spherical harmonics representation of the Li(ω.sub.i, λ.sub.j). The parameter c.sub.m,k,j describes the environment map, and the parameter Y.sub.m describes the spectral parameters.

[00005]$\begin{array}{cc}{C}_k\left(x,{\omega }_o\right)={.Math.}_{j=0}{\mathrm{CIE}}_k\left({\lambda }_j\right){\int }_{{\Omega }^{+}}{.Math.}_{m=0}{c}_{m,k,j}{Y}_m\left({\omega }_i\right)\rho \left(x,{\omega }_i,{\omega }_o,{\lambda }_j,p\right)\left(n.Math.{\omega }_i\right)d{\omega }_i\Delta \lambda & \left(5\right)\end{array}$

[0058] In equation (6), the terms are rearranged. For the given C.sub.k(x, ω.sub.o), the environment map parameter (light) c.sub.m,k,j and the BRDF parameters p may be estimated.

[00006]$\begin{array}{cc}{C}_k\left(x,{\omega }_o\right)={.Math.}_{j=0}{\mathrm{CIE}}_k\left({\lambda }_j\right){.Math.}_{m=0}{c}_{m,k,j}{\int }_{{\Omega }^{+}}{Y}_m\left({\omega }_i\right)\rho \left(x,{\omega }_i,{\omega }_o,{\lambda }_j,p\right)\left(n.Math.{\omega }_i\right)d{\omega }_i\Delta \lambda & \left(6\right)\end{array}$

[0059] The estimation includes the acts of classifying BRDF types on the surface by using the deep neural network. Further, a suitable initial guess for the BRDF p is selected. Following that, the estimation is repeated (e.g., continued) in a loop until convergence is achieved (e.g., error metric). The looping includes estimating the environment map parameter c.sub.m,k,j (e.g., non-parametric estimation using matrices) and estimating the BRDF p using different optimization methods, such as L2-norm or Levenberg-Marquandt-optimization.

[0060] FIG. 2 schematically shows a block diagram of a possible embodiment of a system architecture 10 for retrieval of multi spectral bidirectional reflectance distribution function (BRDF) parameters by using red-green-blue-depth (RGB-D) data.

[0061] As shown in the block diagram of FIG. 2, the system 10 is used for retrieval of multi spectral bidirectional reflectance distribution function (BRDF) parameters by using red-green-blue-depth (RGB-D) data. The system 10 includes, in the illustrated embodiment, an RGB-D camera 11 and a computing unit 12. The RGB-D camera 11 and the computing unit 12 are constructed as two separate entities. In this embodiment, the RGB-D camera 11 and the computing unit 12 are connected via a communication connection 15. The communication connection 15 may be used for exchange of data between the computing unit 12 and the RGB-D camera 11 in both directions. The exchange of data may include the transmit and receive of control signals and/or data signals. The communication connection 15 may include a wireless communication connection such as Wi-Fi, Bluetooth, Bluetooth low energy, NFC, Infrared and/or a wired communication connection such as a serial communication connection including PCI-Express, Gigabit-Ethernet, HDMI, USB-Standard 3.1 and lower, SATA, DVI, or Ethernet. The RGB-D camera 11 and the computing unit 12 include communication connection interfaces for connecting to the corresponding communication connection 15.

[0062] In the illustrated embodiment, the computing unit 12 includes a memory unit 13 and a processing unit 14. The computing unit 12 of the system 10 for retrieval of multi spectral bidirectional reflectance distribution function (BRDF) parameters may be formed as a computer, personal computer, or workstation in a computer network and may include the processing unit 14 (e.g., processor or processors), a memory unit 13, and a system bus coupling various system components including the memory unit 13 to the processing unit 14. The system bus may be one of any number of types of bus structures, including, for example, a memory bus or memory controller, a peripheral bus, and a local bus using any number of bus architectures. The memory unit 13 may contain a read-only memory (ROM) and/or a random access memory (RAM). A basic input/output system (BIOS) containing basic routines that help to transfer information between elements within the PC (e.g., at startup) may be stored in the ROM. The computing unit may also include a hard disk drive for reading from and writing to a hard disk and an optical disk drive for reading or writing to a removable optical disk (e.g., magnetic) such as a compact disk (CD) or other optical media (e.g., magnetic). The drives and associated storage media provide non-volatile storage of machine-readable instructions, data structures, program modules, and other data for the computer. Although the exemplary environment described herein uses a hard disk and a removable optical disk (e.g., magnetic), a skilled person will appreciate that other types of storage media, such as flash memory cards, digital video disks, Random Access Memory (RAMs), Read Only Memory (ROM), and the like may be used in place of or in addition to the storage devices presented above. A number of program modules may be stored on the hard disk, optical disk (e.g., magnetic), ROM, or RAM, such as an operating system, one or more application programs, such as the method of calculating an output and/or other program modules, and/or program data.

[0063] A user may enter commands and information into the computer via input devices such as a keyboard and a pointing device. Other input devices such as a microphone, joystick, scanner, or the like may also be included. These and other input devices are often connected to the processing unit 14 via a serial interface coupled to the system bus. However, input devices may also be connected via other interfaces, such as a parallel port, a game port, or a universal serial bus (USB). A monitor (e.g., a GUI) or other type of display device may also be connected to the system bus via an interface such as a video adapter. In addition to the monitor, the computer may also contain other peripheral output devices such as speakers and printers.

[0064] The computing unit 12 may be operated in a network environment that defines logical connections to one or more remote computers. The remote computer may be another personal computer, a server, a router, a network PC, a peer device, or another shared network node and may contain many or all of the above elements related to the personal computer. Logical connections include a Local Area Network (LAN) and a Wide Area Network (WAN), an intranet, and the Internet.

[0065] Further, the computing unit 12 may be implemented as a system-on-a-chip design on a microcontroller or programmable chip, such as an ASSIC or FPGA.

[0066] The memory unit 13 may store the deep neural network 20.

[0067] The processing unit 12 is adapted to store program instructions, which when executed on the processing unit 14, cause the system to reconstruct the captured at least one image of the one or more objects 30 to one or more 3D reconstructions by using the RGB-D data. Further, the processing unit classifies the BRDF of a surface of the one or more objects 30 based on the 3D reconstructions, using the deep neural network 20. Further, the processing unit 12 retrieves the multi spectral BRDF parameters by approximating the classified BRDF by using an iterative optimization. The processing unit 12 may include a number of processing units that are configured to store and execute program instructions.

[0068] The RGB-D camera 11 may be a specific type of depth sensing camera device that may work in association with an RGB camera that is able to augment the conventional image of an object 30 with depth information (e.g., related with the distance to the sensor) in a per-pixel basis. The camera component may be an infrared sensor, an infrared camera, and/or an RGB camera. The synchronized output stream of depth and color information is converted into spatial information. In the near infrared range, the IR projector emits a coded dot pattern that is visible to the human eye. A CMOS sensor, for example, receives the image reflected by the object and/or scene and calculates the depth matrix with, for example, VGA resolution, to which an RGB image may be assigned, based on the camera distance across the parallaxes of the corresponding points. The calculation may be carried out with parallel algorithms on a chip in the camera, which considerably reduces the load on the host computer. This only describes an exemplary embodiment of an RGB-D camera 11 used in the system 10 according to the present embodiments. The present embodiments are not limited to the above described description. Further configurations and implementations of the RGB-D camera may be provided.

[0069] The RGB-D camera 11 is configured to capture at least one image of one or more objects 30 in a scene. In an embodiment, the RBD-D camera 11 captures a series of images of the one or more objects 30 in a scene depending on the surface (e.g., material) of the object 30. The image and/or series of images may be processed in the camera or by the processing unit 14 of the computer unit 12. In an embodiment, the RGB-D camera 11 may include a memory for storing the image and or the series of images before processing or transmitting the images. The captured at least one image of the one or more objects 30 includes RGB-D data including color and geometry information of the objects 30.

[0070] In an alternative embodiment of the system 10 according to the present embodiments, the system 10 including the RGB-D camera 11 and the computer unit 12 may be constructed in one single entity. The system 10 may be constructed as a handheld, such as a mobile phone, tablet, laptop, and/or PDA including an RGB-D camera 11, a processing unit 13, and a memory unit 13.

[0071] The scope of protection of the present embodiments is specified by the appended claims and is not restricted by the features explained in the description or shown in the drawing.

[0072] In summary, the invention relates to a computer implemented method and apparatus for retrieval of multi spectral bidirectional reflectance distribution function (BRDF) parameters by using red-green-blue-depth (RGBD) data. The method includes capturing S1 by an RGB-D camera 11 at least one image of one or more objects 30 in a scene. The captured at least one image of the one or more objects 30 includes RGB-D data including color and geometry information of the objects 30. The method also includes reconstructing S2, by a processing unit, the captured at least one image of the one or more objects 30 to one or more 3D reconstructions by using the RGB-D data, and classifying S3, by a deep neural network 20, the BRDF of a surface of the one or more objects 30 based on the 3D reconstructions. The deep neural network 20 includes an input layer, an output layer, and at least one hidden layer between the input layer and the output layer. The method includes retrieving S4 the multi spectral BRDF parameters by approximating the classified BRDF by using an iterative optimization method.

[0073] Due to the present embodiments, the multi spectral BRDF parameters of objects in specified environments may be retrieved.

[0074] The elements and features recited in the appended claims may be combined in different ways to produce new claims that likewise fall within the scope of the present invention. Thus, whereas the dependent claims appended below depend from only a single independent or dependent claim, it is to be understood that these dependent claims may, alternatively, be made to depend in the alternative from any preceding or following claim, whether independent or dependent. Such new combinations are to be understood as forming a part of the present specification.

[0075] While the present invention has been described above by reference to various embodiments, it should be understood that many changes and modifications can be made to the described embodiments. It is therefore intended that the foregoing description be regarded as illustrative rather than limiting, and that it be understood that all equivalents and/or combinations of embodiments are intended to be included in this description.