Image processing method, image processing device, and storage medium

Abstract

An image processing device (10) is provided with: an estimation unit (11) for estimating, by using image data and a spectral sensitivity characteristic of a color image sensor in a wavelength range from visible light to near-infrared light, a spectral distribution characteristic of an incident light incident on the color image sensor, based on a model expressing the spectral distribution characteristic of the incident light, the image data including one or more color channels and acquired by capturing, using the color image sensor, the incident light including visible light and near-infrared light; and a generation unit (12) for generating, using the estimated spectral distribution characteristic of the incident light and the spectral sensitivity characteristic of the color image sensor, visible light image data configured only from information representing the visible light and near-infrared light image data configured only from information representing the near-infrared light.

Claims

1. An image processing method comprising: estimating, by using image data and a spectral sensitivity characteristic of a color image sensor in a wavelength range from visible light to near-infrared light, a spectral distribution characteristic of an incident light incident on the color image sensor, based on a model expressing the spectral distribution characteristic of the incident light, the image data including one or more color channels and acquired by capturing, using the color image sensor, the incident light including visible light and near-infrared light; and generating, by using the estimated spectral distribution characteristic of the incident light and the spectral sensitivity characteristic of the color image sensor, visible light image data composed only of information representing visible light and near-infrared light image data composed only of information representing near-infrared light, wherein each integral value of the spectral sensitivity characteristic of the color image sensor in a wavelength range of near-infrared light relating to each of the one or more color channels of the color image sensor satisfies a predetermined condition, wherein spectral distribution characteristic of incident light is modeled by a weighted sum of basis vectors r.sub.i() (i=1 to M), where M is an integer of 1 or greater), is modeled by a weighted sum of basis vectors r.sub.i() (i=1 to M+1), is modeled by the weighted sum of the basis vectors r.sub.i() (i=1 to M) and a mean vector r.sub.0() of the spectral distribution characteristic, or is modeled by the weighted sum of the basis vectors r.sub.i() (i=1 to M+1) and the mean vector r.sub.0().

2. The image processing method according to claim 1, further comprising: generating the visible light image data composed only of information representing visible light, by using a product of the estimated spectral distribution characteristic of the incident light by the spectral sensitivity characteristic of the color image sensor in a wavelength range of visible light; and generating the near-infrared light image data composed only of information representing near-infrared light, by using a product of the spectral distribution characteristic of the incident light by the spectral sensitivity characteristic of the color image sensor in a wavelength range of near-infrared light.

3. The image processing method according to claim 1, further comprising: generating the visible light image data composed only of information representing visible light, by using a product of the estimated spectral distribution characteristic of the incident light by the spectral sensitivity characteristic of the color image sensor in a wavelength range of visible light; and generating the near-infrared light image data composed only of information representing near-infrared light, by subtracting the generated visible light image data from the image data in which visible light and near-infrared light are captured.

4. The image processing method according to claim 1, further comprising: generating the near-infrared light image data composed only of information representing near-infrared light, by using a product of the estimated spectral distribution characteristic of the incident light by the spectral sensitivity characteristic of the color image sensor in a wavelength range of near-infrared light; and generating the visible light image data composed only of information representing visible light, by subtracting the generated near-infrared light image data from the image data in which visible light and near-infrared light are captured.

5. The image processing method according to claim 1, wherein the one or more color channels included in the image data are a red channel, a green channel, and a blue channel.

6. An image processing device comprising: at least one memory storing a program; and at least one processor configured to execute the program to perform: estimating, by using image data and a spectral sensitivity characteristic of a color image sensor in a wavelength range from visible light to near-infrared light, a spectral distribution characteristic of an incident light incident on the color image sensor, based on a model expressing the spectral distribution characteristic of the incident light, the image data including one or more color channels and acquired by capturing, using the color image sensor, the incident light including visible light and near-infrared light; and generating, by using the estimated spectral distribution characteristic of the incident light and the spectral sensitivity characteristic of the color image sensor, visible light image data composed only of information representing visible light and near-infrared light image data composed only of information representing near-infrared light, wherein each integral value of the spectral sensitivity characteristic of the color image sensor in a wavelength range of near-infrared light relating to each of the one or more color channels of the color image sensor satisfies a predetermined condition, wherein spectral distribution characteristic of incident light is modeled by a weighted sum of basis vectors r.sub.i() (i=1 to M, where M is an integer of 1 or greater), is modeled by a weighted sum of basis vectors r.sub.i() (i=1 to M+1), is modeled by the weighted sum of the basis vectors r.sub.i() (i=1 to M) and a mean vector r.sub.0() of the spectral distribution characteristic, or is modeled by the weighted sum of the basis vectors r.sub.i() (i=1 to M+1) and the mean vector r.sub.0().

7. The image processing device according to claim 6, wherein the at least one processor is configured to perform: generating the visible light image data composed only of information representing visible light, by using a product of the estimated spectral distribution characteristic of the incident light by the spectral sensitivity characteristic of the color image sensor in a wavelength range of visible light, and generating the near-infrared light image data composed only of information representing near-infrared light, by using a product of the spectral distribution characteristic of the incident light by the spectral sensitivity characteristic of the color image sensor in a wavelength range of near-infrared light.

8. A non-transitory storage medium recording an image processing program for causing a computer to execute: estimation processing of estimating, by using image data and a spectral sensitivity characteristic of a color image sensor in a wavelength range from visible light to near-infrared light, a spectral distribution characteristic of an incident light incident on the color image sensor, based on a model expressing the spectral distribution characteristic of the incident light, the image data including one or more color channels and acquired by capturing, using the color image sensor, the incident light including visible light and near-infrared light; and generation processing of generating, by using the estimated spectral distribution characteristic of the incident light and the spectral sensitivity characteristic of the color image sensor, visible light image data composed only of information representing visible light and near-infrared light image data composed only of information representing near-infrared light, wherein each integral value of the spectral sensitivity characteristic of the color image sensor in a wavelength range of near-infrared light relating to each of the one or more color channels of the color image sensor satisfies a predetermined condition, wherein spectral distribution characteristic of incident light is modeled by a weighted sum of basis vectors r.sub.i() (i=1 to M, where M is an integer of 1 or greater), is modeled by a weighted sum of basis vectors r.sub.i() (i=1 to M+1), is modeled by the weighted sum of the basis vectors r.sub.i() (i=1 to M) and a mean vector r.sub.0() of the spectral distribution characteristic, or is modeled by the weighted sum of the basis vectors r.sub.i() (i=1 to M+1) and the mean vector r.sub.0().

9. The storage medium according to claim 8, wherein the image processing program causes a computer to execute: first generation processing of generating the visible light image data composed only of information representing visible light, by using a product of the estimated spectral distribution characteristic of the incident light by the spectral sensitivity characteristic of the color image sensor in a wavelength range of visible light; and second generation processing of generating the near-infrared light image data composed only of information representing near-infrared light, by using a product of the estimated spectral distribution characteristic of the incident light by the spectral sensitivity characteristic of the color image sensor in a wavelength range of near-infrared light.

Description

BRIEF DESCRIPTION OF DRAWINGS

(1) FIG. 1 is a block diagram illustrating a configuration example of a first example embodiment of an image processing device 100 according to the present invention.

(2) FIG. 2 is an explanatory diagram illustrating one example of a mean vector and basis vectors of incident light.

(3) FIG. 3 is a flowchart illustrating an operation of image generation processing performed by the image processing device 100 according to the first example embodiment.

(4) FIG. 4 is a block diagram illustrating a configuration example of a second example embodiment of an image processing device 100 according to the present invention.

(5) FIG. 5 is a block diagram illustrating another configuration example of the second example embodiment of the image processing device 100 according to the present invention.

(6) FIG. 6 is a flowchart illustrating an operation of image generation processing performed by the image processing device 100 according to the second example embodiment.

(7) FIG. 7 is a block diagram illustrating a configuration example of a third example embodiment of an image capturing device 200 according to the present invention.

(8) FIG. 8 is a plan view illustrating one example of a photosensor array in which a Bayer array type color filter is incorporated.

(9) FIG. 9 is a plan view illustrating one example of demosaicing processing.

(10) FIG. 10 is a flowchart illustrating an operation of image generation processing performed by the image capturing device 200 according to the third example embodiment.

(11) FIG. 11 is a block diagram illustrating a configuration example of a fourth example embodiment of an image capturing device 300 according to the present invention.

(12) FIG. 12 is a flowchart illustrating an operation of image generation processing performed by the image capturing device 300 of the fourth example embodiment.

(13) FIG. 13 is a block diagram illustrating an overview of the image processing device according to the present invention.

(14) FIG. 14 is an explanatory diagram illustrating one example of a spectral sensitivity characteristic of a general image sensor.

(15) FIG. 15 is an explanatory diagram illustrating one example of spectral transmittance of a color filter.

(16) FIG. 16 is an explanatory diagram illustrating one example of spectral transmittance of an IR cut filter.

EXAMPLE EMBODIMENT

First Example Embodiment

(17) [Description of Configuration]

(18) Example embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram illustrating a configuration example of a first example embodiment of an image processing device 100 according to the present invention.

(19) The image processing device 100 according to the present example embodiment is an image processing device that is simple in configuration and that is able to generate a visible light image signal and a near-infrared light image signal simultaneously from an image signal captured by a color camera in a high-sensitivity capturing mode using near-infrared light, which is used for capturing in a dark place and the like.

(20) Note that the present example embodiment also uses an optical filter exhibiting a spectral transmission characteristic of transmitting not only visible light but also near-infrared light. Further, a ratio of an integral value of a spectral transmission characteristic in a wavelength range of near-infrared light in each channel is a value less than a predetermined threshold value designated by a user.

(21) Further, the image processing device 100 according to the present example embodiment does not use the above-described IR cut filter, but uses image data captured with sufficient use of a spectral sensitivity characteristic of a color image sensor. Thus, color signals of acquired image data include a visible light signal and a near-infrared light signal.

(22) As illustrated in FIG. 1, the image processing device 100 according to the present example embodiment includes an image data acquisition unit 101, an incident light spectrum estimation unit 102, a visible light image generation unit 103, and a near-infrared light image generation unit 104.

(23) Further, as illustrated in FIG. 1, the image processing device 100 is communicably connected with an external means 110. The external means 110 is, for example, an image capturing device such as an image sensor. Further, the external means 110 includes an optical filter incorporated therein.

(24) The image data acquisition unit 101 includes a function of acquiring image data (I.sub.RGB_NIR illustrated in FIG. 1) captured by the external means 110 that exhibits spectral sensitivity of detecting from visible light (VIS) to near-infrared light (NIR). The image data to be acquired may be still image data, or may be moving image data.

(25) The image data to be acquired by the image data acquisition unit 101 of the present example embodiment are composed of M-channel color signals. Note that M is an integer of 1 or greater. Further, each color signal includes both a visible light signal and a near-infrared light signal. Further, a ratio of an integral value of a spectral sensitivity characteristic in a wavelength range of near-infrared light in each channel is a value less than a predetermined threshold value designated by a user.

(26) The image data acquisition unit 101 acquires image data for all pixels. Color information included in the image data is set for the all pixels. As the color information, for example, red (R), green (G), and blue (B), or cyan (C), magenta (M), and yellow (Y) are set. Note that the color information to be set includes information on visible light and information on near-infrared light.

(27) In the present example embodiment, description will be given by assuming that R, G, and B are set as color information included in image data. The image processing device 100 according to the present example embodiment estimates VIS image data and NIR image data respectively from image data in which R, G, and B color information is set for all pixels, and generates each piece of image data.

(28) The image data acquisition unit 101 executes, for example, demosaicing processing in accordance with setting information, and thereby separates the image data I.sub.RGB_NIR into image data I.sub.R_NIR, image data I.sub.G_NIR, and image data I.sub.B_NIR as illustrated in FIG. 1. The image data acquisition unit 101 inputs the separated image data I.sub.R_NIR, the image data I.sub.G_NIR, and the image data I.sub.B_NIR to the incident light spectrum estimation unit 102.

(29) Note that the image data I.sub.RGB_NIR are image data of RGB channels in which visible light and near-infrared light are coexistent. The image data I.sub.R_NIR are image data of R channel in which visible light and near-infrared light are coexistent. The image data I.sub.G_NIR are image data of G channel in which visible light and near-infrared light are coexistent. The image data I.sub.B_NIR are image data of B channel in which visible light and near-infrared light are coexistent.

(30) Note that, the image data acquisition unit 101 itself may acquire image data of RGB channels by capturing, using a color image sensor, incident light including visible light and near-infrared light.

(31) The incident light spectrum estimation unit 102 includes a function of estimating a spectral distribution characteristic (spectrum) of light incident on each pixel composing acquired image data.

(32) The incident light spectrum estimation unit 102 estimates the spectral distribution characteristic of incident light from image data, by using a spectral sensitivity characteristic of the external means 110. The spectrum of incident light estimated by the incident light spectrum estimation unit 102 is associated with light from a wavelength range of visible light to a wavelength range of near-infrared light.

(33) The visible light image generation unit 103 includes a function of generating a visible light image (I.sub.R, I.sub.G, and I.sub.B illustrated in FIG. 1) including only visible light signals, by using the estimated spectrum of light incident on each pixel composing image data and the spectral sensitivity characteristic of the external means 110.

(34) The near-infrared light image generation unit 104 includes a function of generating a near-infrared light image (I.sub.NIR illustrated in FIG. 1) including only near-infrared light signals, by using the estimated spectrum of light incident on each pixel composing image data and the spectral sensitivity characteristic of the external means 110.

(35) Hereinafter, spectral distribution characteristic estimation processing performed by the incident light spectrum estimation unit 102 of the present example embodiment will be described. It is assumed that a spectral sensitivity characteristic of R channel of a color image sensor is C.sub.R(), a spectral sensitivity characteristic of G channel is C.sub.G(), a spectral sensitivity characteristic of B channel is C.sub.B(), and a spectral distribution characteristic of light incident on a pixel composing image data output by the color image sensor is E(), respectively.

(36) The color image sensor exhibits sensitivity to a wavelength range of near-infrared light regarding all of the respective channels. Moreover, a ratio of an integral value of a spectral sensitivity characteristic in a wavelength range of near-infrared light in each channel is a value less than a predetermined threshold value designated by a user. Three color signals R, G, and B of a pixel on which light is incident are respectively represented as follows.
[Mathematical 1]
R=E()C.sub.R()d
G=E()C.sub.G()d
B=E()C.sub.B()dEquation (1)

(37) in Equation (1) represents a wavelength of light. A value of takes any value in a wavelength range to which a color image sensor exhibits sensitivity.

(38) Further, each of spectral sensitivity characteristics C.sub.R(), C.sub.G(), and C.sub.B() of a color image sensor in Equation (1) is a product of a spectral sensitivity characteristic of the image sensor by each spectral transmission characteristic of an RGB optical filter, and is a known function. In Equation (1), integration is performed over a wavelength range to which a color image sensor exhibits sensitivity.

(39) In order to calculate the spectral distribution characteristic E() of incident light on the basis of color signals R, G, and B in image data by using Equation (1), the present example embodiment adopts a model that expresses a spectral distribution characteristic of incident light, which is continuous data being originally infinite-dimensional with respect to a wavelength, with a relatively small number of parameters.

(40) A spectral distribution characteristic of incident light may be modeled by a weighted sum of basis vectors r.sub.i() (i=1 to M) of the spectral distribution characteristic, or may be modeled by a weighted sum of basis vectors r.sub.i() (i=1 to M+1), with use of a relatively small number of parameters. Furthermore, a mean vector r.sub.0() may be added to the model.

(41) First, description will be given to a case in which a spectral distribution characteristic of incident light is modeled by the mean vector r.sub.0() and the basis vectors r.sub.i() (i=1 to M). A spectral distribution characteristic is modeled by a sum of the mean vector and the weighted sum of the basis vectors, as illustrated in Equation (2).
[Mathematical 2]
E()=r.sub.0()+.sub.i=1.sup.Ma.sub.i*r.sub.i()Equation (2)

(42) Note that a basis vector r.sub.i() (i=1 to 3) in Equation (2) is a function capable of sufficiently expressing the spectral distribution characteristic E() of incident light with a small number of parameters. The basis vector r.sub.i() can be acquired by performing principal component analysis on a spectral space formed by a product of a spectral distribution characteristic of a light source by a surface reflectance of an object. Further, a coefficient a.sub.i in Equation (2) is a parameter used for expressing the spectral distribution characteristic E() of incident light.

(43) FIG. 2 is an explanatory diagram illustrating one example of a mean vector and basis vectors of incident light. FIG. 2 illustrates a mean vector r.sub.0() and basis vectors r.sub.1() to r.sub.3().

(44) When Equation (2) is substituted into Equation (1), observation equations relating to an unknown parameter a.sub.i are generated as follows.
[Mathematical 3]
R=(r.sub.0()+.sub.i=1.sup.Ma.sub.ir.sub.i())C.sub.R()d
G=(r.sub.0()+.sub.i=1.sup.Ma.sub.ir.sub.i())C.sub.G()d
B=(r.sub.0()+.sub.i=1.sup.Ma.sub.ir.sub.i())C.sub.B()dEquation (3)

(45) Note that, a right side of Equation (3) originally needs an integral coefficient, but the integral coefficient is omitted in the present example embodiment for simplification of description.

(46) Integral calculation in Equation (3) may be approximated by summation calculation as indicated below.
[Mathematical 4]
R.sub.=400.sup.1000((r.sub.0()+.sub.i=1.sup.Ma.sub.ir.sub.i())C.sub.R())
G.sub.=400.sup.1000((r.sub.0()+.sub.i=1.sup.Ma.sub.ir.sub.i())C.sub.G())
B.sub.=400.sup.1000((r.sub.0()+.sub.i=1.sup.Ma.sub.ir.sub.i())C.sub.B())Equation (4)

(47) Note that, in Equation (4), a sum is calculated in accordance with a spectral sensitivity characteristic of a color image sensor over a wavelength range from 400 nm to 1000 nm.

(48) When an RGB value of image data is acquired, the observation equations in Equation (3) become a simultaneous linear equation with M unknowns relating to the unknown parameter a.sub.i. In other words, it becomes possible to find the unknown parameter a.sub.i.

(49) When the parameter a.sub.i acquired from Equation (4) is substituted into Equation (2), the incident light spectrum estimation unit 102 is able to reconstruct a spectral distribution characteristic E() of light incident on a concerned pixel in image data.

(50) By finding solutions of observation equations such as Equation (4) for all pixels composing image data, the incident light spectrum estimation unit 102 is able to calculate a spectral distribution characteristic E() of incident light regarding the all pixels.

(51) In the above, description has been given to the case in which a spectral distribution characteristic of incident light is modeled by a mean vector r.sub.0() and basis vectors r.sub.i() (i=1 to M). Note that a spectral distribution characteristic of incident light may be modeled by a mean vector r.sub.0() and basis vectors r.sub.i() (i=1 to M+1) in which one basis vector is added. When the basis vectors r.sub.i() (i=1 to M+1) are used, a spectral distribution characteristic is represented as indicated in Equation (5).
[Mathematical 5]
E()=r.sub.0()+.sub.i=1.sup.M+1a.sub.i*r.sub.i()Equation (5)

(52) Similarly, observation equations are generated from Equation (5) as follows.
[Mathematical 6]
R.sub.=400.sup.1000((r.sub.0()+.sub.i=1.sup.M+1a.sub.ir.sub.i())C.sub.R())
G.sub.=400.sup.1000((r.sub.0()+.sub.i=1.sup.M+1a.sub.ir.sub.i())C.sub.G())
B.sub.=400.sup.1000((r.sub.0()+.sub.i=1.sup.M+1a.sub.ir.sub.i())C.sub.B())Equation (6)

(53) In Equation (6), there are (M+1) unknown parameters a.sub.i being coefficients of the basis vectors r.sub.i() (i=1 to M+1). In other words, Equation (6) is in an under-determinant state where there are more unknown parameters than M that is the number of the observation equations, for which unknown parameters a.sub.i cannot be analytically calculated. In order to solve the above problem, a coefficient a.sub.1 of a first basis vector r.sub.1() is calculated in advance by another method.

(54) One example of a method of calculating a coefficient a.sub.1 will be described. As illustrated in FIG. 2, a first basis vector r.sub.1() of incident light is generally a direct-current component. In other words, a coefficient of a first basis vector r.sub.1() is equivalent to a coefficient for representing intensity of incident light. Thus, estimation is made that a sum of M-channel image data is highly correlated with the coefficient of the first basis vector r.sub.1().

(55) When attention is paid to the above property, for example, a regression equation relating to a sum of M-channel image data and a coefficient a.sub.1 of a basis vector r.sub.1() is acquired by single regression analysis. When the sum of the M-channel image data is substituted into the acquired regression equation, the coefficient a.sub.1 of the basis vector r.sub.1() is acquired.

(56) When a.sub.1 calculated by the above method is substituted into Equation (6), the problem of under-determinant is solved. In other words, the observation equations indicated in Equation (6) become a simultaneous linear equation with M unknowns, allowing the remaining unknown parameters a.sub.i to be acquired.

(57) When the parameter a.sub.i acquired as described above is substituted into Equation (5), the incident light spectrum estimation unit 102 is able to reconstruct a spectral distribution characteristic E() of light incident on a concerned pixel in image data.

(58) Hereinafter, visible light image generation processing performed by the visible light image generation unit 103 of the present example embodiment will be described. The visible light image generation unit 103 uses spectral sensitivity characteristics C.sub.R,VIS(), C.sub.G,VIS(), and C.sub.B,VIS() in only a visible light wavelength, out of spectral sensitivity characteristics C.sub.R(), C.sub.G(), and C.sub.B() of a color image sensor. Note that the visible light wavelength is a wavelength range that is approximately from 400 nm to 700 nm.

(59) The visible light image generation unit 103 generates, by using a spectral distribution characteristic E() of incident light acquired by the incident light spectrum estimation unit 102 and spectral sensitivity characteristics C.sub.R,VIS(), C.sub.G,VIS(), and C.sub.B,VIS(), color signals R.sub.VIS, G.sub.VIS, and B.sub.VIS including only visible light signals respectively as follows.
[Mathematical 7]
R.sub.VIS=.sub.=400.sup.700((r.sub.0()+.sub.i=1.sup.Ma.sub.ir.sub.i())C.sub.R,VIS())
G.sub.VIS=.sub.=400.sup.700((r.sub.0()+.sub.i=1.sup.Ma.sub.ir.sub.i())C.sub.G,VIS())
B.sub.VIS=.sub.=400.sup.700((r.sub.0()+.sub.i=1.sup.Ma.sub.ir.sub.i())C.sub.B,VIS())Equation (7)

(60) By respectively calculating the color signals R.sub.VIS, G.sub.VIS, and B.sub.VIS including only visible light signals for all pixels composing image data, the visible light image generation unit 103 is able to generate image data I.sub.R, I.sub.G, and I.sub.B including only visible light signals.

(61) Hereinafter, near-infrared light image generation processing performed by the near-infrared light image generation unit 104 of the present example embodiment will be described. The near-infrared light image generation unit 104 uses spectral sensitivity characteristics C.sub.R,NIR(), C.sub.G,NIR(), and C.sub.B,NIR() in only a near-infrared light wavelength, out of spectral sensitivity characteristics C.sub.R(), C.sub.G(), and C.sub.B() of a color image sensor. Note that the near-infrared light wavelength is a wavelength range that is approximately from 700 nm to 1000 nm.

(62) The near-infrared light image generation unit 104 generates, by using a spectral distribution characteristic E() of incident light acquired by the incident light spectrum estimation unit 102 and spectral sensitivity characteristics C.sub.R,NIR(), C.sub.G,NIR(), and C.sub.B,NIR(), signals R.sub.NIR, G.sub.NIR, and B.sub.NIR including only near-infrared light signals respectively as follows.
[Mathematical 8]
R.sub.NIR=.sub.=700.sup.1000((r.sub.0()+.sub.i=1.sup.Ma.sub.ir.sub.i())C.sub.R,NIR())
G.sub.NIR=.sub.=700.sup.1000((r.sub.0()+.sub.i=1.sup.Ma.sub.ir.sub.i())C.sub.G,NIR())
B.sub.NIR=.sub.=700.sup.1000((r.sub.0()+.sub.i=1.sup.Ma.sub.ir.sub.i())C.sub.B,NIR())Equation (8)

(63) The near-infrared light image generation unit 104 respectively calculates the signals R.sub.NIR, G.sub.NIR, and B.sub.NIR including only near-infrared light signals for all pixels composing image data. Then, the near-infrared light image generation unit 104 adds up the acquired signals R.sub.NIR, G.sub.NIR, and B.sub.NIR regarding each pixel, records an addition result, and is thereby able to generate image data I.sub.NIR including only near-infrared light signals.

(64) Note that, since a user designates a ratio of an integral value of a spectral sensitivity characteristic in a wavelength range of near-infrared light in each channel, for example, R.sub.NIR/G.sub.NIR or R.sub.NIR/B.sub.NIR, which is a ratio of signals, takes a value less than a predetermined value. In other words, a ratio of signals composing image data I.sub.NIR is adjusted.

(65) [Description of Operation]

(66) An operation of generating a visible light image and a near-infrared light image performed by the image processing device 100 according to the present example embodiment will be described below with reference to FIG. 3. FIG. 3 is a flowchart illustrating an operation of image generation processing performed by the image processing device 100 of the first example embodiment.

(67) As illustrated in FIG. 3, first, the image data acquisition unit 101 acquires image data I.sub.RGB_NIR captured by the external means 110 (Step S101). The image data acquisition unit 101 processes the acquired image data, and inputs the processed image data I.sub.R_NIR, I.sub.G_NIR, and I.sub.B_NIR to the incident light spectrum estimation unit 102.

(68) Then, the incident light spectrum estimation unit 102 estimates a spectral distribution characteristic E() of light incident on a pixel in the acquired image data, by using spectral sensitivity characteristics C.sub.R(), C.sub.G(), and C.sub.B() of a color image sensor of the external means 110 (Step S102).

(69) The incident light spectrum estimation unit 102 estimates a spectral distribution characteristic E() regarding all pixels composing the image data. The incident light spectrum estimation unit 102 inputs the estimated spectral distribution characteristic E() to the visible light image generation unit 103 and the near-infrared light image generation unit 104.

(70) Then, the visible light image generation unit 103 calculates color signals R.sub.VIS, G.sub.VIS, and B.sub.VIS including only visible light signals, by using the input spectral distribution characteristic E() and spectral sensitivity characteristics C.sub.R,VIS(), C.sub.G,VIS(), and C.sub.B,VIS() in only a visible light wavelength. Then, the visible light image generation unit 103 generates visible light images I.sub.R, I.sub.G, and I.sub.B on the basis of the calculated color signals (Step S103).

(71) Then, the near-infrared light image generation unit 104 calculates signals R.sub.NIR, G.sub.NIR, and B.sub.NIR including only near-infrared light signals, by using the input spectral distribution characteristic E() and spectral sensitivity characteristics C.sub.R,NIR(), C.sub.G,NIR(), and C.sub.B,NIR() in only a near-infrared light wavelength.

(72) Then, the near-infrared light image generation unit 104 generates a near-infrared light image I.sub.NIR on the basis of the calculated signals (Step S104). After generating the near-infrared light image I.sub.NIR, the image processing device 100 ends the image generation processing.

(73) [Description of Advantageous Effect]

(74) The image processing device 100 according to the present example embodiment is able to acquire image data including only visible light signals and image data including only near-infrared light signals, by using a spectral sensitivity characteristic of a color image sensor exhibiting sensitivity from visible light to near-infrared light and RGB image data including near-infrared light signals captured by the color image sensor.

(75) Specifically, the image data acquisition unit 101 of the present example embodiment acquires, from a color image sensor exhibiting sensitivity from visible light to near-infrared light and outputting a color image in which visible light is captured, M-channel color images in which not only visible light but also near-infrared light are captured.

(76) The above-described color image sensor exhibits sensitivity to a wavelength range of near-infrared light regarding each channel. A ratio of an integral value of a spectral sensitivity characteristic in a wavelength range of near-infrared light in each channel is a value less than a predetermined threshold value designated by a user.

(77) Then, the incident light spectrum estimation unit 102 reconstructs a spectral distribution characteristic of incident light, by using a color signal to be acquired from the acquired M-channel color images, a spectral sensitivity characteristic of the color image sensor, and a mean vector and M basis vectors expressing the spectral distribution characteristic of incident light. Note that the incident light spectrum estimation unit 102 may reconstruct a spectral distribution characteristic of incident light by using (M+1) basis vectors.

(78) Then, the visible light image generation unit 103 generates a visible light image composed only of visible light signals, by using the reconstructed spectral distribution characteristic of incident light and the spectral sensitivity characteristic of the color image sensor. Further, the near-infrared light image generation unit 104 generates a near-infrared light image composed only of near-infrared light signals, by using the reconstructed spectral distribution characteristic of incident light and the spectral sensitivity characteristic of the color image sensor.

(79) The methods described in PTLs 1 to 3 acquire a visible light image and a near-infrared light image by using a special mechanical configuration or a special image sensor, as described above.

(80) The image processing device 100 according to the present example embodiment is able to acquire a visible light image and a near-infrared light image in a simple manner, with use of only a configuration of a general capturing device, rather than use of a special mechanical configuration or a special image sensor. The reason is that the incident light spectrum estimation unit 102 estimates a spectral distribution characteristic E() of incident light by using a spectral sensitivity characteristic of a color image sensor and image data captured by the color image sensor.

Second Example Embodiment

(81) [Description of Configuration]

(82) Next, a second example embodiment of the present invention will be described with reference to the drawings. FIG. 4 is a block diagram illustrating a configuration example of the second example embodiment of an image processing device 100 according to the present invention.

(83) As illustrated in FIG. 4, the image processing device 100 according to the present example embodiment includes an image data acquisition unit 101, an incident light spectrum estimation unit 102, a visible light image generation unit 103, and a signal separation unit 105. Configurations in the second example embodiment other than the image data acquisition unit 101 and the signal separation unit 105 are similar to the configurations in the first example embodiment.

(84) The image data acquisition unit 101 includes a function of acquiring image data (I.sub.RGB_NIR illustrated in FIG. 4) captured by an external means 110 that exhibits spectral sensitivity of detecting from visible light (VIS) to near-infrared light (NIR).

(85) The image data to be acquired by the image data acquisition unit 101 of the present example embodiment are composed of a plurality of color signals. Each color signal composing the image data includes both a visible light signal and a near-infrared light signal.

(86) Further, a ratio of near-infrared light signals of each color channel included in the image data to be acquired by the image data acquisition unit 101 is a value less than a predetermined threshold value designated by a user.

(87) The image data acquisition unit 101 separates the image data I.sub.RGB_NIR into image data I.sub.R_NIR, image data I.sub.G_NIR, and image data I.sub.B_NIR, respectively. The image data acquisition unit 101 inputs the separated image data I.sub.R_NIR, the image data I.sub.G_NIR, and the image data I.sub.B_NIR to the incident light spectrum estimation unit 102.

(88) The incident light spectrum estimation unit 102 includes a function of estimating, on the basis of image data, a spectral distribution characteristic (spectrum) of light incident on each pixel composing the image data, by using a spectral sensitivity characteristic of the external means 110. The spectrum of incident light estimated by the incident light spectrum estimation unit 102 includes from a wavelength range of visible light to a wavelength range of near-infrared light.

(89) The visible light image generation unit 103 includes a function of generating a visible light image (I.sub.R, I.sub.G, and I.sub.B illustrated in FIG. 4) including only visible light signals, by using an estimated spectrum of light incident on each pixel composing image data and a spectral sensitivity characteristic of the external means 110.

(90) The signal separation unit 105 includes a function of outputting, by using two types of image data to be input, first image data and second image data image data acquired by subtracting the second image data from the first image data.

(91) In the example illustrated in FIG. 4, the first image data to be input to the signal separation unit 105 are I.sub.R_NIR, I.sub.G_NIR, and I.sub.B_NIR output from the image data acquisition unit 101. Further, the second image data to be input to the signal separation unit 105 are I.sub.R, I.sub.G, and I.sub.B output from the visible light image generation unit 103.

(92) Thus, the signal separation unit 105 generates a near-infrared light image I.sub.NIR including only near-infrared light signals, by subtracting I.sub.R from I.sub.R_NIR, I.sub.G from I.sub.G_NIR, and I.sub.B from I.sub.B_NIR, respectively.

(93) FIG. 5 is a block diagram illustrating another configuration example of the second example embodiment of the image processing device 100 according to the present invention. FIG. 5 illustrates the image processing device 100 of a case in which the visible light image generation unit 103 of the image processing device 100 illustrated in FIG. 4 is replaced with the near-infrared light image generation unit 104.

(94) The near-infrared light image generation unit 104 includes a function of generating a near-infrared light image (I.sub.NIR illustrated in FIG. 5) including only near-infrared light signals, by using a spectrum of light incident on each pixel composing estimated image data and a spectral sensitivity characteristic of the external means 110.

(95) In the example illustrated in FIG. 5, the first image data to be input to the signal separation unit 105 are I.sub.R_NIR, I.sub.G_NIR, and I.sub.B_NIR output from the image data acquisition unit 101. Further, the second image data to be input to the signal separation unit 105 are I.sub.NIR output from the near-infrared light image generation unit 104.

(96) Thus, the signal separation unit 105 respectively generates visible light images I.sub.R, I.sub.G, and I.sub.B including only visible light signals, by subtracting I.sub.NIR from I.sub.R_NIR, I.sub.NIR from I.sub.G_NIR, and I.sub.NIR from I.sub.B_NIR, respectively.

(97) [Description of Operation]

(98) Hereinafter, an operation of generating a visible light image and a near-infrared light image performed by the image processing device 100 according to the present example embodiment will be described with reference to FIG. 6. FIG. 6 is a flowchart illustrating an operation of image generation processing performed by the image processing device 100 according to the second example embodiment. Note that the image generation processing illustrated in FIG. 6 is executed by the image processing device 100 illustrated in FIG. 4.

(99) Processing of Steps S201 to S203 illustrated in FIG. 6 is similar to processing of Steps S101 to S103 illustrated in FIG. 3, which have been described in the first example embodiment.

(100) As illustrated in FIG. 4, image data I.sub.R_NIR, I.sub.G_NIR, and I.sub.B_NIR are input to the signal separation unit 105 from the image data acquisition unit 101. Further, visible light images I.sub.R, I.sub.G, and I.sub.B are input to the signal separation unit 105 from the visible light image generation unit 103.

(101) As illustrated in FIG. 6, the signal separation unit 105 subtracts I.sub.R from I.sub.R_NIR, I.sub.G from I.sub.G_NIR, and I.sub.B from I.sub.B_NIR, respectively, adds up image data generated as a result of subtraction, and thereby generates a near-infrared light image I.sub.NIR (Step S204).

(102) After generating the near-infrared light image I.sub.NIR, the image processing device 100 ends the image generation processing. Note that, the image processing device 100 illustrated in FIG. 5 generates a visible light image, by also performing the same operation as the image generation processing illustrated in FIG. 6, except that order of generation of a visible light image and generation of a near-infrared light image is swapped.

(103) [Description of Advantageous Effect]

(104) The image processing device 100 of the present example embodiment is able to reduce a generation cost of a visible light image and a near-infrared light image, in comparison with the image processing device 100 of the first example embodiment. The reason is that one image is generated by the signal separation unit 105 subtracting another image from image data.

Third Example Embodiment

(105) [Description of Configuration]

(106) Next, a third example embodiment of the present invention will be described with reference to the drawings. FIG. 7 is a block diagram illustrating a configuration example of the third example embodiment of an image capturing device 200 according to the present invention.

(107) As illustrated in FIG. 7, the image capturing device 200 according to the present example embodiment is constituted by an optical filter 202, a photosensor 203, an image processing unit 204, and a spectral sensitivity characteristic storage memory 205.

(108) Note that the image capturing device 200 of the present example embodiment is an image capturing device from which an infrared light cut filter (IR cut filter) 201 mounted on a normal image capturing device is removed, or an image capturing device on which the infrared light cut filter 201 is not mounted from the beginning.

(109) A dashed rectangle illustrated in FIG. 7 is equivalent to the infrared light cut filter 201 mounted on a normal image capturing device. Note that, in the present example embodiment, no component is present at a location indicated by the dashed rectangle.

(110) For a camera lens illustrated in FIG. 7, a normal camera lens may be used. Further, for the optical filter 202 and the photosensor 203, an optical filter and a photosensor generally used in a color image input device or an image capturing device may be respectively used.

(111) In other words, a spectral transmission characteristic of the optical filter 202 is similar to, for example, the spectral transmission characteristic illustrated in FIG. 16. Further, a spectral transmission characteristic of the photosensor 203 is similar to, for example, the spectral transmission characteristic illustrated in FIG. 15.

(112) Further, a ratio of an integral value of a spectral sensitivity characteristic in a wavelength range of near-infrared light in each channel of the photosensor 203 is a value less than a predetermined threshold value designated by a user.

(113) FIG. 8 is a plan view illustrating one example of a photosensor array in which a Bayer array type color filter is incorporated. FIG. 8 illustrates arrangement of colors in the optical filter 202 and the photosensor 203.

(114) The arrangement of three colors R, G, and B in the optical filter 202 illustrated in FIG. 8 is arrangement called a Bayer array type. Any one of the colors R, G, and B is assigned, respectively, to each pixel in image data output by the photosensor 203, in such a manner as to correspond to R, G, and B arranged in the optical filter 202.

(115) The spectral sensitivity characteristic storage memory 205 illustrated in FIG. 7 includes a function of recording a spectral sensitivity characteristic of the optical filter 202 and the photosensor 203 as an image sensor.

(116) The image processing unit 204 illustrated in FIG. 7 includes a function similar to that of the image processing device 100 according to the first example embodiment, or that of the image processing device 100 according to the second example embodiment. In other words, the image processing unit 204 generates image data I.sub.R including only R color signals, image data I.sub.G including only G color signals, image data I.sub.B including only B color signals, and image data I.sub.NIR including only NIR signals, respectively, on the basis of image data composed of three color signals of R, G, and B.

(117) The image processing unit 204 applies demosaicing processing and the image generation processing of the first example embodiment to image data I.sub.RGB_NIR composed of three color signals of R, G, and B including NIR signals output from the photosensor 203. By applying each processing, the image processing unit 204 generates image data for four channels of R, G, B, and NIR.

(118) Hereinafter, one example of demosaicing processing to be applied to image data I.sub.RGB_NIR composed of three color signals of R, G, and B including NIR signals output from the photosensor 203 will be described with reference to FIG. 9.

(119) FIG. 9 is a plan view illustrating one example of demosaicing processing. Note that the demosaicing processing is equivalent to a part of the processing of separating image data I.sub.RGB_NIR into image data I.sub.R_NIR, image data I.sub.G_NIR, and image data I.sub.B_NIR performed by the image data acquisition units 101 of the first example embodiment and the second example embodiment.

(120) First, the image processing unit 204 acquires data (R, G, and B color signals) on a pixel of coordinates (1,1) in image data illustrated in FIG. 9. The R, G, and B color signals at a point in time of acquisition by the image processing unit 204 include NIR signals, but are simply written as R, G, and B color signals for simplification of description.

(121) As illustrated in FIG. 9, R is assigned to the pixel of coordinates (1,1). Thus, the image processing unit 204 directly sets the data on the pixel of coordinates (1,1) for a signal value of R as follows.
R(1,1)=R(1,1)Equation (9)

(122) The image processing unit 204 interpolates a signal value of G and a signal value of B that are not assigned to the pixel of coordinates (1,1), for example, by using color signals of peripheral pixels as follows.
G(1,1)=(G(2,1)+G(1,2))/2Equation (10)
B(1,1)=B(2,2)Equation (11)

(123) Then, the image processing unit 204 acquires data (R, G, and B color signals) on a pixel of coordinates (1,2) in the image data. As illustrated in FIG. 9, G is assigned to the pixel of coordinates (1,2). Thus, the image processing unit 204 directly sets the data on the pixel of coordinates (1,2) for a signal value of G as follows.
G(1,2)=G(1,2)Equation (12)

(124) Similarly to a case of the pixel of coordinates (1,1), the image processing unit 204 interpolates a signal value of R and a signal value of B that are not assigned to the pixel of coordinates (1,2), for example, by using color signals of peripheral pixels as follows.
R(1,2)=R(1,1)Equation (13)
B(1,2)=B(2,2)Equation (14)

(125) In general, a signal value X(i,j) of a color signal X (=R, B, or G) assigned to a pixel of coordinates (i,j) is data itself on a pixel of X on the coordinates (i,j). On the other hand, a signal value of a color signal Y (=R, B, or G) not assigned to a pixel of coordinates (i,j) is a sum of pieces of weighted data on arbitrary one or more pixels (including Y) around the coordinates (i,j). By repeatedly executing the above-described processing, the image processing unit 204 acquires image data including all R, G, and B color signals regarding all pixels. In other words, a signal value of near-infrared light in each channel included in image data may be a value that is equal for each channel.

(126) Note that, the demosaicing processing according to the present example embodiment is not limited to the above-described processing, but a general demosaicing scheme may be used. With the processing described above, the image processing unit 204 acquires image data in which R, G, and B color signals are set for all pixels.

(127) [Description of Operation]

(128) Hereinafter, an operation of generating a visible light image and a near-infrared light image performed by the image capturing device 200 according to the present example embodiment will be described with reference to FIG. 10. FIG. 10 is a flowchart illustrating an operation of image generation processing performed by the image capturing device 200 according to the third example embodiment.

(129) As illustrated in FIG. 10, light is incident on the image capturing device 200 through a camera lens (Step S301). The incident light, from which near-infrared light is not cut off, is incident on the photosensor 203 including the optical filter 202 incorporated therein in which R, G, and B are arranged in Bayer array.

(130) Then, after the incident light passes through each filter of the optical filter 202, the photosensor 203 generates three color signals of R, G, and B including near-infrared light signals (Step S302).

(131) Then, the image processing unit 204 acquires image data including all R, G, and B color signals regarding all pixels, by executing demosaicing processing (Step S303).

(132) Then, the image processing unit 204 extracts or generates image data composed only of near-infrared light signals, on the basis of image data in which R, G, and B color signals are set for all pixels. In other words, the image processing unit 204 generates image data composed only of visible light signals and image data composed only of near-infrared light signals, on the basis of RGB image data including near-infrared light signals (Step S304).

(133) The processing executed by the image processing unit 204 in Step S304 is the image generation processing illustrated in FIG. 3, or the image generation processing illustrated in FIG. 6. After each piece of image data is generated, the image capturing device 200 ends the image generation processing.

(134) [Description of Advantageous Effect]

(135) The image capturing device 200 according to the present example embodiment is an image capturing device including an optical system from which an IR cut filter included in a general image capturing device is removed. Since the IR cut filter can be removed readily, the image capturing device 200 according to the present example embodiment can be manufactured easily based on a general image capturing device.

(136) In other words, the image capturing device 200 is a device that can be mass-produced at a low cost. In addition, because of a simple configuration, the image capturing device 200 is a compact capturing device, and is expected not to break down frequently. The image capturing device 200 according to the present example embodiment is able to acquire a visible light image and a near-infrared light image simultaneously, even with a configuration similar to a configuration of a general image capturing device.

Fourth Example Embodiment

(137) [Description of Configuration]

(138) Next, a fourth example embodiment of the present invention will be described with reference to the drawings. In the third example embodiment, description has been given to an example of applying the present invention to an image capturing device that splits incident light with a Bayer array type optical filter. In the present example embodiment, description will be given to an example of applying the present invention to a three-CCD image capturing device.

(139) FIG. 11 is a block diagram illustrating a configuration example of the fourth example embodiment of an image capturing device 300 according to the present invention. As illustrated in FIG. 11, the image capturing device 300 is constituted by a prism (color decomposition unit) 301, photosensors 311 to 313, an image processing unit 320, and a spectral sensitivity characteristic storage memory 330.

(140) For a camera lens illustrated in FIG. 11, a normal camera lens may be used. Further, for the prism 301 and the photosensors 311 to 313, a prism and photosensors used generally in a three-CCD image capturing device may be respectively used.

(141) Light incident on the image capturing device 300 through the camera lens is split by the prism 301 into light of three colors of R, G, and B. Outlined arrows illustrated in FIG. 11 represent light.

(142) The photosensor 311 captures color light of R, and outputs image data I.sub.R_NIR. Further, the photosensor 312 captures color light of G, and outputs image data I.sub.G_NIR. Further, the photosensor 313 captures color light of B, and outputs image data I.sub.B_NIR.

(143) Note that color light to be generated by splitting light with the prism 301 includes near-infrared light. Since a near-infrared light cut filter for cutting off near-infrared light is not mounted on the image capturing device 300 according to the present example embodiment, each piece of image data to be output from the photosensors 311 to 313 includes near-infrared light signals.

(144) Further, a ratio of near-infrared light signals included in each piece of image data is a value less than a predetermined threshold value designated by a user.

(145) Each piece of image data I.sub.R_NIR, I.sub.G_NIR, and I.sub.B_NIR of R, G, and B including NIR signals output from the photosensors 311 to 313 is input respectively to the image processing unit 320. Similarly to the image processing unit 204 of the third example embodiment, the image processing unit 320 generates image data I.sub.R including only R color signals, image data I.sub.G including only G color signals, image data I.sub.B including only B color signals, and image data I.sub.NIR including only NIR signals, respectively, on the basis of each piece of R, G, and B image data including NIR signals.

(146) [Description of Operation]

(147) Hereinafter, an operation of generating a visible light image and a near-infrared light image performed by the image capturing device 300 according to the present example embodiment will be described with reference to FIG. 12. FIG. 12 is a flowchart illustrating an operation of image generation processing performed by the image capturing device 300 according to the fourth example embodiment.

(148) As illustrated in FIG. 12, light is incident on the image capturing device 300 through a camera lens (Step S401). The incident light, from which near-infrared light is not cut off, is incident on the prism 301.

(149) Then, the incident light is split by the prism 301 into three rays of visible light of R, G, and B including near-infrared light (Step S402).

(150) Then, the photosensor 311 captures color light of R, and outputs image data I.sub.R_NIR. Further, the photosensor 312 captures color light of G, and outputs image data I.sub.G_NIR. Further, the photosensor 313 captures color light of B, and outputs image data I.sub.B_NIR (Step S403).

(151) Then, the image processing unit 320 generates image data composed only of visible light signals and image data composed only of near-infrared light signals, on the basis of each piece of R, G, and B image data including near-infrared light signals (Step S404).

(152) The processing executed by the image processing unit 320 in Step S404 is the image generation processing illustrated in FIG. 3, or the image generation processing illustrated in FIG. 6. After each piece of image data is generated, the image capturing device 300 ends the image generation processing.

(153) [Description of Advantageous Effect]

(154) The image capturing device 300 according to the present example embodiment includes a prism and photosensors constituting a general three-CCD capturing device. Further, an IR cut filter generally mounted on a three-CCD capturing device is removed.

(155) In other words, the image capturing device 300 can be manufactured by merely making a slight modification to a configuration of a general three-CCD capturing device. Thus, it can be expected that the image capturing device 300 is manufactured at a low cost, and that the image capturing device 300 does not break down frequently. The image capturing device 300 is able to simultaneously acquire a visible light image composed only of visible light signals and a near-infrared light image composed only of near-infrared light signals.

(156) The image processing device 100 of each example embodiment is able to reconstruct a visible light image composed only of visible light signals and a near-infrared light image composed only of near-infrared light signals, from image data captured with use of a general color image sensor, without using a special device or a mechanical drive mechanism for capturing near-infrared light.

(157) In other words, since no special sensor or no mechanical drive mechanism is included, an observation device that observes visible light and near-infrared light to which the technique of each example embodiment is applied is able to generate a visible light image and a near-infrared light image with a simple device configuration. Use of the technique of each example embodiment achieves simplification of circuit and device configurations.

(158) Note that, in each example embodiment, description has been given by assuming that color channels are R channel, G channel, and B channel. However, color channels may be other channels such as cyan (C) channel, magenta (M) channel, and yellow (Y) channel.

(159) Note that, the image processing device 100 of each example embodiment is implemented by, for example, a central processing unit (CPU) that executes processing in accordance with a program stored in a storage medium. In other words, the image data acquisition unit 101, the incident light spectrum estimation unit 102, the visible light image generation unit 103, the near-infrared light image generation unit 104, and the signal separation unit 105 are implemented by, for example, a CPU that executes processing in accordance with program control. Further, only some of the functions may be implemented by a CPU.

(160) Further, each unit in the image processing device 100 of each example embodiment may be implemented by a hardware circuit. As one example, the image data acquisition unit 101, the incident light spectrum estimation unit 102, the visible light image generation unit 103, the near-infrared light image generation unit 104, and the signal separation unit 105 are implemented respectively by large scale integration (LSI). Further, these units may be implemented by one LSI.

(161) Next, an overview of the example embodiments of the present invention will be described. FIG. 13 is a block diagram illustrating an overview of the image processing device according to the example embodiments of the present invention. In FIG. 13, an image processing device 10 according to the example embodiments of the present invention includes an estimation unit 11 (for example, the incident light spectrum estimation unit 102 illustrated in FIG. 1 and the like) that estimates, by using a spectral sensitivity characteristic of a color image sensor in a wavelength range from visible light to near-infrared light and image data including one or more color channels and in which visible light and near-infrared light output from the color image sensor are captured, a spectral distribution characteristic of light incident on the color image sensor, based on a model expressing the spectral distribution characteristic of incident light, and a generation unit 12 (for example, the visible light image generation unit 103 and the near-infrared light image generation unit 104 illustrated in FIG. 1 and the like) that generates, by using the estimated spectral distribution characteristic and the spectral sensitivity characteristic, visible light image data composed only of information representing visible light and near-infrared light image data composed only of information representing near-infrared light, wherein each integral value of the spectral sensitivity characteristic in a wavelength range of near-infrared light relating to each of the one or more color channels of the color image sensor satisfies a predetermined condition.

(162) With such a configuration, the image processing device is able to easily generate a visible light image signal and a near-infrared light image signal respectively from an image signal captured in a wavelength range from visible light to near-infrared light.

(163) Further, the generation unit 12 may generate the visible light image data composed only of information representing visible light, by using a product of the estimated spectral distribution characteristic by the spectral sensitivity characteristic in a wavelength range of visible light, and may generate the near-infrared light image data composed only of information representing near-infrared light, by using a product of the spectral distribution characteristic by the spectral sensitivity characteristic in a wavelength range of near-infrared light.

(164) With such a configuration, the image processing device is able to adjust a wavelength range of visible light and a wavelength range of near-infrared light associated with an image signal to be generated.

(165) Further, the generation unit 12 may generate the visible light image data composed only of information representing visible light, by using a product of the estimated spectral distribution characteristic by the spectral sensitivity characteristic in a wavelength range of visible light, and may generate the near-infrared light image data composed only of information representing near-infrared light, by subtracting the generated visible light image data from the image data in which visible light and near-infrared light are captured.

(166) Further, the generation unit 12 may generate the near-infrared light image data composed only of information representing near-infrared light, by using a product of the estimated spectral distribution characteristic by the spectral sensitivity characteristic in a wavelength range of near-infrared light, and may generate the visible light image data composed only of information representing visible light, by subtracting the generated near-infrared light image data from the image data in which visible light and near-infrared light are captured.

(167) With such a configuration, the image processing device is able to reduce a cost relating to generation of an image signal.

(168) Further, the spectral distribution characteristic of incident light may be modeled by a weighted sum of basis vectors r.sub.i() (i=1 to M) of the spectral distribution characteristic, or may be modeled by a weighted sum of basis vectors r.sub.i() (i=1 to M+1), with use of a relatively small number of parameters. Alternatively, the spectral distribution characteristic of incident light may be modeled by the weighted sum of the basis vectors r.sub.i() (i=1 to M) and a mean vector r.sub.0() of the spectral distribution characteristic, or may be modeled by the weighted sum of the basis vectors r.sub.i() (i=1 to M+1) and the mean vector r.sub.0().

(169) For example, the model expressing the spectral distribution characteristic of incident light may be represented by either a sum of a mean vector of the spectral distribution characteristic of incident light and a weighted sum of basis vectors as many as the number of the one or more color channels included in the image data, or a weighted sum of basis vectors as many as the number incremented by 1 to the number of the color channels.

(170) With such a configuration, the image processing device is able to easily generate a visible light image signal and a near-infrared light image signal respectively by finding a solution of an observation equation.

(171) Further, the one or more color channels included in the image data may be a red channel, a green channel, and a blue channel.

(172) With such a configuration, the image processing device is able to support image data of RGB channels.

(173) Further, the color image sensor may be an image sensor included in a three-CCD image capturing device.

(174) With such a configuration, the image processing device is able to support image data captured with the three-CCD image capturing device.

(175) Further, the image processing device 10 may include an acquisition unit that acquires image data including one or more color channels by capturing, using the color image sensor, incident light including visible light and near-infrared light.

(176) In the above, the invention of the present application has been described with reference to the example embodiments and examples. However, the invention of the present application is not limited to the above-described example embodiments and examples. Various modifications that can be understood by a person skilled in the art can be made to the configurations and details of the invention of the present application within the scope of the invention of the present application.

(177) In the above, the present invention has been described using each of the example embodiments described above as an exemplary example. However, the present invention is not limited to the above-described example embodiments. In other words, various modes that can be understood by a person skilled in the art can be applied to the present invention within the scope of the present invention.

(178) This application is based upon and claims the benefit of priority from Japanese Patent Application No. 2016-249056, filed on Dec. 22, 2016, the disclosure of which is incorporated herein in its entirety.

REFERENCE SIGNS LIST

(179) 10, 100 Image processing device 11 Estimation unit 12 Generation unit 101 Image data acquisition unit 102 Incident light spectrum estimation unit 103 Visible light image generation unit 104 Near-infrared light image generation unit 105 Signal separation unit 110 External means 200, 300 Image capturing device 201 Infrared light cut filter 202 Optical filter 203 Photosensor 204, 320 Image processing unit 205, 330 Spectral sensitivity characteristic storage memory 301 Prism 311 to 313 Photosensor