Image generation apparatus, image generation method, and program

Abstract

Provided is an image generation technology capable of suppressing unpleasantness associated with fluctuation in image quality caused by the viewer's viewpoint movement. A synthesis ratio determination unit of an image generation apparatus includes: an image quality evaluation index calculation unit that generates a synthesis image J.sub.1 from an observation viewpoint image I.sub.1 and an intermediate viewpoint image I.sub.2, and a synthesis image J.sub.3 from an observation viewpoint image I.sub.3 and an intermediate viewpoint image I.sub.2, for a plurality of synthesis ratios, calculates an image quality evaluation index in an observation viewpoint V.sub.1, an image quality evaluation index in an intermediate viewpoint V.sub.2, and an image quality evaluation index in an observation viewpoint V.sub.3 by using the synthesis images J.sub.1 and J.sub.3, and calculates a variation v of an image quality evaluation index by using the image quality evaluation index in the observation viewpoint V.sub.1, the image quality evaluation index in the intermediate viewpoint V.sub.2, and the image quality evaluation index in the observation viewpoint V.sub.3; and an image quality evaluation index comparison unit that determines a synthesis ratio A based on the variation v of the image quality evaluation index.

Claims

1. An image generation apparatus wherein: k as a parameter representing an integer, and P.sub.1 and P.sub.3 as projectors; and a viewpoint corresponding to an installation position of a projector P.sub.2k−1 as an observation viewpoint V.sub.2k−1 (1≤k≤2), a viewpoint corresponding to an intermediate position of installation positions of the two adjacent projectors P.sub.1 and P.sub.3 as an intermediate viewpoint V.sub.2, an image in the observation viewpoint V.sub.2k−1 as an observation viewpoint image I.sub.2k−1 (1≤k≤2), and an image in the intermediate viewpoint V.sub.2 as an intermediate viewpoint image I.sub.2, the image generation apparatus comprising: a synthesis ratio determiner configured to determine a synthesis ratio A by using an observation viewpoint image I.sub.1, the intermediate viewpoint image I.sub.2, and an observation viewpoint image I.sub.3; and an output image generator configured to generate an output image S.sub.1 of the projector P.sub.1 from the observation viewpoint image I.sub.1 and the intermediate viewpoint image I.sub.2, and an output image S.sub.3 of the projector P.sub.3 from the observation viewpoint image I.sub.3 and the intermediate viewpoint image I.sub.2, by using the synthesis ratio A, wherein the synthesis ratio determiner includes: an image quality evaluation index determiner configured to generate a synthesis image J.sub.1 from the observation viewpoint image I.sub.1 and the intermediate viewpoint image I.sub.2, and a synthesis image J.sub.3 from the observation viewpoint image I.sub.3 and the intermediate viewpoint image I.sub.2, for a plurality of synthesis ratios (each synthesis ratio is a real number 0 or greater and 1 or smaller), determine an image quality evaluation index in an observation viewpoint V.sub.1, an image quality evaluation index in the intermediate viewpoint V.sub.2, and an image quality evaluation index in an observation viewpoint V.sub.3 by using the synthesis images J.sub.1 and J.sub.3, and determine a variation v of an image quality evaluation index by using the image quality evaluation index in the observation viewpoint V.sub.1, the image quality evaluation index in the intermediate viewpoint V.sub.2, and the image quality evaluation index in the observation viewpoint V.sub.3; and an image quality evaluation index comparator configured to determine the synthesis ratio A based on the variation v of the image quality evaluation index.

2. The image generation apparatus according to claim 1, wherein the image generation apparatus generates a three-dimensional image with motion parallax using a plurality of projectors.

3. An image generation apparatus wherein: K as an integer greater than or equal to 3, k as an parameter representing an integer, and P.sub.1, P.sub.3, . . . , P.sub.2K−1 as projectors installed aligned in a single row; a viewpoint corresponding to an installation position of a projector P.sub.2k−1 as an observation viewpoint V.sub.2k−1 (1≤k≤K), a viewpoint corresponding to an intermediate position of installation positions of two adjacent projectors P.sub.2k−1 and P.sub.2k+1 as an intermediate viewpoint V.sub.2k (1≤k≤K−1), an image in the observation viewpoint V.sub.2k−1 as an observation viewpoint image I.sub.2k−1 (1≤k≤K), and an image in the intermediate viewpoint V.sub.2k as an intermediate viewpoint image I.sub.2k (1≤k≤K−1); and a set of the observation viewpoint image I.sub.2k−1, the intermediate viewpoint image I.sub.2k, and an observation viewpoint image I.sub.2k+1 as a block B.sub.k (1≤k≤K−1), the image generation apparatus comprising: a synthesis ratio determiner configured to determine a synthesis ratio A.sub.k in the block B.sub.k by using an intermediate viewpoint image I.sub.2k−2, the observation viewpoint image I.sub.2k−1, the intermediate viewpoint image I.sub.2k, the observation viewpoint image I.sub.2k+1, and an intermediate viewpoint image I.sub.2k+2, for the block B.sub.k (1≤k≤K−1) (where I.sub.0=I.sub.2, I.sub.2K=I.sub.2K−2); and an output image generator configured to generate an output image S.sub.2k−1 of the projector P.sub.2k−1 from the intermediate viewpoint image I.sub.2k−2, the observation viewpoint image I.sub.2k−1, and the intermediate viewpoint image I.sub.2k by using synthesis ratios A.sub.k−1, A.sub.k, for k satisfying 1≤k≤K (where A.sub.0=A.sub.1, A.sub.K=A.sub.K−1, I.sub.0=I.sub.2, I.sub.2K=I.sub.2K−2), wherein the synthesis ratio determiner includes: an image quality evaluation index determiner configured to generate a synthesis image J.sub.2k−1 from the intermediate viewpoint image I.sub.2k−2, the observation viewpoint image I.sub.2k−1, and the intermediate viewpoint image I.sub.2k, and a synthesis image J.sub.2k+1 from the intermediate viewpoint image I.sub.2k, the observation viewpoint image I.sub.2k+1, and the intermediate viewpoint image I.sub.2k+2, for a plurality of synthesis ratios (each synthesis ratio is a real number 0 or greater and 1 or smaller), determine an image quality evaluation index in the observation viewpoint V.sub.2k−1, an image quality evaluation index in the intermediate viewpoint V.sub.2k, and an image quality evaluation index in an observation viewpoint V.sub.2k+1 by using the synthesis images J.sub.2k−1, J.sub.2k+1, and determine a variation v.sub.k of an image quality evaluation index in the block B.sub.k by using the image quality evaluation index in the observation viewpoint V.sub.2k−1, the image quality evaluation index in the intermediate viewpoint V.sub.2k, and the image quality evaluation index in the observation viewpoint V.sub.2k+1; and an image quality evaluation index comparator configured to determine the synthesis ratio A.sub.k based on the variation v.sub.k of the image quality evaluation index in the block B.sub.k.

4. The image generation apparatus according to claim 3, wherein the image generation apparatus generates a three-dimensional image with motion parallax using a plurality of projectors.

5. The image generation apparatus according to claim 4, wherein the three dimensional image is viewable by the naked eye based on a linear blending mechanism.

6. The image generation apparatus according to claim 5, wherein the three dimensional image is viewable by the naked eye based on a linear blending mechanism.

7. An image generation apparatus wherein: K as an integer greater than or equal to 3, k as an parameter representing an integer, and P.sub.1, P.sub.3, . . . , P.sub.2K−1 as projectors installed aligned in a circular alignment, the projectors being for projecting an image onto a circular screen; a viewpoint corresponding to an installation position of a projector P.sub.2k−1 as an observation viewpoint V.sub.2k−1 (1≤k≤K), a viewpoint corresponding to an intermediate position of installation positions of two adjacent projectors P.sub.2k−1 and P.sub.2k+1 (where P.sub.2K+1=P.sub.1) as an intermediate viewpoint V.sub.2k (1≤k≤K), an image in the observation viewpoint V.sub.2k−1 as an observation viewpoint image I.sub.2k−1 (1≤k≤K), and an image in the intermediate viewpoint V.sub.2k as an intermediate viewpoint image I.sub.2k (1≤k≤K); and a set of the observation viewpoint image I.sub.2k−1, the intermediate viewpoint image I.sub.2k, and an observation viewpoint image I.sub.2k+1 (where I.sub.2K+1=I.sub.1) as a block B.sub.k (1≤k≤K), the image generation apparatus comprising: a synthesis ratio determiner configured to determine a synthesis ratio A.sub.k in the block B.sub.k by using an intermediate viewpoint image I.sub.2k−2, the observation viewpoint image I.sub.2k−1, the intermediate viewpoint image I.sub.2k, the observation viewpoint image I.sub.2k+1, and an intermediate viewpoint image I.sub.2k+2, for the block B.sub.k (1≤k≤K) (where I.sub.0=I.sub.2K, I.sub.2K+1=I.sub.1, I.sub.2K+2=I.sub.2); and an output image generator configured to generate an output image S.sub.2k2−1 of the projector P.sub.2k−1 from the intermediate viewpoint image I.sub.2k−2, the observation viewpoint image I.sub.2k−1, and the intermediate viewpoint image I.sub.2k by using synthesis ratios A.sub.k−1, A.sub.k, for k satisfying 1≤k≤K (where A.sub.0=A.sub.K, I.sub.0=I.sub.2K), wherein the synthesis ratio determiner includes: an image quality evaluation index determiner configured to generate a synthesis image J.sub.2k−1 from the intermediate viewpoint image I.sub.2k−2, the observation viewpoint image and the intermediate viewpoint image I.sub.2k, and a synthesis image J.sub.2k+1 from the intermediate viewpoint image I.sub.2k, the observation viewpoint image I.sub.2k+1, and the intermediate viewpoint image I.sub.2k+2, for a plurality of synthesis ratios (each synthesis ratio is a real number 0 or greater and 1 or smaller), determine an image quality evaluation index in the observation viewpoint V.sub.2k−1, an image quality evaluation index in the intermediate viewpoint V.sub.2k, and an image quality evaluation index in an observation viewpoint V.sub.2k+1 by using the synthesis images J.sub.2k−1, J.sub.2k+1, and determine a variation v.sub.k of an image quality evaluation index in the block B.sub.k by using the image quality evaluation index in the observation viewpoint V.sub.2k−1, the image quality evaluation index in the intermediate viewpoint V.sub.2k, and the image quality evaluation index in the observation viewpoint V.sub.2k+1 (where J.sub.2K+1=J.sub.1, V.sub.2K+1=V.sub.1); and an image quality evaluation index comparison unit that determines the synthesis ratio A.sub.k based on the variation v.sub.k of the image quality evaluation index in the block B.sub.k.

8. The image generation apparatus according to claim 7, wherein the image generation apparatus generates a three-dimensional image with motion parallax using a plurality of projectors.

9. The image generation apparatus according to claim 8, wherein the three dimensional image is viewable by the naked eye based on a linear blending mechanism.

Description

BRIEF DESCRIPTION OF DRAWINGS

(1) FIG. 1 is a diagram illustrating linear blending.

(2) FIG. 2 is a block diagram illustrating a configuration of an image generation apparatus 100.

(3) FIG. 3 is a flowchart illustrating an operation of the image generation apparatus 100.

(4) FIG. 4 is a block diagram illustrating a configuration of a synthesis ratio determination unit 120.

(5) FIG. 5 is a flowchart illustrating an operation of the synthesis ratio determination unit 120.

(6) FIG. 6 is a diagram illustrating a state of processing performed by the synthesis ratio determination unit 120.

(7) FIG. 7 is a block diagram illustrating a configuration of an image generation apparatus 200.

(8) FIG. 8 is a flowchart illustrating an operation of the image generation apparatus 200.

(9) FIG. 9 is a block diagram illustrating a configuration of a synthesis ratio determination unit 220.

(10) FIG. 10 is a flowchart illustrating an operation of the synthesis ratio determination unit 220.

(11) FIG. 11 is a diagram illustrating a state of processing performed by the synthesis ratio determination unit 220 and the output image generation unit 230 (k=4).

(12) FIG. 12 is a block diagram illustrating a configuration of an image generation apparatus 300.

(13) FIG. 13 is a flowchart illustrating an operation of the image generation apparatus 300.

(14) FIG. 14 is a block diagram illustrating a configuration of a synthesis ratio determination unit 320.

(15) FIG. 15 is a flowchart illustrating an operation of the synthesis ratio determination unit 320.

(16) FIG. 16 is a diagram illustrating a state of processing performed by the synthesis ratio determination unit 320 an output image generation unit 330 (k=4).

(17) FIG. 17 is a block diagram illustrating a configuration of an image generation apparatus 400.

(18) FIG. 18 is a flowchart illustrating an operation of the image generation apparatus 400.

(19) FIG. 19 is a block diagram illustrating a configuration of an image generation apparatus 500.

(20) FIG. 20 is a flowchart illustrating an operation of the image generation apparatus 500.

(21) FIG. 21 is a diagram illustrating an example of a functional configuration of a computer realizing each apparatus according to embodiments of the present invention.

DESCRIPTION OF EMBODIMENTS

(22) Hereinafter, embodiments of the present invention will be described in detail. Components having the same function are denoted by the same reference signs, and redundant description thereof will be omitted.

(23) Prior to describing each embodiment, the method of notation herein will be described.

(24) {circumflex over ( )} (caret) represents the superscript. For example, x.sup.y{circumflex over ( )}z represents y.sup.z is the superscript to x, and x.sub.y{circumflex over ( )}z represents y.sup.z is the subscript to x. _ (underscore) represents the subscript. For example, x.sup.y_z represents y.sub.z is the superscript to x, and x.sub.y_z represents y.sub.z is the subscript to x.

(25) A superscript “{circumflex over ( )}” or “{tilde over ( )}”, such as {circumflex over ( )}x or {tilde over ( )}x to a character x, should be described otherwise above “x”, but are described as {circumflex over ( )}x or {tilde over ( )}x, under the limitations of the written description herein.

(26) The viewpoint corresponding to the installation position of the projector is referred to as an observation viewpoint, and the viewpoint corresponding to the intermediate position of the installation positions of two adjacent projectors is referred to as an intermediate viewpoint. The image in the observation viewpoint is referred to as an observation viewpoint image and the image in the intermediate viewpoint is referred to as an intermediate viewpoint image.

First Embodiment

(27) The image generation apparatus 100 generates an image (hereinafter referred to as an output image) output by two projectors. These two projectors are referred to as P.sub.1 and P.sub.3. The viewpoint corresponding to the installation position of the projector P.sub.2k−1 is referred to as an observation viewpoint V.sub.2k−1 (1≤k≤2), the viewpoint corresponding to the intermediate position of the installation positions of the two adjacent projectors P.sub.1 and P.sub.3 is referred to as an intermediate viewpoint V.sub.2, the image corresponding to the observation viewpoint V.sub.2k−1 is referred to as an observation viewpoint image I.sub.2k−1 (1≤k≤2), and the image in the intermediate viewpoint V.sub.2 is referred to as an intermediate viewpoint image I.sub.2. Here, k is a parameter representing an integer.

(28) Hereinafter, an image generation apparatus 100 will be described with reference to FIGS. 2 and 3. FIG. 2 is a block diagram illustrating a configuration of the image generation apparatus 100. FIG. 3 is a flowchart illustrating an operation of the image generation apparatus 100. As illustrated in FIG. 2, the image generation apparatus 100 includes a viewpoint image acquisition unit 110, a synthesis ratio determination unit 120, an output image generation unit 130, and a recording unit 190. The recording unit 190 is a component configured to appropriately record information required for processing of the image generation apparatus 100.

(29) The operation of the image generation apparatus 100 will be described in accordance with FIG. 3. In step S110, the viewpoint image acquisition unit 110 acquires and outputs an observation viewpoint image I.sub.1, an intermediate viewpoint image I.sub.2, and an observation viewpoint image I.sub.3. Each viewpoint image is, for example, an image capturing a certain photographic subject by a camera installed at a position corresponding to each of the viewpoints.

(30) In step S120, the synthesis ratio determination unit 120 uses the observation viewpoint image I.sub.1, the intermediate viewpoint image I.sub.2, and the observation viewpoint image I.sub.3 acquired at S110 as inputs, and determines and outputs the synthesis ratio A by using the observation viewpoint image I.sub.1, the intermediate viewpoint image I.sub.2, and the observation viewpoint image I.sub.3. Here, the synthesis ratio A is used when the output image generation unit 130 generates the output image. Hereinafter, the synthesis ratio determination unit 120 will be described with reference to FIGS. 4 to 5. FIG. 4 is a block diagram illustrating a configuration of the synthesis ratio determination unit 120. FIG. 5 is a flowchart illustrating an operation of the synthesis ratio determination unit 120. As illustrated in FIG. 4, the synthesis ratio determination unit 120 includes an image quality evaluation index calculation unit 121 and an image quality evaluation index comparison unit 122.

(31) An operation of the synthesis ratio determination unit 120 will be described with reference to FIG. 5. In step S121, the image quality evaluation index calculation unit 121 generates a synthesis image J.sub.1 from the observation viewpoint image I.sub.1 and the intermediate viewpoint image I.sub.2, and a synthesis image J.sub.3 from the observation viewpoint image I.sub.3 and the intermediate viewpoint image I.sub.2 for N synthesis ratios an (where N is an integer of 1 or greater) (0≤a.sub.n≤1, 1≤n≤N, n is a parameter represent an integer). The image quality evaluation index calculation unit 121 uses the synthesis images J.sub.1 and J.sub.3 to calculate the image quality evaluation index in the observation viewpoint V.sub.1, the image quality evaluation index in the intermediate viewpoint V.sub.2, and the image quality evaluation index in the observation viewpoint V.sub.3. The image quality evaluation index calculation unit 121 calculates a variation v (a.sub.n) of the image quality evaluation index by using the image quality evaluation index in the observation viewpoint V.sub.1, the image quality evaluation index in the intermediate viewpoint V.sub.2, and the image quality evaluation index in the observation viewpoint V.sub.3. The variation v (a.sub.n) of the image quality evaluation index can, for example, use an absolute value of the difference between the average value of the image quality evaluation index in the two observation viewpoints and the image quality evaluation in the intermediate viewpoint. Here, the synthesis ratio an is a value that gradually changes from 0 to 1, and is, for example, a value that changes from 0 to 1 by 0.1 (that is, a.sub.1=0, a.sub.2=0.1, a.sub.3=0.2, . . . , a.sub.11=1). For an image quality evaluation index, for example, a mean squared error (MSE), a peak signal-to-noise ratio (PSNR), and a structural similarity (SSIM) can be used. Specific description will be given with reference to FIG. 6. The synthesis image J.sub.1 at the observation viewpoint position V.sub.1 is generated as (1−a.sub.n)×I.sub.1+a.sub.n×I.sub.2. Similarly, the synthesis image J.sub.3 at the observation viewpoint position V.sub.3 is generated as (1−a.sub.n)×I.sub.3+a.sub.n×I.sub.2. Thus, the perceived image at the observation viewpoint position V.sub.1 and the perceived image at the observation viewpoint position V.sub.3 are the synthesis image J.sub.1 at the observation viewpoint position V.sub.1 and the synthesis image J.sub.3 at the observation viewpoint position V.sub.3. The perceived image at the intermediate viewpoint position V.sub.2 is the average of the synthesis image J.sub.1 at the observation viewpoint position V.sub.1 and the synthesis image J.sub.3 at the observation viewpoint position V.sub.3 (that is, ½×(J.sub.1+J.sub.3)). Then, the image quality evaluation index Ind (V.sub.1, a.sub.n) of the perceived image at the observation viewpoint position V.sub.1 with respect to the synthesis ratio a.sub.n, the image quality evaluation index Ind (V.sub.2, a.sub.n) of the perceived image at the intermediate viewpoint position V.sub.2 with respect to the synthesis ratio a.sub.n, and the image quality evaluation index Ind (V.sub.3, a.sub.n) of the perceived image at the observation viewpoint position V.sub.3 with respect to the synthesis ratio a.sub.n are calculated, and the absolute value of the difference with respect to the synthesis ratio a.sub.n, D (a.sub.n)=|Ind(V.sub.2, a.sub.n)−½×(Ind (V.sub.1, a.sub.n)+Ind (V.sub.3, a.sub.n))| is calculated as the variation v (a.sub.n) of the image quality evaluation index. Note that the absolute value D (a.sub.n) of the difference used in the calculation of the variation v (a.sub.n) of the image quality evaluation index is merely an example, and other methods may be used.

(32) In step S122, the image quality evaluation index comparison unit 122 determines a synthesis ratio a.sub.n that minimizes the variation v (a.sub.n) (1≤n≤N) of the image quality evaluation index as the synthesis ratio A. Note that it is an example to determine the synthesis ratio A as the synthesis ratio a.sub.n where the variation v (a.sub.n) (1≤n≤N) in the image quality evaluation index is minimized, and any determination method may be used as long as the image quality evaluation index comparison unit 122 determines the synthesis ratio A based on the variation v (a.sub.n) (1≤n≤N) of the image quality evaluation index. For example, one of variations v (a.sub.n) of the image quality evaluation index smaller than a predetermined threshold value may be determined as the synthesis ratio A, by using the predetermined threshold value (such as an acceptable value as a valuation in the image quality evaluation index defined by an image quality evaluation experiment or the like). Note that in a case where there is no variation v (a.sub.n) in the image quality evaluation index that is smaller than the threshold value, the synthesis ratio a.sub.n where the variation v (a.sub.n) (1≤n≤N) in the image quality evaluation index is minimized may be determined as the synthesis ratio A.

(33) In step S130, the output image generation unit 130 uses the observation viewpoint image I.sub.1, the intermediate viewpoint image I.sub.2, the observation viewpoint image I.sub.3 acquired at S110, and the synthesis ratio A determined at S120 as inputs to generate and output the output image S.sub.1 of the projector P.sub.1 from the observation viewpoint image I.sub.1 and the intermediate viewpoint image I.sub.2 and the output image S.sub.3 of the projector P.sub.3 from the observation viewpoint image I.sub.3 and the intermediate viewpoint image I.sub.2 by using the synthesis ratio A. Here, the output image S.sub.1 in the observation viewpoint position V.sub.1 is generated as (1−A)×I.sub.1+A×I.sub.2. Similarly, the output image S.sub.3 in the observation viewpoint position V.sub.3 is generated as (1−A)×I.sub.3+A×I.sub.2.

Modified Example

(34) In the above description, N synthesis ratio a.sub.n used in the generation of the synthesis image J.sub.1 and the synthesis image J.sub.3 have been described as being the same, but they are not necessary to be the same. For example, N.sub.1 synthesis ratios a.sub.1, n_1 may be used to generate the synthesis image J.sub.1, and N.sub.3 synthesis ratios a.sub.3, n_3 may be used to generate the synthesis image J.sub.3. In other words, in step S121, the image quality evaluation index calculation unit 121 generates the synthesis image J.sub.1 from the observation viewpoint image I.sub.1 and the intermediate viewpoint image I.sub.2 with respect to the N.sub.1 (N.sub.1 is an integer of 1 or greater) synthesis ratios a.sub.1, n1 (0≤a.sub.1, n_1≤1, 1≤n.sub.1≤N.sub.1, n.sub.1 is a parameter representing an integer), and the synthesis image J.sub.3 from the observation viewpoint image I.sub.3 and the intermediate viewpoint image I.sub.2 with respect to the N.sub.3 (N.sub.3 is an integer of 1 or greater) synthesis ratios a.sub.3, n3 (0≤a.sub.3, n_3≤1, 1≤n.sub.3≤N.sub.3, n.sub.3 is a parameter representing an integer). The image quality evaluation index calculation unit 121 uses the synthesis images J.sub.1 and J.sub.3 to calculate the image quality evaluation index in the observation viewpoint V.sub.1, the image quality evaluation index in the intermediate viewpoint V.sub.2, and the image quality evaluation index in the observation viewpoint V.sub.3. The image quality evaluation index calculation unit 121 calculates a variation v (a.sub.1, n_1, a.sub.3, n_3) of the image quality evaluation index by using the image quality evaluation index in the observation viewpoint V.sub.1, the image quality evaluation index in the intermediate viewpoint V.sub.2, and the image quality evaluation index in the observation viewpoint V.sub.3. In step S122, the image quality evaluation index comparison unit 122 determines synthesis ratios a.sub.1, n_1, a.sub.3, n_3 as a synthesis ratio (set) A (=(A.sub.(1), A.sub.(3))) in which a variation v (a.sub.1, n_1, a.sub.3, n_.sub.3) (1≤n1≤N.sub.1, 1≤n.sub.3≤N.sub.3) of the image quality evaluation index is minimized. In step S130, the output image generation unit 130 uses the observation viewpoint image I.sub.1, the intermediate viewpoint image I.sub.2, the observation viewpoint image I.sub.3 acquired at S110, and the synthesis ratio (set) A determined at S120 as inputs to generate and output the output image S.sub.1 of the projector P.sub.1 from the observation viewpoint image I.sub.1 and the intermediate viewpoint image I.sub.2 and the output image S.sub.3 of the projector P.sub.3 from the observation viewpoint image I.sub.3 and the intermediate viewpoint image I.sub.2 by using the synthesis ratio (set) A.

(35) According to the invention of the present embodiment, it is possible to suppress unpleasantness associated with fluctuation in image quality caused by the viewer's viewpoint movement. The reason will be described below. Adding another viewpoint image to the viewpoint image degrades the image quality from the viewpoint image that is originally observed at that viewpoint. That is, by adding an intermediate viewpoint image to the output image, the image quality of the output image deteriorates, while the image quality of the intermediate viewpoint is relatively improved.

Second Embodiment

(36) The image generation apparatus 200 generates an image (hereinafter referred to as an output image) that is output by K projectors (K is an integer of 3 or greater). These K projectors are referred to as P.sub.1, P.sub.3, . . . , P.sub.2K−1. The projectors P.sub.1, P.sub.3, . . . , P.sub.2K−1 are installed aligned in a single row. For example, the projectors P.sub.1, P.sub.3, . . . , P.sub.2K−1 may be installed on a straight line, and each projector may be a projector that projects an image onto a planar screen.

(37) The viewpoint corresponding to the installation position of the projector P.sub.2k−1 is referred to as an observation viewpoint V.sub.2k−1 (1≤k≤K), the viewpoint corresponding to the intermediate position of the installation positions of the two adjacent projectors P.sub.2K−1 and P.sub.2K+1 is referred to as an intermediate viewpoint V.sub.2k (1≤k≤K−1), the image corresponding to the observation viewpoint V.sub.2k−1 is referred to as an observation viewpoint image I.sub.2k−1 (1≤k≤K), and the image in the intermediate viewpoint V.sub.2k is referred to as an intermediate viewpoint image I.sub.2k (1≤k≤K−1). Here, k is a parameter representing an integer.

(38) The set of the observation viewpoint image I.sub.2k−1, the intermediate viewpoint image I.sub.2k, and the observation viewpoint image I.sub.2k+1 are referred to as a block B.sub.k (1≤k≤K−1).

(39) Hereinafter, an image generation apparatus 200 will be described with reference to FIGS. 7 and 8. FIG. 7 is a block diagram illustrating a configuration of the image generation apparatus 200. FIG. 8 is a flowchart illustrating an operation of the image generation apparatus 200. As illustrated in FIG. 7, the image generation apparatus 200 includes a viewpoint image acquisition unit 110, a synthesis ratio determination unit 220, an output image generation unit 230, and a recording unit 190. The recording unit 190 is a component configured to appropriately record information required for processing of the image generation apparatus 200.

(40) The operation of the image generation apparatus 200 will be described in accordance with FIG. 8. In step S110, the viewpoint image acquisition unit 110 acquires and outputs K observation viewpoint images I.sub.2k−1 (1≤k≤K) and K−1 intermediate viewpoint images I.sub.2k (1≤k≤K−1).

(41) In step S220, the synthesis ratio determination unit 220 uses K observation viewpoint images I.sub.2k−1 (1≤k≤K) and K−1 intermediate viewpoint images I.sub.2k (1≤k≤K−1) acquired at S110 as inputs, determines the synthesis ratio A.sub.k in the block B.sub.k by using the intermediate viewpoint image I.sub.2k−2, the observation viewpoint image the intermediate viewpoint image I.sub.2k, the observation viewpoint image I.sub.2k+1, and the intermediate viewpoint image I.sub.2k+2, for the block B.sub.k (1≤k≤K−1), and outputs K−1 synthesis ratios A.sub.k (1≤k≤K−1) (where I.sub.0=I.sub.2, I.sub.2K=I.sub.2K−2). Here, the synthesis ratio A.sub.k is used when the output image generation unit 230 generates the output image. Hereinafter, the synthesis ratio determination unit 220 will be described with reference to FIGS. 9 to 10. FIG. 9 is a block diagram illustrating a configuration of the synthesis ratio determination unit 220. FIG. 10 is a flowchart illustrating an operation of the synthesis ratio determination unit 220. As illustrated in FIG. 9, the synthesis ratio determination unit 220 includes an initialization unit 225, an image quality evaluation index calculation unit 221, an image quality evaluation index comparison unit 222, and an end condition determination unit 226.

(42) An operation of the synthesis ratio determination unit 220 will be described with reference to FIG. 10. In step S225, the initialization unit 225 sets the value of k, which is a parameter representing the number of repetitions, to 1.

(43) In step S221, the image quality evaluation index calculation unit 221 generates the synthesis image J.sub.2k−1 from the intermediate viewpoint image I.sub.2k−2, the observation viewpoint image I.sub.2k−1, and the intermediate viewpoint image I.sub.2k, and the synthesis image J.sub.2k+1 from the intermediate viewpoint image I.sub.2k, the observation viewpoint image I.sub.2k+1, and the intermediate viewpoint image I.sub.2k+2, for N (where N is an integer of 1 or greater) synthesis ratios a.sub.n (0≤a.sub.n≤1, 1≤n≤N, n is a parameter representing an integer), uses the synthesis images J.sub.2k−1 and J.sub.2k+1 to calculate the image quality evaluation index in the observation viewpoint V.sub.2k−1, the image quality evaluation index in the intermediate viewpoint V.sub.2k, and the image quality evaluation index in the observation viewpoint V.sub.2k+1, and uses the image quality evaluation index in the observation viewpoint V.sub.2k−1, the image quality evaluation index in the intermediate viewpoint V.sub.2k, and the image quality evaluation index in the observation viewpoint V.sub.2k+1 to calculate the variation v.sub.k (a.sub.n) of the image quality evaluation index in the block B.sub.k. Detailed description will be given below (see FIG. 11). The synthesis image J.sub.2k−1 at the observation viewpoint position V.sub.2k−1 is generated as (1-2a.sub.n)×I.sub.2k−1+a.sub.n×I.sub.2k−2+a.sub.n×I.sub.2k. Similarly, the synthesis image J.sub.2k+1 at the observation point position V.sub.2k+1 is generated as (1-2a.sub.n)×I.sub.2k+1+a.sub.n×I.sub.2k+a.sub.n×I.sub.2k+2. Thus, the perceived image at the observation viewpoint position V.sub.2k−1 and the perceived image at the observation viewpoint position V.sub.2k+1 are the synthesis image J.sub.2k−1 at the observation viewpoint position V.sub.2k−1 and the synthesis image J.sub.2k+1 at the observation viewpoint position V.sub.2k+1. The perceived image at the intermediate viewpoint position V.sub.2k is the average of the synthesis image J.sub.2k−1 at the observation viewpoint position V.sub.2k−1 and the synthesis image J.sub.2k+1 at the observation viewpoint position V.sub.2k+1 (that is, ½×(J.sub.2k−1+J.sub.2k+1)). Then, the image quality evaluation index Ind (V.sub.2k−1, a.sub.n) of the perceived image at the observation viewpoint position V.sub.2k−1 with respect to the synthesis ratio a.sub.n, the image quality evaluation index Ind (V.sub.2k, a.sub.n) of the perceived image at the intermediate viewpoint position V.sub.2k with respect to the synthesis ratio a.sub.n, and the image quality evaluation index Ind (V.sub.2k+1, a.sub.n) of the perceived image at the observation viewpoint position V.sub.2k+1 with respect to the synthesis ratio a.sub.n are calculated, and the absolute value of the difference with respect to the synthesis ratio a.sub.n, D.sub.k (a.sub.n)=|Ind(V.sub.2k, a.sub.n)−½×(Ind(.sub.2k−1, a.sub.n)+Ind (V.sub.2k+1, a.sub.n))| is calculated as the variation v.sub.k (a.sub.n) of the image quality evaluation index. Note that the absolute value D.sub.k (a.sub.n) of the difference used in the calculation of the variation v.sub.k (a.sub.n) of the image quality evaluation index is merely an example, and other methods may be used.

(44) In step S222, the image quality evaluation index comparison unit 222 determines a synthesis ratio a.sub.n that minimizes the variation v.sub.k (a.sub.n) (1≤n≤N) of the image quality evaluation index in the block B.sub.k as the synthesis ratio A. Note that it is an example to determine the synthesis ratio A.sub.k as the synthesis ratio a.sub.n where the variation v.sub.k (a.sub.n) (1≤n≤N) in the image quality evaluation index is minimized, and any determination method may be used as long as the image quality evaluation index comparison unit 222 determines the synthesis ratio A.sub.k based on the variation v.sub.k (a.sub.n) (1≤n≤N) of the image quality evaluation index in the block B.sub.k. For example, one of variations v.sub.k (a.sub.n) of the image quality evaluation index smaller than a predetermined threshold value may be determined as the synthesis ratio A.sub.k, by using the predetermined threshold value (such as an acceptable value as a valuation in the image quality evaluation index defined by an image quality evaluation experiment or the like). Note that in a case where there is no variation v.sub.k (a.sub.n) in the image quality evaluation index that is smaller than the threshold value, the synthesis ratio a.sub.n where the variation v.sub.k (a.sub.n) (1≤n≤N) in the image quality evaluation index is minimized may be determined as the synthesis ratio A.sub.k.

(45) In step S226, the end condition determination unit 226 increments k by 1, in a case where k reaches K (i.e., k>K−1), outputs K−1 synthesis ratios A.sub.k (1≤k≤K−1), and transitions to the processing of S230, or otherwise the process returns to the processing of S221.

(46) In step S230, the output image generation unit 230 uses the K observation viewpoint images I.sub.2k−1 (1≤k≤K) and the K−1 intermediate viewpoint images I.sub.2k (1≤k≤K−1) acquired at S110 and the K−1 synthesis ratios A.sub.k (1≤k≤K−1) determined at S220 as inputs to generate the output images S.sub.2k−1 of the projector P.sub.2k−1 from the intermediate viewpoint images I.sub.2k−2, the observation viewpoint images I.sub.2k−1, and the intermediate viewpoint images I.sub.2k by using the synthesis ratios A.sub.k−1, A.sub.k for k satisfying 1≤k≤K, and outputs K output images S.sub.2k−1 (1≤k≤K) (where A.sub.0=A.sub.1, A.sub.K=A.sub.K−1, I.sub.0=I.sub.2, I.sub.2K=I.sub.2K−2). Here, the output images S.sub.2k−1 at the observation viewpoint position V.sub.2k−1 is generated as (1−(A.sub.k−1+A.sub.k))×I.sub.2k−1+½×(A.sub.k−1+A.sub.k)×I.sub.2k−2+½×(A.sub.k−1+A.sub.k)×I.sub.2k (see FIG. 11).

(47) Note that in S220 and S230, I.sub.0=I.sub.2 and the intermediate viewpoint image I.sub.2 next to the observation viewpoint image I.sub.1 are used, but instead of using the intermediate viewpoint image I.sub.2, the observation viewpoint image I.sub.1 may be used. That is, I.sub.0=I.sub.1 may be used. Note that in S220 and S230, I.sub.2K=I.sub.2K−2 and the intermediate viewpoint image I.sub.2K−2 next to the observation viewpoint image I.sub.2K−1 are used, but instead of using the intermediate viewpoint image I.sub.2K−2, the observation viewpoint image I.sub.2K−1 may be used. That is, I.sub.2k=I.sub.2K−1 may be used.

Modified Example

(48) In the above description, the N synthesis ratios a.sub.n used in the generation of the synthesis images J.sub.2k−1 and J.sub.2k+1 have been described as being the same, but they are not necessary to be the same in the same manner as in the first embodiment. In this case, K−1 synthesis ratios (set) A.sub.k (1≤k≤K−1) is used.

(49) According to the invention of the present embodiment, it is possible to suppress unpleasantness associated with fluctuation in image quality caused by the viewer's viewpoint movement.

Third Embodiment

(50) The image generation apparatus 300 generates an image (hereinafter referred to as an output image) that is output by K projectors (K is an integer of 3 or greater). These K projectors are referred to as P.sub.1, P.sub.3, . . . , P.sub.2K−1. The projectors P.sub.1, P.sub.3, . . . , P.sub.2K−1 are installed aligned in a circular alignment. For example, the projectors P.sub.1, P.sub.3, . . . , P.sub.2K−1 may be installed on a circle, and each projector may be a projector that projects an image onto a circular screen. For example, K=60 may be used. The present embodiment is not limited to K=60, but generally the smaller the K (i.e., the smaller the number of projectors), the worse the image quality of the perceived image at the intermediate viewpoint position. The circle may not be exactly circular, but for example it may be a predetermined closed curve.

(51) The viewpoint corresponding to the installation position of the projector P.sub.2k−1 is referred to as an observation viewpoint V.sub.2k−1 (1≤k≤K), the viewpoint corresponding to the intermediate position of the installation positions of the two adjacent projectors P.sub.2K-1 and P.sub.2K+1 (where P.sub.2K+1=P.sub.1) is referred to as an intermediate viewpoint V.sub.2k (1≤k≤K), the image corresponding to the observation viewpoint V.sub.2k−1 is referred to as an observation viewpoint image I.sub.2k−1 (1≤k≤K), and the image in the intermediate viewpoint V.sub.2k is referred to as an intermediate viewpoint image I.sub.2k (1≤k≤K). Here, k is a parameter representing an integer.

(52) The set of the observation viewpoint image the intermediate viewpoint image I.sub.2k, and the observation viewpoint image I.sub.2k+1 are referred to as a block B.sub.k (1≤k≤K) (where I.sub.2K+1=I.sub.1).

(53) Hereinafter, an image generation apparatus 300 will be described with reference to FIGS. 12 and 13. FIG. 12 is a block diagram illustrating a configuration of the image generation apparatus 300. FIG. 13 is a flowchart illustrating an operation of the image generation apparatus 300. As illustrated in FIG. 12, the image generation apparatus 300 includes a viewpoint image acquisition unit 110, a synthesis ratio determination unit 320, an output image generation unit 330, and a recording unit 190. The recording unit 190 is a component configured to appropriately record information required for processing of the image generation apparatus 300.

(54) The operation of the image generation apparatus 300 will be described in accordance with FIG. 13. In step S110, the viewpoint image acquisition unit 110 acquires and outputs K observation viewpoint images I.sub.2k−1 (1≤k≤K) and K intermediate viewpoint images I.sub.2k (1≤k≤K).

(55) In step S320, the synthesis ratio determination unit 320 uses K observation viewpoint images I.sub.2k−1 (1≤k≤K) and K intermediate viewpoint images I.sub.2k (1≤k≤K) acquired at S110 as inputs, determines the synthesis ratio A.sub.k in the block B.sub.k by using the intermediate viewpoint image I.sub.2k−2, the observation viewpoint image I.sub.2k−1, the intermediate viewpoint image I.sub.2k, the observation viewpoint image I.sub.2k+1, and the intermediate viewpoint image I.sub.2k+2, for the block B.sub.k (1≤k≤K), and outputs K synthesis ratios A.sub.k (1≤k≤K) (where I.sub.0=I.sub.2K, I.sub.2K+1=I.sub.1, I.sub.2K+2=I.sub.2). Here, the synthesis ratio A.sub.k is used when the output image generation unit 330 generates the output image. Hereinafter, the synthesis ratio determination unit 320 will be described with reference to FIGS. 14 to 15. FIG. 14 is a block diagram illustrating a configuration of the synthesis ratio determination unit 320. FIG. 15 is a flowchart illustrating an operation of the synthesis ratio determination unit 320. As illustrated in FIG. 14, the synthesis ratio determination unit 320 includes an initialization unit 325, an image quality evaluation index calculation unit 321, an image quality evaluation index comparison unit 322, and an end condition determination unit 326.

(56) An operation of the synthesis ratio determination unit 320 will be described with reference to FIG. 15. In step S325, the initialization unit 325 sets the value of k, which is a parameter representing the number of repetitions, to 1.

(57) In step S321, the image quality evaluation index calculation unit 321 generates the synthesis image J.sub.2k−1 from the intermediate viewpoint image I.sub.2k−2, the observation viewpoint image I.sub.2k−1, and the intermediate viewpoint image I.sub.2k, and the synthesis image J.sub.2k+1 from the intermediate viewpoint image I.sub.2k, the observation viewpoint image I.sub.2k+1, and the intermediate viewpoint image I.sub.2k+2, for N (where N is an integer of 1 or greater) synthesis ratios a.sub.n (0≤a.sub.n≤1, 1≤n≤N, n is a parameter representing an integer), uses the synthesis images J.sub.2k−1 and J.sub.2k+1 to calculate the image quality evaluation index in the observation viewpoint V.sub.2k−1, the image quality evaluation index in the intermediate viewpoint V.sub.2k, and the image quality evaluation index in the observation viewpoint V.sub.2k+1, and uses the image quality evaluation index in the observation viewpoint V.sub.2k−1, the image quality evaluation index in the intermediate viewpoint V.sub.2k, and the image quality evaluation index in the observation viewpoint V.sub.2k+1 to calculate the variation v.sub.k (a.sub.n) of the image quality evaluation index in the block B.sub.k (where, J.sub.2K+1=J.sub.1, V.sub.2K+1=V.sub.1). The specific calculation method may be the same as that of the image quality evaluation index calculation unit 221.

(58) In step S322, the image quality evaluation index comparison unit 322 determines a synthesis ratio a.sub.n that minimizes the variation v.sub.k (a.sub.n) (1≤n≤N) of the image quality evaluation index in the block B.sub.k as the synthesis ratio A (see FIG. 16). Note that it is an example to determine the synthesis ratio A.sub.k as the synthesis ratio a.sub.n where the variation v.sub.k (a.sub.n) (1≤n≤N) in the image quality evaluation index is minimized, and any determination method may be used as long as the image quality evaluation index comparison unit 322 determines the synthesis ratio A.sub.k based on the variation v.sub.k (a.sub.n) (1≤n≤N) of the image quality evaluation index in the block B.sub.k. For example, one of variations v.sub.k (a.sub.n) of the image quality evaluation index smaller than a predetermined threshold value may be determined as the synthesis ratio A.sub.k, by using the predetermined threshold value (such as an acceptable value as a valuation in the image quality evaluation index defined by an image quality evaluation experiment or the like). Note that in a case where there is no variation v.sub.k (a.sub.n) in the image quality evaluation index that is smaller than the threshold value, the synthesis ratio a.sub.n where the variation v.sub.k (a.sub.n) (1≤n≤N) in the image quality evaluation index is minimized may be determined as the synthesis ratio A.sub.k.

(59) In step S326, the end condition determination unit 326 increments k by 1, in a case where k exceeds K (i.e., k>K), outputs K synthesis ratios A.sub.k (1≤k≤K), and transitions to the processing of S330, or otherwise the process returns to the processing of S321.

(60) In step S330, the output image generation unit 330 uses the K observation viewpoint images I.sub.2k−1 (1≤k≤K) and the K intermediate viewpoint images I.sub.2k (1≤k≤K) acquired at S110 and the K synthesis ratios A.sub.k (1≤k≤K) determined at S320 as inputs to generate the output images S.sub.2k−1 of the projector P.sub.2k−1 from the intermediate viewpoint images I.sub.2k−2, the observation viewpoint images I.sub.2k−1, and the intermediate viewpoint images I.sub.2k by using the synthesis ratios A.sub.k−1, A.sub.k for k satisfying 1≤k≤K, and outputs K output images S.sub.2k−1 (1≤k≤K) (where A.sub.0=A.sub.K, I.sub.0=I.sub.2K). Here, the output images S.sub.2k−1 at the observation viewpoint position V.sub.2k−1 is generated as (1−(A.sub.k−1+A.sub.k))×I.sub.2k−1+½×(A.sub.k−1+A.sub.k)×I.sub.2k−2+½×(A.sub.k−1+A.sub.k)×I.sub.2k (see FIG. 16).

Modified Example

(61) In the above description, the N synthesis ratios a.sub.n used in the generation of the synthesis images J.sub.2k−1 and J.sub.2k+1 have been described as being the same, but they are not necessary to be the same in the same manner as in the first embodiment. In this case, K−1 synthesis ratios (set) A.sub.k (1≤k≤K−1) is used.

(62) According to the invention of the present embodiment, it is possible to suppress unpleasantness associated with fluctuation in image quality caused by the viewer's viewpoint movement.

Fourth Embodiment

(63) In the image generation apparatus 200, the intermediate viewpoint image is used to generate the output image. Instead of using the intermediate viewpoint image, an output image is generated by using an image (hereinafter referred to as a pseudo viewpoint image) in a pseudo viewpoint corresponding to the intermediate viewpoint generated by using the Hidden Stereo technique described in Reference NPL 1. Here, the pseudo viewpoint image refers to an image generated by adding a parallax inducing edge to the viewpoint image, or an image generated by adding a positive/negative reverse image of the parallax inducing edge to the viewpoint image. The parallax inducing edge is an image obtained by modulating a phase of the viewpoint image such that a phase difference from the viewpoint image becomes φ (0<φ≤π/2). Generating the output image by using the pseudo viewpoint image improves the image quality of the perceived image at the intermediate viewpoint because the parallax inducing edge is cancelled out in the intermediate viewpoint. Reference NPL 1: M. Makiguchi, H. Takada, T. Fukiage, S. Nishida, “Reducing Image Quality Variation with Motion Parallax for Glassless 3D Screens Using Linear Blending Technology,” SID, Vol. 49, Issue 1, pp. 251-254, 2018)

(64) Hereinafter, an image generation apparatus 400 will be described with reference to FIGS. 17 and 18. FIG. 17 is a block diagram illustrating a configuration of the image generation apparatus 400. FIG. 18 is a flowchart illustrating an operation of the image generation apparatus 400. As illustrated in FIG. 17, the image generation apparatus 400 includes a viewpoint image acquisition unit 110, a pseudo viewpoint image generation unit 410, a synthesis ratio determination unit 220, an output image generation unit 230, and a recording unit 190. The recording unit 190 is a component configured to appropriately record information required for processing of the image generation apparatus 400. The image generation apparatus 400 differs from the image generation apparatus 200 only in that the image generation apparatus 400 further includes a pseudo viewpoint image generation unit 410.

(65) The operation of the image generation apparatus 400 will be described in accordance with FIG. 18. In step S110, the viewpoint image acquisition unit 110 acquires and outputs K observation viewpoint images I.sub.2k−1 (1≤k≤K) and K−1 intermediate viewpoint images I.sub.2k (1≤k≤K−1).

(66) In step S410, the pseudo viewpoint image generation unit 410 uses K−1 intermediate viewpoint images I.sub.2k (1≤k≤K−1) acquired at S110 as inputs to generate a parallax inducing edge D.sub.φ having a phase difference from the intermediate viewpoint images I.sub.2k being φ (φ is a real number satisfying 0<φ≤π/2) by using the intermediate viewpoint images I.sub.2k and the observation viewpoint images I.sub.2k+1 for the intermediate viewpoint images I.sub.2k (1≤k≤K−1), generates the pseudo intermediate viewpoint image I.sub.2k.sup.(R) by adding the parallax inducing edge D.sub.φ to the intermediate viewpoint images I.sub.2k, generates the pseudo intermediate viewpoint image I.sub.2k.sup.(L) by adding the positive/negative reverse image of the parallax inducing edge D.sub.φ to the intermediate viewpoint image I.sub.2k, and outputs 2K−2 pseudo intermediate viewpoint images I.sub.2k.sup.(R), I.sub.2k.sup.(L) (1≤k≤K−1).

(67) In step S220, the synthesis ratio determination unit 220 uses K observation viewpoint images I.sub.2k−1 (1≤k≤K) acquired at S110 and 2K−2 pseudo intermediate viewpoint images I.sub.2k.sup.(R), I.sub.2k.sup.(L) (1≤k≤K−1) generated at S410 as inputs, determines the synthesis ratio A.sub.k in the block B.sub.k by using the pseudo intermediate viewpoint image I.sub.2k−2.sup.(R), the observation viewpoint image I.sub.2k−1, the pseudo intermediate viewpoint image I.sub.2k.sup.(L), the pseudo intermediate viewpoint image I.sub.2k.sup.(R), the observation viewpoint image I.sub.2k+1, and the pseudo intermediate viewpoint image I.sub.2k+2.sup.(L), for the block B.sub.k (1≤k≤K−1), and outputs K−1 synthesis ratios A.sub.k (1≤k≤K−1) (where I.sub.0.sup.(R)=I.sub.2.sup.(R), I.sub.2K.sup.(L)=I.sub.2K+2.sup.(L)). Here, the synthesis ratio A.sub.k is used when the output image generation unit 230 generates the output image. Hereinafter, the synthesis ratio determination unit 220 will be described with reference to FIGS. 9 to 10. FIG. 9 is a block diagram illustrating a configuration of the synthesis ratio determination unit 220. FIG. 10 is a flowchart illustrating an operation of the synthesis ratio determination unit 220. As illustrated in FIG. 9, the synthesis ratio determination unit 220 includes an initialization unit 225, an image quality evaluation index calculation unit 221, an image quality evaluation index comparison unit 222, and an end condition determination unit 226.

(68) An operation of the synthesis ratio determination unit 220 will be described with reference to FIG. 10. In step S225, the initialization unit 225 sets the value of k, which is a parameter representing the number of repetitions, to 1.

(69) In step S221, the image quality evaluation index calculation unit 221 generates the synthesis image J.sub.2k−1 from the pseudo intermediate viewpoint image I.sub.2k−2.sup.(R), the observation viewpoint image I.sub.2k−1, and the pseudo intermediate viewpoint image I.sub.2k.sup.(L), and the synthesis image J.sub.2k+1 from the pseudo intermediate viewpoint image I.sub.2k.sup.(R), the observation viewpoint image I.sub.2k+1, and the pseudo intermediate viewpoint image I.sub.2k+2.sup.(I), for N (where N is an integer of 1 or greater) synthesis ratios a.sub.n (0≤a.sub.n≤1, 1≤n≤N, n is a parameter representing an integer), uses the synthesis images J.sub.2k−1 and J.sub.2k+1 to calculate the image quality evaluation index in the observation viewpoint V.sub.2k−1, the image quality evaluation index in the intermediate viewpoint V.sub.2k, and the image quality evaluation index in the observation viewpoint V.sub.2k+1, and uses the image quality evaluation index in the observation viewpoint V.sub.2k−1, the image quality evaluation index in the intermediate viewpoint V.sub.2k, and the image quality evaluation index in the observation viewpoint V.sub.2k+1 to calculate the variation v.sub.k (a.sub.n) of the image quality evaluation index in the block B.sub.k.

(70) In step S222, the image quality evaluation index comparison unit 222 determines a synthesis ratio a.sub.n that minimizes the variation v.sub.k (a.sub.n) (1≤n≤N) of the image quality evaluation index in the block B.sub.k as the synthesis ratio A. Note that it is an example to determine the synthesis ratio A.sub.k as the synthesis ratio a.sub.n where the variation v.sub.k (a.sub.n) (1≤n≤N) in the image quality evaluation index is minimized, and any determination method may be used as long as the image quality evaluation index comparison unit 222 determines the synthesis ratio A.sub.k based on the variation v.sub.k (a.sub.n) (1≤n≤N) of the image quality evaluation index in the block B.sub.k. For example, one of variations v.sub.k (a.sub.n) of the image quality evaluation index smaller than a predetermined threshold value may be determined as the synthesis ratio A.sub.k, by using the predetermined threshold value (such as an acceptable value as a valuation in the image quality evaluation index defined by an image quality evaluation experiment or the like). Note that in a case where there is no variation v.sub.k (a.sub.n) in the image quality evaluation index that is smaller than the threshold value, the synthesis ratio a.sub.n where the variation v.sub.k (a.sub.n) (1≤n≤N) in the image quality evaluation index is minimized may be determined as the synthesis ratio A.sub.k.

(71) In step S226, the end condition determination unit 226 increments k by 1, in a case where k reaches K (i.e., k>K−1), outputs K−1 synthesis ratios A.sub.k (1≤k≤K−1), and transitions to the processing of S230, or otherwise the process returns to the processing of S221.

(72) In step S230, the output image generation unit 230 uses the K observation viewpoint images I.sub.2k−1 (1≤k≤K) acquired at S110 and the 2K−2 pseudo intermediate viewpoint images I.sub.2k.sup.(R), I.sub.2k.sup.(L) (1≤k≤K−1) generated at S410 and the K−1 synthesis ratios A.sub.k (1≤k≤K−1) determined at S220 as inputs to generate the output images S.sub.2k−1 of the projector P.sub.2k−1 from the pseudo intermediate viewpoint images I.sub.2k−2.sup.(R), the observation viewpoint images I.sub.2k−1, and the pseudo intermediate viewpoint images I.sub.2k.sup.(L) by using the synthesis ratios A.sub.k−1, A.sub.k for k satisfying 1≤k≤K, and outputs K output images S.sub.2k−1 (1≤k≤K) (where A.sub.0=A.sub.1, A.sub.K=A.sub.K−1, I.sub.0.sup.(R)=I.sub.2.sup.(R), I.sub.2K.sup.(L)=I.sub.2K−2.sup.(L)).

(73) Note that in S220 and S230, I.sub.0.sup.(R)=I.sub.2.sup.(R) and the pseudo intermediate viewpoint image I.sub.2.sup.(R) next to the observation viewpoint image I.sub.1 are used, but instead of using the pseudo intermediate viewpoint image I.sub.2.sup.(R), the observation viewpoint image I.sub.1 may be used. That is, I.sub.0.sup.(R)=I.sub.1 may be used. Note that in S220 and S230, I.sub.2K.sup.(L)=I.sub.2K−2.sup.(L) and the pseudo intermediate viewpoint image I.sub.2K−2.sup.(L) next to the observation viewpoint image I.sub.2K−1 are used, but instead of using the pseudo intermediate viewpoint image I.sub.2K−2.sup.(L), the observation viewpoint image I.sub.2K−1 may be used. That is, I.sub.2k.sup.(L)=I.sub.2K−1 may be used.

(74) According to the invention of the present embodiment, it is possible to suppress unpleasantness associated with fluctuation in image quality caused by the viewer's viewpoint movement. Because the parallax inducing edge is canceled out in the intermediate viewpoint, the image quality of the perceived image at the intermediate viewpoint is improved.

Fifth Embodiment

(75) In the image generation apparatus 300, similar to the image generation apparatus 200, the Hidden Stereo technique may be used.

(76) Hereinafter, an image generation apparatus 500 will be described with reference to FIGS. 19 and 20. FIG. 19 is a block diagram illustrating a configuration of the image generation apparatus 500. FIG. 20 is a flowchart illustrating an operation of the image generation apparatus 500. As illustrated in FIG. 19, the image generation apparatus 500 includes a viewpoint image acquisition unit 110, a pseudo viewpoint image generation unit 510, a synthesis ratio determination unit 320, an output image generation unit 330, and a recording unit 190. The recording unit 190 is a component configured to appropriately record information required for processing of the image generation apparatus 500. The image generation apparatus 500 differs from the image generation apparatus 300 only in that the image generation apparatus 500 further includes a pseudo viewpoint image generation unit 510.

(77) The operation of the image generation apparatus 500 will be described in accordance with FIG. 20. In step S110, the viewpoint image acquisition unit 110 acquires and outputs K observation viewpoint images I.sub.2k−1 (1≤k≤K) and K intermediate viewpoint images I.sub.2k (1≤k≤K).

(78) In step S510, the pseudo viewpoint image generation unit 510 uses K intermediate viewpoint images I.sub.2k (1≤k≤K) acquired at S110 as inputs to generate a parallax inducing edge D.sub.φ having a phase difference from the intermediate viewpoint images I.sub.2k being φ (φ is a real number satisfying 0<φ≤π/2) by using the intermediate viewpoint images I.sub.2k and the observation viewpoint images I.sub.2k+1 for the intermediate viewpoint images I.sub.2k (1≤k≤K), generates the pseudo intermediate viewpoint image I.sub.2k.sup.(R) by adding the parallax inducing edge D.sub.φ to the intermediate viewpoint images I.sub.2k, generates the pseudo intermediate viewpoint image I.sub.2k.sup.(L) by adding the positive/negative reverse image of the parallax inducing edge D.sub.φ to the intermediate viewpoint image I.sub.2k, and outputs 2K pseudo intermediate viewpoint images I.sub.2k.sup.(R), I.sub.2k.sup.(L) (1≤k≤K) (where I.sub.2K+1=I.sub.1).

(79) In step S320, the synthesis ratio determination unit 320 uses K observation viewpoint images I.sub.2k−1 (1≤k≤K) acquired at S110 and 2K pseudo intermediate viewpoint images I.sub.2k.sup.(R), I.sub.2k.sup.(L) (1≤k≤K) generated at S510 as inputs, determines the synthesis ratio A.sub.k in the block B.sub.k by using the pseudo intermediate viewpoint image I.sub.2k−2.sup.(R), the observation viewpoint image the pseudo intermediate viewpoint image I.sub.2k.sup.(L), the pseudo intermediate viewpoint image I.sub.2k.sup.(R), the observation viewpoint image I.sub.2k+1, and the pseudo intermediate viewpoint image I.sub.2k+2.sup.(L), for the block B.sub.k (1≤k≤K), and outputs K−1 synthesis ratios A.sub.k (1≤k≤K) (where I.sub.0.sup.(R)=I.sub.2K.sup.(R), I.sub.2K+1=I.sub.1, I.sub.2K+2.sup.(L)=I.sub.2.sup.(L)). Here, the synthesis ratio A.sub.k is used when the output image generation unit 330 generates the output image. Hereinafter, the synthesis ratio determination unit 320 will be described with reference to FIGS. 14 to 15. FIG. 14 is a block diagram illustrating a configuration of the synthesis ratio determination unit 320. FIG. 15 is a flowchart illustrating an operation of the synthesis ratio determination unit 320. As illustrated in FIG. 14, the synthesis ratio determination unit 320 includes an initialization unit 325, an image quality evaluation index calculation unit 321, an image quality evaluation index comparison unit 322, and an end condition determination unit 326.

(80) An operation of the synthesis ratio determination unit 320 will be described with reference to FIG. 15. In step S325, the initialization unit 325 sets the value of k, which is a parameter representing the number of repetitions, to 1.

(81) In step S321, the image quality evaluation index calculation unit 321 generates a synthesis image J.sub.2k+1 from the pseudo intermediate viewpoint image I.sub.2k−2.sup.(R), the observation viewpoint image I.sub.2k−1, and the pseudo intermediate viewpoint image I.sub.2k.sup.(L), and a synthesis image J.sub.2k+1 from the pseudo intermediate viewpoint image I.sub.2k.sup.(R), the observation viewpoint image I.sub.2k+1, and the pseudo intermediate viewpoint image I.sub.2k+2.sup.(l) for N synthesis ratios a.sub.n (where N is an integer of 1 or greater) (0≤a.sub.n≤1, 1≤n≤N, n is a parameter representing an integer). The image quality evaluation index calculation unit 321 uses the synthesis images J.sub.2k−1 and J.sub.2k+1 to calculate the image quality evaluation index in the observation viewpoint V.sub.2k−1, the image quality evaluation index in the intermediate viewpoint V.sub.2k, and the image quality evaluation index in the observation viewpoint V.sub.2k+1. The image quality evaluation index calculation unit 321 calculates a variation v.sub.k (a.sub.n) of the image quality evaluation index in the block B.sub.k by using the image quality evaluation index in the observation viewpoint V.sub.2k−1, the image quality evaluation index in the intermediate viewpoint V.sub.2k, and the image quality evaluation index in the observation viewpoint V.sub.2k+1 (where J.sub.2K+1=J.sub.1, V.sub.2K+1=V.sub.1).

(82) In step S322, the image quality evaluation index comparison unit 322 determines a synthesis ratio a.sub.n that minimizes the variation v.sub.k (a.sub.n) (1≤n≤N) of the image quality evaluation index in the block B.sub.k as the synthesis ratio A. Note that it is an example to determine the synthesis ratio A.sub.k as the synthesis ratio a.sub.n where the variation v.sub.k (a.sub.n) (1≤n≤N) in the image quality evaluation index is minimized, and any determination method may be used as long as the image quality evaluation index comparison unit 322 determines the synthesis ratio A.sub.k based on the variation v.sub.k (a.sub.n) (1≤n≤N) of the image quality evaluation index in the block B.sub.k. For example, one of variations v.sub.k (a.sub.n) of the image quality evaluation index smaller than a predetermined threshold value may be determined as the synthesis ratio A.sub.k, by using the predetermined threshold value (such as an acceptable value as a valuation in the image quality evaluation index defined by an image quality evaluation experiment or the like). Note that in a case where there is no variation v.sub.k (a.sub.n) in the image quality evaluation index that is smaller than the threshold value, the synthesis ratio a.sub.n where the variation v.sub.k (a.sub.n) (1≤n≤N) in the image quality evaluation index is minimized may be determined as the synthesis ratio A.sub.k.

(83) In step S326, the end condition determination unit 326 increments k by 1, in a case where k exceeds K (i.e., k>K), outputs K synthesis ratios A.sub.k (1≤k≤K), and transitions to the processing of S330, or otherwise the process returns to the processing of S321.

(84) In step S330, the output image generation unit 330 uses the K observation viewpoint images I.sub.2k−1 (1≤k≤K) acquired at S110 and the 2K pseudo intermediate viewpoint images I.sub.2k.sup.(R), I.sub.2k.sup.(L) (1≤k≤K) generated at S510 and the K synthesis ratios A.sub.k (1≤k≤K) determined at S320 as inputs to generate the output images S.sub.2k−1 of the projector P.sub.2k−1 from the pseudo intermediate viewpoint images I.sub.2k−2.sup.(R), the observation viewpoint images and the pseudo intermediate viewpoint images I.sub.2k.sup.(L) by using the synthesis ratios A.sub.k−1, A.sub.k for k satisfying 1≤k≤K, and outputs K output images S.sub.2k−1 (1≤k≤K) (where A.sub.0=A.sub.K, I.sub.0.sup.(R)=I.sub.2k.sup.(R)).

(85) According to the invention of the present embodiment, it is possible to suppress unpleasantness associated with fluctuation in image quality caused by the viewer's viewpoint movement. Because the parallax inducing edge is canceled out in the intermediate viewpoint, the image quality of the perceived image at the intermediate viewpoint is improved.

SUPPLEMENTS

(86) FIG. 21 is a diagram illustrating an example of a functional configuration of a computer realizing each apparatus mentioned above. The processing in each of the above-described apparatuses can be performed by causing a recording unit 2020 to read a program for causing a computer to function as each of the above-described apparatuses, and operating the program in a control unit 2010, an input unit 2030, an output unit 2040, and the like.

(87) The apparatus according to the present invention includes, for example, as single hardware entities, an input unit to which a keyboard or the like can be connected, an output unit to which a liquid crystal display or the like can be connected, a communication unit to which a communication apparatus (for example, a communication cable) capable of communication with the outside of the hardware entity can be connected, a Central Processing Unit (CPU, which may include a cache memory, a register, and the like), a RAM or a ROM that is a memory, an external storage apparatus that is a hard disk, and a bus connected for data exchange with the input unit, the output unit, the communication unit, the CPU, the RAM, the ROM, and the external storage apparatuses. An apparatus (drive) capable of reading and writing from and to a recording medium such as a CD-ROM may be provided in the hardware entity as necessary. An example of a physical entity including such hardware resources is a general-purpose computer.

(88) A program necessary to implement the above-described functions, data necessary for processing of this program, and the like are stored in the external storage apparatus of the hardware entity (the present invention is not limited to the external storage apparatus; for example, the program may be read out and stored in a ROM that is a dedicated storage apparatus). For example, data obtained by the processing of the program is appropriately stored in a RAM, the external storage apparatus, or the like.

(89) In the hardware entity, each program and data necessary for the processing of each program stored in the external storage apparatus (or a ROM, for example) are read into a memory as necessary and appropriately interpreted, executed, or processed by a CPU. As a result, the CPU implements a predetermined function (each of components represented by xxx unit, xxx means, or the like).

(90) The present invention is not limited to the above-described embodiment, and appropriate changes can be made without departing from the spirit of the present invention. The processing described in the embodiments are not only executed in time series in the described order, but also may be executed in parallel or individually according to a processing capability of an apparatus that executes the processing or as necessary.

(91) As described above, when a processing function in the hardware entity (the apparatus of the present invention) described in the embodiment is implemented by a computer, processing content of a function that the hardware entity should have is described by a program. By executing this program using the computer, the processing function in the hardware entity is implemented on the computer.

(92) The program in which the processing details are described can be recorded on a computer-readable recording medium. The computer-readable recording medium can be any type of medium such as a magnetic recording apparatus, an optical disc, a magneto-optical recording medium, or a semiconductor memory. Specifically, for example, a hard disk apparatus, a flexible disk, a magnetic tape, or the like can be used as a magnetic recording apparatus, a Digital Versatile Disc (DVD), a DVD-Random Access Memory (RAM), a Compact Disc Read Only Memory (CD-ROM), CD-R (Recordable)/RW (ReWritable), or the like can be used as an optical disc, a Magneto-Optical disc (MO) or the like can be used as a magneto-optical recording medium, and an Electronically Erasable and Programmable-Read Only Memory (EEP-ROM) or the like can be used as a semiconductor memory.

(93) The program is distributed, for example, by selling, giving, or lending a portable recording medium such as a DVD or a CD-ROM with the program recorded on it. Further, the program may be stored in a storage apparatus of a server computer and transmitted from the server computer to another computer via a network, so that the program is distributed.

(94) For example, a computer executing the program first temporarily stores the program recorded on the portable recording medium or the program transmitted from the server computer in the own storage apparatus. When processing is executed, the computer reads the program stored in its own storage apparatus and executes the processing in accordance with the read program. As another execution form of the program, the computer may directly read the program from the portable recording medium and execute processing in accordance with the program. Further, each time the program is transmitted from the server computer to the computer, the computer executes processing in order in accordance with the received program. In another configuration, the processing may be executed through a so-called application service provider (ASP) service in which functions of the processing are implemented just by issuing an instruction to execute the program and obtaining results without transmission of the program from the server computer to the computer. The program in this form is assumed to include a program which is information provided for processing of a computer and is equivalent to a program (data or the like that has characteristics regulating processing of the computer rather than a direct instruction for a computer).

(95) Although the hardware entity is configured by a predetermined program being executed on the computer in the present embodiment, at least a part of the processing content of the hardware entity may be implemented in hardware.

(96) The foregoing description of the embodiments of the present invention has been presented for purposes of illustration and description. The foregoing description does not intend to be exhaustive and does not intend to limit the invention to the precise forms disclosed. Modifications and variations are possible from the teachings above. The embodiments have been chosen and expressed in order to provide the best demonstration of the principles of the present invention, and to enable those skilled in the art to utilize the present invention in numerous embodiments and with addition of various modifications suitable for actual use considered. All such modifications and variations are within the scope of the present invention defined by the appended claims that are interpreted according to the width provided justly lawfully and fairly.