Stereo camera and method of controlling stereo camera

Abstract

A first wide angle image, a second wide angle image, a first telephoto image, and a second telephoto image are acquired from a first imaging unit 11.sub.L and a second imaging unit 11.sub.R at the same time, and particularly, optical axes of the wide angle optical system and the telephoto optical system constituting the first imaging optical system 12.sub.L match each other, and the second imaging optical system is similarly configured, and therefore, it is possible to position the main subject at a center position of the first telephoto image and the second telephoto image by independently performing pan and tilt control on the first and the second imaging unit so that the main subject is captured on the respective optical axes of the first imaging optical system and the second imaging optical system on the basis of the first wide angle image and the second wide angle image.

Claims

1. A stereo camera comprising: a first imaging unit including a first imaging optical system including a wide angle optical system and a telephoto optical system having optical axes matching each other and disposed in different regions, and a first directivity sensor having a plurality of pixels that pupil-divide and selectively receive a light beam incident via the wide angle optical system and the telephoto optical system; a second imaging unit including a second imaging optical system having the same configuration as the first imaging optical system and a second directivity sensor having the same configuration as the first directivity sensor, the second imaging unit being disposed to be spaced from the first imaging unit by a baseline length; an image acquisition unit that acquires a first wide angle image and a second wide angle image, and a first telephoto image and a second telephoto image from the first directivity sensor and the second directivity sensor; a pan and tilt mechanism that rotationally moves the first imaging unit and the second imaging unit in a horizontal direction and a vertical direction; a pan and tilt control unit that controls the pan and tilt mechanism on the basis of the first wide angle image and the second wide angle image acquired by the image acquisition unit so that a main subject is captured on the respective optical axes of the first imaging optical system and the second imaging optical system; and a distance information calculation unit that calculates at least distance information of the main subject on the basis of the first telephoto image and the second telephoto image acquired by the image acquisition unit, wherein the distance information calculation unit includes a correspondence point detection unit that detects two correspondence points of which feature quantities match each other on the basis of the first telephoto image and the second telephoto image, and calculates a distance of the correspondence point on the basis of the amount of parallax in the first directivity sensor and the second directivity sensor of the two correspondence points detected by the correspondence point detection unit, the baseline length, an optical axis direction of the first imaging optical system and the second imaging optical system, and a focal length of the telephoto optical system.

2. The stereo camera according to claim 1, wherein the first imaging optical system and the second imaging optical system each include the wide angle optical system including a circular central optical system, and the telephoto optical system including an annular optical system disposed concentrically with the central optical system.

3. The stereo camera according to claim 1, wherein the first directivity sensor and the second directivity sensor each include a microlens array or a light shielding mask that functions as pupil division unit.

4. The stereo camera according to claim 1, further comprising a first focus adjustment unit that performs focus adjustment of the respective telephoto optical systems of the first imaging optical system and the second imaging optical system.

5. The stereo camera according to claim 4, wherein the first focus adjustment unit acquires distance information of the main subject on the basis of the amount of parallax in the first directivity sensor and the second directivity sensor of the main subject included in the first wide angle image and the second wide angle image acquired by the image acquisition unit, the baseline length, the optical axis directions of the first imaging optical system and the second imaging optical system, and a focal length of the wide angle optical system, and performs focus adjustment of the respective telephoto optical systems of the first imaging optical system and the second imaging optical system on the basis of the acquired distance information.

6. The stereo camera according to claim 4, wherein when the pan and tilt mechanism is controlled by the pan and tilt control unit and the main subject is captured on the respective optical axes of the first imaging optical system and the second imaging optical system, the first focus adjustment unit acquires the distance information of the main subject on the basis of the baseline length, the optical axis directions of the first imaging optical system and the second imaging optical system, and the focal length of the wide angle optical system, and performs focus adjustment of the respective telephoto optical systems of the first imaging optical system and the second imaging optical system on the basis of the acquired distance information.

7. The stereo camera according to claim 1, further comprising: a first focus adjustment unit that performs focus adjustment of the respective telephoto optical systems of the first imaging optical system and the second imaging optical system; and a second focus adjustment unit that performs focus adjustment of the respective wide angle optical systems of the first imaging optical system and the second imaging optical system.

8. The stereo camera according to claim 7, wherein the first focus adjustment unit acquires the distance information of the main subject on the basis of in-focus information of the wide angle optical system from the second focus adjustment unit, and performs focus adjustment of the respective telephoto optical systems of the first imaging optical system and the second imaging optical system on the basis of the acquired distance information.

9. The stereo camera according to claim 1, wherein the pan and tilt mechanism includes a first pan and tilt mechanism that rotationally moves the first imaging unit in a horizontal direction and a vertical direction, and a second pan and tilt mechanism that rotationally moves the second imaging unit in a horizontal direction and a vertical direction independently of the first pan and tilt mechanism, and the pan and tilt control unit includes a first pan and tilt control unit that controls the first pan and tilt mechanism on the basis of the first wide angle image, and a second pan and tilt control unit that controls the second pan and tilt mechanism on the basis of the second wide angle image.

10. The stereo camera according to claim 9, wherein the pan and tilt mechanism includes a holding member that holds the first pan and tilt mechanism and the second pan and tilt mechanism, and a pan mechanism that rotationally moves the holding member in the horizontal direction, and the pan and tilt control unit controls the pan mechanism on the basis of the first wide angle image and the second wide angle image so that the first imaging unit and the second imaging unit directly face the main subject.

11. The stereo camera according to claim 1, further comprising: a first subject detection unit and a second subject detection unit that detect the main subject on the basis of the first wide angle image and the second wide angle image, and the pan and tilt control unit controls the pan and tilt mechanism on the basis of position information of the main subject in the first wide angle image and the second wide angle image detected by the first subject detection unit and the second subject detection unit so that the main subject is captured on the respective optical axes of the first imaging optical system and the second imaging optical system.

12. The stereo camera according to claim 11, wherein the first subject detection unit and the second subject detection unit detect a moving body on the basis of the first wide angle image and the second wide angle image successively acquired by the image acquisition unit, and set the detected moving body as the main subject.

13. The stereo camera according to claim 11, wherein the first subject detection unit and the second subject detection unit recognize a specific subject on the basis of the first wide angle image and the second wide angle image and set the recognized specific subject as the main subject.

14. The stereo camera according to claim 11, further comprising: a third subject detection unit and a fourth subject detection unit that detect the main subject on the basis of the first telephoto image and the second telephoto image, wherein the pan and tilt control unit controls the pan and tilt mechanism on the basis of the position information of the main subject in the first telephoto image and the second telephoto image detected by the third subject detection unit and the fourth subject detection unit, and controls the pan and tilt mechanism on the basis of position information of the main subject in the first wide angle image and the second wide angle image detected by the first subject detection unit and the second subject detection unit in a case where the main subject cannot be detected by the third subject detection unit and the fourth subject detection unit.

15. A method of controlling a stereo camera using the stereo camera according to claim 1, the method comprising steps of: acquiring the first wide angle image and the second wide angle image from the first directivity sensor and the second directivity sensor; controlling the pan and tilt mechanism on the basis of the acquired first wide angle image and the acquired second wide angle image so that the main subject is captured on the respective optical axes of the first imaging optical system and the second imaging optical system; acquiring the first telephoto image and the second telephoto image from the first directivity sensor and the second directivity sensor in a state where the main subject is captured on the respective optical axes of the first imaging optical system and the second imaging optical system; calculating at least distance information of the main subject on the basis of the acquired first telephoto image and the acquired second telephoto image; detecting two correspondence points of which feature quantities match each other on the basis of the first acquired telephoto image and the acquired second telephoto image; and calculating a distance of the correspondence point on the basis of the amount of parallax in the first directivity sensor and the second directivity sensor of the two detected correspondence points, the baseline length, an optical axis direction of the first imaging optical system and the second imaging optical system, and a focal length of the telephoto optical system.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) FIG. 1 is an external perspective view of a stereo camera according to the present invention.

(2) FIG. 2 is a cross-sectional view illustrating a first embodiment of a first imaging unit 11.sub.L of a stereo camera.

(3) FIG. 3 is an enlarged diagram of main parts of a microlens array and an image sensor illustrated in FIG. 2.

(4) FIG. 4 is a block diagram illustrating an embodiment of an internal configuration of the stereo camera.

(5) FIG. 5 is a diagram illustrating an example of a wide angle image and a telephoto image captured by a stereo camera.

(6) FIG. 6 is an image diagram illustrating a first telephoto image showing a state in which pan and tilt control is performed so that a specific main subject (a face of a person) is located at a center of the first telephoto image.

(7) FIG. 7 is a diagram used to describe a method of calculating distance information of a main subject or the like.

(8) FIG. 8 is a block diagram illustrating a first focus adjustment unit.

(9) FIG. 9 is a block diagram of a pan and tilt control unit 60.

(10) FIG. 10 is a flowchart showing pan and tilt control for causing a main subject to be captured on optical axes L.sub.1 and L.sub.2 of a first imaging optical system 12.sub.L and a second imaging optical system 12.sub.R.

(11) FIG. 11 is a flowchart showing a process of acquiring distance information of correspondence points of which feature quantities match between a first telephoto image and a second telephoto image.

(12) FIG. 12 is a diagram used to describe an example of a moving body detection method for detecting a main subject that is a moving body.

(13) FIG. 13 is a diagram used to describe an example of a moving body detection method for detecting a main subject that is a moving body.

(14) FIG. 14 is a diagram used to describe an example of a moving body detection method for detecting a main subject that is a moving body.

(15) FIG. 15 is a side view illustrating another embodiment of the directivity sensor.

(16) FIG. 16 is a cross-sectional view illustrating another embodiment of the imaging unit applicable to a stereo camera.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

(17) Hereinafter, embodiments of a stereo camera and a method of controlling the stereo camera according to the present invention will be described with reference to the accompanying drawings.

(18) <Appearance of Stereo Camera>

(19) FIG. 1 is an external perspective view of a stereo camera according to the present invention.

(20) As illustrated in FIG. 1, the stereo camera 1 mainly includes a first pan and tilt camera 10.sub.L for a left eye, a second pan and tilt camera 10.sub.R for a right eye, a holding member 37 that holds the first pan and tilt camera 10.sub.L and the second pan and tilt camera 10.sub.R to be spaced from each other as baseline length D, and a pan device 38 including a pan mechanism that rotationally moves the holding member 37 in a horizontal direction.

(21) The first pan and tilt camera 10.sub.L includes a first imaging unit 11.sub.L, a first pan and tilt device 30.sub.L (FIG. 4), and a first camera body 2.sub.L, and the second pan and tilt camera 10.sub.R includes a second imaging unit 11.sub.R, a second pan and tilt device 30.sub.R (FIG. 4), and a second camera body 2.sub.R.

(22) The first pan and tilt device 30.sub.L includes a base 4.sub.L, and a holding unit 8.sub.L fixed to the base 4.sub.L and holding the first imaging unit 11.sub.L rotationally movably.

(23) The base 4.sub.L is rotatably disposed about an axis of the first camera body 2.sub.L in a vertical direction Z and is rotated (panned) about an axis in the vertical direction Z by a pan driving unit 34 (FIG. 4).

(24) The holding unit 8.sub.L includes a gear 8A provided coaxially with an axis in a horizontal direction X and rotationally moves (tilts) the first imaging unit 11.sub.L in a vertical direction according to a driving force transmitted from the tilt driving unit 36 (FIG. 4) via the gear 8A.

(25) The second pan and tilt device 30.sub.R includes a base 4.sub.R, and a holding unit 8.sub.R fixed to the base 4.sub.R and holding the second imaging unit 11.sub.R rotationally movably.

(26) The base 4.sub.R is rotatably disposed about an axis of the second camera body 2.sub.R in a vertical direction Z and is rotated (panned) about an axis in the vertical direction Z by a pan driving unit 34.

(27) The holding unit 8.sub.R includes a gear 8A provided coaxially with an axis in the horizontal direction X and tilts the second imaging unit 11.sub.R in a vertical direction according to a driving force transmitted from the tilt driving unit 36 via the gear 8A.

(28) Accordingly, the first pan and tilt camera 10.sub.L can cause the imaging direction of the first pan and tilt camera 10.sub.L (a direction of the optical axis L.sub.1 of the first imaging optical system) to be directed in a desired direction through a pan and tilt operation. Similarly, the second pan and tilt camera 10.sub.R can cause the imaging direction of the second pan and tilt camera 10.sub.R (a direction of the optical axis L.sub.2 of the second imaging optical system) to be directed in a desired direction independently of the first pan and tilt camera 10.sub.L through a pan and tilt operation.

(29) [Configuration of First Imaging Unit 11.sub.L]

(30) FIG. 2 is a cross-sectional view illustrating a first embodiment of the first imaging unit 11.sub.L of the stereo camera 1.

(31) As illustrated in FIG. 2, the first imaging unit 11.sub.L includes a first imaging optical system 12.sub.L and a first directivity sensor 17.sub.L.

(32) <First Imaging Optical System 12.sub.L>

(33) The first imaging optical system 12.sub.L includes a wide angle optical system 13 which is a circular central optical system, and a telephoto optical system 14 which is an annular optical system disposed concentrically with respect to the wide angle optical system 13, which are disposed on the same optical axis.

(34) The wide angle optical system 13 is a wide angle lens including a first lens 13a, a second lens 13b, a third lens 13c, a fourth lens 13d, and a common lens 15, and forms a wide angle image on a microlens array 16 constituting the first directivity sensor 17.sub.L.

(35) The telephoto optical system 14 is a telephoto lens including a first lens 14a, a second lens 14b, a first reflecting mirror 14c as a reflecting optical system, a second reflecting mirror 14d, and a common lens 15 and forms a telephoto image on the microlens array 16 constituting the first directivity sensor 17.sub.L.

(36) A light beam incident via the first lens 14a and the second lens 14b is reflected twice by the first reflecting mirror 14c and the second reflecting mirror 14d and then passes through the common lens 15. The light beam is folded back by the first reflecting mirror 14c and the second reflecting mirror 14d, thereby shortening a length in the optical axis direction of the telephoto optical system (telephoto lens) having a long focal length.

(37) <First Directivity Sensor 17.sub.L>

(38) The first directivity sensor 17.sub.L includes a microlens array 16 and an image sensor 18.

(39) FIG. 3 is an enlarged diagram of main parts of the microlens array 16 and the image sensor 18 illustrated in FIG. 2.

(40) The microlens array 16 includes a plurality of microlenses (pupil image formation lenses) 16a that are two-dimensionally arranged. Horizontal and vertical intervals between the respective microlenses correspond to three light reception cells 18a that are photoelectric conversion elements of the image sensor 18. That is, for each microlens of the microlens array 16, a microlens formed in correspondence to a position of every other light reception cell in each of the horizontal direction and the vertical direction is used.

(41) Each microlens 16a of the microlens array 16 forms a circular central pupil image (a first pupil image) 17a and an annular pupil image (a second pupil image) 17b corresponding to the wide angle optical system 13 and the telephoto optical system 14 of the first imaging optical system 12.sub.L on the light reception cell 18a in a corresponding light reception area of the image sensor 18.

(42) According to the microlens array 16 and the image sensor 18 illustrated in FIG. 3, 33 light reception cells 18a in the form of a lattice (a square grid) are allocated to one microlens 16a of the microlens array 16. Hereinafter, one microlens 16a and a light reception cell group (the 33 light reception cells 18a) corresponding to the one microlens 16a is referred to as a unit block.

(43) The central pupil image 17a is formed only on the light reception cell 18a at a center of the unit block, and the annular pupil image 17b is formed on the eight peripheral light reception cells 18a in the unit block.

(44) According to the first imaging unit 11.sub.L having the above configuration, it is possible to capture the first wide angle image corresponding to the wide angle optical system 13 and the first telephoto image corresponding to the telephoto optical system 14 at the same time.

(45) The second imaging unit 11.sub.R of the stereo camera 1 is configured similarly to the first imaging unit 11.sub.L illustrated in FIGS. 2 and 3, and includes a second imaging optical system including a wide angle optical system and a telephoto optical system, and a second directivity sensor. The second imaging unit 11.sub.R can capture the second wide angle image corresponding to the wide angle optical system and the second telephoto image corresponding to the telephoto optical system at the same time.

(46) <Internal Configuration of Stereo Camera 1>

(47) FIG. 4 is a block diagram illustrating an embodiment of an internal configuration of the stereo camera 1. Regarding control of the stereo camera 1, a first image acquisition unit 22.sub.L, a second image acquisition unit 22.sub.R, a first subject detection unit 50.sub.L, a second subject detection unit 50.sub.R, a first pan and tilt control unit 60.sub.L, a second pan and tilt control unit 60.sub.R, a distance image calculation unit 74, an amount-of-parallax detection unit 72, a digital signal processing unit 40, and the like may be configured of a general-purpose CPU or may be configured of a programmable logic device (PLD) that is a processor of which a circuit configuration can be changed after manufacture, such as a field programmable gate array (FPGA), a dedicated electric circuit which is a processor having a circuit configuration provided to be dedicated for execution of a specific process, such as an application specific integrated circuit (ASIC), or a combination thereof. It should be noted that the same applies to a focus control unit 190 and the like in FIG. 8. Further, the stereo camera 1 includes the first pan and tilt camera 10.sub.L for a left eye and the second pan and tilt camera 10.sub.R for a right eye as described with reference to FIG. 1, and the configurations of both of the cameras are the same. Accordingly, the first pan and tilt camera 10.sub.L will be mainly described with reference to FIG. 4, and detailed description of the second pan and tilt camera 10.sub.R will be omitted.

(48) As illustrated in FIG. 4, the first pan and tilt camera 10.sub.L for a left eye constituting the stereo camera 1 includes the first imaging unit 11.sub.L configured of the first imaging optical system 12.sub.L including the wide angle optical system 13 and the telephoto optical system 14 described with reference to FIG. 2, and the first directivity sensor 17.sub.L including the microlens array 16 and the image sensor 18 described with reference to FIG. 3.

(49) It is preferable for the first imaging unit 11.sub.L to include at least a first focus adjustment unit (including a focus adjustment mechanism 19) that automatically performs focus adjustment of the telephoto optical system 14. The focus adjustment mechanism 19 can be configured of, for example, a voice coil motor that moves all or a part of the telephoto optical system 14 in the optical axis direction. In addition, the focus control unit 190 to be described below acquires distance information of a main subject and drives the focus adjustment mechanism 19 on the basis of the acquired distance information, such that the first focus adjustment unit can perform the first focus adjustment of the telephoto optical system 14.

(50) Further, for the wide angle optical system 13, a second focus adjustment unit that automatically performs focus adjustment of the wide angle optical system 13 may be separately provided, or the wide angle optical system 13 may be pan focus. As the second focus adjustment unit applied to the wide angle optical system 13, a focus adjustment unit using a contrast autofocus (AF) scheme or an image plane phase difference AF scheme is conceivable. The contrast AF scheme is a scheme of performing focus adjustment by moving all or some of the wide angle optical system 13 in the optical axis direction so that the contrast of the AF area of the first wide angle image and the second wide angle image is maximized, and the image plane phase difference AF scheme is a scheme of performing focus adjustment of the wide angle optical system 13 by moving all or some of the wide angle optical system 13 in the optical axis direction so that an image plane phase difference of an AF area of the first wide angle image and the second wide angle image becomes 0.

(51) As illustrated in FIG. 1, the first pan and tilt device 30.sub.L includes, for example, a pan mechanism that rotates the first imaging unit 11.sub.L in the horizontal direction (a pan direction) with respect to the first camera body 2.sub.L and a tilt mechanism that rotates the first imaging unit 11.sub.L in a vertical direction (a tilt direction) 32 (hereinafter referred to as a pan and tilt mechanism), a pan driving unit 34, and a tilt driving unit 36.

(52) The pan and tilt mechanism 32 includes a home position sensor that detects a reference position of a rotation angle (a pan angle) in the pan direction, and a home position sensor that detects a reference position of an inclination angle (a tilt angle) in the tilt direction.

(53) Each of the pan driving unit 34 and the tilt driving unit 36 includes a stepping motor and a motor driver, and output driving force to the pan and tilt mechanism 32 to drive the pan and tilt mechanism 32.

(54) The first imaging unit 11.sub.L captures a first wide angle image and a first telephoto image in time series via the first imaging optical system 12.sub.L and the first directivity sensor 17.sub.L, and a subject image formed on a light reception surface of each light reception cell (a photoelectric conversion element) of the first directivity sensor 17.sub.L (the image sensor 18) via the first imaging optical system 12.sub.L is converted into a signal voltage (or charge) of an amount according to the amount of incident light.

(55) The signal voltage (or charge) accumulated in the image sensor 18 is stored in the light reception cell itself or in an attached capacitor. The stored signal voltage (or charge) is read together with selection of a position of the light reception cell using a metal oxide semiconductor (MOS) type imaging element (so-called complementary metal oxide semiconductor (CMOS) sensor) scheme using an XY address system.

(56) Accordingly, it is possible to read an image signal indicating a first wide angle image of a group of central light reception cells corresponding to the wide angle optical system 13 and an image signal indicating a first telephoto image of a group of eight nearby light reception cells corresponding to the telephoto optical system 14 from the image sensor 18. Further, image signals indicating the first wide angle image and the first telephoto image are successively read at a predetermined frame rate (for example, the number of frames of 24p, 30p, or 60p per second) from the image sensor 18. However, in a case where a shutter button (not shown) is manipulated, the image signals indicating the first wide angle image and the first telephoto image which are each one still image, are read.

(57) For the image signal (a voltage signal) read from the image sensor 18, an image signal for each light reception cell is sampled and held through a sampling two correlation pile process (a process of acquiring accurate pixel data by obtaining a difference between a feedthrough component level and a signal component level included in an output signal for each light reception cell for the purpose of reducing noise (especially, thermal noise) or the like included in a sensor output signal), amplified, and then input to analog/digital (A/D) converter 20. Further, the A/D converter 20 converts sequentially input image signals into digital signals and outputs the digital signals to the image acquisition unit 22. Further, an MOS type sensor may include an A/D converter therein, and in this case, a digital signal is directly output from the image sensor 18.

(58) The first image acquisition unit 22.sub.L can acquire the image signal indicating the first wide angle image and the image signal indicating the first telephoto image simultaneously or selectively by selecting the light reception cell position of the image sensor 18 and reading the image signal.

(59) That is, it is possible to acquire an image signal indicating a first wide angle image (an image signal indicating a mosaic image in a Bayer array) of one light reception cell (a central light reception cell among the 33 light reception cells) per microlens by selectively reading the image signal of the light reception cell on which the central pupil image 17a of the image sensor 18 is incident, whereas it is possible to acquire an image signal indicating a first telephoto image of eight light reception cells (light reception cells around the 33 light reception cells) per microlens by selectively reading the image signal of the light reception cell on which the annular pupil image 17b of the image sensor 18 is incident.

(60) Further, all the image signals may be read from the image sensor 18 and temporarily stored in the buffer memory, and the image signals of the two images including the first wide angle image and the first telephoto image may be grouped from the image signals stored in the buffer memory.

(61) The image signals indicating the first wide angle image and the first telephoto image acquired by the first image acquisition unit 22.sub.L are output to the digital signal processing unit 40 and the first subject detection unit 50.sub.L, respectively.

(62) The first subject detection unit 50.sub.L detects the main subject on the basis of the image signal indicating the first wide angle image acquired by the first image acquisition unit 22.sub.L and outputs the position information in the image of the detected main subject to the first pan and tilt control unit 60.sub.L.

(63) An example of a method of detecting the main subject in the first subject detection unit 50.sub.L includes a method of detecting a specific object (a main subject) using an object recognition technique represented by a technique of performing face recognition of a person, or a moving body detection method of detecting a moving body as a main subject.

(64) An example of a method of detecting an object through object recognition includes a method of registering a feature of an appearance of a specific object as an object dictionary in advance, and recognizing an object while comparing an image cut out while changing a position or a size from a captured image with the object dictionary.

(65) FIG. 5 is a diagram illustrating an example of the first wide angle image and the first telephoto image that are captured. Further, an area indicated by a broken line in the first wide angle image indicates an imaging range of the first telephoto image.

(66) In a case where the first wide angle image and the first telephoto image illustrated in FIG. 6 are captured, a face of a person is set as the main subject, and the face of the person is detected using the face recognition technology, the first subject detection unit 50.sub.L can detect the face in the first wide angle image. In a case where the first subject detection unit 50.sub.L detects the face of the person from the first telephoto image, the first subject detection unit 50.sub.L cannot detect the face from the first telephoto image since only a part of the face of the person is included in the first telephoto image illustrated in FIG. 5. In particular, in a case where an imaging magnification of the telephoto optical system 14 is a high magnification (in a case where an angle of view is very small), a probability of a desired main subject being not included in the first telephoto image is high.

(67) Position information in the first wide angle image of the main subject (in this example, the face of the person) detected by the first subject detection unit 50.sub.L is output to the first pan and tilt control unit 60.sub.L.

(68) The first pan and tilt control unit 60.sub.L is a unit that controls the first pan and tilt device 30.sub.L on the basis of the position information of the main subject in the first wide angle image input from the first subject detection unit 50.sub.L. The first pan and tilt control unit 60.sub.L controls the pan and tilt mechanism 32 (that is, an imaging direction of the first imaging unit 11.sub.L) via the pan driving unit 34 and the tilt driving unit 36 so that the position information in the first wide angle image of the main subject (for example, a position of a centroid of a face area in a case where the main subject is a face of a person) is moved to a center position (a position on the optical axis L.sub.1) of the first wide angle image.

(69) In a case where the pan and tilt mechanism 32 is controlled so that the main subject is captured on the optical axis L.sub.1 of the first imaging optical system 12.sub.L (so that the main subject is included at the center position of the first wide angle image), the main subject is included at the center position of the first telephoto image. This is because the optical axes L.sub.1 of the wide angle optical system 13 and the telephoto optical system 14 constituting the first imaging optical system 12.sub.L match each other (see FIG. 2).

(70) FIG. 6 is an image diagram illustrating a first telephoto image showing a state in which pan and tilt control is performed so that a specific main subject (in this example, a face of a person) is at a center of the first wide angle image.

(71) On the other hand, the second pan and tilt camera 10.sub.R for a right eye constituting the stereo camera 1 includes the second imaging unit 11.sub.R including the second imaging optical system 12.sub.R and a second directivity sensor 17.sub.R (not illustrated), and the second pan and tilt device 30.sub.R, similar to the first pan and tilt camera 10.sub.L for a left eye.

(72) The second imaging unit 11.sub.R captures the second wide angle image and the second telephoto image in time series via the second imaging optical system 12.sub.R and the second directivity sensor 17.sub.R. An image signal indicating the second wide angle image and an image signal indicating the second telephoto image can be output from the second directivity sensor 17.sub.R at the same time.

(73) The second image acquisition unit 22.sub.R acquires the image signal indicating the second wide angle image and the image signal indicating the second telephoto image simultaneously or selectively from the second imaging unit 11.sub.R.

(74) The image signals indicating the second wide angle image and the second telephoto image acquired by the second image acquisition unit 22.sub.R are output to the digital signal processing unit 40 and the second subject detection unit 50.sub.R, respectively.

(75) The second subject detection unit 50.sub.R detects the main subject on the basis of the image signal indicating the second wide angle image acquired by the second image acquisition unit 22.sub.R and outputs the position information in the image of the detected main subject to the second pan and tilt control unit 60.sub.R.

(76) The second pan and tilt control unit 60.sub.R is a unit that controls the second pan and tilt device 30.sub.R on the basis of the position information of the main subject in the second wide angle image input from the second subject detection unit 50.sub.R. The second pan and tilt control unit 60.sub.R controls the pan and tilt mechanism 32 (that is, an imaging direction of the second imaging unit 11.sub.R) via the pan driving unit 34 and the tilt driving unit 36 so that the position information in the second wide angle image of the main subject is moved to a center position (a position on the optical axis L.sub.2) of the second wide angle image.

(77) In a case where the pan and tilt mechanism 32 is controlled so that the main subject is captured on the optical axis L.sub.2 of the second imaging optical system 12.sub.R (so that the main subject is included at the center position of the second wide angle image), the main subject is included at the center position of the second telephoto image. This is because the optical axes L.sub.2 of the wide angle optical system 13 and the telephoto optical system 14 constituting the second imaging optical system 12.sub.R match each other (see FIG. 2).

(78) A digital image signal indicating the first wide angle image and the first telephoto image from the first image acquisition unit 22.sub.L and a digital image signal indicating the second wide angle image and the second telephoto image from the second image acquisition unit 22.sub.R are input to the digital signal processing unit 40. The digital signal processing unit 40 performs predetermined signal processing such as offset processing or gamma-correction processing on each input image signal.

(79) In a case where a display unit 42 includes a stereoscopically displaying function, the digital signal processing unit 40 generates data for a display of a stereoscopic wide angle image from an image signal indicating the first wide angle image and the second wide angle image, which is a stereo image or generates data for a display of a stereoscopic telephoto image from an image signal indicating the first telephoto image and the second telephoto image, which is a stereo image, and outputs the generated data for a display to the display unit 42, so that a stereoscopic wide angle image or a three-dimensional telephoto image is displayed on the display unit 42.

(80) Further, in a case where the display unit 42 has no stereoscopically displaying function, the digital signal processing unit 40 generates, for example, display data of the wide angle image or the telephoto image from the image signal indicating the first wide angle image or the first telephoto image and outputs the generated display data to the display unit 42 so that the wide angle image or the telephoto image is displayed on the display unit 42.

(81) Further, in a case where the main subject is captured on the optical axes L.sub.1 and L.sub.2 of the first imaging optical system 12.sub.L and the second imaging optical system 12.sub.R, the digital signal processing unit 40 generates recording data from the image signal indicating the first wide angle image and the second wide angle image which is a stereo image or generates recording data from the image signal indicating the first telephoto image and the second telephoto image which is a stereo image and outputs the generated recording data to a recording unit 44, so that the recording data is recorded on a recording medium (hard disk, memory card, or the like). Further, the recording unit 44 may record only the first telephoto image and the second telephoto image. The display unit 42 can also reproduce a necessary image on the basis of the recording data recorded on the recording unit 44.

(82) <Distance Information Calculation Unit 70>

(83) Next, a distance information calculation unit 70 will be described.

(84) The distance information calculation unit 70 includes the amount-of-parallax detection unit 72 and a distance image calculation unit 74.

(85) The image signal indicating the first telephoto image acquired by the first image acquisition unit 22.sub.L and the image signal indicating the second telephoto image acquired by the second image acquisition unit 22.sub.R are input to the amount-of-parallax detection unit 72. The amount-of-parallax detection unit 72 receives an image signal indicating the first telephoto image and the second telephoto image in a case where the main subject is captured on the optical axes L.sub.1 and L.sub.2 of the first imaging optical system 12.sub.L and the second imaging optical system 12.sub.R (in a case where optical axes L.sub.1 and L.sub.2 intersect and the main subject is located at a cross point at which L.sub.1 and L.sub.2 intersect).

(86) The amount-of-parallax detection unit 72 includes a correspondence-point detection unit 72a that detects correspondence points of which the feature quantities match each other between the first telephoto image and the second telephoto image on the basis of the input image signals indicating the first telephoto image and the second telephoto image, and detects amounts of parallax in the first directivity sensor 17.sub.L and the second directivity sensor 17.sub.R at the two correspondence points detected by the correspondence-point detection unit 72a.

(87) The detection of the correspondence points performed by the correspondence-point detection unit 72a is performed by detecting the pixel position on the second telephoto image corresponding to a target pixel having the highest correlation through block matching between an image having a predetermined block size with reference to a target pixel of the first telephoto image and the second telephoto image.

(88) Information indicating the amount of parallax in the first directivity sensor 17.sub.L and the second directivity sensor 17.sub.R at two correspondence points of the first telephoto image and the second telephoto image, which are detected by the amount-of-parallax detection unit 72, is output to the distance image calculation unit 74.

(89) Angle information indicating the pan angle and the tilt angle of the first pan and tilt device 30.sub.L including the first pan and tilt mechanism and the second pan and tilt device 30.sub.R including the second pan and tilt mechanism (the information indicating the directions of the optical axes L.sub.1 and L.sub.2 of the first imaging optical system 12.sub.L and the second imaging optical system 12.sub.R) is input from the first pan and tilt control unit 60.sub.L and the second pan and tilt control unit 60.sub.R to the other inputs of the distance image calculation unit 74. The distance image calculation unit 74 calculates three-dimensional space information including the distance information of the correspondence point on the basis of the input information.

(90) Next, a method of calculating the distance information of the main subject or the like will be described with reference to FIG. 7.

(91) FIG. 7 illustrates a case where the pan angle and the tilt angle of the first imaging unit 11.sub.L (the first imaging optical system 12.sub.L) and the second imaging unit 11.sub.R (the second imaging optical system 12.sub.R) are controlled, and the main subject is located at a cross point A at which the optical axes L.sub.1 and L.sub.2 of the first imaging optical system 12.sub.L and the second imaging optical system 12.sub.R cross each other. Further, in order to simplify the description, the optical axes L.sub.1 and L.sub.2 of the first imaging optical system 12.sub.L and the second imaging optical system 12.sub.R are assumed to be horizontal.

(92) A meaning of each symbol illustrated in FIG. 7 is as follows.

(93) A: Cross point at which the optical axes L.sub.1 and L.sub.2 intersect (a position of the main subject)

(94) B: Any correspondence point in the first telephoto image and the second telephoto image

(95) .sub.1: Pan angle of the first imaging optical system 12.sub.L

(96) .sub.2. Pan angle of the second imaging optical system 12.sub.R

(97) D: Baseline length

(98) x.sub.1: Amount of parallax in the first directivity sensor 17.sub.L at the correspondence point B

(99) x.sub.2: Amount of parallax in the second directivity sensor 17.sub.R at the correspondence point B

(100) .sub.1: Angle in the horizontal direction of the correspondence point B with respect to the optical axis L1

(101) .sub.2: Angle in the horizontal direction of the correspondence point B with respect to the optical axis L2

(102) f: Focal length of the telephoto optical system 14 included in the first imaging optical system 12.sub.L and the second imaging optical system 12.sub.R

(103) In FIG. 7, considering a triangle consisting of respective positions of the first imaging optical system 12.sub.L and the second imaging optical system 12.sub.R and the cross point A (the main subject), a length of a base side of the triangle is equal to the baseline length D and is known.

(104) In a case where base angles of the triangle are .sub.1 and .sub.2, the base angles .sub.1 and .sub.2 can be calculated on the basis of the pan angle 1 and the pan angle .sub.2, respectively.

(105) Accordingly, distance information of the cross point A (the main subject) of a vertex of the triangle, which is a measurement point, can be calculated from the known base side (the baseline length D) and the two base angles .sub.1 and .sub.2.

(106) On the other hand, the angle .sub.1 in the horizontal direction of the correspondence point B with respect to the optical axis L.sub.1 and the angle .sub.2 in the horizontal direction of the correspondence point B with respect to the optical axis L2 can be calculated by the following equation on the basis of the amounts of parallax x.sub.1 and x.sub.2 in the first directivity sensor 17.sub.L and the second directivity sensor 17.sub.R and the focal length f of the telephoto optical system 14.
.sub.1=arc tan(x.sub.1/f)
.sub.2=arc tan(x.sub.2/f)[Equation 1]

(107) The amounts of parallax x.sub.1 and x.sub.2 of the first directivity sensor 17.sub.L and the second directivity sensor 17.sub.R of the correspondence point B are calculated using the following equation, in which the pixel positions of the pixels at which the correspondence point B is formed as an image from the center positions of the first directivity sensor 17.sub.L and the second directivity sensor 17.sub.R (positions corresponding to the optical axes L.sub.1 and L.sub.2) are n1 and n2 and a pixel pitch of the first directivity sensor 17.sub.L and the second directivity sensor 17.sub.R is p.
x.sub.1=n1p
x.sub.2=n2p[Equation 2]
Since the pixel pitch p of the first directivity sensor 17.sub.L and the second directivity sensor 17.sub.R is known, the amounts of parallax x.sub.1 and x.sub.2 can be calculated using Equation[2] by detecting the position of the correspondence point B on the first directivity sensor 17.sub.L and the second directivity sensor 17.sub.R.

(108) In FIG. 7, considering the triangle consisting of the respective positions of the first imaging optical system 12.sub.L and the second imaging optical system 12.sub.R and any correspondence point B, the distance information of the correspondence point B, which is the vertex of the triangle, can be calculated on the basis of the length (the baseline length D) of the base side of the triangle and the two base angles 1 and 2.

(109) Further, the two base angles 1 and 2 can be obtained using pan angles .sub.1 and .sub.2, and angles .sub.1 and .sub.2 calculated using Equation 1.

(110) Referring back to FIG. 4, the distance image calculation unit 74 can calculate the distance image by calculating the distance information for each correspondence point of the first telephoto image and the second telephoto image.

(111) Here, the distance image is a two-dimensional distribution image of a distance value (distance information) up to the subject obtained by the stereo camera 1, and each pixel of the distance image has distance information. Further, the distance information of the cross point A (the main subject) is distance information of a significant point at which the amounts of parallax x.sub.1 and x.sub.2 are 0.

(112) The distance image calculated by the distance image calculation unit 74 is recorded on a distance image recording unit 76. Through the calculation of the distance image by the distance image calculation unit 74, it is possible to acquire three-dimensional information of a structure, for example, in a case where the structure is the subject, and to appropriately utilize the acquired three-dimensional information by recording the three-dimensional information on the distance image recording unit 76.

(113) <First Focus Adjustment Unit>

(114) Next, the first focus adjustment unit that automatically performs the focus adjustment of the telephoto optical system 14 included in the first imaging optical system 12.sub.L and the second imaging optical system 12.sub.R.

(115) FIG. 8 is a block diagram illustrating the first focus adjustment unit.

(116) The first focus adjustment unit illustrated in FIG. 8 is provided in each of the first imaging unit 11.sub.L and the second imaging unit 11.sub.R. The first focus adjustment unit includes a focus adjustment mechanism 19 that moves all or a part of the telephoto optical system 14 in the optical axis direction, and a focus control unit 190 that controls the focus adjustment mechanism 19.

(117) The focus control unit 190 includes, for example, an amount-of-parallax detection unit 192, a distance information calculation unit 194, and a in-focus position calculation unit 196.

(118) Position information of the main subject in the first wide angle image and the second wide angle image is input from the first subject detection unit 50.sub.L and the second subject detection unit 50.sub.R to the amount-of-parallax detection unit 192. The amount-of-parallax detection unit 192 detects the amount of parallax of the main subject in the first directivity sensor 17.sub.L and the second directivity sensor 17.sub.R on the basis of the input position information of the main subject.

(119) Further, the position information of the main subject in the first wide angle image and the second wide angle image detected by the first subject detection unit 50.sub.L and the second subject detection unit 50.sub.R is used for control of the first pan and tilt device 30.sub.L and the second pan and tilt device 30.sub.R in the first pan and tilt control unit 60.sub.L and the second pan and tilt control unit 60.sub.R as described above, but is used for detection of the amount of parallax in the focus control unit 190. In addition, even before the main subject is captured on the optical axes L.sub.1 and L.sub.2 of the first imaging optical system 12.sub.L and the second imaging optical system 12.sub.R, the focus control unit 190 in the present example can perform the focus adjustment of the telephoto optical system 14. However, when the focus adjustment of the telephoto optical system 14 is performed in a state in which the main subject is captured on the optical axes L.sub.1 and L.sub.2 of the first imaging optical system 12.sub.L and the second imaging optical system 12.sub.R, the amount-of-parallax detection unit 192 becomes unnecessary. This is because the amount of parallax is zero in a case where the main subject is captured on the optical axes L.sub.1 and L.sub.2 of the first imaging optical system 12.sub.L and the second imaging optical system 12.sub.R.

(120) The amount of parallax detected by the amount-of-parallax detection unit 192 is output to the distance information calculation unit 194. Angle information indicating the pan angle and the tilt angle of the first pan and tilt device 30.sub.L (information indicating the directions of the respective optical axes L.sub.1 and L.sub.2 of the first imaging optical system 12.sub.L and the second imaging optical system 12.sub.R) is input from the first pan and tilt control unit 60.sub.L to the other input of the distance information calculation unit 194. The distance information calculation unit 194 calculates the distance information of the main subject on the basis of these input information. Further, the calculation of the distance information of the main subject performed by the distance information calculation unit 194 is performed similarly to the calculation of the distance information of the correspondence point performed by the distance information calculation unit 70 illustrated in FIG. 4, but both differ in that the amounts of parallax x.sub.1 and x.sub.2 shown in Equation 1 described above are the amounts of parallax of the main subject in the first wide angle image and the second wide angle image in the first directivity sensor 17.sub.L and the second directivity sensor 17.sub.R and the focal length f is a focal length of the wide angle optical system 13 included in the first imaging optical system 12.sub.L and the second imaging optical system 12.sub.R.

(121) The distance information of the main subject calculated by the distance information calculation unit 194 is output to the in-focus position calculation unit 196. On the basis of the distance information of the main subject input from the distance information calculation unit 194, the in-focus position calculation unit 196 calculates a movement position in the optical axis direction of all or a part of the telephoto optical system 14 of the first imaging optical system 12.sub.L and the second imaging optical system 12.sub.R, which is an in-focus position focused on the main subject. Further, since the movement position in the optical axis direction of all or a part of the telephoto optical system 14 and the distance information of the subject focused according to the movement position are in one-one correspondence with each other, the in-focus position calculation unit 196 can calculate the in-focus position focused on the main subject in a case where the distance information of the main subject can be acquired.

(122) The focus control unit 190 controls the focus adjustment mechanism 19 on the basis of the in-focus position calculated by the in-focus position calculation unit 196 so that all or a part of the telephoto optical system 14 of the first imaging optical system 12.sub.L and the second imaging optical system 12.sub.R is moved to the in-focus position in the optical axis direction, thereby performing focus adjustment (first focus adjustment) of the telephoto optical system 14.

(123) It should be noted that the focus control unit 190 in the example acquires the distance information of the main subject on the basis of, for example, the amount of parallax of the main subject in the first directivity sensor 17.sub.L and the second directivity sensor 17.sub.R of the main subject included in the first wide angle image and the second wide angle image, and performs focus adjustment of the respective telephoto optical systems 14 of the first imaging optical system 12.sub.L and the second imaging optical system 12.sub.R on the basis of the acquired distance information, but the present invention is not limited thereto, and in a case where a focus adjustment unit (second focus adjustment unit), such as a contrast AF or an image plane phase difference AF, that performs focus adjustment of the wide angle optical system 13 of the first imaging optical system 12.sub.L and the second imaging optical system 12.sub.R is provided, the distance information of the main subject may be acquired on the basis of the in-focus information of the wide angle optical system 13 from the second focus adjustment unit (for example, a movement position in the optical axis direction of all or some of the wide angle optical system 13 in the in-focus state), and the focus adjustment of the respective telephoto optical system 14 of the first imaging optical system 12.sub.L and the second imaging optical system 12.sub.R may be performed on the basis of the acquired distance information.

(124) FIG. 9 is a block diagram of the pan and tilt control unit 60. The pan and tilt control unit 60 includes a pan control unit 62 that controls the pan device 38 illustrated in FIG. 1, in addition to the first pan and tilt control unit 60.sub.L and the second pan and tilt control unit 60.sub.R illustrated in FIG. 4.

(125) The pan control unit 62 controls the pan device 38 including the pan mechanism on the basis of the first wide angle image and the second wide angle image so that the first imaging unit 11.sub.L and the second imaging unit 11.sub.R directly face the main subject.

(126) The first pan and tilt control unit 60.sub.L and the second pan and tilt control unit 60.sub.R control the first pan and tilt device 30.sub.L and the second pan and tilt device 30.sub.R on the basis of the position information of the main subject in the first wide angle image and the second wide angle image, of the main subject detected by the first subject detection unit 50.sub.L and the second subject detection unit 50.sub.R, but the pan control unit 62 acquires angle information indicating the pan angle .sub.1 and pan angle .sub.2 (see FIG. 7) of the first pan and tilt device 30.sub.L and the second pan and tilt device 30.sub.R from the first pan and tilt control unit 60.sub.L and the second pan and tilt control unit 60.sub.R.

(127) In FIG. 7, in a case where pan angle .sub.1 and pan angle .sub.2 are positive in a clockwise direction, the pan control unit 62 rotationally moves the pan device 38 (the holding member 37 that holds the first pan and tilt device 30.sub.L and the second pan and tilt device 30.sub.R) in a horizontal direction so that the pan angle .sub.1 is positive, the pan angle .sub.2 is negative, and an absolute value of the pan angle 1 becomes equal to an absolute value of the pan angle .sub.2, on the basis of the angle information indicating the pan angle .sub.1 and the pan angle .sub.2. That is, in a case where the pan device 38 is rotationally moved by the pan control unit 62, the first pan and tilt device 30.sub.L and the second pan and tilt device 30.sub.R are driven and controlled in the pan direction to capture the main subject on the optical axes L.sub.1 and L.sub.2. Therefore, by controlling a rotationally moving position of the pan device 38, the pan angle 1 can be positive, the pan angle .sub.2 can be negative, and the absolute values of the pan angle 1 and the pan angle .sub.2 can be equal.

(128) By rotationally moving the holding member 37 (the entire stereo camera 1) that holds the first pan and tilt device 30.sub.L and the second pan and tilt device 30.sub.R, in the horizontal direction as described above, a distance between the first imaging unit 11.sub.L and the second imaging unit 11.sub.R with respect to the main subject can be equidistant and the distance information of the main subject or the like can be calculated more accurately.

(129) [Method of Controlling Stereo Camera]

(130) Next, a method of controlling the stereo camera according to the present invention will be described.

(131) FIG. 10 is a flowchart showing pan and tilt control for capturing the main subject on the respective optical axes L.sub.1 and L.sub.2 of the first imaging optical system 12.sub.L and the second imaging optical system 12.sub.R.

(132) As illustrated in FIG. 10, the first wide angle image is acquired via the first imaging unit 11.sub.L and the first image acquisition unit 22.sub.L, and the second wide angle image is acquired via the second imaging unit 11.sub.R and the second image acquisition unit 22.sub.R (step S10).

(133) Subsequently, the first subject detection unit 50.sub.L and the second subject detection unit 50.sub.R detect the main subject in the first wide angle image and the second wide angle image that have been acquired (step S12), and calculates position information of the detected main subject (first and second position information) in the first wide angle image and the second wide angle image (step S14).

(134) The first pan and tilt device 30.sub.L and the second pan and tilt device 30.sub.R are controlled on the basis of the first position information and the second position information of the first wide angle image and the second wide angle image of the main subject calculated in step S14 (step S16).

(135) Then, a determination is made as to whether or not tracking of the main subject has ended (step S18). That is, in a case where the main subject moves and the first wide angle image and the second wide angle image that are stereo images and the first telephoto image and the second telephoto image that are stereo images are acquired as moving images, it is necessary to always capture the main subject (track the main subject) on the optical axes L.sub.1 and L.sub.2 of the first imaging optical system 12.sub.L and the second imaging optical system 12.sub.R.

(136) Therefore, a determination is made as to whether or not the automatic tracking of the main subject has ended (step S18), and in a case where it is determined that the automatic tracking of the main subject has not ended, the process proceeds to step S10. Accordingly, the processes from step S10 to step S18 are repeated, and imaging is performed while automatically tracking the main subject.

(137) On the other hand, in a case where it is determined that the tracking of the main subject has ended, the pan and tilt control causing the main subject to be tracked is ended. Further, the determination as to whether or not the automatic tracking of the main subject has ended may be performed according to turning on/off of a power supply or may be performed according to, for example, a switch input for ending a pan and tilt operation. Further, in a case where the main subject is imaged for a still image, an end point of capturing of the still image can be the end of the automatic tracking of the main subject.

(138) FIG. 11 is a flowchart showing a process of acquiring distance information of correspondence points of which the feature quantities match each other between the first telephoto image and the second telephoto image.

(139) In FIG. 11, in a case where the main subject is captured on the optical axes L.sub.1 and L.sub.2 of the first imaging optical system 12.sub.L and the second imaging optical system 12.sub.R, the first telephoto image is acquired via the first imaging unit 11.sub.L and the first image acquisition unit 22.sub.L and the second telephoto image is acquired via the second imaging unit 11.sub.R and the second image acquisition unit 22.sub.R (step S20).

(140) Subsequently, the correspondence-point detection unit 72a of the amount-of-parallax detection unit 72 detects correspondence points of which the feature quantities match each other between the first telephoto image and the second telephoto image that have been acquired (step S22), and detects the amount of parallax of the two detected correspondence points in the first directivity sensor 17.sub.L and the second directivity sensor 17.sub.R (step S24).

(141) Then, the distance image calculation unit 74 calculates distance information of the correspondence points on the basis of the amount of parallax between the correspondence points calculated in step S26 as described with reference in FIG. 7, the angle information indicating the pan angle and the tilt angle of the first pan and tilt device 30.sub.L and the second pan and tilt device 30.sub.R (information indicating the directions of the optical axes L.sub.1 and L.sub.2 of the first imaging optical system 12.sub.L and the second imaging optical system 12.sub.R), and the baseline length of the first pan and tilt camera 10.sub.L and the second pan and tilt camera 10.sub.R (step S26).

(142) Further, the distance image calculation unit 74 can calculate the distance image by calculating the distance information for each correspondence point of the first telephoto image and the second telephoto image. Further, since the points on the optical axes L.sub.1 and L.sub.2 of the main subject are also correspondence points, the distance information of the main subject is also calculated, but the amount of parallax of the points on the optical axes L.sub.1 and L.sub.2 in this case is zero.

(143) The distance information (the distance image) calculated in step S26 is output to and recorded on the distance image recording unit 76 or is output to an external device (step S28).

(144) <Moving Body Detection Method>

(145) Next, an example of a moving body detection method for detecting the main subject (a moving body) in a case where the main subject is a moving body will be described with reference to FIGS. 12 to 14. Since the first subject detection unit 50.sub.L and the second subject detection unit 50.sub.R have the same function as the moving body detection unit, only the first subject detection unit 50.sub.L that can function as the moving body detection unit will be described.

(146) In a case where the first subject detection unit 50.sub.L functions as a moving body detection unit, the first subject detection unit 50.sub.L acquires two first wide angle images in time series (the first wide angle image (FIG. 12) that has been acquired previous time and the first wide angle image (FIG. 13) acquired this time) as illustrated in FIGS. 12 and 13, and detects a difference image (FIG. 14) that is a difference between the two first wide angle images in time series.

(147) In the examples illustrated in FIGS. 12 and 13, among objects A and B, an object A moves, the object B stops, and the object A is the main subject (a moving body).

(148) Accordingly, difference images A.sub.1 and A.sub.2 are images generated due to the movement of the object A, as illustrated in FIG. 14.

(149) Here, centroid positions of the difference images A.sub.1 and A.sub.2 are calculated, and positions P.sub.1 and P.sub.2 are set, and a middle point of a line segment connecting these positions P.sub.1 and P.sub.2 is defined as a position G. This position G is defined as a position of the object A (the main subject which is a moving body) in the first wide angle image.

(150) The object A is moved (converges) to the center position of the first wide angle image (the first telephoto image) by repeatedly controlling the pan and tilt mechanism 32 (that is, the imaging direction of the first imaging unit 11.sub.L) so that the position G of the object A in the first wide angle image calculated in this way is moved to the center position (a position on the optical axis L1) of the first wide angle image.

(151) In a case where the first imaging unit 11.sub.L is moved (moved by the pan and tilt mechanism 32 or moved by mounting the stereo camera 1 on a vehicle), backgrounds are also moved between images in time series, but in this case, it is possible to detect an object A moving in a real space regardless of the movement of the first imaging unit 11.sub.L by shifting the image so that the backgrounds match each other between the images in time series and taking a difference image between the images after shifting. Further, the moving body detection method is not limited to the above embodiment.

Another Embodiment of Directivity Sensor

(152) FIG. 15 is a side view illustrating another embodiment of the directivity sensor.

(153) A directivity sensor 117 illustrated in FIG. 15 can be used instead of the first directivity sensor 17.sub.L and the second directivity sensor 17.sub.R.

(154) This directivity sensor 117 includes a microlens array 118 and a light shielding member 120 functioning as a light shielding mask, which form pupil division unit, and an image sensor 116 in which part of light reception cells 116a and 116b are shielded by the light shielding member 120. Further, the light reception cell 116a and the light reception cell 116b, which are partly blocked by the light shielding member 120, are provided alternately (in a checker flag shape) in a horizontal direction and a vertical direction of the image sensor 116.

(155) The microlens array 118 includes microlenses 118a corresponding to the light reception cells 116a and 116b of the image sensor 116 in a one-to-one correspondence.

(156) The light shielding member 120 regulates openings of the light reception cells 116a and 116b of the image sensor 116. The light shielding member 120 has an opening shape corresponding to the wide angle optical system 13 and the telephoto optical system 14 of the first imaging optical system 12.sub.L illustrated in FIG. 2.

(157) A light shielding portion 120a of the light shielding member 120 shields a peripheral portion of the opening of the light reception cell 116a, whereas a light shielding portion 120b of the light shielding member 120 shields a central portion of the opening of the light reception cell 116b. Accordingly, a light beam having passed through the wide angle optical system 13 of the first imaging optical system 12.sub.L is pupil-divided by the microlens array 118 and the light shielding portion 120a of the light shielding member 120 and is incident on the light reception cell 116a, whereas a light beam having passed through the telephoto optical system 14 of the first imaging optical system 12.sub.L is pupil-divided by the microlens array 118 and the light shielding portion 120b of the light shielding member 120 and is incident on the light reception cell 116b.

(158) Accordingly, an image signal of a first wide angle image can be read from each light reception cell 116a of the image sensor 116, and an image signal of a first telephoto image can be read from each light reception cell 116b of the image sensor 116.

Another Embodiment of Imaging Unit

(159) Next, another embodiment of the imaging unit to be applied to the stereo camera according to the present invention will be described.

(160) FIG. 16 is a cross-sectional view illustrating another embodiment of the imaging unit (the first imaging unit 11.sub.L and the second imaging unit 11.sub.R) applicable to the stereo camera 1.

(161) This imaging unit includes an imaging optical system 112 and a directivity sensor 17. Since the directivity sensor 17 is the same as that illustrated in FIGS. 2 and 3, the imaging optical system 112 will be described below.

(162) The imaging optical system 112 includes a central optical system 113 at a central portion and an annular optical system 114 at the peripheral portion thereof, which are disposed on the same optical axis.

(163) The central optical system 113 is a telephoto optical system including a first lens 113a, a second lens 113b, and a common lens 115, and has an angle of view .

(164) The annular optical system 114 is a wide angle optical system including a lens 114a and the common lens 115, and has an angle of view (>). The annular optical system 114 is a wider angle system as compared with the central optical system 113.

(165) This imaging optical system 112 as differs from the first imaging optical system 12.sub.L illustrated in FIG. 2 in that the imaging optical system 112 does not use a reflecting mirror, the central optical system 113 is a telephoto optical system, and the annular optical system 114 is a wide angle optical system.

(166) [Others]

(167) In the stereo camera of the embodiment, the pan and tilt mechanism 32 that rotationally moves the first imaging unit 11.sub.L and the second imaging unit 11.sub.R in the pan direction and the tilt direction is provided in the first camera body 2.sub.L and the second camera body 2.sub.R, but the present invention is not limited thereto, and the first imaging unit 11.sub.L and the second imaging unit 11.sub.R may be mounted on two electric panhead devices (pan and tilt devices), respectively.

(168) Further, a third subject detection unit and a fourth subject detection unit that detect the main subject from the first telephoto image and the second telephoto image are provided in addition to the first subject detection unit and the second subject detection unit that detect the main subject from the first wide angle image and the second wide angle image. In a case where the main subject is in the first telephoto image and in the second telephoto image, the third subject detection unit and the fourth subject detection unit detect position information of the main subject in the first telephoto image and the second telephoto image, and the pan and tilt control unit (first and second pan and tilt control units) controls the pan and tilt mechanism on the basis of the detected position information of the main subject in the first telephoto image and in the second telephoto image. In a case where the third subject detection unit and the fourth subject detection unit cannot detect the main subject, the pan and tilt mechanism may be controlled on the basis of the position information of the main subject in the first wide angle image and the second wide angle image detected by the first subject detection unit and the second subject detection unit.

(169) Further, the main subject may be initially set by an operator from the wide angle image displayed on the display unit 42 using a touch panel or the like.

(170) Further, the reflecting mirrors constituting the telephoto optical systems 14 of the first imaging optical system 12.sub.L and the second imaging optical system 12.sub.R are not limited to concave mirrors and convex mirrors, but may be plane mirrors. The number of reflecting mirrors is not limited to two and three or more reflecting mirrors may be provided.

(171) Further, the present invention is not limited to the above-described embodiments, and it is obvious that various modifications can be performed without departing from the spirit of the present invention.

EXPLANATION OF REFERENCES

(172) 1: stereo camera 2.sub.L: first camera body 2.sub.R: second camera body 4.sub.L, 4.sub.R: base 8A: gear 8.sub.L, 8.sub.R: holding unit 10.sub.L: first pan and tilt camera 10.sub.R: second pan and tilt camera 11.sub.L: first imaging unit 11.sub.R: second imaging unit 12.sub.L: first imaging optical system 12.sub.R: second imaging optical system 13: wide angle optical system 13a: first lens 13b: second lens 13c: third lens 13d: fourth lens 14: telephoto optical system 14a: first lens 14b: second lens 14c: first reflecting mirror 14d: second reflecting mirror 15: common Lens 16: microlens array 16a: microlens 17: directivity sensor 17.sub.L: first directivity sensor 17.sub.R: second directivity sensor 17a: central pupil image 17b: annular pupil image 18: image sensor 18a: light reception cell 19: focus adjustment mechanism 20: A/D converter 22: image acquisition unit 22.sub.L: first image acquisition unit 22.sub.R: second image acquisition unit 24p number of frames 30.sub.L: first pan and tilt device 30.sub.R: second pan and tilt device 30p: number of frames 32: pan and tilt mechanism 34: pan driving unit 36: tilt driving unit 37: holding member 38: pan device 40: digital signal processing unit 42: display unit 44: recording unit 50.sub.L: first subject detection unit 50.sub.R: second subject detection unit 60: pan and tilt control unit 60.sub.L: first pan and tilt control unit 60.sub.R: second pan and tilt control unit 62: pan control unit 70: distance information calculation unit 72: amount-of-parallax detection unit 72a: correspondence-point detection unit 74: distance image calculation unit 76: distance image recording unit 112: imaging optical system 113: central optical system 113a: first lens 113b: second lens 114: annular optical system 114a: lens 115: common lens 116: image sensor 116a: light reception cell 116b: light reception cell 117: directivity sensor 118: microlens array 118a: microlens 120: light shielding member 120a: light shielding portion 120b: light shielding portion 190: focus control unit 192: amount-of-parallax detection unit 194: distance information calculation unit 196: in-focus position calculation unit D: baseline length L1: optical axis L2: optical axis S10 to S28: step f: focal length x.sub.1, x.sub.2: amount of parallax .sub.1, .sub.2: pan angle .sub.1, .sub.2, .sub.1, .sub.2: base angle