IMAGE CAPTURING APPARATUS AND IMAGE CAPTURING METHOD
20220279154 · 2022-09-01
Assignee
Inventors
- Ryota Kosakai (Tokyo, JP)
- Katsutoshi Aiki (Kanagawa, JP)
- Nobuyuki Sato (Tokyo, JP)
- Hiroki Nagahama (Tokyo, JP)
- Masatoshi Sase (Kanagawa, JP)
- Yutaka Yoneda (Kanagawa, JP)
Cpc classification
H04N5/772
ELECTRICITY
H04N23/88
ELECTRICITY
G11B27/031
PHYSICS
H04N23/951
ELECTRICITY
H04N23/10
ELECTRICITY
H04N7/0105
ELECTRICITY
H04N5/783
ELECTRICITY
H04N7/0127
ELECTRICITY
H04N9/80
ELECTRICITY
H04N23/667
ELECTRICITY
H04N25/75
ELECTRICITY
G11B27/005
PHYSICS
International classification
H04N9/73
ELECTRICITY
G11B27/00
PHYSICS
G11B27/031
PHYSICS
G11B31/00
PHYSICS
H04N5/783
ELECTRICITY
H04N7/01
ELECTRICITY
H04N9/79
ELECTRICITY
H04N9/80
ELECTRICITY
Abstract
In a high speed image capturing state, a camera signal processing circuit is not needed to perform a signal process at a high screen rate, but at a regular screen rate. In the high speed image capturing mode, raw data of 240 fps received from an image sensor 101 are recorded on a recording device 111 through a conversion processing section 201 and a recording device controlling circuit 210. Raw data that have been decimated and size-converted are supplied to a camera signal processing circuit 203 through a pre-processing circuit 202 and an image being captured is displayed on a display section 112 with a signal for which a camera process has been performed. In a reproducing state, raw data are read from the recording device 111 at a low screen rate according to a display performance of the display section 112 and the raw data that have been read are processed are processed by the pre-processing circuit 202 and the camera signal processing circuit 203 and a reproduced image is displayed by the display section 112.
Claims
1-70. (canceled)
71. An image processing apparatus, comprising: circuitry configured to receive a first number of first frames of image data that have not yet been subjected to white balance processing, gamma correction processing, and YC conversion processing at a first image size or a first frame rate outputted by an image sensor that includes a color filter array; and at least one first storage medium encoded with executable instructions that, when executed by the circuitry, cause the circuitry to: store the image data, without performing white balance processing, gamma correction processing, and YC conversion processing thereon, on a second storage medium while the second storage medium is attached to the image processing apparatus, the second storage medium being configured to be detachable from the image processing apparatus so as to allow the image data to be processed by an external device separate from the image processing apparatus when the second storage medium is detached from the image processing apparatus; and generate a first signal representing a first white-balanced, gamma-corrected, and YC converted image sequence by processing the first frames of image data to generate a second number of second frames of image data, and by performing at least one of white balance processing, gamma correction processing, or YC conversion processing on the second frames of image data at a second image size which is lower than the first image size, or at a second frame rate which is lower than the first frame rate.
72. The image processing apparatus of claim 71, wherein the at least one first storage medium is further encoded with executable instructions that, when executed by the circuitry, cause the circuitry to: in a reproduction process, read the image data from the second storage medium; and generate a second signal representing a second white-balanced, gamma-corrected, and YC converted image sequence to be outputted to the second external device as a slow motion reproduced image by processing the image data.
73. The image processing apparatus of claim 72, wherein the at least one first storage medium is further encoded with executable instructions that, when executed by the circuitry, cause the circuitry to: in the reproduction process, generate the second signal by processing the decompressed image data at a third frame rate which is lower than the first frame rate.
74. The image processing apparatus of claim 73, wherein the third frame rate is the same as the second frame rate.
75. The image processing apparatus of claim 71, wherein the at least one first storage medium is further encoded with executable instructions that, when executed by the circuitry, cause the circuitry to: process the first and second frames of image data such that a ratio of the first number of frames to the second number of frames is the same as a ratio of the first frame rate to the second frame rate.
76. The image processing apparatus of claim 71, wherein the at least one first storage medium is further encoded with executable instructions that, when executed by the circuitry, cause the circuitry to: generate the first signal such that the first white-balanced, gamma-corrected, and YC converted image sequence corresponds to at least some of the image data that is currently being captured by the image sensor.
77. The image processing apparatus of claim 75, wherein the at least one first storage medium is further encoded with executable instructions that, when executed by the circuitry, cause the circuitry to: generate the second number of second frames of image data by decimating the first number of first frames of image data.
78. The image processing apparatus of claim 72, wherein: the image sensor is further configured to output the first number of first frames of image data throughout a first time period; and the at least one first storage medium is further encoded with executable instructions that, when executed by the circuitry, cause the circuitry to, in the reproduction process, generate the second signal so that the second white-balanced, gamma-corrected, and YC converted image sequence is outputted to the second external device throughout a second time period, which is longer than the first time period.
79. The image processing apparatus of claim 71, wherein the at least one first storage medium is further encoded with executable instructions that, when executed by the circuitry, cause the circuitry to: convert the first number of first frames of image data to the second number of second frames of image data by decimating the first number of first frames of image data.
80. The image processing apparatus of claim 79, wherein the at least one first storage medium is further encoded with executable instructions that, when executed by the circuitry, cause the circuitry to: convert the first number of first frames of image data to the second number of second frames of image data by performing a size adjustment processing on at least some frames of image data.
81. The image processing apparatus of claim 71, wherein the second frame rate is one fourth of the first frame rate.
82. The image processing apparatus of claim 71, wherein the at least one first storage medium is further encoded with executable instructions that, when executed by the circuitry, cause the circuitry to: generate the first signal by performing a simultaneous forming process on at least some frames of image data.
83. The image processing apparatus of claim 82, wherein the at least one first storage medium is further encoded with executable instructions that, when executed by the circuitry, cause the circuitry to: generate the first signal by performing each of white balance processing, gamma correction processing, and YC conversion processing on the second frames of image data.
84. The image processing apparatus of claim 71, wherein the external device comprises a liquid crystal display and the at least one first storage medium is further encoded with executable instructions that, when executed by the circuitry, cause the circuitry to: generate the first signal by performing display processing on the second frames of image data, after performing the at least one of white balance processing, gamma correction processing, and YC conversion processing thereon, and providing the first signal to the liquid crystal display so as to cause the liquid crystal display to display the first white-balanced, gamma-corrected, and YC converted image sequence.
85. The image processing apparatus of claim 71, wherein the at least one first storage medium is further encoded with executable instructions that, when executed by the circuitry, control the circuitry such that the image data that is stored on the second storage medium is RAW image data.
86. The image processing apparatus of claim 71, wherein the at least one first storage medium is further encoded with executable instructions that, when executed by the circuitry, cause the circuitry to: generate the first signal by performing white balance processing on the second frames of image data at the second frame rate.
87. The image processing apparatus of claim 71, wherein the at least one first storage medium is further encoded with executable instructions that, when executed by the circuitry, cause the circuitry to: generate the first signal by performing gamma correction processing on the second frames of image data at the second frame rate.
88. The image processing apparatus of claim 71, wherein the at least one first storage medium is further encoded with executable instructions that, when executed by the circuitry, cause the circuitry to: generate the first signal by performing YC conversion processing on the second frames of image data at the second frame rate.
89. The image processing apparatus of claim 71, wherein the first frame rate is approximately 240 fps and the second frame rate is approximately 60 fps.
90. A method for operating an image processing apparatus, comprising: receiving a first number of first frames of image data that have not yet been subjected to white balance processing, gamma correction processing, and YC conversion processing at a first image size or a first frame rate outputted by an image sensor that includes a color filter array; storing the image data, without performing white balance processing, gamma correction processing, and YC conversion processing thereon, on a second storage medium while the second storage medium is attached to the image processing apparatus, the second storage medium being configured to be detachable from the image processing apparatus so as to allow the image data to be processed by an external device separate from the image processing apparatus when the second storage medium is detached from the image processing apparatus; and generating a first signal representing a first white-balanced, gamma-corrected, and YC converted image sequence that is to be outputted to the external device by processing the first frames of image data to generate a second number of second frames of image data, and by performing at least one of white balance processing, gamma correction processing, or YC conversion processing on the second frames of image data at a second image size which is lower than the first image size, or at a second frame rate which is lower than the first frame rate.
Description
BRIEF DESCRIPTION OF DRAWINGS
[0059]
[0060]
[0061]
[0062]
[0063]
[0064]
[0065]
[0066]
[0067]
[0068]
[0069]
[0070]
[0071]
[0072]
[0073]
BEST MODES FOR CARRYING OUT THE INVENTION
[0074] Next, with reference to accompanying drawings, a first embodiment of the present invention will be described. As shown in
[0075] The image sensor 101 converts incident light of an object captured through an optical system (including a lens, an infrared suppression filter, an optical low-pass filter, and so forth) into an electric signal according to the photoelectric conversion. As the image sensor 101, for example, a CMOS (Complementary Metal Oxide Semiconductor) type image capturing device is used. In the CMOS type image capturing device, photo diodes, line-column selection MOS transistors, signal wires, and so forth are two-dimensionally arranged to form a vertical scanning circuit, a horizontal scanning circuit, a noise reduction circuit, a timing generation circuit, and so forth. As the image sensor 101, a CCD (Charge Coupled Device) that can capture images at high speeds may be used.
[0076] The image sensor 101 can be switched over between a high speed image capturing mode in which a signal is read at a first screen rate (also referred to as the frame rate) higher than the regular screen rate (60 fps (fields/sec) that is based on the specifications of the NTSC system and a regular image capturing mode in which a signal is read at a second screen rate that is the regular screen rate. The screen rate of the high speed image capturing mode is needed to be 240 fps that is four times higher than that of the regular rate. The image sensor 101 is internally equipped with a CDS (Correlated Double Sampling), an A/D converter, and so forth and outputs a digitally captured image signal corresponding to the matrix of pixels of the image sensor 101.
[0077] The image sensor 101 uses three image capturing devices that output captured image signals, for example, of three-primary colors and obtains one output line every four output lines of each image capturing device to accomplish a screen rate of 240 fps that is four times higher than that of the regular screen rate (60 fps). Assuming that the number of pixels of one frame at the regular screen rate is, for example, 6.4 million pixels, the number of pixels in the high speed image capturing mode is 1.6 million pixels.
[0078]
[0079] The color filter array shown in
[0080] On the other hand, at the high screen rate, horizontal scanning lines are decimated and read at a rate of one every four horizontal scanning lines. To deal with the high screen rate, as shown in
[0081] The image capturing apparatus 200 includes a conversion processing section 201, a pre-processing circuit 202, a camera signal processing circuit 203, a display processing circuit 208, a recording device control circuit 210, a recording device 111, a display section, and a control section 213.
[0082] The conversion processing section 201 performs signal shunting and display decimating for a digital image signal received from the image sensor 101. The display decimating is performed only when a signal is output to the display processing circuit 208. The display decimating is a decimation of fields that satisfy the number of fields per unit time defined in the display standard in the high speed image capturing mode of the image capturing apparatus 200 (in this case, 60 fps).
[0083] The pre-processing circuit 202 performs an optically correcting process such as a shading correction for a digital image signal that is output from the image sensor 101 and outputs a resultant digital image signal. The camera signal processing circuit 203 performs a camera signal process such as a white balance adjustment process (also referred to as a development process, an image creation process, or the like) for the image signal received from the pre-processing circuit 202. An output signal of the camera signal processing circuit 203 is supplied to the display processing circuit 208.
[0084] The display processing circuit 208 generates an image signal to be displayed on the display section 112 from the image signal received from the camera signal processing circuit 203 and supplies the resultant signal to the display section 112 to cause it to display an image. The display section 112 is composed, for example, of an LCD (Liquid Crystal Display) and displays a camera-through image that is being captured, a reproduced image of data recorded on the recording device 111, and so forth. The display section 112 may be disposed outside the image capturing apparatus 200 and it may be provided with an interface for an external output instead of the display section 112.
[0085] The recording device control circuit 210 connected to the conversion processing section 201 controls writing and reading image data to and from the recording device 111. Data stored in the recording device 111 are captured image data that have not been processed by the foregoing pre-processing circuit 202 and camera signal processing circuit 203 and are referred to as raw data in this specification.
[0086] As the recording device 111, a magnetic tape, a semiconductor memory such as a flash memory, a hard disk, or the like can be used. As the recording device 111, a non-attachable/detachable type is basically used. However, the recording device 111 may be attachable/detachable such that raw data can be retrieved to the outside. When raw data are retrieved to the outside, raw data that have been processed in the pre-processing circuit 202 are preferably retrieved. The camera signal process is performed, for example, according to software of an external personal computer.
[0087] The control section 213 is a microcomputer composed, for example, of a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and so forth and totally controls each section of the image capturing apparatus by executing programs stored in the ROM and so forth.
[0088]
[0089] An output image signal of the size adjustment section 222 of the conversion processing section 201 is supplied to a shading correction circuit 231 of the pre-processing circuit 202. The shading correction circuit 231 corrects the brightness of the vicinity of the screen such that it does not become dark. An output signal of the shading correction circuit 231 is supplied to the camera signal processing circuit 203.
[0090] The camera signal processing circuit 203 is composed, for example, of a simultaneously forming circuit 241, a white balance correction section 242, an aperture correction section 243, a gamma correction section 244, and a YC generation section 245 arranged in the order from the input side. However, the structure of the camera signal processing circuit 203 is not limited to that shown in
[0091] The simultaneously forming circuit 241 interpolates lost pixels of each color component. The simultaneously forming circuit 241 outputs three-primary color signals (R, G, B) in parallel. Output signals of the simultaneously forming circuit 241 are supplied to the white balance correction section 242. The white balance correction section 242 corrects unbalancing of colors caused by a different color temperature environment of an object and different sensitivities of color filters of the sensor.
[0092] An output of the white balance correction section 242 is supplied to the aperture correction section 243. The aperture correction section 243 is to perform a contour correction that extracts a portion where a signal largely changes and emphasizes the portion. An output signal of the aperture correction section 243 is supplied to the gamma correction section 244.
[0093] The gamma correction section 244 corrects input and output characteristics such that the gradation is correctly reproduced when a captured image signal is output to the display section 112. An output signal of the gamma correction section 244 is supplied to the YC generation section 245.
[0094] The YC generation section 245 generates a luminance signal (Y) and a color difference signal (C). The luminance signal is generated by combining the gamma-corrected RGB signals at a predetermined composition ratio. The color difference signal is generated by combining the gamma-corrected RGB signals at a predetermined composition ratio. The generated luminance signal and color difference signal are supplied to the display section 112 through the display processing circuit 208.
[0095] In the image capturing apparatus shown in
[0096] The decimation section 221 of the conversion processing section 201 decimates raw data by ¼ such that the raw data of 60 fps are obtained. The raw data that have been decimated and size-converted are supplied to the camera signal processing circuit 203 through the pre-processing circuit 202. A signal for which the camera signal process has been performed in the camera signal processing circuit 203 is supplied to the display section 112 through the display processing circuit 208 and an image that is being captured is displayed on the display section 112.
[0097] In the image capturing apparatus shown in
[0098] An output signal of the pre-processing circuit 202 is supplied to the display section 112 through the camera signal processing circuit 203 and the display processing circuit 208 and a reproduced image is displayed by the display section 112. For example, when only the screen rate has been changed, the reproduced image becomes a slow motion reproduced image, the time axis of which has been expanded four times than in the recording state. Instead, images may be captured by changing image capturing conditions (exposure condition and so forth) and when they are reproduced, the four types of captured images (any of still images or moving images) may be compared. Instead, a signal that is read from the recording device 111 may be decimated so as to obtain a frame-by-frame reproduction image.
[0099] Next, with reference to
[0100]
[0101] As shown in
[0102] Next, with reference to
[0103]
[0104] As shown in
[0105] Next, with reference to
[0106] In the image capturing apparatus 500, raw data received from the conversion processing section 201 are supplied to the memory 502 through a memory control circuit 501. The memory control circuit 501 controls writing and reading image data to and from the memory 502. The memory 502 is a FIFO (First In First Out) type memory that temporarily stores image data received from the memory control circuit 501, for example an SDRAM (Synchronous Dynamic Random Access Memory) or the like. The memory 502 performs buffering corresponding to throughputs of the pre-processing circuit 202 and the camera signal processing circuit 203.
[0107] An output signal of the camera signal processing circuit 203 is supplied to a terminal r of a switch SW4. A terminal p of the switch SW4 and a terminal p of a switch SW3 are connected in common. The switch SW4 is connected to a recording device control circuit 504. Connected to the recording device control circuit 504 is a recording device 505.
[0108] The recording device control circuit 504 controls writing and reading image data to and from the recording device 505 through the switch SW4. Data stored in the recording device 505 are a luminance signal and a color difference signal processed by the pre-processing circuit 202 and the camera signal processing circuit 203. The recording device 505 may be a magnetic tape, a semiconductor memory such as a flash memory, a recordable optical disc, a hard disk, or the like. The recording device 505 is basically an attachable/detachable type. Instead, the recording device 505 may not be attachable/detachable type and recorded data may be output to the outside through a communication interface.
[0109]
[0110] In a recording pause period in the high speed image capturing mode, for example, in a recoding standby state where a hand is released from a record button, raw data are read from the memory 502 and the raw data are processed by the pre-processing circuit 202 and the camera signal processing circuit 203 and an output signal of the camera signal processing circuit 203 is recorded on the recording device 505 through the terminal r of the switch SW4 and the recording device control circuit 504. The raw data are read from the memory 502 at the regular screen rate of 60 fps or a lower rate.
[0111] As shown in
[0112] The present invention is not limited to the foregoing embodiments. Instead, various modifications of the embodiments can be performed based on the spirit of the present invention. For example, data stored in the recording device 111 of the image capturing apparatus 400 (
[0113] In addition, the present invention can be applied to devices having an image capturing function such as a mobile phone and a PDA (Personal Digital Assistants) as well as a camcorder and a digital still camera. In addition, the present invention can be applied to a processing device and a recording device for a captured image signal of a small camera for a television phone or a game software application connected to a personal computer or the like.