Image processing device, image processing method, reception device, and transmission device

Abstract

The present technology makes it possible for a conventional frame interpolation technology to handle moving image data captured with a high-speed frame shutter and having a high sharpness image component. Moving image data at a predetermined frame rate and a predetermined resolution is acquired. When a ratio of the predetermined frame rate to a camera shutter speed falls below a threshold value, filtering processing for raising the degree of correlation between adjacent frames is performed on the acquired moving image data. For example, the camera shutter speed is estimated on the basis of information on the frame rate and the resolution.

Claims

1. An image processing device comprising: image data acquiring circuitry configured to receive a container and acquire moving image data included in the container at a first frame rate and a first resolution, the moving image data comprising frames from an original image sequence having an original frame rate corresponding to a camera shutter speed, the first frame rate and the original frame rate differing in accordance with a conversion process that generates the moving image data from the original image sequence; and image processing circuitry configured to acquire an indication whether a ratio of the first frame rate to the camera shutter speed falls below a threshold value, when information inserted in a layer of the container indicates that super high-definition video distribution is delivered, perform filtering processing on adjacent frames of the acquired moving image data in response to an indication that the ratio of the first frame rate to the camera shutter speed falls below the threshold value, and in response to an indication that the ratio does not fall below the threshold value, not perform the filtering processing.

2. The image processing device according to claim 1, wherein the image processing circuitry is configured to estimate the camera shutter speed on the basis of information on the first frame rate and the first resolution.

3. The image processing device according to claim 1, wherein the image data acquiring circuitry is configured to receive the container including a video stream obtained by applying encoding processing to the moving image data and acquires the moving image data by applying decoding processing to the video stream.

4. The image processing device according to claim 3, wherein the image processing circuitry is configured to estimate the camera shutter speed on the basis of information on the first frame rate and the first resolution inserted in a layer of the container.

5. The image processing device according to claim 3, wherein the image processing circuitry is configured to estimate the camera shutter speed on the basis of information on the first frame rate and the first resolution inserted in a layer of the video stream.

6. The image processing device according to claim 3, wherein information on the ratio of the first frame rate to the camera shutter speed is inserted in a layer of the container and/or a layer of the video stream, and the image processing circuitry is configured to obtain the ratio of the first frame rate to the camera shutter speed from the information inserted in the layer of the container and/or the layer of the video stream.

7. The image processing device according to claim 1, wherein the image data acquiring circuitry is configured to acquire the moving image data from an external apparatus via a digital interface.

8. The image processing device according to claim 7, wherein the image processing circuitry is configured to estimate the camera shutter speed on the basis of information on the first frame rate and the first resolution inserted in a blanking period of the moving image data.

9. The image processing device according to claim 7, wherein the image processing circuitry is configured to acquire information on the ratio of the first frame rate to the camera shutter speed from the external apparatus via the digital interface.

10. An image processing method comprising: receiving a container and acquiring moving image data included in the container at a first frame rate and a first resolution, the moving image data comprising frames from an original image sequence having an original frame rate corresponding to a camera shutter speed, the first frame rate and the original frame rate differing in accordance with a conversion process that generates the moving image data from the original image sequence; acquiring an indication whether a ratio of the first frame rate to the camera shutter speed falls below a threshold value; when information inserted in a layer of the container indicates that super high-definition video distribution is delivered, performing filtering processing on adjacent frames of the acquired moving image data in response to an indication that the ratio of the first frame rate to the camera shutter speed falls below the threshold value, and in response to an indication that the ratio does not fall below the threshold value, not performing the filtering processing.

11. A reception device comprising: receiving circuitry configured to receive a container including a video stream obtained by applying encoding processing to moving image data having a first frame rate and a first resolution, the moving image data comprising frames from an original image sequence having an original frame rate corresponding to a camera shutter speed, the first frame rate and the original frame rate differing in accordance with a conversion process that generates the moving image data from the original image sequence; and control circuitry configured to control decoding processing of the video stream to obtain the moving image data at the first frame rate and the first resolution, filtering processing on adjacent frames of the moving image data obtained by the decoding processing in response to an indication that a ratio of the first frame rate to the camera shutter speed falls below a threshold value, and interpolation processing of adjusting the first frame rate of the moving image data filtered by the filtering processing to a frame rate corresponding to a display capability by generating an interpolation frame using an interframe motion vector.

12. A reception device comprising: receiving circuitry configured to receive moving image data at a first frame rate and a first resolution from an external apparatus via a digital interface, the moving image data comprising frames from an original image sequence having an original frame rate corresponding to a camera shutter speed, the first frame rate and the original frame rate differing in accordance with a conversion process that generates the moving image data from the original image sequence; and control circuitry configured to control filtering processing on adjacent frames of the received moving image data in response to an indication that a ratio of the first frame rate to the camera shutter speed falls below a threshold value, and interpolation processing of adjusting the first frame rate of the moving image data as filtered by the filtering processing to a frame rate corresponding to a display capability by generating an interpolation frame using an interframe motion vector.

Description

BRIEF DESCRIPTION OF DRAWINGS

(1) FIG. 1 is a block diagram illustrating a configuration example of a transmission/reception system serving as an embodiment.

(2) FIG. 2 is a diagram for explaining a relationship between a camera shutter speed and a frame rate from image sequence production to distribution at a transmitting side.

(3) FIG. 3 is a diagram for explaining a ratio of a frame rate to a camera shutter speed.

(4) FIG. 4 is a block diagram illustrating a configuration example of a transmission device.

(5) FIG. 5 is a diagram illustrating an example of the structure of a coding parameter descriptor.

(6) FIG. 6 is a diagram illustrating the contents of primary information in the example of the structure of the coding parameter descriptor.

(7) FIG. 7 is a diagram illustrating a configuration example of a transport stream TS in the case of performing distribution by a single stream.

(8) FIG. 8 is a diagram illustrating a configuration example of a transport stream TS in the case of performing distribution by multiple streams.

(9) FIG. 9 is a block diagram illustrating a configuration example of a reception device.

(10) FIG. 10 is a diagram schematically illustrating an example of filtering processing in a mid-processing unit.

(11) FIG. 11 is a diagram illustrating an example of a shutter speed estimation table.

(12) FIG. 12 is a flowchart illustrating an example of control processing for the mid-processing unit by a CPU.

(13) FIG. 13 is a diagram for explaining creation of an interpolation frame using an interframe motion vector.

(14) FIG. 14 is a diagram illustrating an example of the structure of a super high-quality descriptor.

(15) FIG. 15 is a diagram illustrating a configuration example of a transport stream TS in the case of performing distribution by a single stream.

(16) FIG. 16 is a diagram illustrating an example of the structure of a frame quality descriptor.

(17) FIG. 17 is a diagram illustrating the contents of primary information in the example of the structure of the frame quality descriptor.

(18) FIG. 18 is a diagram illustrating an example of the structure of a frame quality SEI message.

(19) FIG. 19 is a diagram illustrating a configuration example of a transport stream TS in the case of performing distribution by a single stream.

(20) FIG. 20 is a diagram illustrating a configuration example of a transport stream TS in the case of performing distribution by multiple streams.

(21) FIG. 21 is a flowchart illustrating an example of control processing for a mid-processing unit by a CPU.

(22) FIG. 22 is a block diagram illustrating another configuration example of the transmission/reception system.

MODE FOR CARRYING OUT THE INVENTION

(23) Modes for carrying out the invention (hereinafter, referred to as embodiments) will be described below. Note that the description will be given in the following order.

(24) 1. Embodiments

(25) 2. Variations

1. Embodiments

(26) [Transmission/Reception System]

(27) FIG. 1 illustrates a configuration example of a transmission/reception system 10 serving as an embodiment. This transmission/reception system 10 has a configuration including a transmission device 100 and a reception device 200.

(28) The transmission device 100 incorporates a transport stream TS as a container into a broadcasting wave to transmit. This transport stream TS includes one or a plurality of video streams obtained by encoding moving image data. In this case, for example, encoding such as H.264/AVC or H.265/HEVC is applied.

(29) Here, the moving image data includes, in addition to data of a moving image captured with a normal frame shutter such as 30 Hz or 60 Hz, data of a moving image captured with a high-speed frame shutter such as 120 Hz or 240 Hz or data of a moving image obtained by converting such a moving image into a low-frequency moving image sequence, and the like. The moving image data captured with a high-speed frame shutter has a high sharpness image component. Therefore, when a shutter aperture ratio at a receiving/reproducing side is lowered, such moving image data has factors that cause image quality problems in a conventional frame interpolation technology.

(30) For example, as illustrated in FIG. 2, (1) an original image sequence captured with a high-speed frame shutter has a camera shutter speed of 240 Hz and a 4K resolution of 38402160 and is converted into a distribution image sequence having a frame rate of 120 Hz and a 4K resolution of 38402160 to be transmitted.

(31) (2) In the case of single stream distribution, if one frame is retrieved per two frames when a sequence is generated from 240 Hz to 120 Hz, the original quality at the time of capturing is maintained in regard to the sharpness of the image. However, a ratio of a distribution frame rate to the camera shutter speed is as low as 50% (=120/240).

(32) (3) In the case of multiple stream distribution, here, in the case of two stream distribution (scalability transmission in a time direction), images of the sequence with a ratio to the camera shutter speed of 50% mentioned in (2) are alternately retrieved such that one is assigned to a base group (Base Group) and the other is assigned to an enhanced group (Enhanced Group). The frame rate of the base group is 60 Hz and likewise, the frame rate of the enhanced group is also 60 Hz.

(33) In a case where both of the base group and the enhanced group are decoded and displayed, since the frame rate of the entire image sequence to be displayed is 120 Hz, the ratio to the camera shutter speed is 50%, which is comparable with the distribution by the single stream. However, in a case where only the base group is displayed, the frame rate is 60 Hz and the ratio to the camera shutter speed is further lowered to 25% (=60/240).

(34) FIG. 3 illustrates a ratio of a frame rate to a camera shutter speed. In the case of an original image sequence, the ratio is 100% (=240/240). In the case of single stream distribution, the ratio is 50% (=120/240). In the case of multiple stream distribution, the ratio of each of the base group and the enhanced group is 25% (=60/240) and, when both are combined, the ratio becomes 50% (60*2/240). In a case where only the base group is displayed, the ratio is 25%. As is clear from the illustrated example, the lower the ratio, the lower the correlation between preceding and succeeding frames.

(35) Returning to FIG. 1, the transmission device 100 inserts information on the frame rate and the resolution into a layer of the transport stream TS as a container. In a network abstraction layer (NAL) unit of a sequence parameter set (SPS) of the video stream, information on the frame rate and the resolution is present. As described above, the information on the frame rate and the resolution to be inserted into the layer of the transport stream TS corresponds to the SPS NAL unit of the video stream. By inserting the information on the frame rate and the resolution into the layer of the transport stream TS in this manner, a receiving side can easily acquire this information on the frame rate and the resolution from the layer of the transport stream TS.

(36) In addition, the transmission device 100 inserts information indicating whether super high-definition video distribution is delivered into the layer of the transport stream TS as a container. For example, high-definition video distribution includes distribution of a moving image captured with a high-speed frame shutter such as 120 Hz or 240 Hz as it is, or distribution of a moving image obtained by simply retrieving a predetermined frame from such a moving image to convert into a low-frequency moving image sequence while the sharpness of the image is maintained. By inserting the information indicating whether the super high-definition video distribution is delivered into the layer of the transport stream TS in this manner, a receiving side can easily acquire this information indicating whether the super high-definition video distribution is delivered from the layer of the transport stream TS.

(37) In this embodiment, the transmission device 100 inserts the information on the frame rate and the resolution and the information indicating whether the super high-definition video distribution is delivered that have been described above, for example, into the inside of a video elementary stream loop arranged under a program map table in correspondence with the video stream, as a descriptor. Details of this descriptor will be described later.

(38) The reception device 200 receives the above-described transport stream TS sent from the transmission device 100 by being incorporated into the broadcasting wave. This transport stream TS includes one video stream obtained by encoding moving image data or a plurality of video streams (for example, two video streams of a base stream and an enhanced stream). The reception device 200 applies decoding to the video stream to obtain moving image data at a predetermined frame rate and a predetermined resolution.

(39) In this case, in a case where the moving image data is distributed in a single stream, the reception device 200 applies decoding processing to this single stream to obtain the moving image data at the predetermined frame rate and the predetermined resolution. In addition, in this case, in a case where the moving image data is distributed in multiple streams, the reception device 200 applies decoding processing to a predetermined number of streams according to a decoding capability to obtain the moving image data at the predetermined frame rate and the predetermined resolution.

(40) For example, in a case where the moving image data is distributed in two streams, namely, the base stream and the enhanced stream, only the base stream or both of the base stream and the enhanced stream are decoded to obtain the moving image data at the predetermined frame rate and the predetermined resolution.

(41) The reception device 200 applies interpolation processing for generating an interpolation frame using an interframe motion vector to the moving image data at the predetermined frame rate and the predetermined resolution obtained by the decoding processing and obtains moving image data for display. Here, when the ratio of the predetermined frame rate to the camera shutter speed falls below a threshold value, the reception device 200 carries out, prior to the interpolation processing, filtering processing for raising the degree of correlation between adjacent frames on the moving image data obtained by the decoding processing. By performing this filtering processing, frame interpolation can be satisfactorily performed with a conventional frame interpolation technology.

(42) The reception device 200 estimates the camera shutter speed on the basis of the information on the frame rate and the resolution. For example, the reception device 200 uses the information on the frame rate and the resolution inserted in the layer of the transport stream TS as a container or inserted in the layer of the video stream.

(43) Note that the filtering processing described above focuses on avoiding image quality problems caused in a case where interpolation is performed with a conventional frame interpolation technology while the sharpness of the image owing to high-speed frame shutter capturing is maintained in each frame constituting the moving image obtained by the decoding processing. In this embodiment, in a case where the information indicating whether the super high-definition video distribution is delivered, which is inserted in the layer of the transport stream TS as a container, indicates that the super high-definition video distribution is delivered, the reception device 200 carries out the above-described filtering processing.

(44) [Configuration of Transmission Device]

(45) FIG. 4 illustrates a configuration example of the transmission device 100. This transmission device 100 has a central processing unit (CPU) 101, an encoder 102, a multiplexer 103, and a transmitting unit 104. The CPU 101 is a control unit and controls the action of each member of the transmission device 100.

(46) The encoder 102 accepts the input of moving image data VD that is not compressed and constitutes the distribution image sequence. Then, the encoder 102 applies the encoding processing such as H.264/AVC or H.265/HEVC to this moving image data VD and generates one video stream in the case of the single stream distribution and a plurality of video streams in the case of the multiple stream distribution. Hereinafter, for simplicity of explanation, this embodiment assumes that the plurality of video streams is two video streams of the base stream and the enhanced stream.

(47) The multiplexer 103 converts the video stream generated by the encoder 102 into a packetized elementary stream (PES) packet and further converts the PES packet into a transport packet to multiplex, thereby obtaining the transport stream TS as a multiplexed stream. In this embodiment, this transport stream TS includes only the base stream or both of the base stream and the enhanced stream.

(48) In addition, the multiplexer 103 inserts the information on the frame rate and the resolution and the information indicating whether the super high-definition video distribution is delivered into the layer of the transport stream TS as a container. These pieces of information are inserted as descriptors into the inside of the video elementary stream loop arranged under the program map table in correspondence with the video stream. The transmitting unit 104 incorporates the transport stream TS obtained by the multiplexer 103 into the broadcasting wave to transmit to the reception device 200.

(49) [About Insertion of Information]

(50) The insertion of information in the multiplexer 103 will be further described. The multiplexer 103 inserts a coding parameter descriptor (Coding Parameter_descriptor) to be newly defined. FIG. 5 illustrates an example of the structure of the coding parameter descriptor (Syntax). FIG. 6 illustrates the contents of primary information (Semantics) in the above example of the structure.

(51) An eight-bit field of coding parameter_descriptor_tag represents a descriptor type. coding parameter_descriptor_tag here represents that it is a coding parameter descriptor. An eight-bit field of coding parameter_descriptor length represents a length (size) of the descriptor and indicates the number of subsequent bytes as the length of the descriptor. coding parameter_descriptor length here represents three bytes.

(52) A four-bit field of service_quality_type represents whether the super high-definition video distribution is delivered. 001 indicates that the super high-definition video distribution is delivered. 010 indicates that merely high-definition video distribution is delivered. A one-bit field of temporal_scalablility_flag represents whether a multiple stream configuration having temporal scalability is used. 1 indicates that the multiple stream configuration having the temporal scalability is used. 0 indicates that a single stream configuration not having the temporal scalability is used.

(53) An eight-bit field of Resolution_type represents the resolution. For example, 001 indicates a resolution of 19201080, that is, HD resolution, 002 indicates a resolution of 38402160, that is, 4K resolution, and 003 indicates a resolution of 76804320, that is, 8K resolution. An eight-bit field of FrameRate_type represents the frame rate of the entire distribution. For example, 001 indicates 30 Hz, 002 indicates 60 Hz, and 003 indicates 120 Hz.

(54) [Configuration of Transport Stream TS]

(55) FIG. 7 illustrates a configuration example of the transport stream TS in the case of performing distribution by the single stream. This transport stream TS includes one video stream. That is, according to this configuration example, there is a PES packet video PES1 for the video stream. Encoded image data of each picture included in a PES payload is constituted by NAL units such as AUD, VPS, SPS, PPS, PSEI, SLICE, SSEI, and EOS.

(56) Additionally, the program map table (PMT) is included in the transport stream TS as program specific information (PSI). This PSI is information mentioning which program is the one to which each elementary stream included in the transport stream belongs. The PMT has a program loop (Program loop) stating information relating to the entire program. In addition, the PMT has an elementary stream loop having information relating to each of the elementary streams. According to this configuration example, there is a video elementary stream loop (video ES1 loop).

(57) Information such as a stream type and a packet identifier (PID) is arranged in the video elementary stream loop in correspondence with the video stream (video PES1) and at the same time, a descriptor stating information relating to this video stream is also arranged therein. As one of such descriptors, an HEVC descriptor (HEVC_descriptor) and the above-described coding parameter descriptor (Coding Parameter_descriptor) are inserted.

(58) FIG. 8 illustrates a configuration example of the transport stream TS in the case of performing distribution by multiple streams, here, by two streams. This transport stream TS includes two video streams. That is, according to this configuration example, there is a PES packet video PES1 for the base stream and there also is a PES packet video PES2 for the enhanced stream. Encoded image data of each picture included in a PES payload is constituted by NAL units such as AUD, VPS, SPS, PPS, PSEI, SLICE, SSEI, and EOS.

(59) Additionally, the program map table (PMT) is included in the transport stream TS as program specific information (PSI). This PSI is information mentioning which program is the one to which each elementary stream included in the transport stream belongs. The PMT has a program loop (Program loop) stating information relating to the entire program. In addition, the PMT has an elementary stream loop having information relating to each of the elementary streams. According to this configuration example, there are two video elementary stream loops (video ES1 loop and video ES2 loop) in this configuration example.

(60) Information such as a stream type and a packet identifier (PID) is arranged in the respective video elementary stream loops in correspondence with the video streams (video PES1 and video PES2) and at the same time, descriptors stating information relating to these video streams are also arranged therein. As one of such descriptors, an HEVC descriptor (HEVC_descriptor) and the above-described coding parameter descriptor (Coding Parameter_descriptor) are inserted.

(61) Note that, in the configuration example of the transport stream TS illustrated in FIG. 8, the coding parameter descriptor is inserted in each video elementary stream loop. However, a configuration example in which the coding parameter descriptor is arranged only in the video elementary stream loop corresponding to the base stream is also conceivable.

(62) The action of the transmission device 100 illustrated in FIG. 4 will be briefly described. The moving image data VD that is not compressed and constitutes the distribution image sequence is input to the encoder 102. In the encoder 102, the encoding processing such as H.264/AVC or H.265/HEVC is applied to this moving image data VD and one video stream is generated in the case of the single stream distribution, while two video streams of the base stream and the enhanced stream are generated in the case of the multiple stream distribution. This video stream is supplied to the multiplexer 103.

(63) In the multiplexer 103, the video stream is converted into the PES packet and further converted into the transport packet to be multiplexed, whereby the transport stream TS is obtained as a multiplexed stream. This transport stream TS includes, for example, only the base stream or both of the base stream and the enhanced stream.

(64) In addition, in the multiplexer 103, the information on the frame rate and the resolution and the information indicating whether the super high-definition video distribution is delivered are inserted into the layer of the transport stream TS as a container. These pieces of information are inserted as descriptors into the inside of the video elementary stream loop arranged under the program map table in correspondence with the video stream. Specifically, in the multiplexer 103, the coding parameter descriptor to be newly defined (refer to FIG. 5) is inserted together with the HEVC descriptor.

(65) The transport stream TS generated by the multiplexer 103 is sent to the transmitting unit 104. In the transmitting unit 104, this transport stream TS is modulated by a modulation scheme suitable for broadcasting such as QPSK/OFDM and an RF modulation signal is transmitted from a transmission antenna.

(66) [Configuration of Reception Device]

(67) FIG. 9 illustrates a configuration example of the reception device 200. This reception device 200 has a central processing unit (CPU) 201, a receiving unit 202, a demultiplexer 203, a decoder 204, a mid-processing unit (image processing unit) 205, a post-processing unit (interpolation processing unit) 206, and a display unit 207. The CPU 201 constitutes a control unit and controls the action of each member of the reception device 200.

(68) The receiving unit 202 receives the transport stream TS sent from the transmission device 100 by being incorporated into the broadcasting wave or a packet in a network. In this transport stream TS, one video stream is included in the case of the single stream distribution and the two video streams of the base stream and the enhanced stream are included in the case of the multiple stream distribution.

(69) The demultiplexer 203 takes out one video stream from the transport stream TS in the case of the single stream distribution and takes out only the base stream or both of the base stream and the enhanced stream therefrom in accordance with the decoding capability of the decoder 204 in the case of the multiple stream distribution, through filtering by PID to supply to the decoder 204. In this case, in a case where the multiple stream distribution is delivered and both of the base stream and the enhanced stream are taken out, both of the streams are integrated into one video stream on the basis of decoding timing information and supplied to the decoder 204.

(70) The demultiplexer 203 also extracts section information included in the layer of the transport stream TS to send to the CPU 201. In this case, the coding parameter descriptor (refer to FIG. 5) is extracted as well. As a result, the CPU 201 can obtain the information on the frame rate and the resolution of the moving image according to the distribution and additionally, the information indicating whether the super high-definition video distribution is delivered.

(71) The decoder 204 applies decoding processing to the video stream supplied from the demultiplexer 203 to acquire moving image data at a predetermined frame rate and a predetermined resolution. The post-processing unit 206 adjusts the frame rate of the moving image data acquired by the decoder 204 to the display capability by generating an interpolation frame using an interframe motion vector. For example, when the frame rate of the moving image data acquired by the decoder 204 is 30 Hz, 60 Hz, and 120 Hz and the display capability is 240 Hz, the frame rate is converted into 240 Hz.

(72) The mid-processing unit 205 is interposed between the decoder 204 and the post-processing unit 206. When the ratio of the frame rate of the moving image data obtained by the decoder 204 to the camera shutter speed falls below a threshold value (for example, 50%), the mid-processing unit 205 performs filtering processing for raising the degree of correlation between adjacent frames on this moving image data under the control of the CPU 201. By performing this filtering processing, the post-processing unit 206 can perform frame interpolation satisfactorily with a conventional frame interpolation technology.

(73) FIG. 10 schematically illustrates an example of the filtering processing in the mid-processing unit 205. A current frame of an input image sequence and a predetermined number of frames before and after the current frame are weighted and added to obtain the current frame of an output image sequence. For example, when the current frame is an N frame, individual frames of N3, N2, N1, N, N+1, N+2, N+3 of the input image sequence are weighted by coefficients d, c, b, a, b, c, d, respectively, and added to each other, whereby the N frame of the input image sequence is obtained. For example, 4/8, 2/8, , , 2/4, , , 0, and the like can be considered as an example of the coefficients a, b, c, d. Note that the number of taps and the coefficient values are not limited to this example. The number of taps and the coefficient values can be set in accordance with the number of frames contributing to interpolation and the selection of the filter effect.

(74) When the ratio of the frame rate to the camera shutter speed falls below the threshold value, the mid-processing unit 205 supplies the moving image data as a result of the filtering processing (the output image sequence in FIG. 10) to the post-processing unit 206. On the other hand, when the ratio of the frame rate to the camera shutter speed is equal to or larger than the threshold value, the mid-processing unit 205 supplies the moving image data obtained by the decoder 204 to the post-processing unit 206 as it is.

(75) The CPU 201 determines whether the ratio of the frame rate of the moving image data obtained by the decoder 204 to the camera shutter speed falls below the threshold value. For example, it is also conceivable to directly give information on the camera shutter speed from the transmission device 100 to the reception device 200 by inserting the information into the transport stream TS as a container. Alternatively, it is also conceivable to directly give information on the camera shutter speed to the reception device 200 by a user's input operation.

(76) In this embodiment, the CPU 201 estimates the camera shutter speed on the basis of the information on the frame rate and the resolution obtained from the coding parameter descriptor. The CPU 201 holds a shutter speed estimation table fixed in advance between transmitting and receiving parties and refers to this table to estimate the camera shutter speed.

(77) FIG. 11 illustrates an example of the shutter speed estimation table. In this example, Normal Shutter Speed means that the same speed as the frequency of the frame rate is employed. Meanwhile, Short Shutter Speed means a speed value faster than Normal Shutter Speed. For example, Short Shutter Speed corresponding to a frame rate of 60 Hz and a resolution of 38402160 is assigned to 120 Hz or 240 Hz, and so forth.

(78) The flowchart in FIG. 12 illustrates an example of control processing for the mid-processing unit 205 by the CPU 201. In step ST1, the CPU 201 starts the processing. Thereafter, in step ST2, the CPU 201 determines whether the high-definition video distribution is delivered. The CPU 201 makes this determination on the basis of the information service_quality_type indicating whether the super high-definition video distribution is delivered, which is obtained from the coding parameter descriptor (refer to FIG. 5).

(79) When the high-definition video distribution is not delivered, the CPU 201 controls the mid-processing unit 205 such that the filtering processing is not carried out in step ST3 and thereafter ends the processing in step ST4. In this case, the mid-processing unit 205 is placed in a state of supplying the moving image data obtained by the decoder 204 to the post-processing unit 206 as it is.

(80) On the other hand, when the high-definition video distribution is delivered, the CPU 201 proceeds to processing in step ST5. In this step ST5, the CPU 201 obtains the information on the frame rate and the resolution. The CPU 201 acquires this information from, for example, the coding parameter descriptor (refer to FIG. 5). Note that the CPU 201 also can obtain this information from the NAL unit of the SPS of the video stream.

(81) Next, in step ST6, the CPU 201 refers to the shutter speed estimation table (refer to FIG. 11) and estimates the camera shutter speed on the basis of the information on the frame rate and the resolution obtained in step ST5. Then, in this step ST6, the CPU 201 finds the ratio of the frame rate to the camera shutter speed.

(82) Next, in step ST7, the CPU 201 determines whether to display only the base stream in the case of the multiple stream distribution. In a case where the multiple stream distribution is delivered and only the base stream is decoded by the decoder 204, the CPU 201 proceeds to processing in step ST8. In this step ST8, the CPU 201 multiplies the ratio found in step ST6 by to correct. Thereafter, the CPU 201 proceeds to processing in step 9.

(83) On the other hand, in a case where the single stream distribution is delivered or in a case where the multiple stream distribution is delivered and both of the base stream and the enhanced stream are decoded by the decoder 204, the CPU 201 immediately proceeds to step ST9 from step ST7. In this step ST9, the CPU 201 determines whether the ratio falls below the threshold value.

(84) When the ratio is equal to or larger than the threshold value, the CPU 201 controls the mid-processing unit 205 such that the filtering processing is not carried out in step ST3 and thereafter ends the processing in step ST4. On the other hand, when the ratio falls below the threshold value, the CPU 201 controls the mid-processing unit 205 such that the filtering processing is carried out in step ST10 and thereafter ends the processing in step ST4. In this case, the mid-processing unit 205 is placed in a state of supplying the moving image data as a result of the filtering processing to the post-processing unit 206.

(85) Note that, in the flowchart in FIG. 12, the processing in step ST2 is not necessarily required and control processing excluding this processing is also conceivable. In that case, the CPU 201 immediately proceeds to the processing in step ST5 after starting the processing in step ST1.

(86) FIG. 13 schematically illustrates an example of interpolation frame generation processing using the interframe motion vector. This example depicts an example where an interpolation frame is generated between the N1 frame and the N frame. A moving object and a background coexist in a predetermined motion prediction block. Triangles in the background move from left to right in the screen from the N1 frame to the N frame, while a large circle as the object moves from right to left in the screen.

(87) In a case where an interpolation frame is generated, in the motion prediction for a block A indicated by a one-dot chain line, a texture in the block almost uniformly coincides with the motion vector with respect to prediction for the N1 frame and prediction for the N frame. However, in regard to a block B indicated by a two-dot chain line, a texture in the block is not uniform in disagreement with the motion vector. In this case, although a large circle portion out of the block B coincides with the motion vector, a background portion is equivalent to a motion different from that of the large circle and thus does not coincide with the motion vector.

(88) As a result, in regard to the block B in the interpolation frame, the large circle portion has good image quality, but the background portion other than that has poor image quality. Meanwhile, in the block A in the interpolation frame, the whole inside of the block is the background portion and the image quality thereof is good. In this manner, the image quality of the interpolation frame has such a result that a difference between good quality and poor quality is noticeable in part.

(89) In addition, since the sharpness of the texture itself is raised as the camera shutter speed becomes faster than the frame rate of the image, there is a case where deterioration in image quality in such an interpolation frame causes a more obvious difference in image quality between a portion where motion prediction is true and a portion where motion prediction is not true.

(90) A method of making the size of the motion prediction block smaller such that the texture included in the block coincides with the motion vector is conceivable. However, extremely small sized blocks bring about high implementation cost.

(91) In order not to cause image quality breakdown due to frame interpolation by conventional motion prediction, it is conceivable to lower the sharpness of the texture in a case where the camera shutter speed is fast (refer to the mid-processing unit 205 in FIG. 9).

(92) When an interpolation image is generated by the conventional motion prediction from an image with the sharpness of the custure lowered as described above, the sharpness is reduced and interpolation is performed using a somewhat blurred image. Consequently, a difference in image quality due to non-coincidence with the motion vector between the large circle portion and the background portion in the block B is made smaller and the breakdown of the entire image can be prevented.

(93) Returning to FIG. 9, the display unit 207 displays a moving image according to the moving image data obtained by the post-processing unit 206. This display unit 207 is constituted by, for example, a liquid crystal display (LCD) or an organic electro-luminescence (organic EL) panel. Note that this display unit 207 may be an external apparatus connected to the reception device 200.

(94) The action of the reception device 200 illustrated in FIG. 9 will be briefly described. In the receiving unit 202, the RF modulation signal received by a reception antenna is demodulated and the transport stream TS is acquired. This transport stream TS is sent to the demultiplexer 203. In the demultiplexer 203, a video stream on which decoding processing is to be carried out is taken out from the transport stream TS through filtering by PID.

(95) In this case, one video stream is taken out in the case of the single stream distribution and only the base stream or both of the base stream and the enhanced stream are taken out in accordance with the decoding capability of the decoder 204 in the case of the multiple stream distribution. The video stream taken out in such a manner is supplied to the decoder 204.

(96) In addition, in the demultiplexer 203, the section information included in the layer of the transport stream TS is extracted to be sent to the CPU 201. In this case, the coding parameter descriptor (refer to FIG. 5) is extracted as well. As a result, the CPU 201 can obtain the information on the frame rate and the resolution of the moving image according to the distribution and additionally, the information indicating whether the super high-definition video distribution is delivered.

(97) In the decoder 204, decoding processing is applied to the video stream supplied from the demultiplexer 203 such that moving image data at a predetermined frame rate and a predetermined resolution is obtained. This moving image data is supplied to the post-processing unit 206 via the mid-processing unit 205. In the mid-processing unit 205, when the ratio of the frame rate of the moving image data obtained by the decoder 204 to the camera shutter speed falls below the threshold value (for example, 50%), the filtering processing for raising the degree of correlation between adjacent frames is performed on this moving image data under the control of the CPU 201.

(98) Therefore, when the ratio of the frame rate to the camera shutter speed falls below the threshold value, the moving image data as a result of the filtering processing (the output image sequence in FIG. 10) is supplied from the mid-processing unit 205 to the post-processing unit 206. On the other hand, when the ratio of the frame rate to the camera shutter speed is equal to or larger than the threshold value, the moving image data obtained by the decoder 204 is supplied from the mid-processing unit 205 to the post-processing unit 206 as it is.

(99) In the post-processing unit 206, the frame rate of the moving image data is adjusted to the display capability by generating the interpolation frame using the interframe motion vector. This moving image data processed by the post-processing unit 206 is supplied to the display unit 207 and a moving image is displayed.

(100) As described above, in the transmission/reception system 10 illustrated in FIG. 1, the filtering processing for raising the degree of correlation between adjacent frames is performed by the mid-processing unit 205 in the reception device 200 on the moving image data obtained by the decoder 204 when the ratio of the frame rate thereof to the camera shutter speed falls below the threshold value. Therefore, the post-processing unit 206 can perform frame interpolation satisfactorily with a conventional frame interpolation technology.

(101) In addition, in the transmission/reception system 10 illustrated in FIG. 1, the camera shutter speed is estimated in the reception device 200 on the basis of the information on the frame rate and the resolution and this estimated camera shutter speed is used to find the ratio of the frame rate of the moving image data obtained by the decoder 204 to the camera shutter speed. Therefore, it becomes unnecessary to transmit information on the camera shutter speed from the transmission device 100 to the reception device 200.

(102) Meanwhile, in the transmission/reception system 10 illustrated in FIG. 1, the coding parameter descriptor including the information on the frame rate and the resolution is inserted by the transmission device 200 into the layer of the transport stream TS as a container. Therefore, a receiving side can easily acquire the information on the frame rate and the resolution from this coding parameter descriptor.

(103) Additionally, in the transmission/reception system 10 illustrated in FIG. 1, the coding parameter descriptor including the information indicating whether the high-definition video distribution is delivered is inserted by the transmission device 200 into the layer of the transport stream TS as a container. Therefore, it becomes possible for a receiving side to perform the filtering processing on only the moving image data having a high sharpness image component captured with a high-speed frame shutter using the mid-processing unit 205 on the basis of that information indicating whether the high-definition video distribution is delivered. That is, it is possible to avoid wastefully performing the filtering processing.

2. Variations

(104) Note that the above embodiments have illustrated an example where the transmission device 100 inserts the coding parameter descriptor including the information on the frame rate and the resolution into the layer of the transport stream TS as a container. However, since the reception device 200 can acquire the information on the frame rate and the resolution from the NAL unit of the SPS of the video stream, a configuration in which this coding parameter descriptor is not inserted is also conceivable.

(105) Additionally, in that case, it is also conceivable to insert, into the layer of the transport stream TS as a container, a descriptor with a structure obtained by removing the information on the frame rate and the resolution from the coding parameter descriptor. FIG. 14 illustrates an example of the structure of a super high-quality descriptor (SuperHighQuality_descriptor) to be newly defined in that case (Syntax). In addition, FIG. 15 illustrates an example of a TS structure in that case (single stream distribution).

(106) Meanwhile, the above embodiments have illustrated an example where the reception device 200 estimates the camera shutter speed from the information on the frame rate and the resolution to find the ratio of the frame rate to the camera shutter speed. However, a configuration for transmitting this information on the ratio from the transmission device 100 to the reception device 200 is also conceivable. In this case, this information on the ratio is inserted into the layer of the transport stream TS as a container and/or the layer of the video stream.

(107) For example, this information on the ratio is inserted as a descriptor into the inside of the video elementary stream loop arranged under the program map table in correspondence with the video stream. For example, the multiplexer 103 inserts a frame quality descriptor (FrameQuality_descriptor) to be newly defined. FIG. 16 illustrates an example of the structure of the frame quality descriptor (Syntax). FIG. 17 illustrates the contents of primary information (Semantics) in the above example of the structure.

(108) An eight-bit field of framequality_descriptor_tag represents a descriptor type. framequality_descriptor_tag here represents that it is a frame quality descriptor. An eight-bit field of framequality_descriptor length represents a length (size) of the descriptor and indicates the number of subsequent bytes as the length of the descriptor. framequality_descriptor length here represents three bytes.

(109) A four-bit field of service_quality_type represents whether the super high-definition video distribution is delivered. 001 indicates that the super high-definition video distribution is delivered. 010 indicates that merely high-definition video distribution is delivered. A one-bit field of temporal_scalablility_flag represents whether a multiple stream configuration having temporal scalability is used. 1 indicates that the multiple stream configuration having the temporal scalability is used. 0 indicates that a single stream configuration not having the temporal scalability is used.

(110) A three-bit field of ratio_shutter_speed_vs_frame_rate represents the ratio of an image frame rate to the camera shutter speed. For example, 00 indicates 100% (the shutter speed is the same as the image frame rate), 01 indicates 50% (the shutter speed is twice the image frame rate), 02 indicates 33% (the shutter speed is three times the image frame rate), and 03 indicates 25% (the shutter speed is four times the image frame rate).

(111) In addition, for example, this information on the ratio is inserted as an SEI message into an SEIs portion of an access unit (AU) of the video stream. The encoder 102 inserts a frame quality SEI message (FrameQuality SEI message) to be newly defined. FIG. 18 illustrates an example of the structure of the frame quality SEI message (Syntax).

(112) One-bit flag information of FrameQuality_cancel_flag represents whether this message is to be refreshed. 0 indicates that the message is to be refreshed. 1 indicates that the message is not to be refreshed, that is, a previous message is maintained as it is. Note that information in the respective fields of service_quality_type, temporal_scalablility_flag, and ratio_shutter_speed_vs_frame_rate is the same information as that described in the aforementioned frame quality descriptor (refer to FIG. 17).

(113) FIG. 19 illustrates a configuration example of the transport stream TS in the case of performing distribution by a single stream. FIG. 20 illustrates a configuration example of the transport stream TS in the case of performing distribution by multiple streams, here, by two streams. The frame quality SEI message (refer to FIG. 18) is inserted as an SEI message. In addition, the frame quality descriptor (refer to FIG. 16) is inserted into the video elementary stream loop. Note that, in FIGS. 19 and 20, both of the frame quality SEI message and the frame quality descriptor are inserted, but it is also conceivable to insert only one of them.

(114) The flowchart in FIG. 21 illustrates an example of control processing for the mid-processing unit 205 by the CPU 201 in a case where the information on the ratio of the frame rate to the camera shutter speed is sent from the transmission device 100. This FIG. 21 illustrates members corresponding to those in FIG. 12 with the same reference numerals attached.

(115) When the high-definition video distribution is delivered in step ST1, the CPU 201 proceeds to processing in step ST6A. In this step ST6A, the CPU 201 detects the ratio of the frame rate to the camera shutter speed from the frame quality descriptor or the frame quality SEI message. After this step ST6A, the CPU 201 proceeds to processing in step ST7. Although detailed description is omitted, the other processing is the same as the processing in the flowchart in FIG. 12.

(116) In addition, the above embodiments have indicated the transmission/reception system 10 constituted by the transmission device 100 and the reception device 200. However, the configuration of the transmission/reception system to which the present technology can be applied is not limited thereto. For example, a transmission/reception system 10A as illustrated in FIG. 22 is also conceivable. In this transmission/reception system 10A, a portion for the reception device 200 in the transmission/reception system 10 in FIG. 1 is configured as a set top box 200A and a monitor 200B connected through a digital interface such as the high-definition multimedia interface (HDMI). Note that HDMI is a registered trademark.

(117) In this case, in the configuration of the reception device 200 illustrated in FIG. 9, for example, a portion up to the decoder 204 is included in the set top box 200A and the mid-processing unit 205 and the following members are included in the monitor 200B. In the monitor 200B, the camera shutter speed can be estimated on the basis of the information on the frame rate and the resolution inserted in the blanking period of the moving image data at the predetermined frame and the predetermined resolution transmitted from the set top box 200A and this estimated camera shutter speed can be used to find the ratio of the frame rate of the moving image data to the camera shutter speed.

(118) Note that a configuration in which the information on the ratio of the frame rate to the camera shutter speed is sent from the set top box 200A to the monitor 200B via an HDMI interface is also conceivable. This information on the ratio is inserted, for example, into the blanking period of the moving image data when sent. In this case, it becomes unnecessary for the monitor 200B to estimate the camera shutter speed on the basis of the information on the frame rate and the resolution to find the ratio.

(119) In addition, in this case, it is also conceivable to employ a configuration in which, in the configuration of the reception device 200 illustrated in FIG. 9, for example, a portion up to the mid-processing unit 205 is included in the set top box 200A and the post-processing unit 206 and the following members are included in the monitor 200B.

(120) Additionally, the above embodiments have indicated an example in which the transport stream (MPEG-2 TS) serves as the container. However, the present technology can be similarly applied to a system configured to perform distribution to receiving terminals using a network such as the Internet. In the distribution using the Internet, distribution is often performed in a container of MP4 or a format other than MP4. In other words, containers of various formats such as a transport stream (MPEG-2 TS) adopted in a digital broadcasting standard and MP4 used in Internet distribution fall within the containers.

(121) Note that the present technology can be also configured as described below.

(122) (1) An image processing device including:

(123) an image data acquiring unit that acquires moving image data at a predetermined frame rate and a predetermined resolution; and

(124) an image processing unit that performs filtering processing for raising the degree of correlation between adjacent frames on the acquired moving image data when a ratio of the predetermined frame rate to a camera shutter speed falls below a threshold value.

(125) (2) The image processing device according to (1) above, in which

(126) the image processing unit estimates the camera shutter speed on the basis of information on the frame rate and the resolution.

(127) (3) The image processing device according to (1) or (2) above, in which

(128) the image data acquiring unit receives a container in a predetermined format including a video stream obtained by applying encoding processing to the moving image data and acquires the moving image data by applying decoding processing to the video stream.

(129) (4) The image processing device according to (3) above, in which

(130) the image processing unit estimates the camera shutter speed on the basis of information on the frame rate and the resolution inserted in a layer of the container.

(131) (5) The image processing device according to (3) above, in which

(132) the image processing unit estimates the camera shutter speed on the basis of information on the frame rate and the resolution inserted in a layer of the video stream.

(133) (6) The image processing device according to (3) above, in which

(134) information on the ratio of the frame rate to the camera shutter speed is inserted in a layer of the container and/or a layer of the video stream, and

(135) the image processing unit obtains the ratio of the predetermined frame rate to the camera shutter speed on the basis of the information on the ratio inserted in the layer of the container and/or the layer of the video stream.

(136) (7) The image processing device according to (1) or (2) above, in which

(137) the image data acquiring unit acquires the moving image data from an external apparatus via a digital interface.

(138) (8) The image processing device according to (7) above, in which

(139) the image processing unit estimates the camera shutter speed on the basis of information on the frame rate and the resolution inserted in a blanking period of the moving image data.

(140) (9) The image processing device according to (7) above, in which

(141) the image processing unit acquires information on the ratio of the frame rate to the camera shutter speed from the external apparatus via the digital interface and, on the basis of the information on the ratio, obtains the ratio of the predetermined frame rate to the camera shutter speed.

(142) (10) The image processing device according to any one of (1) to (9) above, in which

(143) the image data acquiring unit receives a container in a predetermined format including a video stream obtained by applying encoding processing to the moving image data and acquires the moving image data by applying decoding processing to the video stream, and

(144) in a case where information indicating whether super high-definition video distribution is delivered, which is inserted in a layer of the container, indicates that the super high-definition video distribution is delivered, the image processing unit performs the filtering processing for raising the degree of correlation between adjacent frames on the acquired moving image data when the ratio of the predetermined frame rate to the camera shutter speed falls below the threshold value.

(145) (11) An image processing method including:

(146) an image data acquiring step of acquiring moving image data at a predetermined frame rate and a predetermined resolution; and

(147) an image processing step of performing filtering processing for raising the degree of correlation between adjacent frames on the acquired moving image data when a ratio of the predetermined frame rate to a camera shutter speed falls below a threshold value.

(148) (12) A reception device including:

(149) a receiving unit that receives a container in a predetermined format including a video stream obtained by applying encoding processing to moving image data;

(150) a decoding unit that applies decoding processing to the video stream to obtain moving image data at a predetermined frame rate and a predetermined resolution;

(151) interpolation processing unit that adjusts the frame rate of the moving image data obtained by the decoding unit to a display capability by generating an interpolation frame using an interframe motion vector; and

(152) an image processing unit that is interposed between the decoding unit and the interpolation processing unit and performs filtering processing for raising the degree of correlation between adjacent frames on the moving image data obtained by the decoding unit when a ratio of the predetermined frame rate to a camera shutter speed falls below a threshold value.

(153) (13) A reception device including:

(154) a receiving unit that receives moving image data at a predetermined frame rate and a predetermined resolution from an external apparatus via a digital interface;

(155) interpolation processing unit that adjusts the frame rate of the moving image data received by the receiving unit to a display capability by generating an interpolation frame using an interframe motion vector; and

(156) an image processing unit that is interposed between the receiving unit and the interpolation processing unit and performs filtering processing for raising the degree of correlation between adjacent frames on the received moving image data when a ratio of the predetermined frame rate to a camera shutter speed falls below a threshold value.

(157) (14) A transmission device including:

(158) an image encoding unit that generates a video stream by applying encoding processing to moving image data;

(159) a transmitting unit that transmits a container in a predetermined format including the video stream; and

(160) an information inserting unit that inserts, into a layer of the container, information on a frame rate and a resolution corresponding to information on a frame rate and a resolution inserted in a layer of the video stream.

(161) (15) The transmission device according to (14) above, in which

(162) the information inserting unit further inserts information indicating whether super high-definition video distribution is delivered into the layer of the container.

(163) (16) A transmission device including:

(164) an image encoding unit that generates a video stream by applying encoding processing to moving image data;

(165) a transmitting unit that transmits a container in a predetermined format including the video stream; and

(166) an information inserting unit that inserts information indicating whether super high-definition video distribution is delivered into a layer of the container.

(167) (17) A transmission device including:

(168) an image encoding unit that generates a video stream by applying encoding processing to moving image data;

(169) a transmitting unit that transmits a container in a predetermined format including the video stream; and

(170) an information inserting unit that inserts information on a ratio of a frame rate to a camera shutter speed into a layer of the container and a layer of the video stream.

(171) (18) The transmission device according to (17) above, in which

(172) the information inserting unit further inserts information indicating whether super high-definition video distribution is delivered into the layer of the container and the video stream.

(173) The main feature of the present technology is that, by performing the filtering processing for raising the degree of correlation between adjacent frames on the moving image data when the ratio of the frame rate thereof to the camera shutter speed falls below the threshold value, it is made possible to perform frame interpolation satisfactorily with a conventional frame interpolation technology (refer to FIGS. 9 to 12).

REFERENCE SIGNS LIST

(174) 10, 10A Transmission/reception system 100 Transmission device 101 CPU 102 Encoder 103 Multiplexer 104 Transmitting unit 200 Reception device 200A Set top box 200B Monitor 201 CPU 202 Receiving unit 203 Demultiplexer 204 Decoder 205 Mid-processing unit 206 Post-processing unit 207 Display unit