Method and apparatus for visualizing information of a digital video stream
10878850 ยท 2020-12-29
Assignee
Inventors
Cpc classification
G11B27/28
PHYSICS
G11B27/005
PHYSICS
International classification
G11B27/00
PHYSICS
H04N7/18
ELECTRICITY
G11B31/00
PHYSICS
Abstract
The present invention relates to a method and apparatus for visualizing information of a digital video stream is disclosed. The apparatus comprises: a memory arranged to store a series of merged image frames which is formed by merging groups of image frames of the digital video stream; a blending module arranged to blend multiple merged frames into a series of blended image frames, wherein each of a plurality of the merged image frames contributes to a plurality of the blended image frames; and a display arranged to display the series of blended image frames.
Claims
1. A method for visualizing information of a digital video stream, the method comprising: merging groups of original image frames of the digital video stream into a series of merged image frames, wherein each merged image frame is generated from a different group of original image frames, and wherein each group of original frames includes no frames of any other group of original frames; storing the series of merged image frames; blending multiple series of merged image frames into a series of blended image frames, wherein each of the merged image frames contributes to a plurality of the blended image frames, and wherein a merged image frame contributing weight is set to be constant for each of the blended image frames to which the merged image frame contributes; and displaying, on a display, the series of blended image frames.
2. The method according to claim 1, further comprising setting number of image frames contributing to each merged frame.
3. The method according to claim 1, further comprising setting number of merged image frames contributing to each blended image frame.
4. The method according to claim 1, wherein the displaying of the series of blended image frames is performed by displaying the blended image frames consecutively as a video stream of blended image frames.
5. The method according to claim 1, wherein the displaying of the series of blended image frames is performed by displaying the blended image frames side by side on the display.
6. The method according to claim 1, further comprising reducing the resolution of the image frames before merging groups of image frames from the video stream into the series of merged image frames.
7. The method according to claim 1, further comprising reducing the resolution of the image frames upon merging groups of image frames from the video stream into the series of merged image frames.
8. The method according to claim 1, further comprising detecting an event of interest in the digital video stream and setting an image frame contributing weight for an image frame of the digital video stream comprising the event differently than an image frame contributing weight for an image frame of the digital video stream not comprising the event.
9. The method according to claim 1, further comprising detecting an event of interest in the digital video stream and setting a merged image frame contributing weight for a merged image frame comprising the event differently than a merged image frame contributing weight for a merged image frame not comprising the event.
10. An apparatus for visualizing information of a digital video stream, the apparatus comprising: a merging module implemented as software code run by a processing unit or dedicated hardware, wherein the merging module is arranged to merge groups of original image frames from the digital video stream into a series of merged image frames, wherein each merged image frame is generated from a different group of original image frames, and wherein each group of original frames includes no frames of any other group of original frames; a memory arranged to store the series of merged image frames; a blending module implemented as software code run by a processing unit or dedicated hardware, wherein the blending module is arranged to blend multiple series of merged image frames into a series of blended image frames, wherein each of the merged image frames contributes to a plurality of the blended image frames, and wherein a merged image frame contributing weight is set to be constant for each of the blended image frames to which the merged image frame contributes; and a display arranged to display the series of blended image frames.
11. A non-transitory computer readable recording medium configured to store a set of instructions that, when executed on a device having processor capabilities, cause the device to perform a method for visualizing information of a digital video stream, the method comprising: merging groups of original image frames of the digital video stream into a series of merged image frames, wherein each merged image frame is generated from a different group of original image frames, and wherein each group of original frames includes no frames of any other group of original frames; storing the series of merged image frames; blending multiple series of merged image frames into a series of blended image frames, wherein each of the merged image frames contributes to a plurality of the blended image frames, and wherein a merged image frame contributing weight is set to be constant for each of the blended image frames to which the merged image frame contributes; and displaying, on a display, the series of blended image frames.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
(1) The above and other aspects of the present invention will now be described in more detail, with reference to appended drawings showing embodiments of the invention. The figures should not be considered limiting the invention to the specific embodiment; instead they are used for explaining and understanding the invention.
(2) As illustrated in the figures, the sizes of layers and regions are exaggerated for illustrative purposes and, thus, are provided to illustrate the general structures of embodiments of the present invention. Like reference numerals refer to like elements throughout.
(3)
(4)
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
(5) The present invention will now be described more fully hereinafter with reference to the accompanying drawings, in which currently preferred embodiments of the invention are shown. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided for thoroughness and completeness, and to fully convey the scope of the invention to the skilled person.
(6)
(7) The apparatus 1 is arranged to produce or receive video data pertaining to the digital video stream. In case the apparatus 1 is arranged to produce video data pertaining to the digital video stream the apparatus 1 comprises an image recording module (not shown). The image recording module may e.g. be a module comprising an image sensor and an image processor arranged to process image data captured by the image sensor. In case the apparatus 1 is arranged to receive video data pertaining to the digital video stream the apparatus 1 comprises an input port (not shown) via which the video data pertaining to the digital video stream is received. The input port may be a digital network port or any other port being suitable for receiving the video data pertaining to the digital video stream. Alternatively, the input port may be a memory port arranged to receive an exchangeable memory such as a USB memory stick or the like.
(8) The merging module 10 is arranged to merge groups of image frames from the digital video stream into a series of merged image frames. This may be visualized by the following example. Consider a digital video stream comprising the following image frames:
(9) A B C D E F G H I J K L M N O P Q R S T U V W X
(10) The merging module 10 may then be arranged to produce a series of merged image frames, MF, according to the following:
(11) MF.sub.1= A+ B+ C
(12) MF.sup.2= D+ E+ F
(13) MF.sub.3= G+ H+ I
(14) MF.sub.4= J+ K+ L
(15) MF.sub.5= M+ N+ O
(16) MF.sub.6= P+ Q+ R
(17) MF.sub.7= S+ T+ U
(18) MF.sub.8= V+ W+ X
(19) According to this example each merged image frame comprises image data from three different original image frames of the digital video stream. However, it shall be understood that other possibilities are also feasible as long as the series of merged image frames is formed by merging groups of image frames of the digital video stream. The series of merged image frames is formed such that the number of merged image frame is less than the number of original image frames. This may for example be made by merging the groups of original image frames such that each original image frame only contribute to a single merged image frame. Upon playback of the series of merged image frames using the same or a similar frame rate as for the digital video stream a fast forward playback of the digital video stream is achieved. This occurs or happens since the number of image frames in the series of merged image frames is reduced as compared to the number of images frames of the digital video stream. The merging module 10 may be arranged so that number of image frames contributing to each merged frame may be set. Number of image frames contributing to each merged frame may be set by a user of the apparatus 1 by interaction by, for example, a keyboard, a mouse, a trackball, a microphone, or any other suitable input unit being connected to the apparatus 1. The number of image frames contributing to each merged image frame may be set to be the same for each merged image frame. Alternatively, the number of image frames contributing to each merged image frame may be set to differ between the merged image frames. The number of image frames contributing to each merged image frame may be set to vary dynamically between the merged image frames. This dynamical change in the number of image frames contributing to each merged image frame may e.g. be controlled by the level of motion in the scene or being triggered by a specific type of event occurring in the scene. In case of a high level of motion in the scene the number of image frames contributing to each merged image frame is set to be low, and in case of a low level of motion in the scene number of image frames contributing to each merged image frame is set to be higher; or vice versa. By changing the number of image frames contributing to each merged image frame the frame rate of the merged frames may be not constant. The not constant frame rate of the merged frames may be compensated for when performing the blending discussed below. Alternatively, the not constant frame rate of the merged frames may not be compensated for and hence the play back speed of the resulting series of blended image frames may vary.
(20) The image data of the original image frames of the digital video stream may be added using various merging schemes, such as Alpha compositing, multiply blend and screen blend. The person skilled in the art also realizes that other merging schemes are also possible to use.
(21) An image frame contributing weight for the contribution from the different image frames to each merged image frame may also be set in the merging module 10. The image frame contributing weight may be set by a user of the apparatus 1 by interaction by, for example, a keyboard, a mouse, a trackball, a microphone, or any other suitable input unit being connected to the apparatus 1. The image frame contributing weight may, as in the example above, be set to be equal for all image frames contributing to a specific merged image frame. Alternatively, the image frame contributing weight may be set to be different for the image frames contributing to a specific merged image frame. The image frame contributing weight may vary dynamically between the merged image frames. This dynamical change of the image frame contributing weight may e.g. be controlled by detecting an event occurring in the image frames. An image frame might get a higher image frame contributing weight if it is considered more interesting for some reason. Image frames being considered more interesting may be found by video analytics used for finding events of interest in the digital video stream. For example, a face detection algorithm could for example indicate one or more image frames where a face is more visible than in others. Other kind of video analytics may also be used in order to find events of interest in the digital video stream.
(22) The memory 12 is arranged to receive and store the series of merged image frames. The memory 12 may be any kind of volatile or non-volatile memory suitable for storing digital data. The series of merged image frames is digital data.
(23) The blending module 14 is arranged to blend multiple merged frames of the series of merged image frames into a series of blended image frames, BF, wherein each of a plurality of the merged image frames contributes to a plurality of the blended image frames. Hence, the series of blended image frames is formed by blending groups of merged image frames. This may be visualized by the following example:
(24) BF.sub.1= MF1+ MF2+ MF3
(25) BF.sub.2= MF2+ MF3+ MF4
(26) BF.sub.3= MF3+ MF4+ MF5
(27) BF.sub.4= MF4+ MF5+ MF6
(28) BF.sub.5= MF5+ MF6+ MF7
(29) BF.sub.6= MF6+ MF7+ MF8
(30) According to this example each blended image frame comprises image data from three different merged image frames of the series of merged image frames. However, it shall be understood that other possibilities are also feasible as long as the series of blended image frames is formed by blending groups of merged image frames of the series of merged image frames. Moreover, at least some of the merged image frames of the series of merged image frames contribute to more than one blended image frame. Accordingly, each of a plurality of the merged image frames contributes to a plurality of the blended image frames. Depending on how the merged image frames are blended into the series of blended image frames the number of blended image frames in the series of blended image frames may be reduced as compared to the number of merged images frames of the series of merged image frames. Hence, upon playback of the series of blended image frames an enhanced playback speed of the digital video stream may be achieved as compared with the playback of the series of merged image frames.
(31) The blending module 14 may be arranged so that number of merged image frames contributing to each blended image frame may be set. Number of merged image frames contributing to each blended image frame may be set by a user of the apparatus 1 by interaction by, for example, a keyboard, a mouse, a trackball, a microphone, or any other suitable input unit being connected to the apparatus 1. The number of merged image frames contributing to each blended images frame may be set to be the same for each blended image frame. Alternatively, the number of merged image frames contributing to each blended image frame may be set to differ between the blended image frames. The number of merged image frames contributing to each blended image frame may be set to vary dynamically between the blended image frames. The number of merged image frames contributing to each blended image frame may be controlled in order to compensate for the change of frame rate arising from having a dynamically set number of image frames contributing to the merged image frames. The number of merged image frames contributing to each blended image frame may also, or in combination, be increased to include a merged image frame comprising an event of particular interest in more blended image frames. Merged image frames comprising an event of particular interest may be found by video analytics used for finding events of interest in the digital video stream. For example, a face detection algorithm could for example indicate one or more image frames where a face is more visible than in others. Other kind of video analytics may also be used in order to find events of interest in the digital video stream.
(32) The image data of the merged image frames of the series of merged image frames may be added using various blending schemes, such as Alpha compositing, multiply blend and screen blend. The person skilled in the art also realizes that other blending schemes are also possible to use.
(33) A merged image frame contributing weight for the contribution from the different merged image frames to each blended image frame may also be set in the blending module 14. The merged image frame contributing may be set by a user of the apparatus 1 by interaction by, for example, a keyboard, a mouse, a trackball, a microphone, or any other suitable input unit being connected to the apparatus 1. The merged image frame contributing weight may, as in the example above, be set to be equal for all merged image frames contributing to a specific blended image frame. Alternatively, the merged image frame contributing weight may be set to be different for the merged image frames contributing to a specific blended image frame. This is illustrated in the following example:
(34) BF.sub.4= MF1+ MF2+ MF3
(35) BF.sub.2= MF2+ MF3+ MF4
(36) BF.sub.3= MF3+ MF4+ MF5
(37) BF.sub.4= MF4+ MF5+ MF6
(38) BF.sub.5= MF5+ MF6+ MF7
(39) BF.sub.6= MF6+ MF7+ MF8
(40) In the above example the merged image frame are blended into the blended image frames in a fading manner. That is, the contribution of a merged image frame is faded in and/or out of the series of blended image frames.
(41) Alternatively or in combination, the merged image frame contributing weight may be set to vary dynamically between the blended image frames. Merged image frames marked as being of interest may have a higher merged image frame contributing weight than non-marked merged image frames. The marking of the merged image frames may be triggered by video analytics events finding events of interest in the digital video stream.
(42) The display 16 is arranged to display the series of blended image frames. The displaying of the series of blended image frames may be performed by displaying the blended image frames consecutively as a video stream of blended image frames. Alternatively, the displaying of the series of blended image frames may be performed by displaying the blended image frames side by side on the display.
(43) The apparatus 1 may also comprise a resolution setting module 18. The resolution setting module 18 may be arranged to set the resolution of the image frames of the digital video stream. The resolution setting module 18 may be arranged to reduce the resolution of at least some of the image frames before the merging module 10 is arranged to merge groups of image frames from the video stream into the series of merged image frames. Alternatively, the resolution setting module 18 and the merging module 10 may be arranged to cooperate so that the resolution of the image frames is reduced upon merging groups of image frames from the video stream into the series of merged image frames. The apparatus 1 may also comprise an event detection module 20. The event detection module 20 is arranged to detect event(s) within the digital video stream containing information of interest. The event may be an object moving within the scene depicted by the digital video stream, e.g. a person, car, animal or another object moving within the scene. Alternatively or in combination, the event may be an object of a specific type being present in the scene depicted by the digital video stream, e.g. a face of a person, a license plate of a car or another object of a specific type being present in the scene.
(44) The information pertaining to the detected event(s) may be used for controlling the image frame contributing weight and/or the merged image frame contributing weight discussed above. Original images frames and/or merged image frames comprising event(s) detected by the event detection module 20 are given higher weights that original images frames and/or merged image frames not comprising event(s) detected by the event detection module 20.
(45) Moreover, the information pertaining to the detected event(s) within the digital video stream may be used by the merging module 10 in order to weight areas of the original image frames comprising the detected event(s) differently than the areas of the original image frame pertaining to a background of the scene. Alternatively or in combination, the information pertaining to the detected event(s) within the digital video stream may be used by the blending module 14 in order to weight areas of the merged image frames comprising the detected event(s) differently than the areas of the merged image frame pertaining to the background of the scene. The background of the scene may be the areas of the original image frames and/or merged image frames not comprising the detected event(s).
(46) The merging module 10, the blending module 14, the image resolution setting module 18 and/or the event detection module 20 of the apparatus 1 may be implemented as software code run by a processing unit, as hardware implemented in the apparatus 1 or as a combination thereof. The apparatus 1 may also comprise an additional memory (not shown). The merging module 10, the blending module 14 and/or the image resolution setting module 18 may be arranged to store and/or retrieve data from/on the additional memory. The additional memory may be any kind of volatile or non-volatile memory suitable for storing digital data.
(47) In
(48) The person skilled in the art realizes that the present invention by no means is limited to the preferred embodiments described above. On the contrary, many modifications and variations are possible within the scope of the appended claims.
(49) For example, the apparatus 1 may instead of comprising the merging module 10 be arranged to receive a series of merged frames from another apparatus. This other apparatus may e.g. be a digital video camera comprising a merging module arranged according to the merging module as discussed above in connection with
(50) The display 16 of the apparatus 1 may form an integral part of the apparatus 1 as shown in
(51) Additionally, variations to the disclosed embodiments can be understood and effected by the skilled person in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims.