METHOD FOR DETERMINING A NOTEWORTHY SUB-SEQUENCE OF A MONITORING IMAGE SEQUENCE

Abstract

The invention relates to a method for determining a noteworthy sub-sequence (114a) of a monitoring image sequence (110) of a monitoring area comprising the following steps: providing an audio signal (S1) from the monitoring area, at least partially including a time period of the monitoring image sequence; providing the monitoring image sequence (S1) of the environment to be monitored, which has been generated by an imaging system; determining at least one segment of the audio signal from the provided audio signal, which has unusual noises (S2); determining at least one segment of the monitoring image sequence having unusual movements within the environment to be monitored (S3); determining a correlation between the at least one segment of the audio signal having unusual noises (114a) and the at least one segment of the monitoring image sequence with unusual movements (114a) in order to determine a noteworthy sub-sequence (114) of the monitoring image sequence (110).

Claims

1-15. (canceled)

16. A method for determining a noteworthy sub-sequence of a monitoring image sequence of a monitoring area, comprising the steps: providing an audio signal from the monitoring area, which at least partially includes a time period of the monitoring image sequence; providing the monitoring image sequence of an environment to be monitored, which has been generated by an imaging system; determining at least one segment of the audio signal from the provided audio signal, which has unusual noises; determining at least one segment of the monitoring image sequence having unusual movements within the environment to be monitored; and determining a correlation between the at least one segment of the audio signal having unusual noises and the at least one segment of the monitoring image sequence having unusual movement to determine the noteworthy sub-sequence of the monitoring image sequence.

17. The method according to claim 16, wherein the at least one noteworthy sub-sequence of the monitoring image sequence is determined by subtracting from the monitoring image sequence at least one sub-sequence in which an expression of the correlation between the at least one segment of the monitoring image sequence having unusual movements and the at least one segment of the audio signal having unusual noises below a limit value is determined.

18. The method according to claim 16, wherein the at least one segment of the audio signal having unusual noises is determined by identifying frequency bands of human voices with respect to unusual amplitudes and/or unusual frequencies in the audio signals.

19. The method according to claim 16, wherein a source location of the provided audio signal is detected and the unusual noises are determined based on the source location.

20. The method according to claim 16, wherein images of the monitoring image sequence are compressed and unusual movements in the monitoring area are determined using the monitoring image sequence based on a change in the amount of effort required to compress successive images of the monitoring image sequence.

21. The method according to claim 16, wherein, for determining unusual movement in the monitoring area, at least one optical flow of images of the monitoring image sequence is determined and unusual movements are determined using the images based on the determined optical flow.

22. The method according to claim 16, wherein characteristic points of persons in the monitoring area are determined, and unusual movements are determined based on a change in the characteristic points within the monitoring image sequence.

23. The method according to claim 22, wherein the characteristic points of persons in the monitoring area are determined using a neural network trained to determine characteristic points.

24. The method according to claim 16, wherein the correlation between the at least one segment of the audio signal having unusual noises and the at least one segment of the monitoring image sequence having unusual movements is determined using a neural network trained to determine a correlation.

25. The method according to claim 24, wherein the neural network trained to determine the correlation is configured to determine the at least one segment of the audio signal that includes unusual noises and/or the at least one segment of the monitoring image sequence having unusual movements.

26. The method according to claim 16, wherein, based on the noteworthy sub-sequence of the monitoring image sequence of the monitoring area, a control signal for controlling an at least partially automated vehicle is provided, and/or, based on the noteworthy sub-sequence, a warning signal for warning a vehicle occupant is provided.

27. A method for training the neural network to determine characteristic points of persons in a monitoring area, with a plurality of training cycles, wherein each of the training cycles comprises the following steps: providing a reference image, wherein characteristic points of persons are labeled in the reference image, and adapting the neural network to determine the characteristic points in order to minimize a deviation from the labeled characteristic points of the respective associated reference image when determining the characteristic points of the persons with the neural network.

28. A monitoring device configured to determine a noteworthy sub-sequence of a monitoring image sequence of a monitoring area, the monitoring device configured to: provide an audio signal from the monitoring area, which at least partially includes a time period of the monitoring image sequence; provide the monitoring image sequence of an environment to be monitored, which has been generated by an imaging system; determine at least one segment of the audio signal from the provided audio signal, which has unusual noises; determine at least one segment of the monitoring image sequence having unusual movements within the environment to be monitored; and determine a correlation between the at least one segment of the audio signal having unusual noises and the at least one segment of the monitoring image sequence having unusual movement to determine the noteworthy sub-sequence of the monitoring image sequence.

29. The method according to claim 29, wherein the monitoring image sequence is provided using an imaging system.

30. A non-transitory computer-readable medium on which is stored a computer program including instructions for determining a noteworthy sub-sequence of a monitoring image sequence of a monitoring area, the instructions, when executed by a computer, causing the computer to perform the following steps: providing an audio signal from the monitoring area, which at least partially includes a time period of the monitoring image sequence; providing the monitoring image sequence of an environment to be monitored, which has been generated by an imaging system; determining at least one segment of the audio signal from the provided audio signal, which has unusual noises; determining at least one segment of the monitoring image sequence having unusual movements within the environment to be monitored; and determining a correlation between the at least one segment of the audio signal having unusual noises and the at least one segment of the monitoring image sequence having unusual movement to determine the noteworthy sub-sequence of the monitoring image sequence.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0085] FIG. 1 shows a schema of the method for determining a noteworthy sub-sequence of a monitoring image sequence, according to an example embodiment of the present invention.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

[0086] FIG. 1 schematically outlines the method 100 for determining a noteworthy sub-sequence 114 a of a monitoring image sequence 110 of a monitoring area.

[0087] The audio signal 120 and the monitoring image sequence 110 from the monitoring area is provided S1, wherein the monitoring image sequence 110 is generated by an imaging system.

[0088] The method 100 is used to determine at least one segment 114a of the audio signal 130 from the provided audio signal 130 S2 that comprises unusual noises, wherein the at least one segment 114a of the audio signal 130 having unusual noises is determined here by identifying frequency bands of human voices with respect to an unusually high amplitude.

[0089] The method is also used to determine movements 140, for example of objects, within the monitoring image sequence 110 and, by means of the movement 140, determine a segment 114a of the monitoring image sequence having unusual movements within the environment to be monitored S3.

[0090] As can be seen from FIG. 1, the audio signal 130 and the movement signal 140 in segment 114 a correlate with one another and thus determine a noteworthy sub-sequence of the monitoring image sequence.

[0091] The segment of the audio signal that comprises unusual noises and/or the segment of the monitoring image sequence having unusual movements can be determined using a neural network trained to make such a determination.

[0092] Alternatively or additionally, the at least one noteworthy sub-sequence 114a of the monitoring image sequence 110 can be determined by subtracting at least one sub-sequence 112 a from the monitoring image sequence 110 in which an expression of the correlation between the at least one segment 112a of the monitoring image sequence 110 having unusual movements and the at least one segment 112a of the audio signal 130 having unusual noises below a limit value is determined.

[0093] A plurality of noteworthy sub-sequences 114a can thus be determined in the monitoring image sequence 110 S4. Alternatively, a plurality of sub-sequences 112a in which the expression of the correlation is determined below a limit value, as described above, can be determined to determine the monitoring image sequence 110. Then, in a step S5, the plurality of sub-sequences 114 of the monitoring image sequence 110 determined to be noteworthy can be uploaded, for example wirelessly, from a vehicle to a cloud.