Method and device for acquiring stream of the precisely time-stamped images

Abstract

Method and device for acquiring stream of the precisely time-stamped images, including the modulated light source, controlled by the absolute global real-time-base (e.g. provided by global navigation satellite system (GNSS) controller) and the image acquisition and processing unit, decoding the light modulation waveform and determining the time-stamp for every image frame. The GNSS time and position messages can be used to provide the full time- and location stamps for each frame. Multi-element light sources can be used to have more informative light modulation in the time domain. For time-stamping of several image streams (e.g. from several cameras) multiple light sources with the same modulation can be used.

Claims

1. A method for acquiring at least one stream of precisely time-stamped image frames, comprising: receiving, by a light modulation controller, a real-time absolute time base signal from an absolute time source; providing, by the light modulation controller, at least one modulation signal based on the real-time absolute time base signal; and modulating, by the light modulation controller using the at least one modulation signal, at least one light source to generate at least one light modulated signal, so that an image acquisition and processing system comprising at least one camera and an image processor is configured to acquire, by the camera, at least one stream of image frames and the at least one light modulated signal, and to generate, by the image processor using the acquired signals, the at least one stream of precisely time-stamped image frames from the at least one stream of image frames.

2. The method of claim 1, wherein, the absolute time source is provided using satellite communications.

3. The method of claim 1, wherein the absolute time source is provided by a global navigation satellite system (GNSS).

4. The method of claim 1, wherein generating the at least one stream of precisely time-stamped image frames comprises calculating an absolute time stamp for each image frame of the at least one stream of image frames using a modulated light intensity in the at least one light modulated signal by decoding a time-domain waveform of variation of the modulated light intensity through the at least one stream of image frames, so that each of the precise time-stamped image frames comprises the calculated absolute time stamp.

5. The method of claim 1, wherein the real-time absolute time base signal is a pulse-per-time-unit signal from a global navigation satellite system (GNSS) receiver/controller.

6. The method of claim 5, wherein each of the precise time-stamped image frames comprises a time-message provided to the image processor directly by the GNSS receiver/controller.

7. The method of claim 5, wherein each of the precise time-stamped image frames comprises a location information provided to the image processor directly by the GNSS receiver/controller.

8. The method according to claim 5, wherein the at least one light modulated signal comprises a binary waveform, controlled by the pulse-per-time unit signal from the GNSS receiver/controller.

9. The method according to claim 5, wherein the at least one light modulated signal comprises a continuously changing time-domain waveform, controlled by the pulse-per-time unit signal from the GNSS receiver/controller.

10. The method of claim 1, wherein the at least one light source comprises one or more light emitting diodes (LEDs).

11. The method of claim 1, wherein the at least one light source contains a plurality of light emitting elements, each modulated by a different unique time-domain waveform.

12. The method of claim 1, wherein the at least one stream of image frames comprises a plurality of streams of image frames, the at least one light source comprises a plurality of light emitting elements, each modulated by an identical time-domain waveform, so that at least one light modulated signal comprises a plurality of light modulated signals from the plurality of light emitting elements, and the at least one camera comprises a plurality of cameras, where each camera of the plurality of cameras is configured to acquire one of the plurality of streams of image frames and one of the of light modulated signals from the corresponding light emitting element of the a plurality of light emitting elements, so that the image processor is configured to provide precisely time-stamped image frames for each of the plurality of streams of image frames.

13. The method of claim 1, wherein a time-domain waveform of the modulated light intensity, after being demodulated by the image processor, is compared with a reference waveform for continuous adjustment to provide a substantially expected timing of acquisition of the image frames.

14. An apparatus for providing at least one stream of precisely time-stamped image frames, comprising: an absolute time source configured to provide a real-time absolute time base signal; a light modulation controller configured, in response to the real-time absolute time base signal from the absolute time source, to provide at least one modulation signal; at least one light source, configured to generate at least one light modulated signal in response to the at least one modulation signal; and an image acquisition and processing system comprising at least one camera and an image processor, configured to acquire, by the camera, at least one stream of image frames and the at least one light modulated signal, and to generate, by the image processor using the acquired signals, the at least one stream of precisely time-stamped image frames from the at least one stream of image frames.

15. The apparatus of claim 14, wherein the absolute time source is provided using satellite communications.

16. The apparatus of claim 14, wherein the absolute time source is provided by a global navigation satellite system (GNSS).

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) FIG. 1 describes the essence of the invention in the form of the block diagram.

(2) FIG. 2 through FIG. 7 shows the various aspects of the implementation examples of the invention.

(3) FIG. 8 gives some possible waveforms of the light modulation.

DETAILED DESCRIPTION OF THE INVENTION

(4) FIG. 1 shows a solution containing the connection 1 to the absolute time source (e.g., the GNSS satellites or other precise time source), a light modulator on the base of the absolute time 2, input of which is connected to the connection 1 and the output of which is connected to the light source 3 (for example, the LED light), image acquisition and processing unit 4 and the captures scene 5, wherein the image acquisition and processing unit 4 identifies the modulated light from a light source 3 and monitors the dynamic scene 5.

(5) As the light modulation is related to the absolute time, it is possible from the image frame sequences to decode the modulation of this light intensity modulation and, thus the image frames of the scene can be associated with the real and absolute time.

(6) FIG. 2 shows that the light intensity modulator based on absolute time may comprise a GNSS (e.g., GPS) receiver 21, and the light modulation controller 22, wherein pulse per unit of time (for example, of pulse per second) output of the GNSS receiver is connected to the input of the light intensity modulation controller 22, in such a way that it generates at every new absolute time unit of (for example, every second), the unique modulation pattern.

(7) In addition, as shown in FIG. 2, the image acquisition and processing unit 4 may include a camera 41 and a computer 42.

(8) It is also shown in FIG. 2, that in addition the GNSS receiver may sent to the computer the full time and/or location information by the respective (e.g., in NMEA format) messages. Thus, it is possible to computer to combine for a particular image frame the approximate time of this message (for typical GNSS receivers, the error is substantially less than one second) and accurate (typically better than 1 microsecond) pulse per unit time, which passes through the route of light modulation from the controller 22 through the light source 3 and the camera 41, and is decoded by the computer (or dedicated image processor) 42. It is also possible by similar messages (such as NMEA format) to add location information to each frame (e.g. as GNSS coordinates). If necessary, the precise time and location values could be interpolated from the decoded waveform of the modulated light intensity.

(9) FIG. 3 shows that the computer 42 may control the frame rate of the camera 41 at the image acquisition. The frame rate could be automatically adjusted so, that into the time unit (for example, one second) a integer number of frames fits. In such case, it is computationally easy for the GNSS receiver to perform the corresponding interpolation of the time and location of the messages for the particular image frame.

(10) FIG. 4 to FIG. 6 show the variants for the implementation of the light source 3, where the light source is consisting of multiple elements 31, 32, . . . 3N), modulation of each is controlled individually. For the implementation example according to FIG. 4, the light source comprises three identical elements (dots or dashes), and status of each lighting element (switched on or off) is found from the images. So, it is possible to modulate the light in a multi-bit way, for example, to detect not only a change of the 1 second time, but a smaller changes per unit time can be indicated, so allowing faster and more accurate determining of the frequency of the frames. Reasonable herein may be to use instead of the combinatorial N-bit codethe Gray or a thermometer code (such as variable length or height of the LED strip).

(11) FIG. 5 shows a three-color light source, by using of the RGB LED elements. This allows to use the color information for multi-bit light modulation.

(12) FIG. 6 shows multi-segment (for example, 7-element) indicator (s) being used as the light source 3. In this case, the immediate absolute time or a part of it, for example, of milliseconds of it, can be directly displayed onto the image, and the computer can by corresponding image processing decode the absolute time value (from the on-off states of these segments and by the corresponding decimal number notation by segments).

(13) FIG. 7 shows a solution, in which the image processing system has multiple (two or more) cameras (411 . . . 41N) and one or many (two or more) of light source elements (31 . . . 3N), wherein the light elements are electrically connected together (in series or in parallel) and so modulated by the same waveform. In this case, each camera image stream is modulated with the same pattern of light and it is possible from the images frames from parallel camera streams to provide the absolute (global) time(and the location) stamps.

(14) Also beneficial could be the solution (not shown on the figures) for which the various light elements (e.g., LEDs 31, . . . , 3N) are connected to a plurality of light modulators, based on the absolute time, e.g. controlled by pulse-per-second signal, as all the light modulation controllers 22 are linked (synchronized), the absolute global time (of, e.g. GNSS system) with very high precision. In such case all individual elements of modulated light are exactly the same, and it is possible to generate accurate global time stamps, from the decoded images of the light modulation. So, the image frames from various camera streams can be compared with each other in time, which is necessary, for example, for stereoscopic (3D) or other applications using various cameras simultaneously.

(15) FIG. 8 shows possible and reasonable waveforms for the light intensity modulation: 61arectangular (binary) waveform with the period corresponding to the GNSS receiver pulse-unit time period of the signal, for example, 1, or 0.1 seconds; 61ba saw-tooth-shaped waveform, also with the pulse per unit time period, enabling by continuously changing modulation waveform to achieve better time-resolution within this period; 61Cpseudo-random binary signal with image frame capturing moments 62, synchronized (triggered) by the pulse per unit signal of the GNSS receiver 21. The pseudo-random modulation with known waveform makes it possible to distinguish the different periods, as well for the specific cases being better to be decoded at various disturbances in the background.

(16) The invention is characterized by the following clauses:

(17) 1. Method for acquiring stream of the precisely time-stamped images, including the time-domain modulated light intensity together with image acquisition and processing, in which the variation of the modulated light intensity of the sequence of the image frames is decoded, characterized that the light modulation is directly related to the real-time absolute time-base and that the absolute time-stamp for every frame is calculated from the time-domain waveform of the variation of the decoded from the sequence of the modulated image frames light intensity waveform

(18) 2. Method according to clause 1, characterized that the absolute real-time time base is directly using the received global navigation satellite system (GNSS) signal.

(19) 3. Method according to clause 2, characterized that the light modulation is directly controlled by the pulse-per-time-unit signal generated by the received GNSS signal.

(20) 4. Method according to clause 3, characterized that the full time-stamp of the image frame is combined together from the outputted by the GNSS solution time-message and decoded modulation time domain waveform of the light intensity

(21) 5. Method according to clause 4, characterized that the full time-stamp includes also the location information, derived from the GNSS messages

(22) 6. Method according to clause 4, characterized that the light intensity modulation has a binary waveform, controlled directly by the pulse-per-time unit signal of the GNSS solution.

(23) 7. Method according to clause 4, characterized that the light modulation has a continuously changing time-domain waveform, controlled directly by the pulse-per-time-unit signal of the GNSS solution.

(24) 8. Method according to clause 1, characterized that the modulated light source contains several light emitting elements, each modulated by the different unique time-domain waveform.

(25) 9. Method according to clause 1, for time-stamping of frames of the several simultaneous continuous image streams, characterized that the light intensity modulation is performed by several light sources, all modulated by the same waveform.

(26) 10. Method according to clause 1, characterized that the demodulated light intensity waveform is compared to the expected (reference) waveform in the time-domain and the frame-rate of the image acquisition is continuously adjusted to have approximately the expected timing of the acquiring of the image frames.

(27) 11. Device for acquiring stream of the precisely time-stamped images, including at least one modulated light source (3) and at least one image acquisition and processing unit (4), containing the functional unit to decode the light modulation waveform, controller (22), output of which is connected to the modulation input of the light source, characterized that the controller (22) is the absolute global real-time controller (e.g using GNSS) and the image acquisition and processing unit includes the functional unit for calculation of the values of the absolute time-stamp for every image frame.