METHOD FOR TRANSMITTING A RAW IMAGE DATA STREAM FROM AN IMAGE SENSOR

Abstract

A device (1) and a method for transferring a raw image data stream (8) from an image sensor (6) via a data transfer connection (4) between a transmitter (2) and a receiver (3). In this case, the latency of the data transfer connection (4) is determined (S1), and the data rate of the raw image data stream (8) at the transmitter (2) is adapted (S3) to the determined latency.

Claims

1. A method for transferring a raw image data stream (8) from an image sensor (6) via a data transfer connection (4) between a transmitter (2) and a receiver (3), the method comprising: determining (S1) a current latency of the data transfer connection (4), and adapting (S3) a data rate of the raw image data stream (8) at the transmitter (2) depending on the determined current latency.

2. The method as claimed in claim 1, further comprising reducing the data rate if the current latency exceeds a latency limit value, in particular wherein the data rate is reduced further as latency increases.

3. The method as claimed in claim 1, further comprising regularly or continuously repeating the determining and adapting steps.

4. The method as claimed in claim 1, further comprising the determining of the current latency being determined at the receiver (3), and providing a supervisory connection (5) between transmitter (2) and receiver (3), the receiver transferring supervisory data via the supervisory connection to the transmitter (2), and the adapting of the data rate at the transmitter (2) is effected based on the supervisory data.

5. The method as claimed in claim 1, wherein adapting the data rate is effected by changing at least one of a resolution, a color depth or a frame rate.

6. The method as claimed in claim 5, wherein adapting the resolution is effected by reconfiguring the image sensor (6).

7. The method as claimed in claim 5, wherein adapting the color depth is effected by digitizing color values with a smaller number of bits.

8. The method as claimed in claim 5, wherein the transmitter (2) adapts the color depth per color value by cutting off one or more less significant ones of the bits and the receiver adds missing bits with virtual noise.

9. The method as claimed in claim 5, wherein adapting the data rate is effected by combining pixels that occur characteristically redundantly in a pixel pattern.

10. The method as claimed in claim 1, wherein the data transfer connection (4) is a wireless data connection.

11. A device (1) for transferring a raw image data stream (8) from at least one image sensor (6) via a data transfer connection (4) between a transmitter (2) and a receiver (3), the device comprising a processor configured to determine a current latency of the data transfer connection (4), and the processor being further configured to adapt a data rate of a raw image data stream (8) at the transmitter (2) depending on the determined current latency.

12. The device (1) as claimed in claim 11, wherein the processor that is configured to determine the current latency is arranged at the receiver (3) and the device further comprises a supervisory connection (5) between transmitter (2) and receiver (3), via which supervisory connection the receiver (3) is configured to transfer supervisory data to the transmitter (2).

13. The device (1) as claimed in claim 12, wherein the transmitter (2) has an image sensor (6) and the image sensor (6) is configurable based on the supervisory data.

14. The method of claim 2, further comprising reducing the data rate further as latency increases.

15. The method of claim 4, wherein the supervisory data comprises at least one of the determined latency or control data.

16. The method of claim 5, wherein the changing is locally weighted in the image.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The invention is explained in greater detail below on the basis of examples with reference to the accompanying drawings.

[0048] In the figures:

[0049] FIG. 1 shows a flow diagram of a method according to the invention,

[0050] FIG. 2A shows a device according to the invention comprising a transmitter having one image sensor and a receiver,

[0051] FIG. 2B shows a device according to the invention comprising a transmitter having two image sensors and a receiver,

[0052] FIG. 3 shows a schematic illustration of the reduction of the resolution by pixel binning,

[0053] FIG. 4 shows a schematic illustration of the reduction of the color depth by selectively omitting color data, and

[0054] FIG. 5 shows a schematic illustration regarding the reduction of the color depth by cutting off less significant bits per color channel.

DETAILED DESCRIPTION

[0055] FIG. 1 shows a flow diagram of a method according to the invention for transferring a raw image data stream from a transmitter to a receiver via a data transfer connection. In a first step S1, the current latency of the data transfer connection is determined. This can be effected for example by ascertaining the transfer rate or by means of time stamps.

[0056] In surgical applications, it is important for the latency to be less than 80 ms. For a user, however, it may likewise be disturbing if the latency frequently changes or fluctuates. This may be disturbing even if the latency overall is low, i.e. for example changes continuously within 40 ms and 80 ms.

[0057] In a second step S2, the determined latency is now compared with a latency limit value. If the determined latency is greater than the latency limit value, in a third step S3 the data rate of the raw image data stream is reduced and the method is continued with the first step.

[0058] If the determined latency is less than the latency limit value, the method likewise continues with the first step S1.

[0059] Alternatively, it is possible here to check whether the data rate has already been reduced and the data rate can be increased again.

[0060] In order to change the data rate of the raw image data stream, the resolution, color depth and/or frame rate of said raw image data stream can be changed. Table 1 shows by way of example the data rate per second as a function of these three parameters and thus the potential for reducing the data rate. In this case, the maximum values for resolution, color depth and frame rate may differ depending on the image sensor and should therefore in no way be understood to be restrictive. In this regard, there are for example also image sensors with a color depth of 14 bits and/or higher resolution, for instance 4K.

TABLE-US-00001 TABLE 1 Resolution Color depth Frame rate Data rate 1920 1080 12 60 1,492,992,000 1920 1080 10 60 1,244,160,000 1920 1080 8 60 995,328,000 1920 540 12 60 746,496,000 960 540 12 60 373,248,000 480 1080 12 60 373,248,000 480 270 12 60 93,312,000 1920 1080 12 30 746,496,000 1920 1080 12 15 373,248,000 480 270 8 15 15,552,000

[0061] FIG. 2A shows a device 1 according to the invention comprising a transmitter 2 and a receiver 3, which are connected to one another via a data transfer connection 4 and a supervisory connection 5.

[0062] The device can be an endoscope, for example, wherein the transmitter 2 can be a camera head and the receiver can be a camera control unit (CCU).

[0063] The data transfer connection 4 and the supervisory connection 5 can be of wired, optical or wireless design. In the example, both are designed as a WiFi connection. Alternatively, the supervisory connection 5 can also be designed as a separate wireless connection, for instance as a Bluetooth connection or as a WiFi connection on a different band or at a different frequency than the data transfer connection 4, in order to have the full bandwidth available for the latter.

[0064] The receiver 3, for instance the CCU of an endoscope, has an image processing unit 10, which receives the raw image data stream 8 and converts the latter into a video image data stream 11 for an image display unit, for instance a monitor. The image processing unit 10 can also be designed for generating the supervisory data and/or control commands 9.

[0065] The receiver 3 additionally has according to the invention means for ascertaining the latency of the data transfer connection 4. This means can for example be integrated in the image processing unit 10, and for example be designed for evaluating time stamps in the raw image data stream.

[0066] The transmitter 2, for example the camera head of an endoscope, has an image sensor 6 and a control unit 7 connected to the image sensor 6.

[0067] In one embodiment, the control unit 7 is an SoC designed for changing a raw image data stream and for driving the image sensor 6.

[0068] In this embodiment, the control unit 7 receives a raw image data stream 8 of the image sensor 6 and transfers it to the receiver 3 via the data transfer connection 4.

[0069] The control unit 7 also receives supervisory data via the supervisory connection 5. From the supervisory data, the control unit ascertains control commands 9 for the image sensor and for changing the raw image data stream. In this embodiment, the control unit 7 can make changes according to the invention to the raw image data stream before the latter is forwarded to the data transfer connection 4.

[0070] In this case, the supervisory data can comprise the determined latency, for example, from which the control unit 7 then carries out suitable measures for changing the data rate according to the invention.

[0071] In an alternative embodiment, the control unit 7 is a simple microcontroller designed for receiving control commands 9 from the receiver and forwarding them to the image sensor 6 and for receiving a raw image data stream 8 from the image sensor 6 and forwarding it to the data transfer connection 4. All of the control commands 9 are generated in the receiver 3.

[0072] This embodiment has the advantage that there is practically no computational complexity in the transmitter and the control unit 7 can accordingly be designed very simply and cost-effectively.

[0073] In both embodiments, the image sensor 6 can be designed to be configurable by means of the control commands 9. As a result, the image sensor 6 can be read for example according to the invention with a lower resolution (pixel binning), color depth or frame rate. Particularly in the case of the last-mentioned embodiment, a variable reduction of the data rate is thus possible, without computing power being necessary or present in the transmitter.

[0074] In this case, what possibilities are available may also depend on the image sensor 6 present. In such a case, it is possible to compensate for a lack of configurability of an image sensor by means of a control unit 7 with an SoC (system on a chip), an ISP (image signal processor) or an FPGA (field programmable gate array) for the signal processing of the raw image data stream.

[0075] FIG. 2B shows a device 1 according to the invention, which device substantially corresponds to the device from FIG. 2A. In this embodiment, however, the transmitter has two image sensors 6, the raw image data of which are transferred via the one data transfer connection 4. These image sensors can be configured for stereoscopic representation, for example. However, by way of example, it is also possible for one image sensor to be designed for real image recording, and the other for capturing instances of fluorescence. The method according to the invention is then applied to both image sensors.

[0076] FIG. 3 shows by way of example a method for reducing the resolution of the image sensor. This reduction of the resolution can be effected by means of configuration of the image sensor or after the digitization by means of image editing.

[0077] In FIG. 3 at (a) a detail of an image sensor 6 with four pixels 12 is shown. The image sensor 6 in the example has a Bayer color filter. Each pixel 12 accordingly has four subpixels 13: one red subpixel R1-R4, one blue subpixel B1-B4 and two green subpixels G1.1-G2.4. In the example, the resolution is reduced to one quarter by means of a 2×2 pixel binning. For this purpose, the corresponding subpixels 13 of the four pixels 12 are in each case combined, which is illustrated in FIG. 3 at (b). FIG. 3 at (c) then shows the resulting pixel 12 such as is received at the receiver 3.

[0078] In this way, the resolution can also be reduced to other values, for example 1×2, 2×1, 2×2, 2×3, . . . , n×m.

[0079] FIG. 4 shows a method for reducing the color depth. The image sensor 6 has a Bayer color filter in this example, too. Here, however, for each pixel 12, the two green values G1 and G2 of the respective subpixels are not transferred via the transmitter 2, rather only one green value Gx is communicated. The latter can be one of the two green values G1 or G2 or a mean value formed therefrom. In the receiver 3, the one green value Gx is set for both green values G1 and G2, such that a Bayer pattern arises again. In this way, however, instead of four color values, only three color values per pixel are to be transferred, as a result of which the data rate can be reduced by 25%.

[0080] FIG. 5 shows a further method for reducing the color depth. In the example, each subpixel 13 of the image sensor 6 is quantized with a color depth of 12 bits (a). The least significant bits often contain image noise that is not relevant to the finished image. In order to reduce the data rate, in the example, the four least significant bits (X) are cut off and only the more significant eight bits are transferred. In the receiver, the four bits cut off are added and padded by white noise or in some other way. The advantage here is that in the example the data rate is reduced by 33% without significant loss of image quality. The advantage over a true 8-bit color depth consists in the higher resolution as a result of the 12-bit digitization.

LIST OF REFERENCE SIGNS

[0081] 1 Device [0082] 2 Transmitter [0083] 3 Receiver [0084] 4 Data transfer connection [0085] 5 Supervisory connection [0086] 6 Image sensor [0087] 7 Control unit [0088] 8 Raw image data stream [0089] 9 Control commands [0090] 10 Image processing unit [0091] 11 Video image data stream [0092] 12 Pixel [0093] 13 Subpixel