IMAGE ACQUISITION METHOD USING A COLOR TRANSFORMATION AND ASSOCIATED MEDICAL IMAGE ACQUISITION SYSTEM

Abstract

Medical image acquisition method for improving the identification of objects using characteristic colors in a color image which has been captured with an image sensor of a medical image acquisition system, wherein firstly a color value of an image area, selected by a user, of the color image is determined at least partially in a computer-implemented manner, and that subsequently, based on the determined color value, a color transformation is applied to the color image, which increases the color distance between image areas of the color image which are identical or similar in color to the determined color value and remaining image areas, not similar in color, of the color image.

Claims

1. An image acquisition method comprising: capturing a sequence of color images with an image sensor of a medical image acquisition system, and subjecting at least one of the color images is to a color transformation to generate a desired representation of the at least one color image on a monitor, wherein the step of subjecting at least one of the color images to a color transformation comprises: selecting an image area in one of the color images of the sequence; determining a color value of the selected image area; and applying a color transformation on the basis of the determined color value such that a color distance between the determined color value and remaining color values of non-selected image areas of the at least one color image is increased.

2. An image acquisition method in accordance with claim 1, wherein the color transformation results in an increase in a color saturation value distance and/or a hue value distance and/or a color brightness value distance, in each case based on a comparison between the selected image area and the non-selected image areas; and wherein the color distance is increased by adapting hue values and/or color saturation values and/or color brightness values of the selected image area and/or of the remaining non-selected image areas, taking into consideration the determined color value in each case.

3. An image acquisition method in accordance with claim 1, wherein the image area is selected using a characteristic hue, and wherein the characteristic hue is generated by means of a dye with which tissue can be stained or that is a natural hue of a tissue, in particular a malignant tissue.

4. An image acquisition method in accordance with claim 1, wherein the image area to be selected by the user or already selected is displayed to the user on the monitor.

5. An image acquisition method in accordance with claim 1, further comprising: automatically selecting additional image areas of the at least one color image, which have a determined color value; applying a color transformation to the automatically selected additional image areas to increase color distance; and wherein the automatically selected, additional image areas have additional image pixels with color values that differ from the determined color value.

6. An image acquisition method in accordance with claim 1, wherein the determined color value is determined using a statistical value calculated from image pixels of the selected image area, in particular an average; and wherein, to this end, RGB values of these image pixels are processed.

7. An image acquisition method in accordance with claim 1, wherein in the color transformation an output signal, in particular a raw data signal, of the image sensor, preferably in the form of an RGB signal, is converted into a signal in a hue-based color space, in particular into an HSV signal; and wherein the determined color value of the selected image area is a hue value, in particular averaged over image pixels of the selected image area; and wherein a saturation value, in particular an average saturation value, of the selected image area is determined using the signal in the hue-based color space, in particular the HSV signal.

8. An image acquisition method in accordance with claim 1, wherein the color distance is increased by adapting, preferably raising, color values of the selected image area, preferably and of the automatically selected additional image areas, and/or by adapting, preferably lowering, color values of the remaining non-selected image areas of the at least one color image, which do not have the determined color value and/or lie outside of a color similarity space of the determined color value.

9. An image acquisition method in accordance with claim 1, wherein the color transformation preserves respective brightness values and/or color values of the selected image areas and/or of the non-selected image areas to enable a detailed representation of image information; or the color transformation preserves relevant relative differences in color values of the selected image areas and/or of the non-selected image areas.

10. An image acquisition method in accordance with claim 1, wherein in order to increase the color distance a wedge-shaped, color similarity space is determined around the color value determined for the selected image area; and wherein, for image pixels of the color image which lie within the color similarity space, the respective color values, in particular color saturation values and/or hue values, are raised and/or for image pixels of the color image which lie outside of the color similarity space, respective color values, in particular color saturation values and/or hue values, are lowered.

11. An image acquisition method in accordance with claim 10, wherein for image areas lying within the determined color similarity space, a respective color distance to the determined color value of the selected image area is increased by raising or lowering color values in each case.

12. An image acquisition method in accordance with claim 10, wherein all image pixels which lie within the determined color similarity space are selected as image pixels to be highlighted; and wherein the color similarity space is subsequently elongated by extending color values of individual pixels out of those to be highlighted beyond the color similarity space.

13. An image acquisition method in accordance with claim 10, wherein the color similarity space is determined in a hue-based color space, in particular in the HSV space, and/or a color saturation adaptation is performed by taking, for individual image pixels of the at least one color image, an absolute value of a difference between a color value of the respective image pixel and the color value determined for the selected image area; by comparing the absolute value with a threshold value; and by increasing a color saturation value associated with the relevant image pixel, if the threshold value is exceeded, and/or reducing a color saturation value associated with the respective image pixel, if the absolute value falls below the threshold value.

14. An image acquisition method in accordance with claim 1, wherein, to further improve the representation, a color space of the at least one color image is rotated in such a way that the color value determined for the selected image area comes to rest on a nearest primary color, for example pure blue, red or green, or on a nearest secondary color, for example pure cyan, magenta or yellow; and wherein the selected image area and the automatically selected additional image areas, is/are displayed on the monitor in the primary color, while the remaining non-selected image areas are displayed exclusively in color values deviating from the primary color.

15. An image acquisition method in accordance with claim 1, wherein the color value to be determined for the selected image area is first automatically pre-determined by the endoscopy system by means of a statistical analysis of color values of the selected image area, and displayed to a user of the endoscopy system.

16. An image acquisition method in accordance with claim 15, wherein the user subsequently confirms or discards the pre-determined color value via the monitor, and/or re-adjusts a presented color value, and/or a presented saturation value adaptation, with the aid of a color value scale and/or saturation value scale shown graphically on the monitor as an overlay of an additional fine adjustment scale for fine adjustment of the color and/or saturation values.

17. An image acquisition method in accordance with one of the preceding claims, wherein a user selects the image area manually on the monitor such that the color value of the selected image area is determined, and/or the user is presented with a fixed target area for computer-assisted selection of the image area.

18. An image acquisition method in accordance with claim 1, wherein the color transformation is applied successively, preferably in real time, to multiple color images of the sequence, in a manner wherein a color distance between the determined color value and non-selected image areas of the respective transformed color images is increased.

19. A medical image acquisition system comprising: an image sensor; a camera control unit; and wherein the image acquisition system includes a controller, such as an FPGA with a controlling microprocessor, which is set up to perform, in combination with an external monitor, an image acquisition method in accordance with claim 1; and wherein the controller is set up for performing a color transformation with which a color distance between a selected image area of a color image captured with the image acquisition system and non-selected image areas of this color image is increased.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0086] The following are shown by the figures:

[0087] FIG. 1 a first color image of an image sequence that has been captured with an endoscope,

[0088] FIG. 2 a further color image of the sequence, wherein the observed image segment has shifted,

[0089] FIG. 3 the color image from FIG. 2, following the superimposition of a fixed target area,

[0090] FIG. 4 the color image from FIG. 3, after this has undergone a color transformation,

[0091] FIG. 5 an illustration of the HSV color space,

[0092] FIG. 6 a cross section through the HSV color space at a particular brightness value;

[0093] FIG. 7 left: the color image from FIG. 2 and right: a particular color value within the HSV color space from FIG. 6,

[0094] FIG. 8 left: the color image from FIG. 4 and right: an illustration of the color transformation performed in the color image from FIG. 4 within the HSV color space,

[0095] FIG. 9 an analogous representation to that in FIG. 8, wherein here the color transformation comprises an adaptation of color saturation values and of hue values, and

[0096] FIG. 10 the representation from FIG. 8, after elongating the color similarity space in which the selected image areas lay and

[0097] FIG. 11 a schematic diagram of an endoscopy system in accordance with the invention.

DETAILED DESCRIPTION OF THE DRAWINGS

[0098] FIG. 1 shows a first digital color image 1 of a sequence of color images which was captured with an image sensor 23 of an endoscope of an endoscopy system, in other words a medical image acquisition system within the meaning of the invention. Color image 1 consists of a plurality of uniform image pixels and is observed by an surgeon as the user of the endoscopy system as a live video image on an external monitor 25. The video color image 1 reflects a typical endoscopic image acquisition situation during a surgical procedure. More specifically, in color image 1 of FIG. 1 both healthy tissue 16 and malignant tissue 15 can be identified. The malignant tissue 15 has been stained with the aid of methylene blue, which is illustrated by the hatching with solid lines.

[0099] FIG. 2 shows the same scene from FIG. 1, but recorded in a subsequent color image 1 of the sequence after the surgeon has moved the endoscope to shift a region of interest (ROI) 26 into the center of the image. The surgeon does this as long as the ROI 26 lies within a fixed target area 4 superimposed by means of a sight 7. After this, the surgeon operates a control key of the endoscopy system, by which means he selects the rectangular image area 2 which is defined by the target area 4 in the center of the image.

[0100] When making this selection, the surgeon is guided by the characteristic blue hue of the malignant tissue 15, i.e. he selects the image area 2 using the characteristic hue of the dye methylene blue with which the malignant tissue 15 has been selectively stained.

[0101] In addition the surgeon is shown on the monitor 25, with the aid of the target area 4, the image area 2 to be selected by him. After the selection, the target area 4 still remains superimposed and hence shows the surgeon the image area 2 selected by him at that particular time.

The selection of the image area 2 triggers a color value determination of the image area 2 manually selected by the surgeon. To this end, the color values, for example hue values, of all the image pixels which lie within the selected image area 2 are averaged in order to determine the color value 6 of the selected image area 2 as an average. To this end, an image processing unit of the endoscopy system processes RGB values of the image pixels within the selected image area 2.

[0102] The color value 6 of the selected image area 2 determined automatically by the endoscopy system through image processing and statistical analysis 6 is, in a subsequent step, firstly only displayed to the surgeon as a pre-determined color value 6. This takes place by means of the vertical value scale 8 illustrated in FIG. 3, which visually illustrates the pre-determined color value 6 to the surgeon with the aid of a display element 27.

[0103] In a next step, the surgeon can now accept the pre-determined color value 6 by operating the control key again.

[0104] He may also, however, firstly compare the color value 6 visually illustrated by means of the value scale 8 with the ROI 26. If, here, he identifies an unsatisfactory color deviation, he may with the aid of a fine adjustment scale 9—shown horizontally in FIG. 3—readjust the presented pre-determined color value 6. The display element 27 stylized as a triangle in FIG. 3 shows the surgeon what color value he has just set.

[0105] This fundamental approach for determining a color value can be applied to hue values, color saturation values and color brightness values.

[0106] If the surgeon has achieved, through readjustment, satisfactory matching between the now readjusted color value 6 and the ROI 26, he can then accept the color value 6 currently presented with the control key. With these steps, the surgeon has thereby, assisted by computer, determined the color value 6 of the image area 2 selected by him within the ROI 26 with high precision.

[0107] This high precision is significant for a subsequent step in which the endoscopy system, with the aid of the image processing unit, now automatically identifies additional image areas 10 within the color image 3 which—in terms of color—are very similar or even identical to the selected area 2.

[0108] The colors are identical if for example an image pixel shows the determined color value 6. The colors are similar, on the other hand, if an image pixel shows a color value which lies within a color similarity space 5 which was calculated around the determined color value 6 by the image processing unit using pre-set parameters or parameters readjusted by the surgeon. The color similarity space may for example take into consideration both hue values and color saturation values.

[0109] Using the color value 6 determined precisely by the surgeon and once readjustment of the color similarity space 5 has taken place, the endoscopy system now automatically selects the additional image areas marked with reference symbol 10 in FIG. 3. These additional image areas 10 have image pixels with color values that deviate from the determined color value 6. However, all of the pixels of the image areas now selected in total (and shown hatched in FIG. 4) lie in the color similarity space 5 with respect to the determined color value 6 of the image area 2 selected at the start (cf. FIG. 2), as will be explained more precisely using FIGS. 5 to 7.

[0110] In a final step, which is illustrated in FIG. 4, the endoscopy system now increases the color distance between the determined color value 6 of the selected image area 2 and the remaining color values of the non-selected image areas 3 of the color image 1 displayed in FIG. 4 by applying a color transformation to the entire color image 1. FIG. 4 can thus be viewed as being representative of a whole sequence of consecutive color images 1, which are captured successively with the image sensor 23 and to which said color transformation is successively applied.

[0111] This even extends so far that—if the surgeon moves the endoscope again—new additional image areas 10, which move into the color image 1 at the edge of the respective current color image 1, are again automatically selected by the endoscopy system, such that the color transformation is applied to these successively newly selected additional image areas 10. In simple terms, therefore, additional malignant tissue is successively highlighted in color by increasing the color distance, also when the surgeon actively changes the captured image scene by moving the endoscope. This repositioning of the color highlighting permits very simplified working, because a color highlighting of image areas selected using the determined color value, once set, can be successively extended to additional new image areas, if these new image areas are similar in color to the determined color value.

[0112] In the color transformation, an RGB signal 32 from the image sensor 23, which characterizes the particular color image 1 to be transformed, is initially converted into an HSV signal 40. After this RGB-to-HSV transformation 30, the image colors of all the image pixels of the particular color image 1 are no longer described by RGB coordinates but by HSV coordinates.

[0113] The color transformation, however, goes beyond pure coordinate transformation. This is because owing to the color transformation, a color saturation of the image area 2 selected by the surgeon at the beginning and of the additional image areas 10 automatically selected using the determined color value 6, all of which are shown hatched in FIG. 4, is subsequently raised. At the same time, a color saturation of the remaining non-selected image areas 3 of the color image 1, to which the transformation is applied, is lowered; as a result, these areas become pale in color and recede into the background.

[0114] Accordingly, the increased color distance that is recognizable in FIG. 4 between the selected image areas 2 and 10 and the remaining non-selected image areas 3 of the current color image 1 arises. These areas correspond, as a very good match, specifically to the malignant tissue 15 and the healthy tissue 16 respectively, as is visible in FIG. 4. As a result, on the monitor 25 the surgeon is therefore shown the malignant tissue 15 in the overall color image 1 as structures stained in dark blue with a high color distance from the remaining healthy tissue 16, which moves into the background as a result of the reduced color saturation.

[0115] The color transformation is set up in just such a way that the relevant brightness values and/or color values both of the selected image areas 2, 10 and also of the non-selected image areas 3 are retained. This has the effect that despite color highlighting, patterns and fine details remain recognizable to the surgeon in the whole color image 1.

[0116] For a better understanding of the concept of the invention, FIG. 5 illustrates the known HSV color space which has the three coordinates color angle 12 (or color value), color saturation 13, and brightness value 14, as illustrated by the arrows in FIG. 5. Each image pixel of the color image 1 may essentially reproduce a color which can be located within the HSV color space.

[0117] As FIG. 6 illustrates, primary colors such as red 17, yellow 18, green 19, cyan 20, blue 21 and magenta 22 correspond to the color angles 0°/60°/120°/180°/240°/300°.

[0118] The purer the relevant color, the higher its respective color saturation 13 and the further outside, in a radial direction, the associated color point lies within the color space 5 shown in FIG. 5. Here the representation of FIG. 6 constitutes a cross section through the HSV color space from FIG. 5, specifically for a constant brightness value 14.

[0119] Fundamentally, the (color) saturation describes how strongly a colored stimulus differs from an achromatic stimulus regardless of its brightness, in other words its distance from the achromatic axis (black-white axis), which specifically corresponds to the central axis of the cylindrical color space in FIG. 5.

[0120] Hence all hues can have a saturation of up to 100%, whilst white, grey and black show a saturation of 0% respectively.

[0121] As can be recognized in FIG. 7, the image processing unit has determined the color value 6 of the selected image area 2 within the HSV color space. To this end, as already previously described, firstly an average was calculated from RGB values of the image pixels of the selected image area 2 and subsequently this value transformed into the HSV space. Consequently the determined color value 6 is a hue value, averaged over image pixels of the selected image area 2. As can be recognized in the right-hand half of FIG. 7, the determined color value 6 lies in the vicinity of pure blue 21.

[0122] In the same manner it is possible, for the selected image area 2, to also determine an averaged saturation value which specifically corresponds to the radius of the pixel indicated with reference symbol 6 within the HSV space on the right-hand side of FIG. 7. Naturally, the same also applies to an average color brightness value which can also be determined for the selected image area 2 using averaging.

[0123] A first possibility for increasing the color distance on the basis of the color transformation performed by a controller of the endoscopy system is illustrated by the right-hand diagram in FIG. 8. There one can first recognize a wedge-shaped color similarity space 5 which is formed around the determined color value 6, i.e. calculated using pre-set parameters from the image processing unit.

[0124] For all pixels of color image 1 which lie within the calculated color similarity space 5, the color saturation was increased as part of the color transformation, which corresponds to a migration of these pixels outwards in a radial direction. Accordingly, clear, striking blue hues result for these image areas 2, 10. This can be clearly identified from the arrow directed outwards in a radial direction within the color similarity space 5, which indicates the increase in the color saturation value of the determined color value 6.

[0125] Since the color transformation is applied in an analogous manner to all pixels within the color similarity space, in particular also to pixels within the automatically selected additional image areas, there results an analogous increase in the color saturation values of all these pixels (not shown in FIG. 8). Here differences remain however, in the color saturation values and in the hue values, meaning that after the color transformation, too, structures remain recognizable within the image areas transformed by the color transformation.

[0126] For the remaining pixels, for example the pixel illustrated by means of the dotted arrow line, which lie outside of the color similarity space 5 and hence in non-selected image areas 3 of the color image 1, the color saturation has, by contrast, been lowered, which corresponds to a movement inwards in a radial direction in the HSV space and in FIG. 8 is illustrated by several arrows directed at the center of the HSV space.

[0127] After adapting the saturation values of the individual pixels/image pixels of the color image 1, the entire color image 1 was then transformed back again into the RGB space by means of an HSV-to-RGB transformation 29 and displayed on the monitor 25.

[0128] As has already been mentioned, all of these individual color transformation steps can be applied repeatedly to several successive color images 1 of the sequence.

[0129] FIG. 9, which is designed in an analogous manner to FIG. 8, illustrates a further possible design of the color transformation which is used to increase the color distance: as can be recognized in the right-hand part of FIG. 8, firstly a color value 6 and a color similarity space 5 similar to the example from FIG. 8 were determined.

[0130] Subsequently, a color transformation was applied to all pixels within the color similarity spaces, including those of the automatically selected image areas. In order to increase the color distance firstly hue values, measured as color angles or hue values, were increased, which is indicated by the curved arrow on the right-hand side of FIG. 9. By this means, the hues of the respective pixels were thus increased by approx. 35°, so for instance for the determined color value from approx. 235° to 270°, which corresponds to a hue shift towards magenta (300°). This is clearly shown by the rotation of the color similarity space 5, as illustrated on the right-hand side of FIG. 9.

[0131] In a subsequent step, in addition to the hue shift, the color saturation values of the selected (including the automatically selected) image areas 2 were then increased and this was done analogously to the example of FIG. 8, which corresponds to a migration of the color values in a radial direction outwards in the right-hand picture in FIG. 9.

[0132] In addition, both of these steps were applied in reverse to pixels within the non-selected image areas. As is recognizable with the aid of the color value at approx. 95° color angle, to this end firstly the hue values of these pixels were reduced (which corresponds in the HSV space to a rotation in a clockwise direction, so for example from 95° to 90°, as illustrated on the right-hand side of FIG. 9) and subsequently their color saturation values were reduced (which in the HSV space corresponds to a migration inwards in a radial direction—cf. the radially inwardly directed arrow on the right-hand side of FIG. 9).

[0133] The first step of this two-stage color transformation may consequently be understood as a hue value elongation by means of which the color distances, measured in hue values, between selected and non-selected image areas is increased. By this means there may also be a hue shift of the selected area, meaning that this is displayed in a false color on the monitor after color transformation.

[0134] The color transformation was carried out both for the selected image areas 2 and for the non-selected image areas 3 in just such a way that the respective relative differences in the color values were preserved. As a result, the selected areas 2 and 10 shown hatched on the left-hand side of FIG. 9 appear in different colors following the transformation (shifted in the direction of magenta) and also with increased color saturation, but image structures are still recognizable in these areas because the relative differences in the hue values and the color saturation values between individual image pixels were preserved.

[0135] It goes without saying that an analogous color transformation also on the basis of color brightness values can be used alternatively or additionally to increase the color distance, as has already been explained above.

[0136] As a further optional step it is then possible—as shown in FIG. 10—for the color similarity space in which the selected image areas lie to be extended or elongated. To this end color values, in particular color saturation values but also hue values, of individual pixels of those which originally lay within the color similarity spaces, and which are to be highlighted in color (so the “image pixels to be highlighted”), may be extended beyond the color similarity space. As the HSV diagram in the right-hand part of FIG. 10 shows, for this purpose the color values, in particular hue values, of individual image pixels at the edge of the color similarity space are lowered/raised so much that they now lie outside of the original color similarity space, as shown by a comparison of the dotted with the dashed line. Through the elongation of the color space that takes place, the selected additional image areas 10 together with the originally selected image area 2 (both shown hatched in the left-hand diagram of FIG. 10) now cover a color space which, in terms of the hue values covers a larger color angle range than the original color similarity space 5. It should also be mentioned here that, in this case, color space which previously was occupied by non-selected image areas 3 is deliberately used to display the image areas 2, 10 to be highlighted. This is advantageous since the color values of the non-selected image areas 3, in particular their saturation values, were previously lowered meaning that these image areas 3, simply speaking, free up color space.

[0137] The diagram from FIG. 11 ultimately describes, schematically, the structure of an endoscopy system in accordance with the invention. This initially generates, with the aid of an image sensor 23, a sequence of color images 1 each of which are present as RGB image data 39. By means of a function “select image area” 36, in the RGB image data 39, firstly an image area 2 is selected by a user or by the endoscopy system itself for one of the color images and this is analyzed with an image processing system 37 in order to determine the color value 6.

[0138] After this, the entire color image 1 runs through a cascade of image processing steps such as edge filtering 33, noise filtering 34 and scaling 35, which help to improve the image quality before an RGB-to-HSV transformation 30 is applied to the color image 1, by which means an HSV signal 40 is generated. Using the color value 6—determined with the aid of image processing 37—of the selected image area 2, the color image 1 is then processed further in the HSV space, wherein the color distance is increased (=image processing in the HSV space 38—c.f. also the illustration on the right-hand side of FIG. 8 already outlined). In order to carry out this color transformation, use is made of a ‘color matrix’ which defines the transformation that is to be carried out.

[0139] This image processing in the HSV space 38 is followed by an HSV-to-RGB transformation 29, before the color image 1 is then transmitted as an RGB signal 32 by means of an on-screen display function 28 to the monitor 25 for display.

[0140] With this type of image processing or color transformation, the light source 24 that is used, or more precisely its color temperature, is taken into account. This is because the information from the light source 24 relevant to the image processing or the color matrix used in the process consists of the color temperature of the light emitted by the light source 24. The color temperature may for example be determined with the aid of a white fader. Also, in particular coefficients of the color matrix used for color transformation can be adapted on the basis of the determined color temperature of the light source 24.

[0141] In summary, in order to improve the recognition of objects using characteristic colors in a color image 1, which was captured with an image sensor 23 of a medical image acquisition system, it is proposed that firstly a color value 6 of an image area 2 of the color image 1 selected by a user is at least partially determined in a computer-implemented way and that subsequently, based on the color value 6 determined, a color transformation is applied to the color image 1 which increases the color distance between image areas 2, 10 of the color image 1, which are identical or similar in color to the determined color value 6, and other image areas 3 of the color image 1 that are not similar in color (cf. FIG. 8).