ASSISTANCE DEVICE AND METHOD FOR PROVIDING IMAGING SUPPORT TO AN OPERATING SURGEON DURING A SURGICAL PROCEDURE INVOLVING AT LEAST ONE MEDICAL INSTRUMENT

Abstract

Described is an assistance device for providing imaging support to an operating surgeon during a surgical procedure involving at least one medical instrument. The assistance device comprises a camera, a display unit, a manipulator coupled to the camera, a manipulator controller and an image processing unit. The image processing unit includes an instrument detection module for detecting at least one target structure that represents the instrument being used in the frame in question by identifying a predetermined distinguishing feature, and for extracting position information that indicates the position of the target structure in the frame. The instrument detection module identifies an image segment as the predetermined distinguishing feature, said image segment being characterized by a color saturation that is equal to or less than a predefined color saturation and by a contour line that delimits the image segment and has at least one rectilinear section.

Claims

1. An assistance device for providing imaging support to an operating surgeon during a surgical procedure involving at least one medical instrument, said assistance device comprising a camera for generating a video signal that contains a sequence of image frames; a display unit for displaying the image sequence on the basis of the video signal; an image processing unit having an instrument detection module for detecting at least one target structure that represents the instrument being used in the frame in question by identifying a predetermined distinguishing feature and extracting position information that indicates the position of the target structure in the frame; a manipulator, which is coupled to the camera and can be actuated via a control signal to move the camera; and a manipulator controller for generating the control signal from the position information and for actuating the manipulator via the control signal; characterized in that the instrument detection module identifies an image segment as the predetermined distinguishing feature, said image segment being characterized by a color saturation that is equal to or less than a predefined color saturation, and by an outline that delimits the image segment and has at least one rectilinear section.

2. The assistance device according to claim 1, wherein the image processing unit includes a segmentation module, which generates at least one binary image on the basis of the frame in question, in which the instrument detection module then identifies the image segment.

3. The assistance device according to claim 2, wherein the image processing unit includes a preprocessing module, which generates a grayscale image on the basis of the frame in question, the pixels of which are each assigned a grayscale value that represents the color saturation of the corresponding pixel of the frame; and on the basis of the grayscale image, the segmentation module generates the at least one binary image, the binary pixels of which are assigned a first binary value if the associated grayscale values are equal to or less than a threshold value that corresponds to the predefined color saturation, and the binary pixels of which are assigned a second binary value if the associated grayscale values are greater than the threshold value.

4. The assistance device according to claim 2, wherein the instrument detection module combines a plurality of collinearly spaced rectilinear sections of the outline to form a continuous edge line that represents an edge of the instrument.

5. The assistance device according to claim 4, wherein for detection of the target structure, the instrument detection module pairs two edge lines that are arranged parallel to one another in each case.

6. The assistance device according to claim 5, wherein the instrument detection module identifies an instrument tip on the basis of the two paired edge lines.

7. The assistance device according to claim 2, wherein the segmentation module generates a plurality of binary images on the basis of the respective grayscale image, and uses different threshold values for the generation of said images.

8. The assistance device according to claim 7, wherein the instrument detection module identifies mutually corresponding rectilinear sections in the binary images generated on the basis of the frame in question, and combines these sections to form a single edge line that represents an edge of the instrument.

9. The assistance device according to claim 1, wherein the image processing unit includes a tracking module, which tracks the target structure detected by the instrument detection module over multiple successive frames; and the manipulator controller generates the control signal related to the frame in question, using the position information about the target structure, for the purpose of actuating the manipulator only if said target structure has already been tracked by the tracking module over multiple successive frames.

10. The assistance device according to claim 9, wherein the tracking module assigns a tracker to a target structure that is detected for the first time in the frame in question by the instrument detection module, and uses the tracker to track said target structure that is detected in the subsequent frames.

11. The assistance device according to claim 1, wherein the image processing unit includes a flow module, which detects an optical flow of the image sequence, which represents movement information contained in the image sequence.

12. The assistance device according to claim 9, wherein the tracking module comprises a first submodule, located upstream of the instrument detection module, and a second submodule, located downstream of the instrument detection module; the first submodule factors in the optical flow detected by the flow module to make a prediction regarding the position information about the tracker for the next frame, which has not yet been processed by the instrument detection module; and for said next frame, which has been processed by the instrument detection module, the second submodule verifies the prediction made by the first submodule based on the position information for the tracker detected by the instrument detection module.

13. The assistance device according to claim 3, wherein the preprocessing module is designed to perform a white balance adjustment of the frame in question.

14. The assistance device according to claim 13, wherein the image processing unit includes a parameter optimization module, which processes the frames asynchronously to the preprocessing module and from them generates actuation information, which specifies whether or not the preprocessing module should perform the white balance adjustment.

15. The assistance device according to claim 14, wherein the parameter optimization module predefines the threshold values for the segmentation module on the basis of the asynchronous processing of the frames.

16. A method for providing imaging support to an operating surgeon during a surgical procedure involving at least one medical instrument, comprising: a camera that generates a video signal that includes a sequence of image frames; the image sequence that is displayed on a display unit on the basis of the video signal; at least one target structure that represents the instrument being used in the frame in question being detected by identifying a predefined distinguishing feature, and position information that indicates the position of the target structure in the frame is extracted; and a control signal being generated from the position information, and a manipulator coupled to the camera being actuated via the control signal to move the camera; wherein an image segment is identified as the predefined distinguishing feature, said image segment being characterized by a color saturation that is equal to or less than a predefined color saturation, and by an outline that delimits the image segment and has at least one rectilinear section.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In the following, the invention will be described in greater detail in reference to the figures. The figures show:

[0026] FIG. 1 is a block diagram illustrating the overall structure of an assistance device according to the invention;

[0027] FIG. 2 is a block diagram illustrating the structure of an image processing unit of the assistance device;

[0028] FIG. 3 is a block diagram showing the process flow of image preprocessing carried out in the method of the invention;

[0029] FIG. 4 is a block diagram showing the process flow of instrument detection carried out in the method of the invention;

[0030] FIG. 5 is a block diagram showing the process flow of instrument tracking carried out in the method of the invention;

[0031] FIG. 6 is a binary image generated in the method of the invention; and

[0032] FIG. 7 is an outline obtained from the binary image of FIG. 6.

DETAILED DESCRIPTION AND INDUSTRIAL APPLICABILITY

[0033] FIG. 1 is a block diagram of an assistance device 10 according to the invention.

[0034] Assistance device 10 comprises a camera 12, which is part of an endoscope, not explicitly shown, which is held by a manipulator 14, for example a robotic arm. Manipulator 14 has mechanical degrees of freedom that enable a repositioning of camera 12.

[0035] Camera 12 captures a video image of a target area inside the human body in which the anatomical structure to be treated is located. Camera 12 thus generates a video signal that contains a sequence of image frames and is output in the form of a data stream to a camera controller 16. Camera controller 16 transmits the video signal to a display unit 18, for example a monitor, on which a video image of the anatomical structure being treated, corresponding to the video signal, is displayed.

[0036] Camera controller 18 feeds the video signal to a controller 20 via an image acquisition module, for example a “frame grabber”, which is not shown in FIG. 1. Controller 20 includes an image processing unit 22, which uses the video signal supplied to it as an input signal for carrying out an instrument identification in a manner that will be explained in detail later. During this instrument identification, the surgical instruments that are visible in the video image are identified and tracked by the image processing unit 22. In that process, position information is obtained and is fed to a control unit 24. Control unit 24 includes a manipulator controller 26. Said controller functions as a path controller, which uses the position information supplied to it to generate a control signal, which is used to actuate manipulator 14. With this actuation, for example, the manipulator 14 repositions the camera 12 that is held on it such that the tip of an instrument in the video image being displayed on the display unit 18 is always located at the center of the image.

[0037] Assistance device 10 further has a trigger switch 28, which is coupled to an interface control unit 30 contained in control unit 24. Actuating the trigger switch 28 causes the interface control unit 30 to activate manipulator controller 26 so as to reposition camera 12.

[0038] Assistance device 10 further has a graphic user interface 32, which is coupled on one side to image processing unit 22 and interface control unit 30, and on the other side to display unit 18. Assistance device 10 further comprises additional input devices that are of minor importance for an understanding of the present invention and are generally denoted in FIG. 1 by reference sign 34.

[0039] FIG. 2 shows a block diagram illustrating the structure of image processing unit 22.

[0040] Image processing unit 22 includes a preprocessing module 36, a parameter optimization module 38 and a flow module 40. The video signal, which contains a sequence of image frames, is fed to each of modules 36, 38 and 40 by camera controller 16. Parameter optimization module 38 is coupled to preprocessing module 36.

[0041] Image processing unit 22 further includes a first submodule 42, which, together with a second submodule 44, forms a tracking module, denoted generally in FIG. 2 by reference sign 46. The first submodule 42 is coupled on the input side to preprocessing module 36 and to flow module 40. On the output side, it is connected to a segmentation module 48. Segmentation module 48 is in turn coupled on the output side to an instrument detection module 50. The output of instrument module 50 is coupled to the second submodule 44 of tracking module 46. Finally, the second submodule 44 forms the output of image processing unit 22 and is therefore connected to manipulator controller 26, which generates the control signal for actuating the manipulator 14 from the signal supplied to it by image processing unit 22.

[0042] The operating principle of modules 36 to 50 included in image processing unit 22 will be apparent in the following from the flow charts shown in FIGS. 3, 4 and 5, which illustrate the image processing according to the invention by way of example. In FIGS. 3 to 5, rectangular symbols denote the processing steps performed by the respective modules, while diamond-shaped symbols represent storage and readout operations.

[0043] The flow chart of FIG. 3 depicts image preprocessing as carried out according to the invention, in which preprocessing module 36, parameter optimization module 38 and flow module 40 of image processing unit 22 are used, in particular.

[0044] In step S1, preprocessing module 36, parameter optimization module 38 and flow module 40 receive a buffered frame of the image sequence contained in the video signal. In the present exemplary embodiment, it is assumed in the following that said frame is a conventionally produced individual RGB image, frequently often referred to as an RGB frame.

[0045] In step S2, preprocessing module 36 performs various adjustments and optimizations of the RGB frame, aimed at adjusting the image size, brightness and contrast appropriately, for example. In particular, if necessary, preprocessing module 36 also performs an automatic white balance adjustment in step S2. Whether or not a white balance adjustment is performed in S2 is determined in step S3 by parameter optimization module 38 via an actuation signal AWB Y/N, which parameter optimization module 38 supplies to preprocessing module 36. For this purpose, parameter optimization module 38 performs a corresponding check of the image sequence supplied to it. Parameter optimization module 38 operates asynchronously to preprocessing module 36 as part of an iterative parameter optimization, for example in that it does not perform the aforementioned check for each frame.

[0046] In step S2, preprocessing module 36 also detects a camera mask in the frame, which represents an image area that cannot be analyzed. Preprocessing module 36 then stores the corresponding mask information which enables the non-usable image area that corresponds to the mask to be filtered out of the frame, as it were, as the process continues.

[0047] Once the frame has been optimized in step S2 for further processing, it is stored in an image memory.

[0048] In step S4, preprocessing module 36 uses the RGB frame to calculate a grayscale image, the pixels of which are each assigned a grayscale value that represents the color saturation of the corresponding pixel of the RGB frame. The grayscale image generated in step S4 is thus a color saturation image. In S4, this color saturation image is stored for subsequent processing steps.

[0049] In step S5, preprocessing module 36 performs a quality check of the RGB frame. If this quality check results in a negative assessment, the RGB frame is discarded and processing continues with the next frame. In contrast, if the quality of the RGB frame is assessed as positive in step S5, then preprocessing ends and the process flow continues with the processing steps according to FIG. 4.

[0050] In step S6 according to FIG. 3, a process that runs in parallel with the process steps described above is executed by flow module 40. Flow module 40 is designed to detect the optical flow of the image sequence contained in the video signal. The optical flow represents movement information contained in the image sequence, for example in the form of a vector field that indicates the magnitude and direction of the speed of pixels in the image sequence. In detecting the optical flow, in addition to considering the current RGB frame, flow module 40 also considers the preceding RGB frame. The detected optical flow is then stored. A comparison with previous detection results is also carried out in step S6.

[0051] In step S7, the preprocessing according to FIG. 3 ends.

[0052] The image processing shown in FIG. 4 is carried out in image processing unit 22 primarily by tracking module 46, which comprises the two submodules 42, 44, by segmentation module 48 and by instrument detection module 50. After image processing starts in step S11, in step S12 segmentation module 48 reads in the grayscale image that was calculated and stored by preprocessing module 36, on the basis of the RGB frame, in step S4 of FIG. 3. Based on this grayscale image, segmentation module 48 generates N binary images using different threshold values, which are predefined, for example, by parameter optimization module 38. Each of these threshold values corresponds to a predefined color saturation, with which the grayscale values of the pixels of the grayscale image are compared. Those pixels of the grayscale image whose grayscale values are equal to or less than the threshold value that corresponds to the predefined color saturation are assigned binary pixels that have a first binary value (for example, 1) in the respective binary image. Correspondingly, those pixels of the grayscale image whose grayscale values are greater than the threshold value are assigned binary pixels that have a second binary value (for example, 0) in the respective binary image. Such a binary image generated in step S12 is shown purely by way of example in FIG. 6, in which binary pixels having a binary value of 1 are assigned the color white and binary pixels having a binary value of 0 are assigned the color black. For purposes of illustration, in FIG. 6 the color black is indicated as dots. Thus, in the binary image of FIG. 6, white image segments represent image areas in which the color saturation is equal to or less than the respectively predefined color saturation. In contrast, black image segments represent image areas in which the color saturation is greater than the predefined color saturation. Since the image processing according to the invention assumes that the color saturation of the instruments to be identified in the video image is lower than that of the anatomical structure being treated, in the binary image of FIG. 6 the white image segments represent the instruments to be identified, while the black image segments represent the anatomical structure.

[0053] In step S12, segmentation module 48 therefore generates N binary images of the type shown in FIG. 6, however these images differ more or less from one another, since they are based on N different threshold values. The greater the difference between the N threshold values, the more the binary images generated using these threshold values will differ from one another.

[0054] In step S13, instrument detection module 50 extracts outlines from each of the N binary images and stores these in N outline data sets. In the exemplary binary image of FIG. 6, these outlines extend at the junctions between the white and black image segments.

[0055] In step S14, instrument detection module 50 uses the N outline data sets to identify rectilinear sections, hereinafter referred to as line segments. Instrument detection module 50 stores the identified line segments in N line segment data sets.

[0056] The process step according to S14 is illustrated in the diagram of FIG. 7. There, the line segments are shown as dashed lines.

[0057] In step S14, instrument detection module 50 also detects curved sections of the outlines, to eliminate these from the outlines to be further processed. These curved sections are shown as dotted-dashed lines in FIG. 7.

[0058] To avoid any adulteration by the camera mask, in step S14 instrument detection module 50 factors in the mask information that was ascertained and stored by preprocessing module 36 in step S2 of FIG. 2.

[0059] Proceeding from the N line segment data sets generated in step S14, in step S15 the instrument detection module 50 generates a single data set in which mutually corresponding line segments from the N binary images are combined to form a single line segment. This composite line segment, also referred to in the following as a compact line segment, represents an edge line of the instrument that results, as it were, from the superimposition of all N binary images. In step S15, based on predefined criteria, instrument detection module 50 ascertains whether line segments identified in the N binary images correspond to one another in the sense described above.

[0060] One example of such a criterion is the parallelism of the line segments in question. For example, an angle may be specified as a threshold value, and a check may be made to determine whether the line segments in question have an angular deviation in their alignment that is below this threshold value. If so, the criterion of parallelism is considered to be met. A further criterion may be, for example, a so-called overlap, for which a threshold value can again be predefined. If the segments in question have an overlap with one another that exceeds this threshold value, this criterion is also considered to be met. A further criterion may be, for example, the distance between the line segments in question. If this value is below a predefined threshold value, this criterion is considered to be met. The grayscale value combined with the aforementioned segment spacing also represents a suitable criterion.

[0061] The above criteria are essentially aimed at reliably determining whether the line segments in question occur along one and the same gradient within the grayscale image that forms the basis for the N binary images.

[0062] Thus, in step S15 the instrument detection module filters the result set generated in step S14 based on the criteria described above.

[0063] In step S16, instrument detection module 50 produces edge lines from the compact line segments in which mutually corresponding line segments are combined. Said edge lines represent the edges of the detected instrument. In so doing, instrument detection module 50 factors in detection results that were obtained from the preceding frame.

[0064] In step S17, instrument detection module 50 performs a pairing of two edge lines that are arranged parallel to one another and were ascertained in step S16. Once again, this edge pairing is performed on the basis of predefined criteria. One example of such criteria is the orientation of two vectors that extend perpendicular to each of the two edge lines in question.

[0065] In process step S17, information regarding the positions of instruments from the preceding frame is used. Also taken into account are position data from so-called trackers, each of which is regarded as a representation of an instrument detected in the frame.

[0066] Typically, it is not possible to pair each edge with a different edge in step S17. Therefore, in step S18, instrument detection module 50 assigns leftover edges to supposed instruments located near the edge of the frame. For this purpose, the instrument detection module again factors in the mask information provided in step S2 of FIG. 3.

[0067] In step S19, instrument detection module 50 determines the instrument axis as well as the length and orientation of the associated instrument based on a respective edge pair.

[0068] In step S20, instrument detection module 50 identifies the tip of the instrument, factoring in geometric characteristics that instrument detection module 50 assumes from the outset as given. One such geometric characteristic might be, for example, a conical shape of the instrument, assumed from the outset. The detection results obtained in step S20 are stored for further use in the subsequent processing steps.

[0069] In step S21, instrument detection module 50 filters out false-positively identified instruments. One criterion that may be applied for this purpose is the quality of the edge pairing, for example. Presupposed regularities of certain color properties may also be applied as a criterion. For example, a certain color property exists either in an instrument or in the anatomical structure, but not in the instrument and in the anatomical structure. By factoring in such a regularity, false-positively identified instruments can be filtered out.

[0070] Image processing according to FIG. 4 ends with step S22.

[0071] The image processing according to FIG. 4 results in stored information about the length, orientation and position of the instrument tip. Supplemental information, for example about the quality of the edge pairing, is also provided.

[0072] FIG. 5 is a flow chart depicting the functioning of tracking module 46, which is contained in image processing unit 22 and comprises the two submodules 42 and 44.

[0073] After starting in step S30, in step S31 the first submodule 42 of tracking module 46 corrects the positions of the trackers ascertained in the last frame, on the basis of the optical flow detected by flow module 40 in step S6 of FIG. 3. In this process, the first submodule 42 makes a prediction as to where the trackers will move to in the present frame. This prediction of the tracker movement serves to prevent gaps and jumps.

[0074] In step S32, segmentation module 48 and instrument detection module 50 then perform the processing steps shown in FIG. 4 for the present frame.

[0075] In step S33, the second submodule 44 of tracking module 46 updates the positions of the trackers based on the detection results obtained in step S20 of FIG. 4, and generates new trackers when needed, if new instruments are detected in the frame. The second submodule 44 thereby verifies the prediction of the tracker positions made by the first submodule 42 in step S31. By applying certain criteria, the second submodule 44 determines the reliability of the trackers currently in use.

[0076] In step S33, the second submodule 44 assigns markers (labels) to the trackers, which are then displayed on display unit 18, for example, in different colors. Such a marking of different trackers is advantageous, for example, when an instrument that is first being tracked by a particular tracker A drops out of tracking, causing tracker A to be deleted, and ultimately tracking of the same instrument is begun again so that a new tracker B is generated for this purpose. In that case, the two trackers A and B are assigned one and the same marker.

[0077] The second submodule 44 then stores the detection results for the next run-through in step S33.

[0078] The process according to FIG. 5 ends in step S34.

[0079] The image processing described above in reference to FIGS. 3 to 5 is to be understood merely as an example. For instance, it is not absolutely necessary to generate multiple binary images based on the respective grayscale image using different threshold values. Often, the generation of a single binary image using a suitably chosen threshold value is sufficient for separating image segments of low color saturation, which represent an instrument in the frame, from image segments of high color saturation, which represent the anatomical structure. This is true, for example, in cases in which only instruments made of the same material and thus having the same color saturation are used.

[0080] It will be apparent to those skilled in the art that various modifications and variations can be made to the present disclosure without departing from the scope of the disclosure. Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the teachings disclosed herein. It is intended that the specification and embodiments described herein be considered as exemplary only.