DEVICE AND METHOD FOR IMAGE-BASED SUPPORT OF A USER

Abstract

Devices and methods for image-based treatment with an instrument configured to apply variable amounts of treatment medium at different locations. Devices having an image caption device that is configured to capture the area to be treated. The device comprises an evaluation device that is configured to determine a treatment trace of the treatment medium relative to the area to be treated from data of the image capture device and to determine a spatially resolved dosage for the area to be treated based on the treatment trace. Devices having a representation device that is configured to display the treatment trace and/or the spatially resolved dosage optically for a user, relative to the tissue. Methods to optically indicate the locations of the area to be treated of the tissue surface that have already been treated and/or the dosage with which the respective treated locations have been treated.

Claims

1. A device for image-based support of a user comprising: an instrument that is configured to apply a treatment medium at alterable locations in an area to be treated; an image capture device that is configured to capture the area to be treated; an evaluation device that is configured to determine a treatment trace of the treatment medium relative to the area to be treated from data of the image capture device and to determine a spatially resolved dosage for the area to be treated based on the treatment trace; and a representation device that is configured to optically display the treatment trace and/or the spatially resolved dosage in relation to the area to be treated for the user.

2. The device according to claim 1, characterized in that the treatment medium that can be applied is a plasma.

3. The device according to claim 1, wherein the image capture device is configured to capture the area to be treated at least partly by an image sequence comprising a multiplicity of individual images.

4. The device according to claim 1, wherein the evaluation device is configured to recognize the treatment medium and/or a head of the instrument in the image sequence and to trace it over multiple individual images in the image sequence.

5. The device according to claim 3, wherein the evaluation device is configured to determine a position and/or a size of the applied treatment medium in a current individual image of the image sequence.

6. The device according to claim 3, wherein the evaluation device is configured to determine a position, an orientation and/or a dimension of the head of the instrument in the current individual image of the image sequence.

7. The device according to claim 1, wherein the evaluation device is configured to recognize the treatment medium in data of the image capture device based on a color and/or shape feature, at least based on one of the following features: a color value range, a brightness value range and a saturation value range.

8. The device according to claim 1, wherein the evaluation device is configured to recognize the head in data of the image capture device based on a color and/or shape feature or based on a marker attached on the head.

9. The device according to claim 4, wherein evaluation device is configured to recognize the treatment medium in a search area in the current individual image by means of a search routine, whereby the search area for the treatment medium is modified based on the position, the orientation and/or the dimension of the head.

10. The device according to claim 9, wherein the evaluation device is configured to determine a predicted position of the treatment medium in the current individual image based on a position and/or velocity of the treatment medium from a previous individual image, wherein preferably the search area in which the search routine recognizes the treatment medium (P) in the current individual image is modified based on the predicted position and/or the velocity.

11. The device according to claim 1, wherein the evaluation device is configured so that the area to be treated can be defined by the user prior to the application of treatment medium.

12. The device according to claim 1, wherein the evaluation device is configured to recognize characteristic image features of the area (B) to be treated in data of the image capture device or to recognize reference markers applied in the area of the tissue surface and to assign the treatment trace to the area to be treated in a spatially accurate manner.

13. The device according to claim 1, wherein the evaluation device is communicatively connected with a supply unit of the instrument, and further wherein the evaluation device is configured to control the application of the treatment medium by the instrument depending on the local dosage.

14. The device according to claim 1, wherein the evaluation device is communicatively connected with a supply unit of the instrument, whereby the evaluation device is configured to determine the spatially resolved dosage in addition based on measurement parameters that are transmitted by the supply unit to the evaluation device.

15. A method for image-based support of a user, comprising: capturing an area to be treated in which a treatment medium can be applied at alterable locations; determination of a treatment trace of the treatment medium relative to the area; determination of a spatially resolved dosage for the area to be treated based on the treatment trace; representation of the treatment trace and/or the spatially resolved dosage with reference to the area to be treated.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Additional details of advantageous embodiments of the invention are apparent from the drawings, the description as well as the claims. In the drawings show:

[0045] FIG. 1 is a schematic view of a device for image-based support of a user in accordance with an embodiment of the invention;

[0046] FIG. 2 is a schematic view of another device for image-based support of the user in accordance with an embodiment of the invention;

[0047] FIG. 3 is a schematic view of another device for image-based support of a user in accordance with an embodiment of the invention′

[0048] FIGS. 4a to 4c are examples of an area to be treated with examples of associated live images of this area in which the treatment trace is illustrated;

[0049] FIG. 5 is an example for a live image of an area to be treated;

[0050] FIG. 6 is a flow diagram of a method for image-based support of a user in accordance with an embodiment of the invention; and

[0051] FIG. 7 is another example for a live image of an area to be treated with predicted position for the treatment medium.

DETAILED DESCRIPTION

[0052] In FIG. 1, a device 1 is illustrated that serves to support a user of a (medical) instrument 2 by means of imaging.

[0053] The device 1 comprises an instrument 2 by which a user can apply a treatment medium P in an area to be treated B of a patient. The instrument 2 can be moved freely in the space by the user. The movement of the instrument can follow practically arbitrary paths in the space x, y, z and can thereby also be pivoted in arbitrary directions. By moving and/or rotating the instrument 2, the user can vary the location at which the treatment medium P impacts onto the tissue surface in an area to be treated B.

[0054] At a distal end 7 of instrument 2, the instrument 2 comprises a head 6 for generating a treatment medium, for example a plasma, particularly a non-equilibrium plasma (cold plasma). The plasma exits the distal end 7 of instrument 2 and impacts on the tissue surface. The distal end 7 of instrument 2 is the end which is closer to the tissue surface when a treatment medium is applied. Conversely, the proximal end 8 of instrument 2 is the end that is farther away from the tissue surface when a treatment medium is applied. The tissue surface can be, for example, skin, mucosa or wounds in the skin or mucosa. The instrument 2 is supplied by a supply unit 9 comprising an RF generator with electrical power and a suitable gas, for example argon.

[0055] The area to be treated B is captured in this embodiment by means of an image capture device 3 at least partly in a field of view (FoV) of the image capture device 3. For example, the image capture device can be a (video) camera. Preferably, the image capture device 3 can be configured to create an image sequence comprising many individual images, wherein the image capture device 3 can also produce a current individual image (live image) of the area to be treated B. The image capture device 3 can be arranged immovably or moveably (handheld) relative to the area to be treated B. If the image capture device 3 is movably arranged relative to the area B, the field of view FoV can move relative to the area B during the treatment.

[0056] The image capture device 3 is communicatively connected with an evaluation device 4. The evaluation device 4 is configured to receive and evaluate data from the image capture device 3. In the evaluation device 4, a treatment trace T of the treatment medium P relative to the area B to be treated can be determined from data of the image capture device 3. Based on the treatment trace T a spatially resolved dosage D for the area to be treated can then be determined in the evaluation device 4. If the image capture device 3 is arranged movably relative to the area B, in the evaluation device 4, for example, characteristic features 11 of the area B to be treated can be recognized in the individual images of the image sequence and the movements of the image capture device 3 can be compensated.

[0057] The evaluation device 4 is in addition communicatively connected with a representation device. In this embodiment, the representation device 5 is configured as display device 5a, for example as a screen. In the present example, the representation device 5 is configured to represent the treatment trace T for the user optically virtually on the image of the tissue. The representation device 5 can modify the transparency or color intensity of the shown treatment trace T according to the local dosage D. Thus, the influence of the medium on the tissue that is per se traceless is made visible in form of a trace.

[0058] In FIG. 1, the treatment trace is shown in a live image of the area to be treated spatially accurately. In the live image in FIG. 1, the characteristic features 11 are also shown. The evaluation device 4 is in addition configured to recognize the characteristic features 11 in the individual images of the image sequence and to calculate the treatment trace T relative to the position of the characteristic features for a current individual image (live image) of the image sequence and to show it therein.

[0059] In addition or as an alternative, a reference marker 12 can be adhesively attached on the tissue surface in the area B to be treated. For example, the reference marker 12 can have an optical pattern that can be remarkably easily recognized by the evaluation device 4. In a preferred embodiment, two or more reference markers are adhesively attached onto the tissue surface in order to allow an unambiguous determination of the spatial orientation of each individual image.

[0060] In FIG. 2, a second device in accordance with an embodiment of the invention is illustrated. Instead of the display device 5a, in this example the treatment trace T and the spatially resolved dosage is directly projected into the area B to be treated by means of a projection device 5b. For example, the projection device 5b can comprise a laser or light emitting diode as light source as well as additional optical components such as lenses and mirrors in order to illustrate the treatment trace T respectively.

[0061] Alternatively or additionally, the representation device 5 can also comprise augmented reality (AR) glasses 5c that can also be communicatively connected with the evaluation device 4, for example in a wired or wireless manner. The AR glasses 5c are configured to project the spatially resolved dosage of the area to be treated in a spatially accurate manner into the field of view of the glasses.

[0062] A third device in accordance with an embodiment of the invention is illustrated in FIG. 3. The example of FIG. 3 corresponds substantially to the embodiment of FIG. 1 with the difference that the evaluation device 4 is communicatively connected with the supply unit 9 comprising, amongst others, an RF generator as well as a gas supply and supplies instrument 2.

[0063] Due to this communicative connection, the evaluation device 4 can access measurement and control data of the supply unit 9 and use them for the calculation of the dosage. For example, the measurement or control data can be the ignition detection, the power, the output power, the current, the voltage or the resistance for the RF generator or in case of the gas supply unit, the gas flow or the gas pressure.

[0064] In a possible alternative embodiment, values of ideal dosage and/or limit values for a minimum and/or a maximum dosage of the tissue can be stored in the evaluation device. The values can either be preset in a non-variable manner or defined by an appropriate input device by the user manually, for example adapted to the patient or the intervention that has to be carried out.

[0065] The evaluation device 4 can be configured in this embodiment to vary the application of the treatment medium P depending on the determined local dosage for the region in which the head 6 of instrument 2 is currently located. In this manner, for example, the application of the treatment medium by the instrument can be reduced if the currently treated region is already close to the stored, desired or maximum dosage, that is, if already 70% or 80% or 90% of the desired or maximum dosage are achieved, for example. The application of the treatment medium can also be completely switched off, if the area to be treated has already been treated with the desired or maximum dosage. If the head of the instrument is subsequently moved over a tissue region where the desired or maximum dosage has not been achieved yet, the application of the treatment medium is again increased or allowed. For example, this can be achieved in that the evaluation device 4 provides information about the treatment trace T or the local dosage D to a control unit of the RF generator in the supply unit 9. It is also possible that the evaluation device 4 sends a signal to the control unit for reducing or increasing the power or also for switching off or switching on the power that is supplied to the instrument 2 by the RF generator of the supply unit 9.

[0066] The spatially resolved dosage can be calculated from the residence time of the treatment medium P at the individual locations inside the area B to be treated and the adjusted treatment parameters, such as a treatment mode and/or a treatment power. In a simplified embodiment, also only the residence time can be used for calculation of the spatially resolved dosage, if it can be presumed that a predefined treatment mode and/or a predefined treatment power is used for the intervention.

[0067] In FIGS. 4a to 4c, an example for an area B to be treated is illustrated together with the associated live image.

[0068] On the left side in FIGS. 4a to 4c, the area to be treated B is shown respectively, wherein the plasma P is applied on the tissue surface G by means of the instrument 2. On the right side in FIGS. 4a to 4c, the associated live image is illustrated respectively.

[0069] In the live image in FIG. 4a, the current position of the plasma P relative to the area B to be treated has been determined by the evaluation device 4. In addition, in the live image, the treatment trace T is illustrated, that is the time dependent progress of the position of the plasma P inside the area B to be treated.

[0070] Alternatively or additionally, instead of the treatment trace T also the dosage D can be shown in the live image. This is for example achieved in that the color or the transparency of the shown treatment trace T in the live image is varied dependent on the local dosage D.

[0071] The example of the area B to be treated of FIG. 4a is shown in FIG. 4b. Compared with FIG. 4a, the instrument 2 has been moved further so that the location at which the instrument 2 applies the treatment medium P has been changed. In FIG. 4b, the treatment medium P is obscured by head 6 of instrument 2. For the evaluation device 4 it is for this reason not possible to directly find the treatment medium P in the current individual image of the image sequence. In this case, the evaluation device 4 determines the position and the orientation of head 6 of instrument 2 relative to the area B to be treated.

[0072] For this purpose, the evaluation device 4 determines, for example, a main axis A of head 6 of the instrument 2 and, based on the main axis A, a distal direction of the head 6 is determined. Based on the position and the orientation of head 6, the evaluation device 4 determines the position of the obscured treatment medium P. In an embodiment, the evaluation device 4 can use the distal end 7 of head 6 of instrument 2 as position of the treatment medium P on the tissue surface.

[0073] In another embodiment, the evaluation device can determine the position of the treatment medium P by extrapolation of the position of the head 6 along the orientation of instrument 2 characterized by the main axis A of instrument head 6. In doing so, also in case of obscured treatment medium P in the current individual image, a precise determination of the position of a treatment medium can be achieved.

[0074] The area B to be treated of FIGS. 4a and 4b is shown in FIG. 4c, whereby the instrument 2 has been moved by the user so that the current location has been changed at which the instrument applies the treatment medium P.

[0075] In FIG. 5, an enlarged section of a live image is illustrated schematically. The area B to be treated shown in the live image is separated into a multiplicity of resolution elements C11 to Cmn. For each resolution element a dosage D is determined that is calculated based on the treatment trace T. For example, in FIG. 5 no treatment medium P has been applied in the resolution element C11, because the treatment trace T does not extend through the resolution element C11. Therefore the dosage D11 of resolution element C11 has the value 0. The treatment trace T extends in this example partly through the resolution element C12. For the resolution element C12, the evaluation device 4 can therefore determine a measure for the dosage D12 from the time duration during which the treatment medium P is located inside the resolution element C12 on one hand. On the other hand, the evaluation device 4 can also use the portion of the area of the resolution cell on which the treatment medium P has been applied for the measure. This can be advantageous, particularly for the accuracy of the dosage D12 in an edge region of the treatment medium P. For example, if in the current individual image the position of the treatment medium P is determined as center point of the detected treatment medium P and a border of treatment medium P is placed around the determined center point in the current individual image, the border of treatment medium P can extend between multiple resolution cells so that particularly in the edge region of some resolution cells, an area portion is treated by the treatment medium P. Preferably, a combination of both ways is advantageous, in which the time dependent residence time of the treatment medium P in the resolution element as well as the treated area portion is considered. In FIG. 5, the evaluation device 4 can determine the current position 13 of the treatment medium P as center point of the at least substantially circular region, for example, in which the treatment medium P is currently applied. Also, for example, the evaluation device 4 can determine the diameter 14 of this range as size of the currently applied treatment medium.

[0076] Also, the evaluation device 4 can use measurement values of the control of instrument 2, for example an ignition recognition, a current, a voltage, a power or a resistance, to improve the accuracy of the position progress of the treatment medium P (treatment trace T) and the dosage determined therefrom. A relation between the measurement values and the treatment trace can be obtained by means of time information.

[0077] In FIG. 6, a flow diagram of an example of a method for image-based support of a user according to the invention carried out by the evaluation device 4 and the other components of device 2 is shown.

[0078] In method step V1, an area B to be treated is captured, inside which treatment medium P can be applied at manually alterable locations. For example, the area B can be captured in an image sequence comprising a multiplicity of individual images. For the further processing steps of the trace loop (tracking loop), a current individual image is now used, for example (method step V11).

[0079] In method step V2, a treatment trace T of treatment medium P relative to the area B can be determined. For this purpose, it can be first determined in a method step V20, whether the field of view FoV of the image capture device 3 has been moved relative to the area B in the current individual image. A movement of the field of view FoV can occur, for example, due to a movement, such as a translation and/or rotation, of the image capture device 3 relative to the tissue surface or vice versa. The movement can be determined by comparison of the position and location of characteristic features 11 in the current individual image with the position and location of the characteristic features 11 in a previous individual image. In this manner, occurring movements can be determined quantitatively and can be compensated. Subsequently, the treatment medium P can be searched in the current individual image and position 13 and/or the size 14 of the currently applied treatment medium P can be determined (method step V21). If the treatment medium P cannot be found in the current individual image (query in method step V22) the head 6 of instrument 2 can be searched and the position, the orientation and/or the size of head 6 can be determined (method step V23).

[0080] If the treatment medium P has been found in the current individual image, that is a current position 13 and/or size 14 of treatment medium P has been determined for the current individual image, the treatment trace T can be determined or updated in method step V25. For this purpose, for example, the determined, current position 13 as well as the size of the treatment medium P can be stored for each respective individual image so that a treatment trace T results in the image sequence from the positions 13 and sizes 14 of the individual images as a whole. The spatially resolved dosage D can now be determined by an evaluation of the spatial progress and preferably the time dependent progress of the position and size of the treatment medium (treatment trace) (further on method step V25). If the position of the treatment medium P has not been found directly in a current individual image, but the position of head 6 of instrument 2 has been determined (query in method step V24), the position of the treatment medium P can be determined (estimated) based on the position, the orientation and/or the dimension of head 6 of instrument 2. If a position of head 6 of instrument 2 can neither be found, the method is continued in method step V11, in which now a new current individual image is used as database for the substeps V21 to V25 of method step V2.

[0081] It is also possible that the steps V21/V22 and V23/V25 are exchanged in their order. First, the position, the orientation and/or the size of head 6 can be determined. Based on these parameters, a search area in which the search routine searches the treatment medium P in the current individual image can now be varied. Preferably, the search area is now limited to the region adjacent to the distal end 7 of head 6. In doing so, it is no longer necessary to search through the entire current individual image for the treatment medium P.

[0082] In addition, it is possible to carry out the method steps V21 and V23 in parallel.

[0083] In a simplified embodiment, instead of the position of the treatment medium P (substeps V21 and V22), also only the position of head 6 (substep V23) can be used as measure for the position of the treatment medium P. Therefore, in this example, only the substeps V23 to V25 of method step V2 are carried out.

[0084] The treatment trace T is preferably updated in method step V25 according to the estimated position of the treatment medium P.

[0085] Based on the treatment trace T, a spatially resolved dosage of the area B to be treated can be determined in method step V3.

[0086] In method step V4, the treatment trace T and/or the spatially resolved dosage D can now be illustrated in relation to the area B to be treated, in that the treatment trace is shown in the live image.

[0087] After method step V4, method step V11 follows again in which a new current individual image is used as data base for the method steps of the trace loop.

[0088] In FIG. 7, an enlarged section of a live image is schematically illustrated. In this embodiment, the evaluation device is configured to determine a predicted position Pos*_t of treatment medium P for the current individual image. The predicted position Pos*_t can be determined based on the previous position Pos*_t−1 and/or the previous velocity V_t−1 of the treatment medium P. In this example, the search area S in which the search routine recognizes the treatment medium in the current individual image is defined around the predicted position Pos*_t. The position Pos_t of treatment medium P detected in the current individual image is located inside the search area S.

[0089] Alternatively, the defined search area S can also be used to eliminate wrong positive mismatches of detections fp in which treatment medium P is detected outside the search area S, for example due to light reflections.

[0090] In addition, the possibility exists that the size of the search area S is adapted depending on the velocity V_t−1. If the velocity V_t−1 is high, the search area S is made larger. However, if the velocity V_t−1 is low, the search area S can also be smaller.

[0091] An embodiment of the operation of the device 1 for image-based support follows:

[0092] For carrying out a treatment, the user can apply plasma with the instrument 2, for example a plasma applicator, in a handheld manner in an area to be treated on a tissue surface in a desired dosage. During the treatment, the user can, for example, look at the display device 5a, on which the area to be treated of the tissue surface is shown and/or can wear AR glasses 5c. During the treatment, in addition, the treatment trace T and/or the dosage D can be projected in the area to be treated onto the tissue surface in a manner visible for the user.

[0093] While the treatment is carried out it is now indicated to the user which locations of the area to be treated have already been treated with plasma. Preferably the dosage that the individual locations inside the area to be treated have been received is in addition indicated to the user. The user is therefore enabled to apply plasma on non-treated locations or locations with a too low dosage.

[0094] For example, in case a treatment of a cervical intraepitheleal neoplasia (CIN), an area of the mucosa surface to be treated is monitored by a video coloscope. In this case, the evaluation device can be connected with the usually already present video coloscope so that the evaluation device can evaluate the (video)data of the video coloscope. Similarly, in other interventions, the camera of an endoscope or laparoscope can be used, for example.

[0095] The treatment of a CIN can be carried out, for example, in that a non-thermal plasma is applied onto the mucosa surface. The data of the video coloscope can now be used by the evaluation device in order to determine the time dependent progress of the position of the plasma (treatment trace) and therefrom the spatially resolved dosage. The treatment trace or spatially resolved dosage can accordingly be displayed on a screen of the video coloscope and/or on other screens.

[0096] The device 1 according to an embodiment of the invention serves for image-based support of an instrument 2. The instrument 2 is configured to apply a treatment medium at alterable locations in an area to be treated of a tissue surface. In addition, the device comprises an image capture device that is configured to capture the area to be treated. The device 1 comprises an evaluation device 4 that is configured to determine a treatment trace of the treatment medium relative to the area to be treated from data of the image capture device and to determine a spatially resolved dosage for the area to be treated based on the treatment trace. The device 1 also comprises a representation device 5 that is configured to display the treatment trace and/or the spatially resolved dosage optically for the user, relative to the tissue. With the device 1 according to the invention, it is possible to optically indicate to the user which locations of the area to be treated of the tissue surface have already been treated and/or the dosage with which the respective treated locations have been treated.