Method for Acquiring Annotated Data with the Aid of Surgical Microscopy Systems

Abstract

A method for acquiring annotated data with the aid of surgical microscopy systems comprises obtaining desired criteria which are intended to be satisfied by desired data to be annotated, and storing the set of desired criteria in a plurality of surgical microscopy systems. In each surgical microscopy system, images are then recorded and current criteria which correspond to the recorded images are determined. The current criteria are compared with the desired criteria. If the desired criteria sufficiently correspond to the current criteria, a confirmation is requested from a user as to whether said user would like to annotate data. If the user provides the confirmation, annotations for images are received from the user and stored together with the images.

Claims

1. Method for acquiring annotated data with the aid of surgical microscopy systems, comprising: obtaining at least one set of desired criteria which are intended to be satisfied by desired data to be annotated; storing the set of desired criteria in a plurality of surgical microscopy systems; in each surgical microscopy system of the plurality of surgical microscopy systems: recording images by means of a microscope of the surgical microscopy system and determining a set of current criteria which correspond to the recorded images, comparing the set of current criteria with the stored sets of desired criteria, if a set of the desired criteria sufficiently corresponds to the set of current criteria: requesting a confirmation from a user of the surgical microscopy system as to whether said user would like to annotate data; if the user provides the confirmation, receiving at least one annotation for at least one image from the user and storing the at least one image together with the at least one annotation.

2. Method according to claim 1, wherein the desired criteria specify a type of a medical intervention, an age, a sex and/or a state of health of a patient on whom the medical intervention is performed, a quality of tissue imaged in the images, a characteristic of the user of the surgical microscopy system, a location at which the surgical microscopy system is situated, and/or a use history of the surgical microscopy system.

3. Method according to claim 1, wherein determining the set of criteria which correspond to the recorded images comprises requesting at least one criterion from the user, and/or obtaining at least one criterion from an information management system of a hospital in which the surgical microscopy system is situated, and/or of a company which manufactured the surgical microscopy system, and/or determining at least one criterion on the basis of the recorded images.

4. Method according to claim 1, wherein the set of the desired criteria sufficiently corresponds to the set of current criteria if an intersection of the criteria has a number of criteria that satisfies a predetermined relation with respect to a number of the criteria of the set of desired criteria.

5. Method according to claim 1, wherein the annotation comprises a designation indicating a quality of tissue imaged in the images, and/or comprises at least one graphical marking within a recorded image.

6. Method according to claim 1, wherein the at least one image is stored together with the at least one annotation in a database that is shared by all the surgical microscopy systems.

7. Method according to claim 1, furthermore comprising analysing the images stored together with the annotations.

8. Method according to claim 7, wherein the analysing comprises generating a classifier with which at least one criterion of the set of desired criteria is determinable on the basis of an image which is recorded by a surgical microscope and which is different from the images stored together with the annotations.

9. System for acquiring annotated data, comprising: a management unit; and a plurality of surgical microscopy systems; wherein the management unit comprises: a communication unit configured for obtaining at least one set of desired criteria which are intended to be satisfied by desired data to be annotated; transmitting the at least one set of desired criteria obtained to the plurality of surgical microscopy systems; a database configured for storing the at least one set of desired criteria obtained; wherein each of the plurality of surgical microscopy systems comprises: a communication unit configured for receiving the at least one set of desired criteria transmitted by the management unit; a storage unit configured for storing the at least one received set of desired criteria; a microscope configured for recording images; a data processing apparatus configured for determining a set of current criteria which correspond to the recorded images, comparing the determined set of current criteria with the at least one set of desired criteria stored in the storage unit, requesting a confirmation from a user of the surgical microscopy system as to whether said user would like to annotate data if the comparing reveals that a set from the at least one set of the desired criteria sufficiently corresponds to the determined set of current criteria; a user interface configured for receiving the confirmation from the user, and, if the user provides the confirmation, receiving at least one annotation for at least one image from the user; wherein the data processing apparatus is furthermore configured for storing the at least one image together with the at least one annotation.

10. System according to claim 9, wherein the desired criteria specify a type of a medical intervention, an age, a sex and/or a state of health of a patient on whom the medical intervention is performed, a quality of tissue imaged in the images, a characteristic of the user of the surgical microscopy system, a location at which the surgical microscopy system is situated, and/or a use history of the surgical microscopy system.

11. System according to claim 9, wherein the data processing apparatus of the respective surgical microscopy system is configured, for the purpose of determining the set of current criteria which correspond to the recorded images, to carry out requesting at least one criterion from the user, and/or obtaining at least one criterion from an information management system of a hospital in which the surgical microscopy system is situated, and/or of a company which manufactured the surgical microscopy system, and/or determining at least one criterion on the basis of the recorded images.

12. System according to claim 9, wherein the data processing apparatus of the respective surgical microscopy system is furthermore configured to determine that the at least one set of the desired criteria sufficiently corresponds to the set of current criteria if an intersection of the criteria has a number of criteria that satisfies a predetermined relation with respect to a number of the criteria of the set of desired criteria.

13. System according to claim 9, wherein the annotation comprises a designation specifying a quality of tissue imaged in the images, and/or comprises at least one graphical marking within a recorded image.

14. System according to claim 9, furthermore comprising: a database that is shared by all the surgical microscopy systems; wherein the data processing apparatus of the respective surgical microscopy system causes the communication unit of the respective surgical microscopy system to transfer the at least one image together with the at least one annotation to the database; wherein the database is configured to store the received at least one image together with the at least one annotation.

15. System according to claim 9, furthermore comprising: a computer-implemented analyser configured for analysing the images stored together with the annotations.

16. System according to claim 15, wherein the computer-implemented analyser is configured, for the purpose of analysing the images stored together with the annotations, to carry out generating a classifier with which at least one criterion of the set of desired criteria is determinable on the basis of an image which is recorded by a surgical microscope and which is different from the images stored together with the annotations.

Description

DESCRIPTION OF DRAWINGS

[0017] Embodiments of the disclosure are explained below with reference to figures, in which:

[0018] FIG. 1 shows a flow diagram of a method for acquiring annotated data in accordance with one embodiment;

[0019] FIG. 2 shows one example of an annotated image;

[0020] FIG. 3 shows a detail from FIG. 1;

[0021] FIG. 4 shows a schematic illustration of a management unit in a system for acquiring annotated data;

[0022] FIG. 5 shows a schematic illustration of a surgical microscopy system in the system shown in FIG. 4; and

[0023] FIG. 6 shows a schematic illustration of an analyser.

DETAILED DESCRIPTION

[0024] FIG. 1 is a diagram for elucidating a method for acquiring annotated data with the aid of surgical microscopy systems. In a step 3, a set of desired criteria is obtained by a user who desires annotated data inputting this set of desired criteria via a suitable user interface. In this case, inputting the set of the desired criteria can also comprise a selection from predetermined criteria. The desired criteria can specify for example the type of the desired data. Examples thereof are “image”, “video”, “fluorescence”, “OCT” and the like. The desired criteria can furthermore specify for example the type of the desired annotations. Examples thereof are “tissue designation”, “contour line”, “arrow marking” and the like. The desired criteria can furthermore specify for example the type of the desired medical intervention. Examples thereof are “neuro”, “vascular”, “aneurism”, “tumour”, “glioblastoma”, “meningioma”, “spinal column”, “eye”, “anterior chamber of the eye”, “posterior chamber of the eye”, “cataract”, “retina”, “membrane peeling” and the like. The desired criteria can furthermore specify for example the desired phase of a medical intervention. The desired criteria can furthermore specify a desired characteristic of the patient. Examples thereof are “male”, “female”, “20-30 years old”, “25-35 years old”, “pre-existing disease X” and the like. The desired criteria can furthermore specify a desired characteristic of the user. Examples thereof are “5-10 years of professional experience” and the like. The desired criteria can furthermore specify a desired state of the surgical microscopy system. Examples thereof are “microscope is in the fluorescence mode”, “set magnification is greater than 10 fold” and the like.

[0025] The set of desired criteria that has been input in step 3 is subsequently stored in a database 5 for desired criteria. Further sets of desired criteria may also already have been stored or will subsequently be stored in said database 5 for desired criteria. The sets of desired criteria that have been stored in the database 5 for desired criteria are communicated to a plurality of surgical microscopy systems 7. This communication can be effected for example in such a way that each surgical microscopy system 7 obtains a copy of the database 5 for desired criteria. However, it is also possible that each of the surgical microscopy systems 7 as necessary can access the database 5 for desired criteria in order to obtain the sets of desired criteria.

[0026] The surgical microscopy systems 7 typically comprise a surgical microscope 73 with the microscopy optics and with sensors, such as cameras or OCT scanners, for instance, a stand for mounting the surgical microscope 73 and a control system for controlling the surgical microscope 73 and for providing functionalities. The control system can provide user interfaces, databases and remote data connections, inter alia. Not all of the components of the surgical microscopy system 7 have to be arranged together with the surgical microscope 73 thereof in a room, for example.

[0027] The database 5 can be a database system comprising a plurality of hardware and software modules, which can be arranged in a distributed manner. By way of example, each surgical microscopy system 7 can comprise a software module of the database system which runs on the hardware of the surgical microscopy system and enables the interaction of the surgical microscopy system 7 with the database system 5. The data sets can be stored in data memories of the database system which are arranged at locations which differ from the locations at which the surgical microscopes of the surgical microscopy systems 7 are arranged.

[0028] During the operation of the plurality of surgical microscopy systems 7, as described below, annotated data are acquired, which are stored in a database 9 for annotated data. In a step 11, the annotated data that have been stored in the database 9 for annotated data are analysed in order, on the basis of the analysis, to create a classifier 13, which is one example of a machine learning tool created on the basis of the annotated data acquired.

[0029] FIG. 2 schematically illustrates one example of annotated data which can be acquired using a surgical microscopy system 7. FIG. 2 shows an image 15 which was recorded by the surgical microscope 73 of the surgical microscopy system 7. In the image 15, lines 17 represent structures of body tissue 19 of a patient on whom a medical intervention is performed by means of the surgical microscopy system 7. Part of the tissue 19 contains a tumour, which is recognized by the experienced user of the surgical microscopy system in the image 15. The user uses a mouse of the user interface of the surgical microscopy system 7 in order to insert a line 21 into the image 15, said line identifying a contour of the tumour in the tissue 19. In addition, the user inputs a designation 22 for the tissue identified by the line 21 as a series of letters, which is represented by “ABC” in FIG. 3 and is connected to the line 21 via a curved connecting line 23. However, it is also possible to represent the series of letters in a different way in association with the line 21. The line 21 and the series of letters 22 thus together form an annotation of data provided by the image 15. A multiplicity of such annotated data comprising different images of different patients and different annotations can subsequently be used to train a machine learning instrument which is able, in new tissue images, to recognize and highlight the regions containing tumours.

[0030] FIG. 3 is a flow diagram for elucidating part of the method illustrated in FIG. 1, which proceeds in a surgical microscopy system 7 of the plurality of surgical microscopy systems.

[0031] The method is started in a step 31 at the beginning of an intervention which is performed on a patient and which is visualized by means of the surgical microscopy system 7. A video stream 33 generated by a camera of the surgical microscope of the surgical microscopy system is then analysed in a step 35 in order to determine current criteria. Besides the video stream 33, other sources of information, such as, for example, a database of the hospital in which the surgical microscopy system 7 is installed, can also be used for determining the current criteria in step 35.

[0032] The current criteria can furthermore comprise the instantaneous state of the surgical microscope 73. Said state can be ascertained by reading sensors and comprises for example the type of illumination used, filters used in the illumination beam path and in the analysis beam path in order to capture specific types of fluorescence examinations. Furthermore, by way of example, position sensors of the stand of the surgical microscopy system can be read in order to deduce the type of the intervention currently being carried out. Furthermore, the images recorded by cameras of the surgical microscope 73 of the surgical microscopy system can be analysed in order to recognize therein for example typical types of tissue, such as blood vessels, for example, which in turn allows the type of the intervention carried out to be deduced.

[0033] In a step 37, the current criteria determined in step 35 are compared with the desired criteria stored in the database 5 for desired criteria. If sufficient correspondence is not established in step 37, step 35 is repeated in order to make a new attempt, on the basis of the video stream 33, to determine subsequently possibly other current criteria which may result in correspondence in step 37.

[0034] If sufficient correspondence is found in step 37, in a step 39 a confirmation 41 from a user of the surgical microscopy system is requested, obtained and checked. If the confirmation obtained comprises the fact that the user does not agree to annotate data, the method is ended in a step 43.

[0035] By contrast, if the user agrees to annotate data, said user is given the opportunity to do this by the provision of the suitable user interface. In a step 45, an annotation 47 can then be obtained from the user. The annotation 47 can consist, for example, of the input—illustrated in FIG. 3—of the line 21 and the series of letters 22 with respect to an image 15 of the video stream 33. A step 49 then involves checking whether the user has input a termination 51 via the user interface. If this is not the case, further annotations 47 with respect to further images 15 of the video stream 33 are obtained in step 45. If the user terminates the further input of annotations 47 in step 49, a step 53 involves storing the annotated images 15 together with the annotations 21, 22 or the entire video stream 33 together with the annotations in the database 9 for annotated data.

[0036] As soon as enough annotated data have been stored in the database 9 for annotated data, a machine learning instrument can be trained using these data and annotations. One example of such a machine learning instrument is a classifier, the algorithm of which comprises for example a neural network, a decision tree or the like. Furthermore, a clinical hypothesis can be validated on the basis of the annotated data. Furthermore, a new classifier can also be determined on the basis of the annotated data, which new classifier can be used to determine the current criteria in step 35. The new classifier can be transferred to the surgical microscopy system in order to replace the existing classifier there.

[0037] FIG. 4 shows a schematic illustration of a system for acquiring annotated data. The system 1 comprises a management unit 100 and a plurality of surgical microscopy systems 7.

[0038] In particular, FIG. 4 shows a schematic illustration of the management unit 100 used to make possible the input of a new set of desired criteria and the storage and management thereof. The management unit 100 comprises for example a communication unit 101 and the database for desired criteria 5. The database can be implemented by a storage unit.

[0039] The communication unit 101 serves for communication with other components of the overall system and the user. In this regard, the communication unit 101 can comprise input and output units, for example, which allow the user to input and display the sets of desired criteria.

[0040] The database for desired criteria 5 stores the sets of desired criteria that have been input. In the example in FIG. 4, three different sets A, B and C of desired criteria have been stored in the database for desired criteria 5.

[0041] The communication unit 101 is furthermore configured to transmit one or more sets of desired criteria that have been stored in the database for desired criteria 5 to the surgical microscopy systems 7.

[0042] FIG. 5 shows a schematic illustration of one of the surgical microscopy systems 7 of the system 1. The surgical microscopy system comprises a communication unit 71, a storage unit 72, a surgical microscope 73, a data processing apparatus 74 and a user interface 75. The components of the surgical microscopy system 7 are connected to one another via a communication network.

[0043] The communication unit 71 serves for communication with other components of the overall system. For example, the communication unit 71 is configured to receive one or more sets of desired criteria from the management unit 100.

[0044] The storage unit 72 stores the sets of desired criteria that have been received by the communication unit 71. In the present example, not all of the sets stored in the database for desired criteria 5 were transferred to the surgical microscopy system 7, but rather only the sets A and B. The selection and management of the sets of desired criteria to be used in a surgical microscopy system 7 can for example be performed by the management unit 100 and be dependent on the constitution of the respective surgical microscopy system 7.

[0045] The surgical microscope 73 serves for recording images.

[0046] The data processing apparatus 74 is configured to carry out the method described in FIG. 2.

[0047] The user interface 75 serves for communication with the user. In this regard, the user interface 75 can comprise input and output units, for example, which make it possible to display the images recorded by the surgical microscope 73 for a user and to receive user-effected annotations and other inputs and to pass them on to the data processing apparatus 74.

[0048] FIG. 6 shows a schematic illustration of an analyser 10 configured to carry out the analysis 11. For this purpose, the analyser 10 receives annotated data from the database for annotated data 3 and analyses them. The result of the analysis 11 is an improved classifier 13, for example, which is transmitted to the surgical microscopy systems 7.