METHOD FOR OPERATING AN AUGMENTED REALITY OBSERVATION SYSTEM IN A SURGICAL APPLICATION, AND AUGMENTED REALITY OBSERVATION SYSTEM FOR A SURGICAL APPLICATION

Abstract

The invention relates to a method for operating an augmented reality observation system in a surgical application, wherein a viewing direction of a user is registered by means of a viewing direction sensor system of an AR observation apparatus, wherein the registered viewing direction is evaluated by means of a control device, and wherein at least one property of at least one controllable light source in the environment is altered by means of the control device by means of a control signal on the basis of the registered viewing direction. Further, the invention relates to an augmented reality observation system for a surgical application.

Claims

1. A method for operating an augmented reality observation system in a surgical application, wherein a viewing direction of a user is registered by means of a viewing direction sensor system of an AR observation apparatus, wherein the registered viewing direction is evaluated by means of a control device, and wherein at least one property of at least one controllable light source in the environment is altered by means of the control device by means of a control signal on the basis of the registered viewing direction.

2. The method as claimed in claim 1, wherein the at least one property is altered if the viewing direction is determined during the evaluation as being directed in the direction of a region in the environment lit or illuminated by the at least one controllable light source or if a viewing direction previously directed at such a region has departed from the latter again.

3. The method as claimed in claim 1, wherein the at least one property of the at least one controllable light source is altered such that an image which is registrable by the user of the AR observation apparatus and which is of the region lit or illuminated by the at least one controllable light source satisfies at least one specified criterion.

4. The method as claimed in claim 3, wherein the at least one specified criterion defines a value range of at least one image parameter of the registrable image.

5. The method as claimed in claim 2, wherein an environment sensor system of the AR observation apparatus is used to register a region of the environment that is registrable by the user, wherein lit or illuminated regions of the at least one controllable light source are identified based on sensor data that correspond to the registered region.

6. The method as claimed in claim 2, wherein whether a region lit or illuminated by the at least one controllable light source is in the registered viewing direction is determined in consideration of a three-dimensional model of the at least one controllable light source and/or the environment.

7. The method as claimed in claim 2, wherein at least one light property of light which is registrable by a user from a direction of the registered viewing direction and which emanates from the region lit or illuminated by the at least one controllable light source is registered by means of the environment sensor system, wherein the change of the at least one property of the at least one light source is implemented in consideration of the registered at least one light property.

8. The method as claimed in claim 2, wherein at least one light property of light which is registrable by a user from a direction of the registered viewing direction and which emanates from the region lit or illuminated by the at least one controllable light source is determined on the basis of control data and/or state data of the at least one controllable light source, wherein the change of the at least one property of the at least one controllable light source is implemented in consideration of the determined at least one light property.

9. The method as claimed in claim 1, wherein the at least one property of the at least one controllable light source is additionally altered in consideration of at least one transmission property of the AR observation apparatus.

10. An augmented reality observation system for a surgical application, comprising: an augmented reality observation apparatus with a viewing direction sensor system, wherein the viewing direction sensor system is configured to register a viewing direction of a user of the AR observation apparatus, and a control device, wherein the control device is configured to evaluate the registered viewing direction and to alter at least one property of at least one controllable light source in the environment on the basis of the registered viewing direction by means of a control signal.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The invention is explained in greater detail below on the basis of preferred exemplary embodiments with reference to the figures. In the figures:

[0037] FIG. 1 shows a schematic illustration of an embodiment of the augmented reality observation system for a surgical application; and

[0038] FIG. 2 shows a schematic flowchart of an exemplary embodiment of the method for operating an AR observation system during a surgical application.

DETAILED DESCRIPTION

[0039] FIG. 1 shows a schematic illustration of an embodiment of the augmented reality observation system 1 for a surgical application. The AR observation system 1 is used in particular in the context of surgery on a patient in an operating room. In this case an environment 20 of the AR observation system 1 corresponds to a typical environment in an operating room, in particular. Two controllable light sources 21 are arranged in the environment 20 in exemplary fashion. One of the controllable light sources 21 is a display device 22 with a lit region 23. By way of example, the display device 22 can be a computer or appliance monitor or a freely positionable (3-D) visualization monitor of a robotic visualization system (not shown) used during surgery. The other controllable light source 21 is a lighting device 24 of a robotic visualization system (not shown) with an illuminated region 25. By way of example, the illuminated region 25 of the lighting device 24 coincides with a situs. The two controllable light sources 21 each have a light source controller 26, 27, by means of which at least one property of the controllable light sources 21 can be controlled in each case by stipulation of a corresponding control signal 31, 32.

[0040] The AR observation system 1 comprises an augmented reality observation apparatus 2 and a control device 3. The method for operating the augmented reality observation system 1 in a surgical application is described in more detail below on the basis of the augmented reality observation system 1.

[0041] The augmented reality observation apparatus 2 is configured as a pair of AR glasses and is worn on the head of a user during application. The augmented reality observation apparatus 2 comprises a viewing direction sensor system 4, which comprises a line-of-sight sensor system (not shown), for example an eye/gaze tracking device, and/or a head viewing direction sensor system (not shown), for example a head viewing direction sensor.

[0042] The control device 3 comprises a computing device 3-1 and a memory 3-2. The computing device 3-1 is configured to be able to carry out computational operations on data stored in the memory 3-2 and can, as a result thereof, carry out measures required to implement the method. By way of example, the computing device 3-1 comprises a microprocessor which for the purposes of carrying out parts of the method described in this disclosure can execute program code stored in the memory 3-2.

[0043] The viewing direction sensor system 4 of the AR observation apparatus 2 registers a viewing direction A, B, C of the user in the environment 20 by virtue of using the augmented reality observation system 1. In this case, the viewing direction A, B, C is determined and provided from the sensor data provided by the line-of-sight sensor system, for example by the eye/gaze tracking device, and by the head viewing direction sensor system, for example by the head viewing direction sensor. The registered viewing direction A, B, C is fed to the control device 3, for example as viewing direction signal 30, and is received for example by way of an interface 3-3 of the control device 3 configured to this end. By way of example, the viewing direction signal 30 comprises information regarding a direction and/or a solid angle which corresponds to the registered viewing direction A, B, C in relation to a relative position of the AR observation apparatus 2 or in relation to another coordinate system.

[0044] The control device 3 evaluates the registered viewing direction A, B, C. To this end, the registered viewing direction A, B, C is compared to for example specified directions and/or solid angles, in the case of which there should be a change in the at least one property 40, 41 of the at least one controllable light source 21.

[0045] If the registered viewing direction A, B, C is directed in such a specified direction and/or a specified solid angle, the control device 3 generates a control signal 31, 32 for altering at least one property 40, 41 of the at least one controllable light source 21. The type of the at least one property 40, 41 and a scope of change are linked to the specified direction and/or the specified solid angle, for example. By way of example, the specified directions and/or solid angles can be stored, together with the properties 40, 41 to be changed in each case, in a lookup table or database in the memory 3-2 of the control device 3 and can be queried from the latter when necessary. In a simple example, provision can be made for a luminous intensity of the at least one light source 21 to be reduced, in particular to a specified value, for specified directions and/or specified solid angles. The associated control signal 31, 32 is generated accordingly by the control device 3 and fed to the respective light source controller 26, 27 by way of interfaces 3-4, 3-5 configured to this end. The light source controller 26, 27 then alters the at least one property 40, 41 in accordance with the control signal 31, 32.

[0046] In particular, provision can be made for the at least one property 40, 41 to be altered if the viewing direction A, B, C is determined during the evaluation as being directed in the direction of a region 23 in the environment 20 lit, or a region 25 in the environment 20 illuminated, by the at least one controllable light source 21 or if a viewing direction A, B, C previously directed at such a region 23, 25 has departed from the latter again. To this end, the received viewing direction signal 30 is compared for example to solid angle ranges which, based on a relative position of the AR observation apparatus 2, coincide with the lit or illuminated regions 23, 25.

[0047] If it is determined by way of an evaluation result that the viewing direction A, B, C is directed in the direction of a region 23, 25 in the environment 20 lit or illuminated by the controllable light sources 21 then the at least one property 40, 41 of the controllable light source 21 corresponding to the region 23, 25 is altered by means of a control signal 31, 32 on the basis of the registered viewing direction A, B, C.

[0048] An example for the viewing direction A is the following: A user of the AR observation apparatus 2 gazes at displayed content of the display device 22, for example to register a live video data stream from a robotic visualization system (not shown) (i.e., there is no augmentation at this time). Since some of a luminous intensity emanating from the lit region 23 is absorbed by optical layers of the AR observation apparatus 2, an image registrable to this end by a user has a lower luminous intensity or a lower brightness behind the AR observation apparatus 2. To compensate this, the control device 3 generates a control signal 31, which increases a brightness of the display device 22 such that a luminous intensity is increased and the absorption by the optical layers of the AR observation apparatus 2 can be compensated, after determining that the viewing direction A is directed in the direction of the region 23 illuminated by the display device 22 through the AR observation apparatus 2.

[0049] An example for the viewing direction B is the following: During surgery, a user of the AR observation apparatus 2 gazes through the AR observation apparatus 2 at content of the display device 22 (i.e., there is no augmentation at this time) and carries out a surgical intervention with the aid of a robotic visualization system (not shown). To ensure an optimal illustration on the display device 22 by the robotic visualization system, the associated lighting device 24 is set to a high luminous intensity. To obtain a short, non-magnified view of the overall situs, the user briefly gazes directly at the overall situs in the illuminated region 25. This can be implemented by a head movement, with the user continuing to gaze through the AR observation apparatus 2, or by way of an eye movement by virtue of the user looking past the AR observation apparatus 2 at the bottom, corresponding to the viewing direction B illustrated in exemplary fashion. Then, it is determined for the registered viewing direction B that the viewing direction B is directed past the AR observation apparatus 2 at the bottom and directly on the illuminated region 25 of the lighting device 24, that is to say the overall situs. Since the high luminous intensity emanating from the illuminated region 25 would dazzle the user and an adaptation of the eyes would be required, the control device 3, following the determination that the registered viewing direction B is directed in the direction of the illuminated region 25, generates a control signal 32 which reduces the luminous intensity of the lighting device 24 such that the user is no longer dazzled and can register the overall situs in the illuminated region 25 without interruption or impediment. If the user subsequently gazes on the display device 22 again, in order to register the situs magnified by means of the robotic visualization system once again, the control device 3 once again generates a control signal 32 which prompts the lighting device 24 to increase the luminous intensity back to the previous value in order to ensure optimal illumination during the registration by means of the robotic visualization system.

[0050] By contrast, for the viewing direction C there is no change in properties of the controllable light sources 21 since it is directed in neither of the two regions 23, 25.

[0051] Provision can be made for the at least one property 40, 41 of the at least one controllable light source 21 to be altered such that an image which is registrable by the user of the AR observation apparatus 2 and which is of the region 23, 25 lit or illuminated by the at least one controllable light source 21 satisfies at least one specified criterion 10. As a result, images registered by the user from different information sources (environment, controllable light sources, AR content, etc.) via and/or through the AR observation apparatus 2 can be altered and more particularly optimized in view of a uniform impression. By way of example, the registrable image can be estimated on the basis of optical properties (transmission properties, reproduction properties during the reproduction of AR content, etc.) of the AR observation apparatus 2. In this case, the control device 3 checks whether or not the at least one criterion 10 is satisfied, and alters the at least one property 40, 41 of the at least one controllable light source 21 until the at least one criterion 10 is satisfied.

[0052] In a development, provision can be made, in particular, for the at least one criterion 10 to define a value range 11 of at least one image parameter of the registrable image. By way of example, such an image parameter can be a luminous intensity, a brightness, a contrast ratio and/or a color property, for example a color spectrum.

[0053] Provision can be made for the AR observation apparatus 2 to comprise an environment sensor system 5, for example a camera that registers the environment 20. An environment sensor system 5 can also be a depth sensor which registers and provides a three-dimensional image representation of the environment 20. The environment sensor system 5 is used to register a region of the environment 20 that is registrable by the user, wherein lit or illuminated regions 23, 25 of the at least one controllable light source 21 are identified based on sensor data that correspond to the registered region. In the process, use can be made, for example, of an object recognition by means of methods of computer vision and/or machine learning, in order to recognize the controllable light sources 21 and/or the associated regions 23, 25. In the process, provision can be made for the use of markers on the controllable light sources 21 in order to assist the recognition.

[0054] Provision can be made for whether a region 23 lit, or a region 25 illuminated, by the at least one controllable light source 21 is in the registered viewing direction A, B, C to be determined in consideration of a three-dimensional model 12 of the at least one controllable light source 21 and/or the environment 20. The AR observation apparatus 2 and the registered viewing direction A, B, C can be located within the three-dimensional model 12 such that, proceeding therefrom, it is possible to determine the direction in which the registered viewing direction A, B, C is directed. In this case, use can be made of computer vision and/or machine learning, for example for recognizing the lit and/or illuminated regions 23, 25 in consideration of or using the three-dimensional model 12.

[0055] Provision can be made for at least one light property 50 of light which is registrable by a user from a direction of the registered viewing direction A, B, C and which emanates from the region 23 lit, or region 25 illuminated, by the at least one controllable light source 21 to be registered by means of the environment sensor system 5, for example by means of a camera, wherein the change of the at least one property 40, 41 of the at least one light source 21 is implemented in consideration of the registered at least one light property 50. The registered at least one light property 50 can comprise a luminous intensity or brightness, a contrast ratio or a color or frequency spectrum, for example. The registered at least one light property 50 is fed to the control device 3 by the AR observation apparatus 2. Based on the registered at least one light property 50, the control device 3 determines the at least one property 40, 41 of the controllable light source 21 that should be altered, and the scope of the change. In the examples for viewing directions A and B, described above, the environment sensor system 5 can register, for example, a luminous intensity (or brightness) emanating from the regions 23, 25 as a light property 50, wherein the control device 3 controls or regulates the luminous intensity (or the brightness) of the display device 22 or of the lighting device 24 to a respectively required value proceeding therefrom.

[0056] Provision can be made for at least one light property 50 of light which is registrable by a user from a direction of the registered viewing direction A, B, C and which emanates from the region 23 lit, or region 25 illuminated, by the at least one controllable light source 21 to be determined on the basis of control data 51 and/or state data 52 of the at least one controllable light source 21, wherein the change of the at least one property 40, 41 of the at least one controllable light source 21 is implemented in consideration of the determined at least one light property 50. To this end, the control device 3 queries the control data 51 and/or state data 52 at the light source controllers 26, 27 via the interfaces 3-4, 3-5 and takes these into account when generating the control signals 31, 32. The control data 51 and/or state data 52 of the display device 22 can comprise a brightness, a contrast ratio or a color setting, for example. The control data 51 and/or state data 52 of the lighting device 24 can comprise a luminous intensity, for example.

[0057] Provision can be made for the at least one property 40, 41 of the at least one controllable light source 21 to be additionally altered in consideration of at least one transmission property 13 and/or absorption property of the AR observation apparatus 2. By way of example, the at least one transmission property 13 comprises a transmission spectrum which describes a frequency-dependent or wavelength-dependent transmission of the AR observation apparatus 2, in particular of a screen of the AR observation apparatus 2. With the aid of the transmission spectrum it is possible to estimate an image of the environment and, in particular, of the controllable light sources 21 registrable by a user, and so for example the at least one property 40 of the display device 22 can be altered in such a way that the frequency-dependent or wavelength-dependent absorption of the AR observation apparatus 2 can be compensated for. As a result, it is possible in particular to maintain a color fidelity of content displayed on the display device 22. By way of example, the at least one transmission property 13 and/or absorption property is taken from a datasheet of the AR observation apparatus 2 or determined empirically, and stored in the memory 3-2 of the control device 3.

[0058] Provision can be made for an opacity of the AR observation apparatus 2 to be additionally altered at least in one portion of a field of view on the basis of the registered viewing direction A, B, C. This allows light from non-controllable light sources (not shown) to be suppressed. By way of example, if a door to a darkened operating room is opened, disturbing light can reach the operating room from the outside and can disturb the user of the AR observation apparatus 2 in their workflow. If the viewing direction A, B, C is directed in the direction of the non-controllable light source, a position, corresponding herewith, of a screen of the AR observation apparatus 2 that is changeable in respect of its opacity is darkened such that the light of the non-controllable light source is absorbed more strongly than light reaching the AR observation apparatus 2 from surrounding regions of the environment 20.

[0059] With the aid of the AR observation system 1 for a surgical application described in this disclosure and of the method for operating the AR observation system 1 in a surgical application, it is possible to reduce or even prevent an impediment to the workflow of a user of an AR observation apparatus 2 by light sources in the environment of the surgical application.

[0060] Shown in FIG. 2 is a schematic flowchart of an exemplary embodiment of the method for operating an AR observation system during a surgical application.

[0061] In a measure 100, a viewing direction of a user is registered by means of a viewing direction sensor system of an AR observation apparatus of the AR observation system. In the process, both a line of sight and a head viewing direction are registered in particular, and so it is possible to also determine a relative viewing direction of the eyes in relation to the head. Therefore, the registered viewing direction in particular also comprises information in respect of whether a user of the AR observation apparatus gazes through or gazes past the latter.

[0062] In a measure 101, the registered viewing direction is evaluated. To this end, a check is carried out as to whether the registered viewing direction is directed in the direction of a lit or illuminated region of at least one controllable light source in the environment.

[0063] Should this be the case, a measure 102 is carried out. The measure 102 comprises measures 103 to 106. In measure 103, a luminous intensity of light emanating from the lit or illuminated region is registered by means of an environment sensor system, in particular by means of a camera, of the AR observation apparatus. The extent to which the registered luminous intensity deviates from a target value specified as a specified criterion, in particular, is determined in measure 104. In this case, the specified target value corresponds to an optimal brightness of an image which is registered by a user of the AR observation apparatus and which is of the controllable light sources in the environment. In measure 105, a control signal corresponding to the target value is generated by means of the control device and fed to the controllable light source. In measure 106, the luminous intensity of the controllable light source is altered in accordance with the control signal such that it corresponds to the target value for the luminous intensity.

[0064] By contrast, if an evaluation result in measure 101 yields that the viewing direction is not directed in the direction of a lit or illuminated region, a check is carried out in a measure 107 as to whether the registered viewing direction had previously been directed in the direction of such a region. Measure 108, which comprises measures 109 and 110, is carried out should this be the case. In measure 109, a control signal corresponding to the original value of the luminous intensity of the controllable light source is generated by means of the control device and fed to the controllable light source. In measure 110, the luminous intensity of the controllable light source is altered in accordance with the control signal such that it corresponds to the original value for the luminous intensity again.

[0065] The method is repeated cyclically such that a smooth workflow with an always optimal light intensity of the controllable light sources in the environment of the AR observation apparatus is facilitated.

[0066] The shown embodiments are merely exemplary. In particular, it is also possible to alter different properties of the at least one controllable light source to the properties described and, in particular, a plurality of properties of the at least one controllable light source.

LIST OF REFERENCE SIGNS

[0067] 1 Augmented reality (AR) observation system [0068] 2 Augmented reality (AR) observation apparatus [0069] 3 Control device [0070] 3-1 Computing device [0071] 3-2 Memory [0072] 3-3 Interface [0073] 3-4 Interface [0074] 3-5 Interface [0075] 4 Viewing direction sensor system [0076] 5 Environment sensor system [0077] 10 Specified criterion [0078] 11 Value range (image parameters) [0079] 12 Three-dimensional model [0080] 13 Transmission property [0081] 20 Environment [0082] 21 Controllable light source [0083] 22 Display device [0084] 23 Lit region [0085] 24 Lighting device [0086] 25 Illuminated region [0087] 26 Light source controller [0088] 27 Light source controller [0089] 30 Viewing direction signal [0090] 31 Control signal [0091] 32 Control signal [0092] 40 Property [0093] 41 Property [0094] 50 Light property [0095] 51 Control data [0096] 52 State data [0097] A, B, C Viewing direction [0098] 100-110 Method measures