Surgical microscope with gesture control and method for a gesture control of a surgical microscope

Abstract

The present invention relates to a surgical microscope with a field of view and comprising an optical imaging system which images an inspection area which is at least partially located in the field of view, and to a method for a gesture control of a surgical microscope having an optical imaging system. The surgical microscope further comprises a gesture detection unit for detection of a movement of a surgical instrument, the gesture detection unit having a detection zone which is located between the inspection area and the optical imaging system and is spaced apart from the inspection area, and the gesture detection unit being configured to output a control signal to the optical imaging system depending on the movement of the surgical instrument in the detection zone, the optical imaging system being configured to alter its state depending on the control signal.

Claims

1. A surgical microscope comprising: an optical imaging system including an objective defining an optical axis, wherein the optical imaging system images a field of view determined by a numerical aperture of the optical imaging system and a working distance of the objective relative to an object being observed along the optical axis; a camera which images an inspection area, wherein the field of view is a portion of the inspection area; wherein the surgical microscope further comprises a gesture detection unit for detection of a movement of a surgical instrument, the gesture detection unit being configured to receive data from the camera, the gesture detection unit being connected to the optical imaging system and configured to output a control signal to the optical imaging system depending on the movement of the surgical instrument, the optical imaging system being configured to alter its state depending on the control signal; wherein the gesture detection unit has a detection zone which is located between the inspection area and the objective along the optical axis and which detection zone is spaced apart from the inspection area and from the objective along the optical axis; wherein gesture detection of the gesture detection unit is limited to movements of the surgical instrument in the detection zone such that movements of the surgical instrument in the inspection area and in a space between the detection zone and the objective will not alter the state of the optical imaging system.

2. The surgical microscope according to claim 1, wherein the gesture detection unit is configured to detect a further type of object, the further type of object being one of a finger and a tool.

3. The surgical microscope according to claim 2, wherein gesture detection of the gesture detection unit is limited to movements of the further object in the detection zone.

4. The surgical microscope according to claim 1, wherein the gesture detection unit further comprises a movement detection module for differentiating 3-dimensional movements of the surgical instrument in the detection zone, wherein the movement detection module is connected to the gesture detection unit via a movement-data line and the movement detection module is adapted to output a movement-data signal via the movement-data line to the gesture detection unit depending on the movement of an object.

5. The surgical microscope according to claim 4, wherein the gesture detection unit further comprises a relationship mapping unit which is connected to the movement detection module and which relationship mapping unit is adapted to map relationships between a movement of the surgical instrument and a control signal, which control signal is provided via the signal line to the optical imaging system.

6. The surgical microscope according to claim 2, wherein the gesture detection unit further comprises a movement detection module for differentiating 3-dimensional movements of the further object and the surgical instrument in the detection zone, wherein the movement detection module is connected to the gesture detection unit via a movement-data line and the movement detection module is adapted to output a movement-data signal via the movement-data line to the gesture detection unit depending on the movement of the object.

7. The surgical microscope according to claim 6, wherein the gesture detection unit further comprises a relationship mapping unit which is connected to the movement detection module and which relationship mapping unit is adapted to map relationships between a movement of the surgical instrument and a movement of the further object and a control signal, which control signal is provided via the signal line to the optical imaging system.

8. The surgical microscope according to claim 7, wherein the relationship map for the further type of object provides control signals for coarse adjustment of the optical imaging system, and the relationship map for the surgical instrument provides control signals for fine adjustment of the optical imaging system.

9. The surgical microscope according to claim 7 wherein the relationship map for the further type of object and the relationship map for the surgical instrument provide control signals in dependence on a detected combination of movements.

10. The surgical microscope according to claim 1, wherein the optical imaging system comprises at least one moveable mechanical component, wherein a position of the at least one moveable mechanical component depends on the control signal.

11. The surgical microscope of claim 10 wherein the optical imaging system being configured to alter its state comprises the optical imaging system being configured to change one or more of pan, zoom or tilt in response to the control signal.

12. The surgical microscope according to claim 1, further comprising a projection unit optically connected to the optical imaging system, wherein the projection unit is adapted to project optional and/or interactive images on at least one interaction portion, wherein the at least one interaction portion at least partially overlaps with the imaged inspection area.

13. A method for a gesture control of a surgical microscope having an optical imaging system and a camera which images an inspection area which includes a field of view of the optical imaging system, the method comprising: detecting by the surgical microscope a movement of a surgical instrument with respect to a reference point based on data from the camera; determining by the surgical microscope whether the movement of the surgical instrument is in the inspection area or in a detection zone which is located between the inspection area and the optical imaging system and is spaced apart from the optical imaging system; when the movement of the surgical instrument is determined by the surgical microscope as being in the detection zone, controlling by the surgical microscope the optical imaging system to alter its state depending on the movement of the surgical instrument; when the movement of the surgical instrument is determined by the surgical microscope as being in the inspection area, not controlling by the surgical microscope the optical imaging system to alter its state depending on the movement of the surgical instrument; and when the movement of the surgical instrument is determined by the surgical microscope as being between the detection zone and the optical imaging system, not controlling by the surgical microscope the optical imaging system to alter its state depending on the movement of the surgical instrument.

14. The method of claim 13, further comprising the step of detecting by the surgical microscope a movement of a further type of object with respect to the reference point based on the data from the camera, the further type of object being one of a finger and a tool.

15. The method of claim 14, further comprising the steps of: determining by the surgical microscope whether the movement of the further type of object is in the inspection area or in the detection zone; when the movement of the further type of object is determined by the surgical microscope as being in the detection zone, controlling by the surgical microscope the optical imaging system to alter its state depending on the movement of the further type of object; and when the movement of the further type of object is determined by the surgical microscope as being in the inspection area, not controlling the optical imaging system to alter its state depending on the movement of the further type of object.

16. The method of claim 13, further comprising the step of comparing by the surgical microscope the detected movement of the surgical instrument with predetermined movement patterns.

17. The method of claim 13, wherein the data from the camera includes distance and/or movement data relative to the reference point.

18. The method of claim 13, further comprising: when the movement of the surgical instrument is determined by the surgical microscope as being in the detection zone, controlling by the surgical microscope at least one moveable mechanical component of the surgical microscope depending on the movement of the surgical instrument; and when the movement of the surgical instrument is determined by the surgical microscope as being in the inspection area or between the detection zone and the optical imaging system, not controlling at the least one moveable mechanical component of the surgical microscope depending on the movement of the surgical instrument.

19. The method of claim 13, wherein controlling by the surgical microscope the optical imaging system to alter its state comprises controlling the optical imaging system by the surgical microscope to change one or more of pan, zoom or tilt depending on the movement of the surgical instrument.

20. The method of claim 13, further comprising: projecting at least one image on at least one interaction portion, wherein the at least one interaction portion at least partially overlaps with the inspection area; comparing by the surgical microscope the movement of the surgical instrument with predetermined interaction movement patterns; and controlling by the surgical microscope the optical imaging system and/or the surgical microscope depending on the movement of the surgical instrument within the at least one interaction portion.

Description

BRIEF DESCRIPTION OF THE DRAWING VIEWS

(1) In the Figures,

(2) FIG. 1 shows a schematic side view of a first embodiment of the inventive surgical microscope;

(3) FIG. 2 shows a schematic front view of a second embodiment of the inventive surgical microscope;

(4) FIG. 3 shows different possible movement patterns of the object;

(5) FIG. 4 shows an exemplary adjustment of the working distance of the inventive surgical microscope; and

(6) FIG. 5 shows an exemplary application of one of four interaction portions for selection of an imaging note.

DETAILED DESCRIPTION OF THE INVENTION

(7) FIG. 1 shows a schematic side view of a first embodiment of an inventive surgical microscope 1, which surgical microscope 1 comprises an optical imaging system 3, which optical imaging system 3 comprises, for instance, a stereoscopic eyepiece 4 comprising eyepieces 5, a stereoscopic assistant's eyepiece 4a comprising assistant's eyepieces 5a located at the side of the surgical microscope 1 and an objective 7. The optical imaging system 3 determines a field of view 9 which is a portion of the inspection area 11 which may be a patient 13.

(8) The field of view 9 is determined by the numerical aperture of the optical imaging system 3 and a working distance 15, which working distance 15 is measured along a z-axis. The field of view 9 extends along the x-axis and y-axis.

(9) The surgical microscope 1 further comprises a gesture detection unit 17 which is attached to the housing 19 of the surgical microscope 1. The gesture detection unit 17 acts as a reference point 38 in the embodiments shown in FIG. 1. In other embodiments the reference point 38 may for instance be located in the inspection area 11.

(10) The eyepieces 5 and the assistant's eyepieces 5a comprise sensors 6 which detect whether one of the eyepieces 5, 5a is used for viewing the imaged field of view 9. The sensors 6 are proximity sensors 6a in the embodiment shown.

(11) The gesture detection unit 17 comprises two signal lines 76, wherein one signal line 76 is an electrical line (solid) and a second signal line 76 is a wireless connection (dashed). Via those two signal lines 76, a control signal 77, illustrated by an arrow, is transmitted.

(12) The signal line 76 indicated by the solid line is connected to a moveable mechanical component which is a joint 82a of a holding arm 82b which holds the surgical microscope 1.

(13) The gesture detection unit 17 is tilted around the x-axis and faces towards the field of view 9, such that a gesture detection zone 21 at least partially overlaps with the viewing section 23 of the optical imaging system 3. The viewing section 23 is indicated by a dashed surrounding line.

(14) The objective 7 defines an optical axis 25 which is oriented parallel to the z-axis. It is noted that the optical axis 25 is oriented according to the objective 7 and not necessarily to further components (e.g. the eyepiece 5) of the optical imaging system 3.

(15) In FIG. 1, it is shown that the detection zone 21, which is a 3D-volume, is located at a distance to the field of view 9 of the inspection area 11. Only the movement of an object (not shown) within the surrounded volume inside the dashed line is detected, i.e. movements of an object above (with reference to the z-axis) or below the detection zone 21 will be ignored and will not generate a control signal.

(16) FIG. 2 shows a schematic front view of a second embodiment of the inventive surgical microscope 1 in which the gesture detection unit 17 is embodied integrally in the housing 19 of the surgical microscope 1. This embodiment does not comprise the stereoscopic assistant's eyepiece 4a.

(17) In the embodiment shown in FIG. 2, the 3D-camera 27 of the surgical microscope 1 provides the distance information of objects 29 via a signal line 76.

(18) The gesture detection unit 17 further comprises a movement detection module 78, which is provided by the signal line 76a with data from the 3D-camera 27.

(19) The movement detection unit 78 provides a movement-data signal 80 via a movement-data line 79 to the gesture detection unit 17.

(20) Via the signal line 76, the gesture detection unit 17 outputs a control signal 77 to the optical imaging system 3. The control signal 77 and the movement-data signal 80 are indicated by arrows.

(21) The gesture detection unit 17 further comprises a relationship mapping unit 81, which maps a detected movement 65 with an operation to be performed by the optical imaging system 3 or the moveable mechanical component 82.

(22) The surgical microscope 1 further comprises a projection unit 83, which is adapted to project additional and/or interactive images 71 (not shown) onto the imaged inspection area. In the embodiment shown, the projection unit 83 utilizes the beam splitters 33 for being optically coupled to the optical imaging system 3.

(23) In FIG. 2, the object 29 is a scalpel 31, which is located inside the detection zone 21 and whose movements are detected by the gesture detection unit 17.

(24) The 3D-camera 27 is only schematically shown, whereas only the beam splitters 33 and the 2D-sensors 35 are indicated. The 2D-sensors may be CCD cameras. Due to the different viewing sections 23a and 23b of the two channels, a three-dimensional image of the field of view 9 may be generated, imaged and viewed via the eyepiece 5.

(25) In the embodiment of the surgical microscope 1 shown in FIG. 2, the gesture detection unit 17 receives data from the 3D-camera 27, in particular from the 2D-sensors 35 and assesses the distance data of the scalpel 31, whereas the distance data corresponds to the distance 37 of the tip of the scalpel 31.

(26) The surgical microscope 1 of FIG. 2 does not comprise a stereoscopic assistant's eyepiece 4a.

(27) FIG. 3 shows a schematic drawing illustrating different movement patterns of the object (not shown) whereas one may distinguish one hand-gestures 39 and two-hand gestures 41.

(28) Possible one-hand gestures 39 are movements of the objects along the y-axis, for instance a vertical gesture 43, a horizontal gesture 45, a combined 2D-gesture 47, a circular gesture 49 and an upward gesture 51 during which the object is moved along the z-direction in a positive sense of the z-axis.

(29) Possible two-hand gestures 41 are displacement gestures 53a and 53b, a zoom-in gesture 55 and a zoom-out gesture 57.

(30) A possible two-hand gesture 41 along the z-axis is shown in case of the second upward gesture 59, in which both objects (not shown) are moved along the z-direction, whereas the drawn arrows are pointing in different directions due to the perspective chosen for the figure.

(31) FIG. 3 furthermore shows a rotational gesture 49a, which is one possible one-hand gesture 39 and a circular gesture 49 as well. The rotational gesture 49a is performed in the x-y plane by rotating the scalpel 31 from a first rotational position 31a to a second rotational position 31b and into a third rotational position 31c.

(32) FIG. 4 shows an exemplary operation which is performed by the optical imaging system depending on the movement of a scalpel 31.

(33) FIG. 4 shows the image of the field of view 9 in two states of the optical imaging system 3. In the unfocused state 61 the image is blurred due to an incorrect working distance and consequently misaligned focus of the surgical microscope 1.

(34) In the unfocused state 61, the scalpel 31 is located in the detection zone 21, which is, however not visible in FIG. 4 (see for instance FIG. 2), i.e. movements of the scalpel 31 are detected by the gesture detection unit 17 (not shown), which in turn generates a control signal for altering the state of the optical imaging system 3.

(35) The scalpel 31 performs a movement 65 which is indicated by an arrow, which movement 65 starts at a start position 67 and ends at an end position 69 of the scalpel 31.

(36) The detected movement generates a control signal which controls the working distance of the optical imaging system 3 and which control signal initiates change of the working distance 15 of the optical imaging system 3, which in turn adjusts the focus of the surgical microscope 1.

(37) If the scalpel 31 is moved from the start position 67 to the end position 69, the movement 65 of the scalpel 31 is detected as an online gesture, i.e. the surgeon directly sees the change of the state of the surgical microscope 1 during the movement 65.

(38) Upon arriving at a focused state 63 of the field of view 9 imaged by the optical imaging system 3, the surgeon stops the movement of the scalpel, which stopped position corresponds to the end position 69.

(39) FIG. 5 shows an exemplary application of interaction portions for selecting one out of several imaging modes.

(40) In FIG. 5, the same field of view 9 of a focused state 63 is shown, where a projection unit (not shown) projects four opaque images 71 in an overlaying manner on the imaged inspection area, which images 71 define interaction portions 73 of the field of view 9. The images 71 and interaction portions 73 are solely labeled in the first view 75a shown in FIG. 5.

(41) The images 71 contain a visual feedback of the operation of the optical imaging system 3 which can be activated by the corresponding interaction portion 73.

(42) In FIG. 5, the shown interaction portions 73 correspond to different imaging modes, wherein in the first view 75a, the scalpel 31 is positioned outside the interaction portions 73 and subsequently moved into the left lower interaction portion 73a, which is shown in the second view 75b.

(43) In the third view 75c, a movement 65 of the scalpel 31 out of the drawing plane, i.e. along the z-axis is indicated by an arrow. This gesture is detected by the gesture detection unit 17 (not shown), which gesture detection unit 17 generates a control signal processed by a computing unit (not shown), which subsequently initiates change of the imaging mode to the one corresponding to the according left lower interaction portion 73a.

(44) The imaging mode selected in FIG. 5 in the third view 75c is the mode ‘MFL 400’ and after recognition of the movement 65 of the scalpel 31, the detection of the movement 65 for activating the function corresponding to the left lower interaction portion 73a is indicated by projecting an alternate, i.e. modified, image 71 in the left lower interaction portion 73a.

(45) The movement 65 shown in the third view 75c is an offline movement, i.e. the operation corresponding to the activation of the according interaction portion 73 is only performed after completion of the movement 65.

(46) As already mentioned in the beginning of the explanation of the figures, the embodiment shown in FIGS. 1-5 are to be understood as exemplary and do not limit the scope of protection of the present disclosure. Therefore, any number of interaction portions 73, one-hand gestures 39, two-hand gestures 41 and functional sections of the field of view 9 are conceivable.

REFERENCE NUMERALS

(47) 1 surgical microscope 3 optical imaging system 4 stereoscopic eyepiece 4a stereoscopic assistant's eyepiece 5 eyepiece 5a assistant's eyepiece 6 sensor 6a proximity sensor 7 objective 9 field of view 11 inspection area 13 patient 15 working distance 17 gesture detection unit 19 housing 21 detection zone 23, 23a, 23b viewing section 25 optical axis 27 3D-camera 29 object 31 scalpel 31a first rotational position 31b second rotational position 31c third rotational position 33 beam splitter 35 2D-sensor 37 distance 38 reference point 39 one-hand gestures 41 two-hand gestures 43 vertical gesture 45 horizontal gesture 47 combined 2D-gesture 49 circular gesture 49a rotational gesture 51 upward gesture 53a, 53b displacement gestures 55 zoom-in gesture 57 zoom-out gesture 59 second upward gesture 61 unfocused state 63 focused state 65 movement 67 start position 69 end position 71 image 73 interaction portion 73a left lower interaction portion 75a first view 75b second view 75c third view 76, 76a signal line 77 control signal 78 movement detection module 79 movement-data line 80 movement-data signal 81 relationship mapping unit 82 moveable mechanical components 82a joint 82b holding arm 83 projection unit x x-axis y y-axis z z-axis