Technique For Determining A Visualization Based On An Estimated Surgeon Pose

Abstract

A method, a system, and a computer program for determining a visualization of content to be displayed to a surgeon on a display of a portable display device are provided. First pose data indicative a first pose of an anatomical feature of a patient in an operating room and second pose data indicative of a second pose of a portable display device in the operating room are obtained. Based on the first pose data and the second pose data, a third pose of a surgeon relative to the patient is estimated. Based on the estimated third pose, a visualization of content to be displayed to the surgeon on a display of the portable display device is determined.

Claims

1. A method for determining a visualization of content to be displayed to a surgeon on a display of a portable display device, the method comprising: obtaining first pose data indicative a first pose of an anatomical feature of a patient in an operating room; obtaining second pose data indicative of a second pose of a portable display device in the operating room; estimating, based on the first pose data and the second pose data, a third pose of a surgeon relative to the patient; and determining, based on the estimated third pose, a visualization of content to be displayed to the surgeon on a display of the portable display device.

2. The method of claim 1, further comprising: obtaining a fourth pose of the surgeon relative to the portable display device, wherein the third pose is estimated further based on the fourth pose.

3. The method of claim 2, wherein the fourth pose is determined based on a predefined spatial criterion associated with the portable display device.

4. The method of claim 3, wherein the predefined spatial criterion defines that the fourth pose lies on a trajectory having a predefined spatial relationship relative to the portable display device.

5. The method of claim 3, wherein the fourth pose is determined based on sensor data acquired by a sensor of the portable display device.

6. The method of claim 5, wherein an approximate fourth pose is determined based on the predefined spatial criterion, and the approximate fourth pose is refined based on the sensor data to determine the fourth pose.

7. The method of claim 5, wherein the sensor comprises a camera and the sensor data comprises image data acquired by the camera.

8. The method of claim 1, wherein at least one of the first pose data and the second pose data is obtained from a surgical tracking system and wherein at least one of the following conditions is fulfilled: (a) the first pose data is obtained from the surgical tracking system that is configured to track the anatomical feature or one or more trackers fixedly attached relative to the anatomical feature; and (b) the second pose data is obtained from the surgical tracking system that is configured to track the portable display device or one or more trackers fixedly attached relative to the portable display device.

9. The method of claim 1, wherein the visualization is determined based on a viewing direction of the surgeon onto the patient.

10. The method of claim 9, wherein the content is associated with one or more anatomical features of the patient having a predefined spatial relationship to the viewing direction.

11. The method of claim 10, wherein the visualization is an augmented view of (i) the content and (ii) the one or more anatomical features or a representation thereof.

12. The method of claim 1, further comprising triggering display of the visualization on the display of the portable display device.

13. The method of claim 7, further comprising triggering display of the visualization on the display of the portable display device, wherein the display and the camera of the portable display device face in a similar direction.

14. The method of claim 1, further comprising: comparing at least one pose selected from the second pose, the third pose and the fourth pose with a first criterion; and based on a result of the comparing, performing one or more predefined actions associated with the result.

15. The method of claim 1, further comprising: obtaining user input entered via the portable display device; and performing one or more actions based on the obtained user input.

16. The method of claim 5, further comprising: obtaining user input entered via the portable display device; and performing one or more actions based on the obtained user input, wherein the user input comprises a gesture command entered via the sensor.

17. A system for determining a visualization of content to be displayed to a surgeon on a display of a portable display device, the system comprising at least one processor configured to: obtain first pose data indicative a first pose of an anatomical feature of a patient in an operating room; obtain second pose data indicative of a second pose of a portable display device in the operating room; estimate, based on the first pose data and the second pose data, a third pose of a surgeon relative to the patient; and determine, based on the estimated third pose, a visualization of content to be displayed to the surgeon on a display of the portable display device.

18. The system of claim 17, further comprising the portable display device.

19. The system of claim 17, further comprising: a surgical tracking system configured to track at least one entity selected from (i) the anatomical feature, (ii) one or more trackers fixedly attached relative to the anatomical feature, (iii) the portable display device, and (iv) one or more trackers fixedly attached relative to the portable display device, wherein the surgical tracking system is configured to provide the at least one processor with at least one input selected from the first pose data and the second pose data.

20. A non-transitory computer readable medium storing a computer program comprising instructions which, when executed on at least one processor, cause the at least one processor to: obtain first pose data indicative a first pose of an anatomical feature of a patient in an operating room; obtain second pose data indicative of a second pose of a portable display device in the operating room; estimate, based on the first pose data and the second pose data, a third pose of a surgeon relative to the patient; and determine, based on the estimated third pose, a visualization of content to be displayed to the surgeon on a display of the portable display device.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Further details, advantages and aspects of the present disclosure will become apparent from the following embodiments taken in conjunction with the drawings, wherein:

[0035] FIG. 1 shows a system in accordance with the present disclosure;

[0036] FIG. 2 shows a method in accordance with the present disclosure;

[0037] FIG. 3a-3b show exemplary visualizations in accordance with the present disclosure; and

[0038] FIG. 4 shows exemplary coordinate systems of a system in accordance with the present disclosure.

DETAILED DESCRIPTION

[0039] In the following description, exemplary embodiments of a system and a method will be explained with reference to the drawings. The same reference numerals will be used to denote the same or similar structural features.

[0040] FIG. 1 shows a system 100 in accordance with the present disclosure. The system 100 comprises a computing system 200. The computing system 200 comprises a processor 2 coupled to a memory 4 and an interface 6. The system 100 further comprises a surgical tracking system 300 connected to the interface 6 via a wired or wireless communication connection 8, and a portable display device 400 connected to the interface 6 via a wired or wireless communication connection 10. Other variants of connecting the processor 2 with the surgical tracking system 300 and the portable display device 400 (e.g., using a plurality of interfaces or relay devices) are also possible.

[0041] The surgical tracking system 300 is an optical tracking system configured to track a first tracker 12 fixed to a patient support 14 holding a patient 16. The first tracker 12 has a fixed spatial relationship to the anatomical feature 20 of the patient 16. The surgical tracking system 300 is further configured to track a second tracker 18 that is attached to the portable display device 400.

[0042] In one variation, the surgical tracking system 300 may be configured to track (e.g., the anatomical feature 20 of) the patient 16 directly, instead of tracking the first tracker 12. Alternatively or additionally, the surgical tracking system 300 may be configured to track the portable display device 400 directly, instead of tracking the second tracker 18. In these cases, one or both of the respective trackers 12, 18 may be omitted. For tracking (e.g., anatomical feature 20 of) the patient 16 and the portable display device 400 directly, the surgical tracking system 300 may comprise one or more image sensors configured to acquire an image using visible light. The (e.g., anatomical feature 20 of the) patient 16 and the portable display device 400 may then be detected in the image using appropriate image analysis techniques (e.g., object recognition). Other variants for directly tracking (e.g., the anatomical feature 20 of) the patient 16 and the portable display device 400 are also possible (e.g., using a time-of-flight sensor, a thermal camera or the like).

[0043] The portable display device 400 is a tablet computer and comprises a display 21 that is configured as a touch screen. The portable display device 400 further comprises a frontal selfie camera 22 facing a person looking at the display 21. Alternatively or in addition to the camera 22, the portable display device 400 comprises a depth sensor 24 configured to measure a distance between the portable display device 400 and the person looking at the display 21. The portable display device 400 may further comprise various buttons 26 as part of a user interface. The portable display device 400 is configured to display, on the display 21, content to a surgeon 28. The portable display device 400 may be configured to be sterilized. Alternatively or additionally, the portable display device 400 may be configured to display the content to the surgeon 28 while being covered by a surgical drape 27. In FIG. 1, the head of the surgeon 28 is illustrated from behind to indicate that the surgeon is looking toward the patient 16 along the viewing direction 30 or onto the display 21 along the line of sight 32.

[0044] The portable display device 400 may be placed in close proximity to (e.g., <1 m, <50 cm or even <30 cm away from) the surgical field 34. This may minimize the amount of head and eye movement required by the surgeon to change his field of view from the surgical field 34 to the display 21 and back. Furthermore, the surgeon 28 does not need to wear a HMD to be provided with the visualization. Instead, the surgeon 28 may simply look at the portable display device 400 whenever (s)he intends to do so.

[0045] FIG. 2 shows a method in accordance with the present disclosure. The method may be performed by the processor 2. The memory 4 may store instructions which, when executed by the processor 2, cause the processor 2 to perform the method described herein.

[0046] In step 202, first pose data indicative of a first pose of the anatomical feature 20 of the patient 16 in the operating room is obtained. The first pose data may alternatively or additionally be indicative of a first pose of the patient 16 in the operating room. The first pose data is obtained from the surgical tracking system 300 and is indicative of a position and orientation of (e.g., the anatomical feature 20 of) the patient 16 relative to the surgical tracking system 300. The first pose data may be determined by tracking, with the surgical tracking system 300, the first tracker 12 or by directly tracking (e.g., the anatomical feature 20 of) the patient 16 with the surgical tracking system 300.

[0047] In step 204, second pose data indicative of a second pose of the portable display device 400 in the operating room is obtained. The second pose data is obtained from the surgical tracking system 300 and is indicative of a position and orientation of the portable display device 400 relative to the surgical tracking system 300. The second pose data may be determined by tracking, with the surgical tracking system 300, the second tracker 18 or by directly tracking the portable display device 400 with the surgical tracking system 300. In a still further variant, the second pose data may be obtained from the portable display device 400 configured to determine its position and orientation in the operating room using one or more of its sensor(s) (e.g., the camera 22 and/or the depth sensor 24).

[0048] In step 206, (e.g., only) based on the first pose data and the second pose data, a third pose of the surgeon 28 relative to (e.g., the anatomical feature 20 of) the patient 16 is estimated.

[0049] Due to the estimation of the third pose based on the first pose data and the second pose data, the surgeon 28 does not need to be tracked by the surgical tracking system 300. In other words, the only data obtained from the surgical tracking system 300 for determining the third pose may be the first pose data and the second pose data. There is no need to attach an additional tracker to the surgeon 28 to obtain the third pose of the surgeon 28.

[0050] The method may comprise obtaining a fourth pose of the surgeon 28 relative to the portable display device 400. The third pose of the surgeon 28 relative to the patient 16 may then be estimated further based on the fourth pose.

[0051] Based on the fourth pose and the second pose, a position and orientation of the surgeon 28 relative to the surgical tracking system 300 can be determined. The first pose is indicative of a position and orientation of (e.g., the anatomical feature 20 of) the patient 16 in the operating room. Thus, the position and orientation of (e.g., the anatomical feature 20 of) the patient 16 and the position and orientation of the surgeon 28 are each known relative to the surgical tracking system 300. Based on these positions and orientations, the fourth pose of the surgeon relative to the portable display device 400 can be determined.

[0052] An approximate fourth pose, also referred to as roughly estimated fourth pose, may be determined to lie on a trajectory having a predefined spatial relationship relative to the portable display device 400. The trajectory is parallel to a normal of the display of the portable display device and intersects a center of the display of the portable display device. In other words, it may be assumed that the surgeon 28 positions the display 21 such that it faces towards him.

[0053] In some cases, the fourth pose of the surgeon 28 lies exactly on the trajectory. In these cases, the roughly estimated fourth pose may be taken as the fourth pose. Otherwise, the roughly estimated fourth pose may be refined based on sensor data acquired by the camera 22 or the depth sensor 24 of the portable display device 400, thereby determining the fourth pose. For instance, a precise location of the surgeon 28 may be determined in an image acquired by the camera 22. The approximate fourth pose may be used for detecting the precise location of the surgeon 28 in the image, for example by detecting a human face that is closest to the trajectory.

[0054] In step 208, based on the estimated third pose, a visualization of content to be displayed to the surgeon 28 on the display 21 of the portable display device 400 is determined.

[0055] The visualization may be determined based on the viewing direction 30 of the surgeon 28 onto the patient 16. The viewing direction 30 is determined as a direction from the surgeon 28 towards (e.g., the anatomical feature 20 of) the patient 16. The viewing direction 30 may be determined based on an orientation of the surgeon's head relative to the portable display device 400, the orientation being detected in an image acquired by the camera 22.

[0056] The content to be displayed to the surgeon 28 may include pre-planned objects such as implants, planned trajectories or body portions of interest. The content to be displayed is associated with one or more anatomical features of the patient 16, including the anatomical feature 20, which may have a predefined spatial relationship to the viewing direction 30. The content may include a model, segmentation or representation of the associated anatomical feature(s) of the patient 16. For example, content associated with all anatomical features that are intersected by the viewing direction 30 or that lie within a predefined distance around the viewing direction 30 may be displayed. The surgeon 28 may adapt the contents to be displayed.

[0057] The visualization may be an augmented view. The content may be overlaid onto the patient 16 or on an image thereof. The augmented view may comprise the content and the associated anatomical features or a representation (e.g., an image, model or segmentation) thereof.

[0058] The visualization to be displayed mimics the view of the surgeon 28 onto the patient 16, in particular a view of the surgeon 28 onto the surgical field 34 including the anatomical feature 20. The visualization may thus correspond to a view of the surgeon 28 onto the patient 16 and comprise the content overlaid onto the view. In other words, the visualization is determined such that the displayed (e.g., representation of the) anatomical feature(s) and the content associated therewith comply with the viewing direction 30 of surgeon 28 onto the patient 16.

[0059] The surgeon 28 may thus not need to transpose (e.g., rotate) the displayed content mentally when changing his field of view from the surgical field 34 to the display 21 and vice versa.

[0060] The visualization may be determined further based on the line of sight 32 of the surgeon 28 onto the display 21 of the portable display device 400. The visualization may be determined such that a projection of the visualization along the line of sight 32 complies with a view along the viewing direction 30 of the surgeon 28. Thereby, a potential distortion of the displayed content perceived by the surgeon 28 due to a sharp viewing angle onto the display 21 may be compensated.

[0061] In step 210, display of the visualization is triggered on the display 21 of the portable display device 400. The processor 2 may send a control command to the portable display device 400 to trigger display of the visualization. The processor 2 may send information (e.g., image data) representing or defining the visualization to be displayed to the portable display device 400.

[0062] In one example, only if the second pose, the third pose and/or the fourth pose exceed a preset maximum deviation from a previously obtained instance of the respective second, third or fourth pose, the visualization is determined and triggered to be displayed. This may avoid determining and updating the displayed visualization too often, thereby lowering processing time and reducing disturbance for the surgeon 28.

[0063] The method may further comprise obtaining user input entered via the portable display device 400. One or more predefined actions may be performed corresponding to the user input. For example, a navigation workflow may be started or the visualization may be adapted based on the user input. The user input may comprise a gesture command recognized based on image data acquired by the camera 22 or based on depth data acquired by the depth sensor 24. The user input may be entered via the touch screen 21 or the button(s) 26.

[0064] FIGS. 3a and 3b show exemplary visualizations in accordance with the present disclosure. In these figures, the anatomical feature 20 is schematically illustrated as a cube having distinguishable sides.

[0065] As illustrated in FIG. 3a, the surgeon 28 may be positioned on a trajectory 38 parallel to a normal of the display 21, the trajectory 38 originating at the middle of the display 21. The fourth pose of the surgeon 28 relative to the portable display device 400 can thus be determined by assuming that the surgeon 28 is positioned on the trajectory 38. The pose of the portable display device 400 and the pose of the anatomical feature 20 are obtained (e.g., in steps 202 and 204). A spatial relationship between the portable display device 400 and the anatomical feature 20 can be derived from these obtained poses. This allows estimating the pose of the surgeon 28 relative to the anatomical feature 20 (e.g., in step 206) based on the obtained first and second pose and based on the fourth pose. The visualization may then be determined (e.g., in step 208) such that it complies with a viewing direction of the surgeon 28 onto the anatomical feature 20. In the illustrated example, the determined visualization is displayed on the display 21 such that the surgeon 28 is presented with a view of a side of the anatomical feature 20 that is in his field of view.

[0066] As illustrated in FIG. 3b, the surgeon 28 may not be positioned on the trajectory 38. An approximate fourth pose of the surgeon 28 relative to the portable display device 400 may be determined by assuming that the surgeon 28 is located on the trajectory 38. This may be advantageous in case a plurality of persons are located in proximity of the portable display device 400. In the illustrated example, a nurse 40 is located in the operating room and may view the display 21. Nevertheless, the nurse 40 in this example is located further from the normal 38 than the surgeon 28. In other words, the display 21 is more suitably oriented for the surgeon 28 than for the nurse 40. The approximate fourth pose is thus closer to the real pose of the surgeon 28 than to that of the nurse 40.

[0067] To refine the approximate fourth pose of the surgeon 28, sensor data of the portable display device 400 may be obtained. For example, an image acquired by the camera 22 or depth data acquired by the depth sensor 24 may be used to detect a human (e.g., a human face). The surgeon 28 may be identified using facial recognition or object recognition (e.g., recognition of an identification patch worn by the surgeon or of clothing worn by the surgeon). A detected person located closest to the trajectory 38 may be assumed to be the surgeon 28, and its detected pose may be taken as the pose of the surgeon 28 relative to the portable display device 400. The approximate fourth pose may be refined by starting the facial recognition or the object recognition in an area of the image that is closest to the trajectory 38. Alternatively, the detected position of the surgeon 28 may be directly used as the fourth pose of the surgeon 28 without considering the approximate fourth pose or the trajectory 38.

[0068] Again, the pose of the anatomical feature 20 and the pose of the portable display device 400 are obtained (e.g., in steps 202 and 204), such that their relative positions and orientations are known. Based on the (e.g., refined) fourth pose of the surgeon 28 relative to the portable display device 400, the third pose of the surgeon 28 relative to the anatomical feature 20 is determined (e.g., in step 206). In the illustrated example, the surgeon 28 is presented with a view of two sides of the anatomical feature 20 that are in his field of view. If the approximate fourth pose were used instead of the refined fourth pose, the surgeon 28 would instead be presented with a view of a single side of the anatomical element 20 as illustrated in FIG. 3a.

[0069] FIG. 4 shows exemplary coordinate systems in accordance with the present disclosure.

[0070] The first pose of (e.g., the anatomical feature 20 of) the patient 16 may be obtained as pose P.sub.1 in a coordinate system C.sub.tracking system of the surgical tracking system 300. The first pose may be determined based on a tracked pose P.sub.2 of the first tracker 12 in the coordinate system C.sub.tracking system and based on a known relative pose P.sub.3 between (e.g., the anatomical feature 20 of) the patient 16 and the first tracker 12. Alternatively, the first pose P.sub.1 may be obtained by directly tracking (e.g., the anatomical feature 20 of) the patient 16 by the surgical tracking system 300 in the coordinate system C.sub.tracking system.

[0071] The second pose of the portable display device 400 may be obtained as the pose P.sub.5 in the coordinate system C.sub.tracking system of the surgical tracking system 300. The second pose may be determined based on a tracked pose P.sub.6 of the second tracker 18 in the coordinate system C.sub.tracking system and based on a known relative pose P.sub.7 between the portable display device 400 and second tracker 18. Alternatively, the second pose P.sub.5 may be determined by directly tracking the portable display device 400 by the surgical tracking system 300 in the coordinate system C.sub.tracking system.

[0072] The fourth pose P.sub.8 of the surgeon 28 relative to the portable display device 400 may be obtained as described above. This may involve determining the approximate fourth pose and refining the approximate fourth pose based on the sensor data.

[0073] The poses P.sub.5 and P.sub.8, or of the poses P.sub.6, P.sub.7 and P.sub.8 may be combined to obtain the pose P.sub.9 of the surgeon 28 in the coordinate system C.sub.tracking system. The third pose P.sub.10 between the surgeon 28 and (e.g., the anatomical feature 20 of) the patient 16 may be determined based on the poses P.sub.1 and P.sub.9, or based on the poses P.sub.2, P.sub.3 and P.sub.9. Instead of the pose P.sub.9, a combination of the poses P.sub.5 and P.sub.8, or of the poses P.sub.6, P.sub.7 and P.sub.8 may be used.

[0074] Using the approach described herein, no tracking of the surgeon 28 by the surgical tracking system 300 is required. Instead, the pose P.sub.9 is estimated based on the poses P.sub.5 and P.sub.8, or based on the poses P.sub.6, P.sub.7 and P.sub.8.

[0075] Various techniques of determining a registration between the patient 16 and patient image data comprising at least one image of the patient's body including the anatomical feature 20 exist. For instance, a probe tracked by the surgical tracking system 300 may be placed on (e.g., the anatomical feature 20 of) the patient 16 to obtain the first pose of (e.g., the anatomical feature 20 of) the patient 16 in the coordinate system C.sub.tracking system. The first pose may then be correlated with or mapped to the patient image data to determine the registration between (e.g., the anatomical feature 20 of) the patient 16 and the patient image data. A relative pose P.sub.4 between the coordinate system C.sub.patient image data of the patient image data and the coordinate system C.sub.tracking system of the surgical tracking system 300 may then be determined (e.g., based on the registration).

[0076] The content to be displayed to the surgeon 28 may be part of the patient image data or defined (e.g., planned) in the coordinate system C.sub.patient image data. Based on the third pose P.sub.10, the viewing direction 30 of the surgeon 28 onto (e.g., the anatomical feature 10 of) the patient 16 can be determined. The visualization may then be determined such that the content to be displayed is arranged in a similar orientation relative to the surgeon 28 like the patient 16 relative to the surgeon 28. In other words, the visualization of the content may be determined such that it corresponds or complies with a view of the surgeon 28 onto the patient 16.

[0077] As will become apparent from the above, the technique disclosed herein may provide numerous advantages. For example, the technique may allow for using a tracked portable display device that may be arranged very close to a surgical field. The portable display device may be a mobile tablet device. Contents such as surgical navigation information may be provided in close proximity to the surgical field. Compared with a HMD, the display of the portable display device may not block the surgeon's view. The display of the portable display device may not need to be arranged in a completely different viewing direction (e.g., on a cart), but may be arranged close to the surgical field. The surgeon may not need to adjust the focus of his eyes when shifting his field of view from the surgical field on the patient to the display or vice versa.

[0078] By tracking the positioning and orientation of the portable display device, a spatial relationship (e.g., the third pose) between the surgeon and the patient may be estimated. This information may be used to reduce the mental load for the surgeon.

[0079] The displayed content may be adjusted based on the pose of the surgeon relative to the portable display device. In particular, (e.g., navigated) content may be adjusted by rotating a screen perspective to comply with a real perspective of the surgeon. The surgeon may not need to mentally rotate the displayed information (e.g., the content and the one or more anatomical features or the representation thereof) when shifting his field of view from the surgical field to the display or vice versa.

[0080] The surgeon may not need to wear a tracker. His viewing direction may still be simulated on the display using the technique described herein. The surgeon may not need to wear a HMD to be provided with the displayed content.

[0081] The portable display device may be made sterile (e.g., via a surgical drape or other techniques) and may be used as a remote control through its touch screen capabilities. A gesture command may be detected by the portable display device as user input.

[0082] Based on the pose of the surgeon in relation to other tracked devices (e.g., surgical instruments), additional predefined actions may be performed (e.g., activating a workflow in a navigation software or adjusting the visualization).

[0083] Additional advantages may be apparent to those skilled in the art in view of the above. Various modifications to the technique disclosed herein are possible. For example, the first pose data may be obtained from a first tracking system and the second pose data may be obtained from a separate second tracking system (e.g., using a different tracking modality than the first tracking system). In another example, the second pose data may be obtained from the portable display device in case the portable display device is configured to determined its spatial pose relative on its own (e.g., using the sensor(s) comprised in the portable display device).