SYSTEM AND METHOD FOR ENHANCED DATA ANALYSIS WITH VIDEO ENABLED SOFTWARE TOOLS FOR MEDICAL ENVIRONMENTS

Abstract

Medical software tools platform utilizes a surgical display to provide access to specific medical software tools, such as medically-oriented applications or tools, that can assist those in the operating room, such as a surgeon or surgical team, with a surgery. In particular, an optical sensor located within an endoscopic camera may register subtle differences in color characteristics reflected from a tissue surface and in turn transmit the information to a medical image processing system. Moreover, the new tools are intended to help surgeons better determine the boundaries between healthy and diseased regions during surgical procedures. Various tools are intended to be used in procedures where indocyanine green (ICG) fluorescent dye is used. Intraoperative fluorescence imaging is commonly used during minimally invasive procedures to enable surgeons to visualize tissue perfusion and anatomical structures. The term perfusion refers to the passage of blood and tissue fluid through the capillary bed. Intraoperative fluorescence imaging can also be used to improve visualization of vessels and structures, which, in turn, may reduce the risk of complications during minimally invasive surgeries.

Claims

1. (canceled)

2. A system for medical software tools, comprising: an image stream interface module configured to receive an image stream from a surgical camera, wherein said module includes a CPU, GPU and a FPGA; a user interface overlay module configured to provide a user interface overlay adapted for presentation over the image stream; an optical sensor located corresponding with said surgical camera for registering momentary changes in spectral characteristics reflected from a tissue surface under inspection; a medical software tools module configured to provide a medical software tool through the user interface, the medical software tool being configured to perform an operation with respect to the image stream and provide an output adapted to be presented over the image stream; a medical image processing system for processing patient medical data and corresponding said patient medical data with said momentary changes in spectral characteristics for generating optical signature data indicative of various patient conditions, wherein the medical software tool measures: a. changes in color intensity and b. rates at which said color intensities change in response to: heartbeat pushed blood, breathing pushed oxygen or light intensity or modulation from a light source; wherein a contrast dye is introduced associated with said tissue under inspection in the assessment of the perfusion of tissues; and, wherein an image stream enhancement function with either a de-haze function, a de-blur function, a shadow function, or a thermal turbulence function for enabling system users to obtain an optimized view of a surgical procedure.

3. A system according to claim 2 wherein said contrast dye is fluorescent dye.

4. A system according to claim 2 wherein said spectral characteristics associated with said patient medical data may be displayed and then subsequently displayed over and over as desired by an operator of said system so that said operator can evaluate said patient medical data without regard for patient location.

5. A system according to claim 2 wherein said patient data is shared via a cloud based data system.

6. A system according to claim 2 wherein grid lines are output along with images associated with patient medical data and wherein said grid lines are adjustable to be useful to a user of the system, and wherein the number of grid lines may be established along any axis as selected by said user.

7. A system according to claim 6 wherein said grid lines correspond with prior surgical outcomes and wherein surgeons may utilize said grid lines for enhanced surgical outcomes.

8. A system according to claim 6 wherein said grid lines may correspond to a value corresponding to a cloud based artificial intelligence database for enhancing surgical results.

9. A system according to claim 6 wherein said grid lines may correspond to a degree of opacity.

10. A system according to claim 2 wherein a user may select a perfusion visualization and quantification tool to generate a visual panel for quantifying a degree and a rate for selected patient tissue to absorb and dissipate said contrast fluorescent fluid injected into a patient's blood stream for optimizing said images associated with patient medical data.

11. A system according to claim 2 wherein a user may select a color collaboration tool for providing said user a choice of views of patient tissue surfaces each corresponding with a color map choice.

12. A system according to claim 11 wherein said choice is based on user preference wherein said user selects a color based on margins of diseased tissue.

13. A method for providing medical software tools to surgeons, comprising: an image stream interface module configured to receive an image stream from a surgical camera, wherein said module includes a CPU, GPU and a FPGA; a user interface overlay module configured to provide a user interface overlay adapted for presentation over the image stream; an optical sensor located corresponding with said surgical camera for registering momentary changes in spectral characteristics reflected from a tissue surface under inspection; a medical software tools module configured to provide a medical software tool through the user interface, the medical software tool being configured to perform an operation with respect to the image stream and provide an output adapted to be presented over the image stream; a medical image processing system for processing patient medical data and corresponding said patient medical data with said momentary changes in spectral characteristics for generating optical signature data indicative of various patient conditions, wherein the medical software tool measures: a. changes in color intensity and b. rates at which said color intensities change in response to: heartbeat pushed blood, breathing pushed oxygen or light intensity or modulation from a light source; wherein a contrast dye is introduced associated with said tissue under inspection in the assessment of the perfusion of tissues; and, wherein an image stream enhancement function with either a de-haze function, a de-blur function, a shadow function, or a thermal turbulence function for enabling system users to obtain an optimized view of a surgical procedure.

14. A method according to claim 13 wherein said contrast dye is fluorescent dye.

15. A method according to claim 13 wherein said spectral characteristics associated with said patient medical data may be displayed and then subsequently displayed over and over as desired by an operator of said system so that said operator can evaluate said patient medical data without regard for patient location.

16. A method according to claim 13 wherein said patient data is shared via a cloud based data system.

17. A method according to claim 13 wherein grid lines are output along with images associated with patient medical data and wherein said grid lines are adjustable to be useful to a user of the system, and wherein the number of grid lines may be established along any axis as selected by said user.

18. A method according to claim 17 wherein said grid lines correspond with prior surgical outcomes and wherein surgeons may utilize said grid lines for enhanced surgical outcomes.

19. A method according to claim 17 wherein said grid lines may correspond to a value corresponding to a cloud based artificial intelligence database for enhancing surgical results.

20. A method according to claim 17 wherein said grid lines may correspond to a degree of opacity.

21. A method according to claim 13 wherein a user may select a perfusion visualization and quantification tool to generate a visual panel for quantifying a degree and a rate for selected patient tissue to absorb and dissipate said contrast fluorescent fluid injected into a patient's blood stream for optimizing said images associated with patient medical data.

22. A method according to claim 13 wherein a user may select a color collaboration tool for providing said user a choice of views of patient tissue surfaces each corresponding with a color map choice.

23. A method according to claim 22 wherein said choice is based on user preference wherein said user selects a color based on margins of diseased tissue.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0015] FIG. 1 is a block diagram illustrating an example of the overall processing system that may be used in implementing various features of embodiments of the disclosed technology.

[0016] FIG. 2 is a block diagram illustrating an example of the image processing system that may be used in implementing various features of embodiments of the disclosed technology.

[0017] FIG. 3 is a block diagram illustrating an example medical software tools platform in accordance with some embodiments of the technology described herein.

[0018] FIG. 4. ICG Visualization, a tool that enables the visualization of perfusion of the ICG agent to help identify healthy tissue.

[0019] FIG. 5. Instant Replay, a tool that enables a surgeon to replay a portion of the surgical video and to slow down the visual presentation of the absorption process to better characterize the differences in neighboring tissue.

[0020] FIG. 6. Height Mapping, a tool that allows mapping of green intensity to reveal greater detail in the presence of ICG.

[0021] FIG. 7. Grid Tool, a tool that overlays a dynamically adjustable grid over a selected area of the surgical video. The number of grid lines along the x and y axes can be increased or decreased by user on the fly.

[0022] FIG. 8. Perfusion Visualization and Quantification, a tool that brings up a panel that quantifies the degree and rate of selected tissue to absorb and dissipate contrast fluid (ICG) injected into a patient's blood stream.

[0023] FIG. 9. Color Collaboration, a tool that allow the surgeon to personalized gamma adjustments in real time. The colors are adjusted and displayed to a standard unique to each individual user.

[0024] FIG. 9.1 is an example of different types of color processing for the Color Collaboration tool.

[0025] FIG. 10. is a block diagram illustrating the ICG Visualization module within the medical software tools platform.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

[0026] FIG. 1 is a block diagram illustrating an example of the overall processing system that may be used in implementing various features of embodiments of the disclosed technology. In accordance with the preferred embodiment of the present invention, the processing system 100 consists of processor elements such as: a central processing unit (CPU) 102; a graphics processing unit (GPU) 104; and a field programmable gate array (FPGA) 106. The processing system 100 may be used to retrieve and process raw data derived from a surgical camera 110 or a data storage device, such as a medical archive 108. The surgical camera 110 or medical archive 108 transmits a data stream to the processing system 100, whereby that data is processed by the CPU 102. The FPGA 106, connected to the CPU 102 and the GPU 104, simultaneously processes the received data by using a series of programmed system algorithms 118, thus functioning as an image clarifier within the processing system 100. The GPU 104 communicates with the user interface 112 to display the received data from the medical archive 108. The GPU 104 enables the user interface to then communicate the data to connected input devices 114 and output devices 116. The user interface 112 can communicate to multiple input 114 and output devices 116 simultaneously. An input device 114 can include, for example, a keyboard, touchscreen or voice activated device. An output device 116 can include, for example, a video display, a digital video recorder (DVR) or universal serial bus (USB).

[0027] FIG. 2 is a block diagram illustrating an example of the image processing system that may be used in implementing various features of embodiments of the disclosed technology. In accordance with the preferred embodiment of the present invention, the image processing system 200 consists of three components that process image data received from a sensor 202 in order to send that data to a display or video router 210. The three components of the image processing system 200 are: camera head video pre-processing 204; real time video enhancement 206; and the video display transport 208 function. Image data is collected by a sensor imaging device 202, and is then transmitted to the camera head video pre-processing component 204 within the image processing system 200. This data may be, for example, a raw video image that is pre-processed using various image processing algorithms. Image pre-processing may also include software modules for image registration and segmentation to optimize the video data and communicate via the system bus 212 with the internal system processors 214: the CPU; GPU; and FPGA.

[0028] The pre-processed image data is transmitted to the real time video enhancement 206 component, whereby the image data is enhanced to improve clarity or highlight certain details. Once the image data resolution has been enhanced, the video display transport 208 component completes image post-processing, formatting from the initial sensor resolution to the eventual display resolution, for example, enhancing the video data to 1080p HD or 4K display resolution or using software modules such as video cross conversion, scaling and adding graphic overlays. The processed image data is then transmitted from the image processing system 200 to the display or video router 210. The video display transport also saves the processed image data to the processing system memory 216 that can consist of internal and external memory storage.

[0029] FIG. 3 is a block diagram illustrating an example medical software tools platform system in accordance with some embodiments of the technology described herein. The medical software tools platform provides access to medically-oriented applications (apps) or tools that can assist members of the surgical team during an operation. For example, many surgeries utilize ICG to enable surgeons to visualize tissue perfusion and anatomical structures, however, the camera sensor captures much more color detail that the human eye can discern. Therefore, the use of an ICG visualization tool can provide a more accurate and objective measure of the rate of perfusion. An instant replay tool enables the users to replay a portion of the surgical video in slow motion in order to more closely observe the process of perfusion. A height mapping tool provides a simulated 3D visualization that enables the user to quickly and accurately identify areas of highest absorption of the ICG fluorescent dye. A grid tool brings up an overlay that draws grid lines to provide an x,y location reference. This enables a surgeon to precisely mark the location of an anomaly or area of interest for future reference. A perfusion visualization and quantification tool precisely measure the rates of rise and decay of the of fluoresced pixel intensity. This is intended to assist a surgeon in identifying the borders between healthy and diseased tissue. A color calibration tool allows users to configure a color preference so that colors are adjusted and displayed to a standard unique to each individual. This enables color blind surgeons to better visualize information on the surgical display and perceive subtle changes in color.

[0030] In accordance with the preferred embodiment of the present invention, the medical software tools platform system 300 includes: an image stream interface module 302; a user interface overlay module 304; medical software tools 310; a medical device interface module 306; and an image stream processing system interface module 308. The medical software tools platform system 300 may be integrated, in whole or in part, into a video display or an image stream processing system utilized in an operating room. The image stream interface module 302 may receive an image stream acquired by a surgical camera or the like. Depending on the embodiment, the image stream may be received directly from the surgical camera, or may be provided by way of one or more components, such as an image stream processing system. The image stream received from the image stream interface module 302 may vary in resolution, frame rate, format, and protocol according to the surgical camera or the image stream processing system providing the image stream.

[0031] The user interface overlay module 304 may provide a user interface to the medical software tools platform system 300, which may include one or more graphical user interface (GUI) elements presented over the image stream received through the image stream interface module 302. For some embodiments, the user interface comprises a bottom toolbar configured to be presented over the image stream, and configured to provide access to various medical software tools 310 available through the medical software tools platform system 300.

[0032] The medical software tools platform system 300 may include one or more medical software tools, such as medically-oriented applications or widgets, which can be utilized with respect to the image stream being received through the image stream interface module 302. The medical software tools 310 include but are not limited to: a ICG Visualization module 312; an instant replay module 314; a height mapping module 316; a grid tool module 318; an perfusion visualization and quantification module 320; a color calibration module 322.

[0033] The medical device interface module 306 may facilitate communication between the medical software tools platform system 300, one or more of the medical software tools 310, and one or more various medical devices utilized in an operating room. The image stream processing system interface module 308 may facilitate communication between the medical software tools platform system 300 and an image stream processing system utilized to process an image stream acquired by a surgical camera or the like. Through the communication, the image stream processing system interface module 308 may transmit control data to an image stream processing system, or receive an image stream from a surgical camera as processed by the image stream processing system. The image stream processing system interface module 308 may include various data interfaces, including wired or wireless network interfaces and serial communication interfaces.

[0034] ICG Visualization (FIG. 4) tool enables the visualization of perfusion of the ICG agent to help identify healthy tissue. The ICG Visualization tool can be used in conjunction with other tools to enable the following capabilities: 1. Slow Motion—replaying the video in slow motion 2. Intensity—quantifying the degree of ICG absorption for defined areas of interest 3. Speed—measuring the time to full absorption for defined areas of interest 4. ICG Map—simulated 3D views of ICG absorption using intensity or speed of ICG data.

[0035] In a preferred embodiment the ICG Visualization tool can be used in conjunction with the Slow Motion Replay tool 402 and the Height Mapping tool 404, which can be integrated into the ICG Visualization user interface. In this implementation, a surgeon might use the ICG Visualization in conjunction with the Slow Motion Replay tool and the Height Mapping tools in a four step process as follows.

[0036] In a first step, a user, such as the surgeon or an assistant, uses the mouse to sweep an area of interest. The selected image segment is scaled and centered in the yellow working box as show in the upper left corner 408.

[0037] As ICG is administered, the computer measures time to reach full luminance. The example in FIG. 4 shows this time to be 12 seconds 410.

[0038] Video frames are cached in memory a full frame rate. Slow Motion replays are available at selected speeds and use motion-compensated frame interpolation for a smooth playback. An elevator bar on the right side is used to control image zoom.

[0039] In a second step, an operator uses the mouse to create a box around one or more regions of interest, as shown in FIG. 4 on the right where three yellow boxes are selected 406.

[0040] The intensity and the time to reach full intensity for each region is graphed using the ICG channel to provide a quantitative comparison 410.

[0041] Selected regions are described by outline boxes (dashed yellow areas in the example 406). Unchecking the form box labeled, “Show Regions” hides the outlines 414. Region outlines can be moved and resized with the graphs updated in real time.

[0042] In a third step, the system quantifies perfusion based on intensity (absorption) and speed 404.

[0043] In a fourth step, the user has the option to invoke slow motion instant replays at selectable speed so that the details of the process of perfusion can be reviewed and studied 402.

[0044] The user also has the option to invoke Height Map tool which shows ICG data in simulated 3D to demonstrate areas of highest perfusion. Both intensity and speed can be mapped and the image can be rotated 404.

[0045] Instant Replay (FIG. 5) tool enables a surgeon to replay a portion of the surgical video. An example above described how Instant Replay can be used to review the speed of ICG contrast agent perfusion to provide the surgeon an opportunity to slow down the visual presentation of the absorption process to better characterize the differences in neighboring tissue. FIG. 5 shows how Instant Replay works. Video from the endoscope is recorded and stored in a special purpose computer that implements various image processing algorithms 500. The last n seconds of video frames are stored on a circular video frame buffer within the computer 502. Instant Replay uses video frames from the computer circular memory buffer, playing frames at selected frame rates. The user sets the recall time period and speed from slow motion to speed forward 504. The user can select algorithms to apply to the video as it is being displayed 506. The user can also select levels of contrast and brightness for the replay 508.

[0046] The Height Map tool is shown in FIG. 6. To use the tool, the user first sweeps an area of interest with the mouse. The selected area is enlarged and rendered as a “Picture-In-Picture” insert 600. The insert simulates a 3D view by mapping the area's pixels onto an elevation grid using the pixel intensities to create a height map. The user can experiment with different views and degrees of detail by rotating the insert in 3D using a mouse, adjusting for more or less detail with the window/level sliders 602, and by scaling with the vertical bar 604. The Pulse button 606 animates the pixel heights in relation to the intensity change resulting from the pulse of pumping blood into the local vascular network. This creates a visual effect of pixels dancing in synchronization with the patient's heartbeat. The rise and fall height and rate of change in pixel intensity demonstrate the different responses of diseased and healthy tissue.

[0047] The Grid Tool is shown in FIG. 7. Selecting the Grid tool brings up an overlay that draws grid lines to provide an x,y location reference 710. The icon includes a “minus” and a “plus” touch point to grow or shrink the grid resolution 700. Double clicking brings up a sub-menu 702 to select full or border grid lines and select line color and opacity. Default settings from the Personal Preferences are used when the tool is initialized.

[0048] The grid is dynamically adjustable and the number of grid lines in x and y can be increased or decreased by user on the fly. The Grid Lines are selectable such that the user can select lines at borders 704, across full image 708, or none. Users can configure the attributes of the grid lines to set line weight, color and opacity 702. In multi-display environments, the user can select which displays have grid lines. The user can configure the systems to display the coordinates of the visible cursor location 712. This visual confirmation of cursor location can eliminate doubt and ease cognitive burden for the user. The system also offers the ability to save the personal settings for each user, including an option to recall “Last Used” settings.

[0049] The Perfusion Visualization and Quantification tool is shown in FIG. 8. Selecting the Perfusion Visualization and Quantification tool brings up a panel that quantifies the degree and rate of selected tissue to absorb and dissipate contrast fluid (ICG) injected into a patient's blood stream 800. The tool displays both endoscopic video channels (RGB 802 and IR 804) and a third fused image with the IR illuminated tissue transformed to fluorescent green for overlay on the RGB image 806. The purpose is to assist the surgeon in characterizing the state of the tissue. Default settings from the Personal Preferences are used when the tool is initialized.

[0050] The user sweeps an area of interest 806 to see a graph of the average pixel intensity (Y-axis) plotted against time (X-axis) 800. The curve provides a quantitative measure of the tissue's absorption and decay rate for comparison with other areas to locate the healthiest tissue. Graphs respond in real time and may be moved about the image to explore responses. Graph results are plotted in different colors for comparison purposes 800.

[0051] The graph 800 provides an objective method to quantify the perfusion process using the degree of absorption and also the time to reach maximum intensity and the time to dissipate the agent. Without this tool, surgeons must rely on subjective measurements of color intensity and time.

[0052] The tool enables a user to measuring rise time, the time it takes for a fluoresced pixel intensity to achieve maximum luminance, and the decay time, the time it take for a fluoresced pixel intensity to achieve initial state. The instant replay feature allows the user to view the fluorescing period at selective speeds to better observe the process. A graphical representation shows the amplitude and time of the fluorescing event in real time 800. The tool is able to capture and compare multiple samples using software mechanisms to collect and compare multiple samples 806, 800.

[0053] The Color Collaboration tool, shown in FIG. 9, presents the surgeon with a choice of views of the tissue surface with a choice of color maps 900. The goal is to more clearly characterize the margins of diseased tissue, based on the individual color preference of a user.

[0054] FIG. 9.1 is an example of different types of color processing for the Color Collaboration tool. In accordance with the preferred embodiment of the present invention, surgeons use color to identify and characterize tissue in many ways. However individual humans perceive color differently and surgery frequently involves a team of specialists, each perceiving color in their own way. Some form of color blindness effects ˜8% of men and 0.5% of women and there are many types of color blindness—FIG. 9.1 illustrates three common forms of color blindness.

[0055] By first sampling a surgeon's color perception against reference images 900, the colors in the endoscopic video can be adjusted to present an image closer to a standard reference. Team members can have unique individual adjustments effected simultaneously by using dedicated displays, or special eyeglasses equipped with miniature color displays, or a primary display that can switch between color tables on command.

[0056] Main Menu settings personalize the default choices for each surgeon. The defaults include the presentation format and color translation tables to display when the Color Map icon is selected 902. Mouse clicking a selection brings it to the primary display window where it is subject to other tools for further analysis as shown in FIG. 9.

[0057] FIG. 10 is a block diagram illustrating the ICG Visualization module 1012 within the medical software tools platform. In accordance with the preferred embodiment of the present invention, an optical sensor 1002 located within an endoscopic camera lens 1000 registers the color characteristics reflected from a tissue surface. This image data 1004 is transmitted to the medical image processing system 1006, which can also receive data from a device such as a patient heart rate monitor 1008. The medical image processing system 1006 processes and transmits the image data 1004 from the optical sensor with data from the heart rate monitoring device 1008 to the user interface 1010, whereby all sets of data are received through the optical signature measurement module 1012 displaying the rate of change 1014 in the color characteristics simultaneously with the patient vitals data 1016. This data is analyzed 1018 in the module and the results of any anomaly detection 1020 are displayed to the user. The slight changes in one of more of the color components, and the speed at which they change, in relation to a source stimulus (heartbeat, breath, external stimulus) indicates the arrival of blood, contrast agents, or oxygen absorption. Combinations of the components can be used to characterize different types and states of disease and to identify the margins of tumors and diseased areas.

[0058] Although the disclosed technology is described above in terms of various exemplary embodiments and implementations, it should be understood that the various features, aspects and functionality described in one or more of the individual embodiments are not limited in their applicability to the particular embodiment with which they are described, but instead may be applied, alone or in various combinations, to one or more of the other embodiments of the disclosed technology, whether or not such embodiments are described and whether or not such features are presented as being a part of a described embodiment. Thus, the breadth and scope of the technology disclosed herein should not be limited by any of the above-described exemplary embodiments.

[0059] Terms and phrases used in this document, and variations thereof, unless otherwise expressly stated, should be construed as open ended as opposed to limiting. As examples of the foregoing: the term “including” should be read as meaning “including, without limitation” or the like; the term “example” is used to provide exemplary instances of the item in discussion, not an exhaustive or limiting list thereof; the terms “a” or “an” should be read as meaning “at least one,” “one or more” or the like; and adjectives such as “conventional,” “traditional,” “normal,” “standard,” “known” and terms of similar meaning should not be construed as limiting the item described to a given time period or to an item available as of a given time, but instead should be read to encompass conventional, traditional, normal, or standard technologies that may be available or known now or at any time in the future. Likewise, where this document refers to technologies that would be apparent or known to one of ordinary skill in the art, such technologies encompass those apparent or known to the skilled artisan now or at any time in the future.

[0060] The presence of broadening words and phrases such as “one or more,” “at least,” “but not limited to” or other like phrases in some instances shall not be read to mean that the narrower case is intended or required in instances where such broadening phrases may be absent. The use of the term “module” does not imply that the components or functionality described or claimed as part of the module are all configured in a common package. Indeed, any or all of the various components of a module, whether control logic or other components, may be combined in a single package or separately maintained and can further be distributed in multiple groupings or packages or across multiple locations.

[0061] Additionally, the various embodiments set forth herein are described in terms of exemplary block diagrams, flow charts and other illustrations. As will become apparent to one of ordinary skill in the art after reading this document, the illustrated embodiments and their various alternatives may be implemented without confinement to the illustrated examples. For example, block diagrams and their accompanying description should not be construed as mandating a particular architecture or configuration.

[0062] Embodiments presented are particular ways to realize the invention and are not inclusive of all ways possible. Therefore, there may exist embodiments that do not deviate from the spirit and scope of this disclosure as set forth by appended claims, but do not appear here as specific examples. It will be appreciated that a great plurality of alternative versions are possible.