IMAGE-BASED INSTRUMENT IDENTIFICATION AND TRACKING

Abstract

Disclosed is a computer-implemented method of transmitting identification information of a medical instrument. The method encompasses comparing a digital image of an instrument tray and an instrument to a digital image of just the instrument tray to determine the identity of the instrument. A characteristic geometry such as its envelope is assigned to the instrument, and a characteristic quantity of the envelope such as its aspect ratio may be used to identify the instrument. Based on determining, from the image of the instrument and the instrument tray, the relative position between those two entities, the method determines whether the instrument has been taken from the instrument tray, and accordingly instructs a medical computing system about this determination. The medical computing system may then determine whether the correct instrument has been taken from the instrument has been taken from the instrument tray, for example by comparison with medical procedure planning data.

Claims

1. A computer-implemented method of transmitting identification information of a medical instrument, the method comprising: acquiring initial state image data by a computer and from an associated camera operatively coupled with the computer, the initial image data being representative of an image of an instrument tray in an initial state; acquiring instrument state image data by the computer from the associated camera, the instrument state image data being representative of an image of an instrument and the instrument tray, with the instrument being located on the instrument tray; determining instrument envelope data based on the initial state image data and the instrument state image data, wherein the instrument envelope data describes a position of an envelope around the image representation of the instrument in the instrument state image data relative to a position of the instrument tray; acquiring instrument identity data and assigning the instrument identity data to the instrument envelope data based on a user input; determining instrument usage data based on the instrument envelope data, wherein the instrument usage data describes that the envelope around the image representation of the instrument has attained a predetermined position relative to the position of the instrument tray; and transmitting the instrument identity data by the computer, based on the instrument usage data, to a further associated computer of an associated medical computing system.

2. The method according to claim 1, wherein: the acquiring the initial state image data and the instrument state image data comprises acquiring the initial state image data and the instrument state image data by the computer from an associated camera configured to image the instrument tray in an initial state and the instrument being located on the instrument tray in an infrared wavelength range.

3. The method according to claim 1, wherein the determining the instrument envelope data comprises determining a difference between the image of the instrument tray in the initial state and the image of the instrument being located on the instrument tray.

4. The method according to claim 1, further comprising tracking a position of the instrument.

5. The method according to claim 1, wherein the determining the instrument envelope data comprises determining instrument envelope data that describes a generic bounding box shape that is independent of a physical shape of the instrument.

6. The method according to claim 5, wherein the determining the instrument envelope data comprises determining instrument envelope data that describes an envelope having a predetermined distance around a circumference of the image representation of the instrument.

7. The method according to claim 1, further comprising; determining instrument position data based on the instrument envelope data, wherein the instrument position data describes the position of the instrument relative to the position of the instrument tray.

8. The method according to claim 7, wherein the acquiring the instrument identity data comprises acquiring the instrument identity data based on the instrument position data.

9. The method according to claim 1, further comprising: extracting from the instrument envelope data an image signature of the instrument comprising an aspect ratio of an envelope around the image representation of the instrument.

10. The method according to claim 9, wherein the determining the instrument identity data comprises determining the instrument identity data based on the image signature.

11. The method according to claim 1, further comprising acquiring infrared image data that describes an infrared image comprising an image representation of the instrument tray.

12. The method according to claim 1, wherein the acquiring the instrument state image data comprises acquiring an image representation of an optical code disposed on the instrument, wherein the optical code comprises information content, and wherein the instrument identity data is determined further based on the information content of the optical code.

13. A computer program comprising instructions that, when the program is executed by a processor device of an associated computer, cause the associated computer to perform a method of transmitting identification information of a medical instrument, the method comprising: acquiring initial state image data by a computer from an associated camera operatively coupled with the computer, the initial image data being representative of an image of an instrument tray in an initial state: acquiring instrument state image data by the computer from the associated camera, the instrument state image data being representative of an image of an instrument and the instrument tray, with the instrument being located on the instrument tray: determining instrument envelope data based on the initial state image data and the instrument state image data, wherein the instrument envelope data describes a position of an envelope around the image representation of the instrument in the instrument state image data relative to a position of the instrument tray: acquiring instrument identity data and assigning the instrument identity data to the instrument envelope data based on a user input; determining instrument usage data based on the instrument envelope data, wherein the instrument usage data describes that the envelope around the image representation of the instrument has attained a predetermined position relative to the position of the instrument tray; and transmitting the instrument identity data by the computer, based on the instrument usage data, to a further associated computer of an associated medical computing system.

14. A medical system comprising: a computer comprising a processor, a storage medium device operatively coupled with the processor, and a computer program stored in the storage medium device; and a camera operably coupled with the computer and being operable for generating initial state image data representative of an image of an instrument tray in an initial state and instrument state image representative of an image of an instrument and the instrument tray with the instrument being located on the instrument tray, wherein the processor is operable to execute the computer program to perform a method comprising: acquiring the initial state image data by the camera: acquiring the instrument state image data by the camera: determining instrument envelope data based on the initial state image data and the instrument state image data, wherein the instrument envelope data describes a position of an envelope around the image representation of the instrument in the instrument state image data relative to a position of the instrument tray: acquiring instrument identity data and assigning the instrument identity data to the instrument envelope data based on a user input; determining instrument usage data based on the instrument envelope data, wherein the instrument usage data describes that the envelope around the image representation of the instrument has attained a predetermined position relative to the position of the instrument tray; and transmitting the instrument identity data by the computer, based on the instrument usage data, to a further associated computer of an associated medical computin system.

15. (canceled)

16. The method according to claim 1, wherein: the acquiring the instrument identity data comprises labelling the instrument by receiving via a user interface an instruction from the user that assigns a name or other identifier to the image representation of the instrument in the instrument state.

17. The computer program according to claim 13, further comprising: determining instrument position data based on the instrument envelope data, wherein the instrument position data describes the position of the instrument relative to the position of the instrument tray.

18. The computer program according to claim 17, wherein the acquiring the instrument identity data comprises acquiring the instrument identity data based on the instrument position data.

19. The computer program according to claim 13, wherein: the acquiring the instrument identity data comprises labelling the instrument by receiving via a user interface an instruction from the user that assigns a name or other identifier to the image representation of the instrument in the instrument state.

20. The medical system according to claim 14, wherein the processor is operable to execute the computer program to: determine instrument position data based on the instrument envelope data, wherein the instrument position data describes the position of the instrument relative to the position of the instrument tray, wherein the acquiring the instrument identity data comprises acquiring the instrument identity data based on the instrument position data.

21. The medical system according to claim 14, wherein: the acquiring the instrument identity data comprises labelling the instrument by receiving via a user interface an instruction from the user that assigns a name or other identifier to the image representation of the instrument in the instrument state.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In the following, the invention is described with reference to the appended figures which give background explanations and represent specific embodiments of the invention. The scope of the invention is however not limited to the specific features disclosed in the context of the figures, wherein

[0038] FIG. 1 illustrates the basic flow of the method according to the first aspect; and

[0039] FIG. 2 is a schematic illustration of the system according to the fifth aspect.

DESCRIPTION OF EMBODIMENTS

[0040] FIG. 1 illustrates the basic steps of the method according to the first aspect, in which step S11 encompasses acquisition of the initial state image data, step S12 encompasses acquisition of the instrument state image data and subsequent step S13 encompasses determination of the instrument envelope data. Then, step S14 continues with acquisition of the instrument identity data. Step S15 is directed to determining the instrument usage data, and step S16 to the transmission of the instrument identity data to a further computer.

[0041] FIG. 2 is a schematic illustration of the medical system 1 according to the fifth aspect. The system is in its entirety identified by reference sign 1 and comprises a computer 2, a camera 3 for generating the initial state image data and the instrument state image data and a medical computing system 4 comprising a further computer. The components of the medical system 1 have the functionalities and properties explained above with regard to the fifth aspect of this disclosure.

[0042] In one example, the disclosed method encompasses a labelling process during setup of the instrument tray by operation room technicians.

[0043] This labelling process can be used for verification that the necessary instrument is present by comparing to a list of instruments during the later workflow.

[0044] The idea is to identify an instrument that is picked up from the instrument tray not by their shape or their marker (such as their envelope), but by their position inside the instrument tray.

[0045] Once the position-identity relation is established, the shape or marker can be stored and later used during the same procedure, once the instrument is put somewhere else in the instrument tray.

[0046] The relation between the position and the instrument can be obtained by either [0047] A manual labelling during the setup (filling) of the instrument tray: the user identifies the instruments optically or they are identified using markers; optional: the user verifies during this manual step, that all instruments are present. [0048] By using a standard setup from a database for the type of surgery. In that database, the type of surgery, the instrument and the position in the instrument tray is stored. [0049] The relation is stored by assigning a two-dimensional position or area relative to the tray coordinates to an instrument type.

[0050] An optical camera is provided above the instrument tray. When by image analysis it is found that an instrument is taken out from a position/area of interest, the identity of the instrument is known and can be transferred to the tracking and display system, e.g. for more accurate tracking because the shape of the instrument is known or for more accurate augmentation or visualization of the correct instrument. The type used can also be stored for reporting or can be used for identifying errors.

[0051] The method can be outlined as follows: [0052] Define position and identification data in tray [teach in] [0053] When instrument is picked up, measure position (x,y) and identify instrument [tracking] [0054] Use instrument identity e.g. for tracking the tip of the instrument, e.g. when the tip position is calculated from a marker position using the exact geometry of the instrument, visualization of the instrument, e.g. when the way the instrument is displayed depends on the identity, augmentation, e.g. when the information displayed in an augmented reality device depends on the type of instrument, e.g. a drill diameter, a warning, e.g. when the type of instrument doesn't fit to the type of procedure, reporting of the type of instruments used, plausibility check, e.g. if the combination of different instruments can be used in a surgery, e.g. a drill bit and a drill, e.g. a power drill.

[0055] Optional variations of the method are: [0056] When instrument is put back in, identify risk of accidental exchange, e.g. provide guidance where to put it or give a warning if it was misplaced. [0057] Store instrument shape or other instrument identifiers once it is used, so it can be identified by the shape or identifier during the next use. [0058] Trigger re-teach if necessary, e.g. if the user misplaces it or if the identification doesn't match the database entry. [0059] Give augmented reality guidance during instrument tray setup from predefined database of position/identity relations [0060] Store preferred instrument tray setups per indication and/or user; then augmented guiding for instrument tray setup can be provided [0061] Ask user for confirmation that the right instrument is present during teach in.