Removing ghost markers from a measured set of medical markers
10699409 ยท 2020-06-30
Assignee
Inventors
Cpc classification
A61B2090/3983
HUMAN NECESSITIES
A61B6/5258
HUMAN NECESSITIES
A61B2090/3966
HUMAN NECESSITIES
International classification
Abstract
A method for removing ghost markers from a measured set of medical markers, wherein the measured set of markers represents the positions of real markers and the positions of ghost markers, comprising the steps of: calculating all possible minimal marker sets, wherein a minimal marker set is a subset of the measured set and a minimal marker set consists of the smallest possible number of markers which would cause the measured set to be measured; calculating the smallest extent of each minimal marker set, wherein the smallest extent is the smallest extent among the extents in three orthogonal directions; and selecting the minimal marker set having the smallest out of the smallest extents as a marker set without ghost markers.
Claims
1. A method for removing ghost markers from a measured set of medical markers, wherein the measured set of markers represents the positions of real markers and the positions of ghost markers, comprising the steps of: measuring the set of medical markers with a first and a second camera; calculating all possible minimal marker sets, wherein each of the calculated minimal marker sets: is a subset of the measured set, and consists of the smallest possible number of markers which would cause the measured set to be measured; calculating the smallest extent of each minimal marker set, wherein the smallest extent is the smallest extent among the extents in three orthogonal directions; and selecting the minimal marker set having the smallest extent out of the smallest extents as a marker set without ghost markers.
2. The method of claim 1, wherein the three orthogonal directions are set such that the smallest extent of a minimal marker set is minimized.
3. The method of claim 1, wherein the step of calculating the smallest extent involves calculating the principal axes of the minimal marker set and using the directions of the principal axes as the three orthogonal directions.
4. The method of claim 3, wherein the principal axes correspond to eigenvectors of a covariance matrix of the positions of the markers in the measured set in a measurement coordinate system.
5. The method of claim 4, wherein the eigenvalues of the covariance matrix are calculated and the smallest extent is calculated on the basis of at least one of the eigenvalues.
6. The method of claim 5, wherein the smallest eigenvalue of each minimal marker set is used as the smallest extent of the minimal marker set.
7. The method of claim 5, wherein the product of the eigenvalues is calculated for each minimal marker set and this product is used as the smallest extent.
8. The method of claim 1, wherein the measured set of medical markers is measured using two cameras and calculating the minimal marker sets involves the steps of: acquiring a first line set for the first camera and a second line set for the second camera, wherein each line set consists of a set of straight lines in space, each straight line runs through the front lens of the corresponding camera and at least one real marker lies on a straight line; finding a ghost marker susceptible line set, wherein a ghost marker susceptible line set comprises all lines of the first and second line sets which lie in the same plane and each of those lines intersects at least two lines of the respective other line set in the ghost marker susceptible line set, and comprises at least two lines of each line set; determining the minimal marker sets as those marker sets for which each line of the line set having more of its lines in the ghost marker susceptible line set intersects exactly one line of the other line set and there is at least one marker on each line of the other line set.
9. The method of claim 1, wherein the measured set of medical markers is measured using a stereoscopic camera comprising two cameras and calculating the minimal marker sets involves the steps of: acquiring, from the stereoscopic camera, a marker matrix assigning each marker in the measured set to one line in a first line set for the first camera and to one line in a second line set for the second camera, wherein each line set consists of a set of straight lines in space, each straight line runs through the front lens of the corresponding camera and at least one real marker lies on a straight line; finding all sub-matrices of the marker matrix in which every line of the larger one of the two line sets has exactly one marker assigned thereto and the lines of the other line set only have markers assigned thereto which correspond to the markers assigned to the lines in the larger line set; determining the minimal marker sets from the sub-matrices.
10. The method of claim 1, wherein the measured set of medical markers is measured using a stereoscopic camera comprising two cameras and calculating the minimal marker sets involves the steps of: acquiring, from the stereoscopic camera, a marker matrix assigning each marker in the measured set to one line in a first line set for the first camera and to one line in a second line set for the second camera, wherein each line set consists of a set of straight lines in space, each straight line runs through the front lens of the corresponding camera and at least one real marker lies on a straight line; finding all sub-matrices of the marker matrix in which every line of the first line set and every line of the second line set has exactly one marker assigned thereto and every marker in the sub-matrix is assigned to exactly one line of the first line set and to exactly one line of the second line set; and determining the minimal marker sets from the sub-matrices.
11. A method for removing ghost markers from a measured set of medical markers, wherein the set of measured markers represents the positions of real markers and the positions of ghost markers and was obtained by use of a stereoscopic camera comprising two cameras, comprising the steps of: finding a potential ghost marker subset of the measured set, the potential ghost marker subset comprising real markers and ghost markers caused by the real markers; calculating the distance of each marker in the potential ghost marker subset to the stereoscopic camera; removing the marker with the smallest distance and the marker with the largest distance from the measured set and the potential ghost marker subset; repeating the removing step if the potential ghost marker subset comprises further potential ghost markers.
12. The method of claim 11, wherein the distance between a marker and the stereoscopic camera is the distance between the marker and the middle of a straight line which connects the two cameras.
13. The method of claim 12, wherein the step of finding a potential ghost marker subset involves the steps of: acquiring a first line set for the first camera and a second line set for the second camera, wherein each line set consists of a set of straight lines in space, each straight line runs through the front lens of the corresponding camera and at least one real marker lies on a straight line; finding a ghost marker susceptible line set, wherein a ghost marker susceptible line set comprises all lines of the first and second line sets which lie in the same plane and each of those lines intersects at least two lines of the respective other line set in the ghost marker susceptible line set, and comprises at least two lines of each line set; adding all markers of the measured set to the potential ghost marker subset that lie on an intersection of two lines of the ghost marker susceptible line set.
14. At least one non-transitory computer-readable medium comprising instructions that, in response to execution of the instructions by one or more processors, cause the one or more processors to perform the following operations to remove ghost markers from a measured set of medical markers, wherein the measured set of markers represents the positions of real markers and the positions of ghost markers: measure the set of medical markers with a first and a second camera; calculating all possible minimal marker sets, wherein each of the calculated minimal marker sets: is a subset of the measured set, and consists of the smallest possible number of markers which would cause the measured set to be measured; calculate the smallest extent of each minimal marker set, wherein the smallest extent is the smallest extent among the extents in three orthogonal directions; and select the minimal marker set having the smallest extent out of the smallest extents as a marker set without ghost markers.
15. A system to remove ghost markers from a measured set of medical markers, wherein the measured set of markers represents the positions of real markers and the positions of ghost markers: at least one computer having at least one processor and associated memory; a first and a second camera operably connected to the at least one computer; computer instructions stored in the associated memory, which when executed by the at least one processor, cause the at least one processor to: measure the set of medical markers with the first and a second camera; calculate all possible minimal marker sets, wherein each of the calculated minimal marker sets: is a subset of the measured set, and consists of the smallest possible number of markers which would cause the measured set to be measured; calculate the smallest extent of each minimal marker set, wherein the smallest extent is the smallest extent among the extents in three orthogonal directions; and select the minimal marker set having the smallest extent out of the smallest extents as a marker set without ghost markers.
Description
BRIEF DESCRIPTION OF DRAWINGS
(1) In the following, the invention is described with reference to the enclosed figures which represent preferred embodiments of the invention. The scope of the invention is not however limited to the specific features disclosed in the figures, which show:
(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9)
(10)
(11)
(12)
(13)
(14)
(15)
DETAILED DESCRIPTION
(16)
(17) The system 1 comprises a computer 2 connected to two cameras 3a and 3b with a known relative position between the two cameras 3a, 3b. The cameras 3a and 3b constitute a stereoscopic camera which captures images which depict medical markers. The location of the medical markers relative to the position of the cameras 3a, 3b can be obtained by image analysis as known in the art. This analysis can be performed in the cameras 3a, 3b, the computer 2 or a combination thereof.
(18) The computer 2 comprises a central processing unit (processor) 4, a memory 5 and an interface 6. Via the interface 6, the computer can be coupled to external devices, such as the cameras 3a, 3b. The processor 4 executes program code in order to perform the methods described herein. The memory 5 stores the program code and/or processed data and/or data to be processed.
(19) The computer 2 is connected to an input device 7, such as a mouse, a keyboard, a touchpad or a touchscreen, and an output device 8, such as a monitor.
(20)
(21) A measured set of medical markers also comprises so-called ghost markers M4 and M5 as explained below.
(22) The principle of detecting markers by use of cameras 3a and 3b is explained with reference to
(23)
(24) The marker M3 can be unambiguously identified at the intersection of lines L3 and R1. However, lines L1, L2, R2 and R3 lie in the same plane. This means that there are not only intersections of lines L2 and R2 at the spatial location of marker M2 and of lines L1 and R3 at the spatial location of marker M1, but there is also an intersection of lines L1 and R2 and an intersection of lines L2 and R3. This means that markers M4 and M5 are detected at the spatial locations of those intersections. Since only markers M1 to M3 are real markers, markers M4 and M5 are referred to as ghost markers. The measured set of medical markers therefore comprises real markers M1 to M3 and ghost markers M4 and M5 and their respective locations in a reference system of the stereoscopic camera. The present invention aims at removing the ghost markers from the measured set of medical markers.
(25)
(26)
(27) In step S02, all possible minimal marker sets are calculated from the marker matrix. As can be seen from
(28) Step S02 therefore finds all subsets of the measured set which would create ghost markers such that the measured set results. In the present example, a first minimal marker set comprises markers M1, M2 and M3 and a second minimal marker set comprises markers M3, M4 and M5.
(29) In the present embodiment, a minimal marker set is calculated by iterating through all lines of the larger one of the line sets comprised in the marker matrix. If both line sets comprise the same number of lines, then any one of the two line sets is used as the larger line set. In the following explanation, the process iterates through lines L1 to L3. Details of this process are shown in
(30) It shall be noted that step S02 starts with an empty minimal marker set. Step S02a involves selecting one of the lines, for example line L1. Step S02b involves selecting a marker on the selected line. For example, marker M1 is selected. In step S02c, the selected marker is added to the minimal marker set and the other entries in the selected line are removed in step S02d, thus removing all other markers from this line. In this example, the entry for marker M4 is removed from line L1.
(31) In step S02e, the line of the other line set in the marker matrix to which the selected marker (M1 in the present example) is also assigned is identified. In the present example, this is line R3. Step S02f then involves finding all deletable markers on the other line. A deletable marker is a marker which is not assigned to a line not selected yet (lines L2 and L3 at the present stage of this example) and which is not comprised in the minimal marker set. At the present stage of this example, there is no deletable marker on line R3 because marker M5, which is assigned to line R3, is also assigned to line L2, which was not selected so far. Step S02g then involves removing all entries belonging to the deletable markers found in step S02f from the marker matrix.
(32) Step S02h involves determining whether or not there are more lines to be tested. This is the case (with lines L2 and L3 which have not been selected yet), such that the process returns to step S02a for a new iteration and selects a new line, for example line L2. In step S02b, for example marker M2 on line L2 is selected and added to the minimal marker set in step S02c. The matrix entry for marker M5 is then deleted for line L2 in step S02d. Then line R2 is found in step S02e and marker M4 is identified as a deletable marker in step S02f. Marker M4 is not assigned to a yet unselected line (line L3) and is not comprised in the minimal marker set. The matrix entry for marker M4 in line R4 is then removed in step S02g.
(33) It is then determined step S02h that there is still another line to be analyzed. In the next iteration, line L3 is then selected in step S02a and marker M3 on this line is then selected in step S02b. In step S02c, marker M3 is added to the minimal marker set. It is determined in step S02d that there are no other matrix entries to be removed from line L3. Line R1 is then determined in step S02e, and it is found in step S02f that there are no deletable matrix entries in line R1, such that step S02g does not remove any matrix entries.
(34) It is then determined in step S02d that there is no more line to be analyzed, such that the process proceeds to step S02i, in which invalid matrix entries are removed. In particular, matrix entries belonging to markers which are not comprised in the minimal marker set are removed. In the present example, those entries assign marker M5 to lines L2 and R3. The result is the marker matrix shown in
(35) It is then determined in step S02j whether or not all minimal marker sets have been found. This is not the case in the present stage of this example, such that a new, empty minimal marker set is created in step S02k and the process starts again at step S02a, but with the original marker matrix having all measured entries. The process is repeated until all possible permutations of sub-matrices have been found. In the present example, another sub-matrix as shown in
(36) However, the determined plurality of sub-matrices may also comprise sub-matrices corresponding to invalid minimal marker sets. A minimal marker set is invalid if the corresponding sub-matrix comprises at least one line to which no marker is assigned. As can be seen from the example in
(37) Instead of removing invalid minimal marker sets, it is possible to avoid the creation of invalid minimal marker sets. For example, the selection of a marker in step S02b can be amended appropriately.
(38) Step S03 of
(39) If the arrangement of real markers is considered to be flat, then the real markers basically lie in a plane. However, due to the nature of the intersecting lines in space, the detected ghost markers do not lie in this plane. It can therefore be assumed that the flattest out of the possible minimal marker sets only comprises real markers and no ghost markers.
(40) This can be seen from
(41) The geometric size of a minimal marker set means the size of a space occupied by the markers in the minimal marker set. This size is defined in the three orthogonal directions. It is necessary to find those three orthogonal directions such that the smallest of the three extents along those three directions is minimized. This can be achieved by a statistical analysis of the minimal marker set in terms of finding the principal axes.
(42) In one implementation, the three orthogonal directions define the axes of the marker reference system, which is a coordinate system aligned with the markers in the minimal marker set such that the extent of the volume occupied by the markers in one of the axes of the coordinate system is minimized. The camera coordinate system is shifted into the center of the markers comprised in the minimal marker set. Then a covariance matrix of the locations of the markers of the minimal marker set in the marker reference system is created. The eigenvectors of the covariance matrix are calculated next. Those eigenvectors constitute a transformed reference system, which is the marker reference system. This is shown in
(43) Step S04 then involves selecting the minimal marker set having the smallest out of the smallest extents. This selected minimal marker set is the flattest of all possible minimal marker sets and is therefore considered as a ghost marker free marker set.
(44) The justification for selecting the flattest of the possible minimal marker sets is explained with reference to
(45) The pseudocode listed above will now be explained with reference to the marker configuration of
(46) ActiveRowsLeft and ActiveRowsRight each have the value 3 and MarkerSize is 5. MarkerSizeLimit is thus 31 and therefore has 5 bits. MarkerMask in loop1 then counts from 1 to 31. In the present example, the most significant bit of a variable represents marker M1 and the least significant bit represents marker M5.
(47) For values of MarkerMask between 1 and 6, the binary representation has less than three bits set, such that loop1 is left in step 9. For MarkerMask=7, the binary representation is 00111 and thus has three bits set.
(48) In steps 10 and 11, MarkersFoundLeft and MarkersFoundRight are each set to 0 (binary 00000), and starting in line 12, loop2 counts from 1 to 3.
(49) Markers M1 and M4 lie on line L1 and marker M3 lies on line R1. For LineNumber=1, LeftCameraRow is therefore 10010 and RightCameraRow is therefore 00100. This means that LeftCameraRowResult becomes 00010 in step 15 and RightCameraRowResult becomes 00100 in step 19. In both variables, no more than 1 bit is set, such that the process continues with step 22. In this step, the OR combination changes MarkersFoundLeft from 00000 to 00010. In analogy, step 23 changes MarkersFoundRight from 00000 to 00100.
(50) loop2 is then repeated with LineNumber=2. LeftCameraRow is then 01001 and Right CameraRow is 01010. LeftCameraRowResult then becomes 00001 and RightCameraRowResult becomes 00010. MarkersFoundLeft then changes from 00010 to 00011 and MarkersFoundRight changes from 00100 to 00110.
(51) loop2 is then repeated with LineNumber=3. LeftCameraRow is then 00100 and Right CameraRow is 10001. LeftCameraRowResult then becomes 00100 and RightCameraRowResult becomes 00001. MarkersFoundLeft then changes from 00011 to 00111 and MarkersFoundRight changes from 00110 to 00111.
(52) The AND combination of MarkersFoundLeft and MarkersFoundRight in step 25 then results in MarkersFound=00111. In this variable, three bits corresponding to three markers M3, M4 and M5 are set, which equals the number of lines in the two line sets. The set of M3, M4 and M5 is therefore found as a minimal marker set.
(53) loop1 is then repeated for the next values of MarkerMask. Details are now explained for MarkerMask=14, which is binary 01110.
(54) MarkersFoundLeft and MarkersFoundRight are each set to 00000 in steps 10 and 11. loop2 counts LineNumber from 1 to 3.
(55) For LineNumer=1, LeftCameraRow is 10010 and RightCameraRow is 00100.
(56) LeftCameraRowResult becomes 00010 and RightCameraRowResult becomes 00100. MarkersFoundLeft changes from 00000 to 00010 and MarkersFoundRight from 00000 to 00100.
(57) For LineNumer=2, LeftCameraRow is 01001 and RightCameraRow is 01010. LeftCameraRowResult becomes 01000 and RightCameraRowResult becomes 01010. RightCameraRowResult this has two bits set, such that loop2 is quit in step 21 and loop1 continues with MarkerMask=14. As a result, the set of markers M2, M3 and M4 represented by MarkerMask=14 is not a minimal marker set, because it has two markers in line R2.
(58) The code then identifies marker sets represented by MarkerMask from 15 to 27 as not being minimal marker sets.
(59) For MarkerMask=28, the binary representation is 11100. In steps 10 and 11, MarkersFoundLeft and MarkersFoundRight are thus each set to 0 (binary 00000), and starting in line 12, loop2 counts from 1 to 3.
(60) For LineNumer=1, LeftCameraRow is 10010 and RightCameraRow is 00100. LeftCameraRowResult becomes 10000 and RightCameraRowResult becomes 00100. MarkersFoundLeft changes from 00000 to 10000 and MarkersFoundRight from 00000 to 00100.
(61) For LineNumer=2, LeftCameraRow is 01001 and RightCameraRow is 01010. LeftCameraRowResult becomes 01000 and RightCameraRowResult becomes 01000. MarkersFoundLeft changes from 10000 to 11000 and MarkersFoundRight from 00100 to 01100.
(62) For LineNumer=3, LeftCameraRow is 00100 and RightCameraRow is 10001. LeftCameraRowResult becomes 00100 and RightCameraRowResult becomes 10000. MarkersFoundLeft changes from 11000 to 11100 and MarkersFoundRight from 01100 to 11100.
(63) The AND combination of MarkersFoundLeft and MarkersFoundRight in step 25 then results in MarkersFound=11100. In this variable, three bits corresponding to three markers M3, M4 and M5 are set, which equals the number of lines in the two line sets. The set of M1, M2 and M3 is therefore found as a minimal marker set.
(64) The further iterations of loop1 for MarkerMask from 29 to 31 does not find any further minimal marker sets, such that the identified minimal marker sets are M1, M2, M3 and M3, M4, M5.
(65)
(66)
(67) Step S11 involves measuring the medical markers, thus resulting in a measured set of markers comprising markers M1 to M5, that is real markers M1 to M3 and ghost markers M4 and M5, together with the locations of all those markers in a camera reference system. The measured set of medical markers can be represented by a marker matrix as shown in and explained with reference to
(68) Step S12 involves finding a potential ghost marker subset of the measured set of medical markers. The potential ghost marker subset only comprises real markers which can have caused ghost markers and those ghost markers. In the example shown in
(69) Using a marker matrix, which is output by the stereoscopic camera, as shown in
(70) In an optional step subsequent to step S12, the potential ghost marker subset is pruned. This means that only markers in the potential ghost marker subset which lie in the same plane remain in the potential ghost marker subset and all other markers are removed therefrom. This is motivated by the fact that the constellation in which two lines corresponding to the first camera and two lines corresponding to the second camera all lie in a first plane causes ghost markers in the first plane, and two lines corresponding to the first camera and two lines corresponding to the second camera lying in a second plane cause ghost markers in the second plane.
(71) In the subsequent steps of this embodiment, only ghost markers in a particular plane are considered. If there are ghost markers in different planes, different potential ghost marker subsets are created and processed iteratively.
(72) Step S13 involves calculating the distances of all markers in the potential ghost marker subset to the reference point R. It is of course also possible to not calculate the distances for a particular potential ghost marker subset, but once for all potential ghost markers in the measured set of medical markers and to retrieve the distances for the markers comprised in the potential ghost marker subset.
(73) Step S14 involves removing the marker having the smallest distance to the reference R from the potential ghost marker subset and the measured set. Step S15 then involves removing the marker having the largest distance to the reference R from the potential ghost marker subset and the measured set. In the example shown in
(74) Step S16 involves determining whether or not there are more ghost markers in the potential ghost marker subset. If this is the case, the process returns to step S14 and removes the markers which are now the closest and the farthest from the reference R from the potential ghost marker subset and the measured set. If there are no more ghost markers in the potential ghost marker subset, the process ends at step S17.
(75) In one example, the number of ghost markers in the initial potential ghost marker subset found in step S12 is determined. The number of markers in the potential ghost marker subset is the product of the number of lines of the first line set and the number of lines in the second line set which lie in the same plane. The minimal number of real markers which could cause the potential ghost marker subset equals the larger one of the number of lines of the first line set and the number of lines of the second line set lying in this plane. In the example shown in
(76) If there is an odd number of ghost markers in the initial potential ghost marker subset, there are two options. In the first option, one of the steps S14 and S15 is skipped in the last iteration of the method. For example, the average distance of all remaining markers in the potential ghost marker subset or all remaining markers in the potential ghost marker subset without the markers having the smallest and the largest distance to the reference R is calculated and the one of the closest and the farthest markers in the potential ghost marker subset having the larger difference to this average is removed from the potential ghost marker subset. In the second option, both steps S14 and S15 are performed anyway. This means that a real marker is removed from the measured set, but at the same time, all ghost markers are reliably removed.
(77) As explained above, the present invention works best if the constellation of the real markers is basically flat, which means that it widely extends over a two-dimensional area, but hardly in a direction perpendicular thereto. One potential test whether or not this situation is given is explained with reference to
(78) In