IMAGE SEGMENTATION SYSTEM
20230074125 · 2023-03-09
Inventors
- FRANK MICHAEL WEBER (NORDERSTEDT, DE)
- JOCHEN PETERS (NORDERSTEDT, DE)
- ROLF JÜRGEN WEESE (NORDERSTEDT, DE)
Cpc classification
G06T19/00
PHYSICS
G06T2207/20101
PHYSICS
International classification
G06T19/00
PHYSICS
Abstract
Disclosed herein is a medical system (100, 300) comprising a display (112) and a user interface (114). The execution of machine executable instructions (120) causes a processor (104) to: receive (200) three-dimensional medical image data (122) of an anatomical structure (128, 322); receive (202) a three-dimensional segmentation (124) with one or more reference locations (800); display (204) at least one two-dimensional slice (126) of the three-dimensional medical image data; render (206) a cross section (134) of the three-dimensional segmentation, provide (208) a control element (130) of the user interface that is configured for receiving a one-dimensional position of the at least one reference location along a predetermined one-dimensional path (806); receive (210) the one-dimensional position (137) from the control element; adjust (212) the three-dimensional segmentation (124) using the one-dimensional position; and update (214) the rendering of the cross section of the three-dimensional segmentation.
Claims
1. A medical system, comprising: a display; a user interface; a memory containing machine executable instructions; and a processor for controlling the medical system, wherein execution of the machine executable instructions causes the processor to: receive three-dimensional medical image data descriptive of an anatomical structure; receive a three-dimensional segmentation of the anatomical structure, wherein the three-dimensional segmentation comprises one or more reference locations, wherein the one or more reference locations comprise an anatomical landmark; display at least one two-dimensional slice of the three-dimensional medical image data using the display; render a cross section of the three-dimensional segmentation within the at least one two-dimensional slice on the display; provide a control element of the user interface for the at least one reference location on the user interface, wherein the control element is configured for receiving a one-dimensional position of the at least one reference location along a predetermined one-dimensional path; receive the one-dimensional position from the control element; adjust the three-dimensional segmentation using the one-dimensional position; and update the rendering of the cross section of the three-dimensional segmentation within the at least one two-dimensional slice on the display.
2. The medical system of claim 1, wherein the one-dimensional path is defined within the three-dimensional segmentation.
3. The medical system of claim 1, wherein the three-dimensional segmentation is adjusted by: calculating a vector translation of the reference location using the one-dimensional position; and updating the three-dimensional segmentation by inputting the vector translation into a three-dimensional editing engine.
4. The medical system of claim 3, wherein the vector translation of the reference location is input into the three-dimensional editing engine as virtual mouse motion.
5. The medical system of claim 1, wherein the three-dimensional segmentation is a cardiac segmentation, wherein the at least one reference location comprises at least one of: a left ventricular heart apex, a right ventricular heart apex, a ventricular apex, a mitral valve plane, a tricuspid plane, a valve plane.
6. The medical system of claim 1, wherein the three-dimensional segmentation is a prostate segmentation, wherein the at least one reference location comprises at least one of: a prostate base, a prostate apex, a prostate mid-gland plane location.
7. The medical system of claims1, wherein the one or more reference locations further comprises at least one of: a vertex, a triangle, a plane.
8. The medical system of claim 1, wherein the at least one two-dimensional slice is multiple two-dimensional slices, wherein a plurality of the multiple two-dimensional slices on the display are displayed simultaneously.
9. The medical system of claim 1, wherein the user interface is further configured for receiving a modification of the cross section of the three-dimensional segmentation within the at least one two-dimensional slice on the display, wherein execution of the machine executable instructions further causes the processor to: receive the modification of the cross section from the user interface; adjust the three-dimensional segmentation using the modification of the cross section; and update the rendering of the cross section of the three-dimensional segmentation within each of the multiple two-dimensional slices on the display.
10. The medical system of claim 1, wherein execution of the machine executable instructions further causes the processor to provide the three-dimensional segmentation by segmenting the three-dimensional medical imaging data using an image segmentation module.
11. The medical system of claim 1, wherein the medical system further comprises a medical imaging system configured for acquiring the three-dimensional medical imaging data from an imaging zone, wherein execution of the machine executable instructions are further configured for controlling the medical imaging system to acquire the three-dimensional medical image data.
12. The medical system of claim 11, wherein the medical imaging system is at least one of: a magnetic resonance imaging system, a computed tomography system, and an ultrasound imaging system.
13. The medical system of claim 1, wherein the control element of the user interface for the at least one reference location on the user interface comprises at least one of: a slider, a dial, a graphical user interface control located outside of the rendering of the cross section of the three-dimensional segmentation.
14. (canceled)
15. A method of operating a medical system, comprising: receiving three-dimensional medical image data descriptive of an anatomical structure; receiving a three-dimensional segmentation of the anatomical structure, wherein the three-dimensional segmentation comprises one or more reference locations, wherein the one or more reference locations comprise an anatomical landmark; displaying at least one two-dimensional slice of the three-dimensional medical image data on ausing the display; rendering a cross section of the three-dimensional segmentation within the at least one two-dimensional slice on the display; providing a control element of a user interface for the at least one reference location on the user interface, wherein the control element is configured for receiving a one-dimensional position of the at least one reference location along a predetermined one-dimensional path; receiving the one-dimensional position from the control element; adjusting the three-dimensional segmentation using the one-dimensional position; and updating the rendering of the cross section of the three-dimensional segmentation within the at least one two-dimensional slice on the display.
16. A non-transitory computer-readable medium for storing executable instructions, which cause a method to be performed to operate a medical system, the method comprising: receiving three-dimensional medical image data descriptive of an anatomical structure; receiving a three-dimensional segmentation of the anatomical structure, wherein the three-dimensional segmentation comprises one or more reference locations, wherein the one or more reference locations comprise an anatomical landmark; displaying at least one two-dimensional slice of the three-dimensional medical image data on a display; rendering a cross section of the three-dimensional segmentation within the at least one two-dimensional slice on the display; providing a control element of a user interface for the at least one reference location on the user interface, wherein the control element is configured for receiving a one-dimensional position of the at least one reference location along a predetermined one-dimensional path; receiving the one-dimensional position from the control element; adjusting the three-dimensional segmentation using the one-dimensional position; and updating the rendering of the cross section of the three-dimensional segmentation within the at least one two-dimensional slice on the display.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
[0079] In the following preferred embodiments of the invention will be described, by way of example only, and with reference to the drawings in which:
[0080]
[0081]
[0082]
[0083]
[0084]
[0085]
[0086]
[0087]
DETAILED DESCRIPTION OF THE EMBODIMENTS
[0088] Like numbered elements in these figures are either equivalent elements or perform the same function. Elements which have been discussed previously will not necessarily be discussed in later figures if the function is equivalent.
[0089]
[0090] The memory 110 is shown as containing machine-executable instructions 120. The machine-executable instructions 120 contain instructions which enable the processor 104 to perform various control, data processing and image processing tasks. The memory 110 is further shown as containing three-dimensional medical image data 122 and a segmentation 124. The segmentation 124 contains a segmentation of an anatomical structure. In the graphical user interface 114 there is a window which shows a rendering of a two-dimensional slice 126 of the three-dimensional medical image data 122. Within the slice 126 the anatomical structure 128 can be shown.
[0091] The graphical user interface 114 is also shown as containing a control element 130. In this example the control element 130 is a slider. The box 132 represents the slider in a first position. The dot-dash line 134 represents the cross-section of the three-dimensional segmentation 124. In this example the cross-section 134 does not fit the anatomical structure 128 well. The user then moves the slider into the second slider position 136. The segmentation is then updated such that the segmentation is now the dotted line 138. This is the adjusted cross-section 138 of the three-dimensional segmentation 124.
[0092] When the slider 130 is moved from the first position 132 to the second position 136 a one-dimensional position 137 is stored in the memory 110. This one-dimensional position 137 is used to update the segmentation by moving the at least one reference location along a predetermined one-dimensional path.
[0093]
[0094] Next in step 206 the cross-section of the three-dimensional segmentation 134 is rendered within the two-dimensional slice 126. Next in step 208 the control element 130 is provided on the user interface 114. In this example there is only one reference location that is provided. In other models there may be additional sliders or additional controls. The user then adjusts the slider from the first position 132 to the second position 136 and in step 216 the one-dimensional position is received from the control element 130. Next, in step 212, the three-dimensional segmentation is adjusted using this one-dimensional position. Then, in step 214, the rendering of the cross-section 138 is updated on the two-dimensional slice 126.
[0095]
[0096] The magnetic resonance imaging system 302 comprises a magnet 304. The magnet 304 is a superconducting cylindrical type magnet with a bore 306 through it. The use of different types of magnets is also possible; for instance it is also possible to use both a split cylindrical magnet and a so called open magnet. A split cylindrical magnet is similar to a standard cylindrical magnet, except that the cryostat has been split into two sections to allow access to the iso-plane of the magnet, such magnets may for instance be used in conjunction with charged particle beam therapy. An open magnet has two magnet sections, one above the other with a space in-between that is large enough to receive a subject: the arrangement of the two sections area similar to that of a Helmholtz coil. Open magnets are popular, because the subject is less confined. Inside the cryostat of the cylindrical magnet there is a collection of superconducting coils.
[0097] Within the bore 306 of the cylindrical magnet 304 there is an imaging zone 308 where the magnetic field is strong and uniform enough to perform magnetic resonance imaging. A region of interest 309 is shown within the imaging zone 308. The magnetic resonance data that is acquired typically acquried for the region of interest. A subject 318 is shown as being supported by a subject support 320 such that at least a portion of the subject 318 is within the imaging zone 308 and the region of interest 309. Within the region of interest 309 there is an anatomical structure 322. In this example the anatomical structure 322 of the subject 318 is the subject's heart. Other organs or structures may also be imaged such as the prostate.
[0098] Within the bore 306 of the magnet there is also a set of magnetic field gradient coils 310 which is used for acquisition of preliminary magnetic resonance data to spatially encode magnetic spins within the imaging zone 308 of the magnet 304. The magnetic field gradient coils 310 connected to a magnetic field gradient coil power supply 312. The magnetic field gradient coils 310 are intended to be representative. Typically magnetic field gradient coils 310 contain three separate sets of coils for spatially encoding in three orthogonal spatial directions. A magnetic field gradient power supply supplies current to the magnetic field gradient coils. The current supplied to the magnetic field gradient coils 310 is controlled as a function of time and may be ramped or pulsed.
[0099] Adjacent to the imaging zone 308 is a radio-frequency coil 314 for manipulating the orientations of magnetic spins within the imaging zone 308 and for receiving radio transmissions from spins also within the imaging zone 308. The radio frequency antenna may contain multiple coil elements. The radio frequency antenna may also be referred to as a channel or antenna. The radio-frequency coil 314 is connected to a radio frequency transceiver 316. The radio-frequency coil 314 and radio frequency transceiver 316 may be replaced by separate transmit and receive coils and a separate transmitter and receiver. It is understood that the radio-frequency coil 314 and the radio frequency transceiver 316 are representative. The radio-frequency coil 314 is intended to also represent a dedicated transmit antenna and a dedicated receive antenna. Likewise the transceiver 316 may also represent a separate transmitter and receivers. The radio-frequency coil 314 may also have multiple receive/transmit elements and the radio frequency transceiver 316 may have multiple receive/transmit channels. For example if a parallel imaging technique such as SENSE is performed, the radio-frequency could 314 will have multiple coil elements.
[0100] The transceiver 316 and the gradient controller 312 are shown as being connected to the hardware interface 106 of a computer system 102.
[0101] The memory 110 is additionally shown as containing pulse sequence commands 330. The pulse sequence commands enable the processor 104 to control the magnetic resonance imaging system to acquire magnetic resonance imaging data 332. The magnetic resonance imaging data 332 may be reconstructed by the processor into the three-dimensional medical image data 122. The memory 110 is also shown as optionally containing an image segmentation module 334. The image segmentation module 334 may be configured to take the three-dimensional medical image data 122 as input and in response output the three-dimensional segmentation 124. As mentioned earlier above, there are a variety of means which this may be accomplished by.
[0102]
[0103] Some examples may relate to interactive 3D mesh editing along a normal/out-of-plane direction of a view plane by additional GUI controls.
[0104] A possible problem for 3D editing tools in MR applications is that only slice-by-slice editing may be available in the user interface. This means that often results are only edited in-plane, for example in cardiac MR or prostate MR. In cardiac MR, when short-axis planes are shown, this does not allow for changing the height of the apex or the valve planes. As an example, the left ventricle (LV) could be segmented too short, i.e., the segmented apex position is above the actual apex slice. In that case, the mesh is not even visible in the actual apex slice. Without a displayed mesh contour, no editing would be possible that could shift the apex down to the correct location in the image.
[0105] In another example the adjustment of the three-dimensional segmentation may be split into two components. In-plane interactions with the 3D mesh may still done using the mouse. Edits along the normal direction are done by combining anatomical information from the 3D mesh and a one-dimensional slider GUI control element or other control element. The start point and direction of the editing are defined from the anatomical context of the model. For example, to move the apex, the apex position is the starting point, and the connection line between apex and mitral valve defines the direction. When the slider is being moved, the editing algorithm is fed with information as if the mouse had dragged the apex position along that direction. This way, the mesh is being updated as if the apex had been dragged upwards/downwards.
Overall, this enables to control in-plane and out-of-plane edits by a consistent editing toolkit preserving the 3D mesh.
[0106] Examples may split the interaction editing into two components. In-plane interactions with the 3D mesh may be done in some examples using the mouse. Edits with a (non-negligible) component in the normal direction may be done by combining anatomical information from the 3D mesh and a one-dimensional slider GUI or other control element. The edits do not have to be exactly along the normal direction. The start point and direction of the editing may be defined from the anatomical context of the model. For example, to move the apex, the apex position is the starting point, and the connection line between apex and mitral valve defines the direction. When the slider is being moved, the editing algorithm is fed with information as if the mouse had dragged the apex position along that direction. This way, the mesh is being updated as if the apex had been dragged upwards/downwards.
[0107]
[0108] The behavior of the control elements 130 may be modified in different examples. For example, in one case after the slider has been released and the mesh has been updated, the slider 130 is directly reset again to a central location from which a new edit can start.
[0109] In another case, the slider is kept at the target location even after releasing it such that the user can:
[0110] inspect other anatomical locations (e.g. scroll through slices) for the correctness of the current mesh position, and
[0111] further adjust the slider. This then modifies the previous slider editing step (i.e. with the same initial mesh as in the previous step, just with another target position along the one-dimensional axis),
[0112] In this example, the slider is reset to a center location on the next manual edit or when changing one of the potentially present other sliders. Then, on the next slider movement, the editing is performed based on the mesh that is valid at that point, also recalculating all landmarks from the mesh.
[0113]
[0114]
[0115]
[0116] Examples may comprise one or more of the following components:
[0117] Display unit: displays a cardiac MR image together with an overlay of the segmentation result (i.e. the segmented 3D mesh). An example display is shown in
[0118] Interactive 3D editing unit: Allows to interact with the 3D mesh by clicking and dragging directly on the display image/mesh. Typical usage:
[0119] User looks at image with overlaid segmentation result and identifies a region to edit.
[0120] User clicks into the center of the region to edit and holds down the mouse button.
[0121] User drags the mouse to the desired location, while the displayed mesh is being dynamically deformed, i.e., a region around the start point is shifted towards the new mouse position. The mesh is deformed in 3D space.
[0122] The user releases the mouse button at the desired location.
[0123] The deformed mesh is displayed and used for further analysis such as volume calculation.
[0124] As can be seen from the description, the mesh can be edited in directions that correspond to mouse movements in the displayed plane. If only a short axis cut-plane is shown, the LV wall can, for example, be moved in and out, but not up and down.
[0125] Slider editing unit: Allows to modify the mesh along directions which are not enabled by shown short-axis cut-planes. This comprises:
[0126] Controls in the graphical user interface that correspond to anatomically defined mesh corrections. Typically, this can be slider controls, see
[0127] Move LV apex up and down
[0128] Move mitral valve plane up and down
[0129] Move tricuspid valve plane up and down
[0130] An anatomical unit that translates interactions with the slider control into virtual mouse movements. These virtual mouse movements are understood by the 3D editing engine, but cannot be performed directly by the user due to the missing view planes.
[0131] From the anatomical context of the segmentation model, the position of the apex p.sub.apex and the mitral valve p.sub.mv are determined.
[0132] The direction of the correction (virtual mouse motion) n.sub.slider is calculated as normalized difference vector
[0133] The interactive editing is initialized as if the user had clicked On P.sub.apex
[0134] The user then adjusts the slider value d.sub.slider which determines the length of the editing for the apex (virtual mouse movement). From this, a target mouse point is calculated as P.sub.current=P.sub.apex+d.sub.slider*n.sub.slider. This point P.sub.current is then fed into the 3D editing engine as if the user had moved the mouse there.
[0135] Note: In practice, the direction of n.sub.slider can be inverted, such that positive slider values make the ventricle longer.
[0136] In some examples only slices that are parallel (or roughly parallel) are displayed. For example, a long axis view (or other view) may be presented, but editing may still only be possible in the other views (i.e. in the short-axis views). Therefore, the general absence of a particular view itself is not a requirement for examples to become effective.
[0137]
[0138] While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive; the invention is not limited to the disclosed embodiments.
[0139] Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measured cannot be used to advantage. A computer program may be stored/distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems. Any reference signs in the claims should not be construed as limiting the scope.
LIST OF REFERENCE NUMERALS
[0140] 100 medical system
[0141] 102 computer
[0142] 104 processor
[0143] 106 hardware interface
[0144] 108 user interface
[0145] 110 memory
[0146] 112 display
[0147] 114 graphical user interface
[0148] 120 machine executable instructions
[0149] 122 three-dimensional medical image data
[0150] 124 three-dimensional segmentation
[0151] 126 rendering of two-dimensional slice
[0152] 128 anatomical structure
[0153] 130 control element (slider)
[0154] 132 first slider position
[0155] 134 cross section of three-dimensional segmentation
[0156] 136 second slider position
[0157] 137 one dimensional position
[0158] 138 adjusted cross section of three-dimensional segmentation
[0159] 200 receive three-dimensional medical image data descriptive of an anatomical structure
[0160] 202 receive a three-dimensional segmentation of the anatomical structure, wherein the three-dimensional segmentation comprises one or more reference locations;
[0161] 204 display at least one two-dimensional slice of the three-dimensional medical image data using the display
[0162] 206 render a cross section of the three-dimensional segmentation within the at least one two-dimensional slice on the display
[0163] 208 provide a control element of the user interface for the at least one reference location on the user interface
[0164] 210 receive the one-dimensional position from the control element
[0165] 212 adjust the three-dimensional segmentation using the one-dimensional position
[0166] 214 update the rendering of the cross section of the three-dimensional segmentation within the at least one two-dimensional slice on the display
[0167] 300 medical instrumnet
[0168] 302 magnetic resonance imaging system
[0169] 304 magnet
[0170] 306 bore of magnet
[0171] 308 imaging zone
[0172] 309 region of interest
[0173] 310 magnetic field gradient coils
[0174] 312 magnetic field gradient coil power supply
[0175] 314 radio-frequency coil
[0176] 316 transceiver
[0177] 318 subject
[0178] 320 subject support
[0179] 322 anatomica structure
[0180] 330 pulse sequence commands
[0181] 332 magnetic resonance imaging data
[0182] 334 image segmentation module
[0183] 400 controlling the medical imaging system to acquire the three-dimensional medical image data
[0184] 402 segmenting the three-dimensional medical imaging data using an image segmentation module
[0185] 500 short axis images rendered (two-dimensional slices)
[0186] 502 long axis images not not rendered
[0187] 504 original segmentation cross secton
[0188] 506 adjusted segmentation cross section
[0189] 510 before slider correction
[0190] 512 after small slider correcton
[0191] 514 after large slider correction
[0192] 516 after manual mouse edits in plane
[0193] 700 path for vertex
[0194] 702 path for tricuspid valve
[0195] 704 path for mitral valve
[0196] 800 location of vertex
[0197] 802 segmentation
[0198] 804 vector translation
[0199] 806 one dimensional path
[0200] 808 location of mitral valve
[0201] 810 one dimensional location