Method and system for dynamic multi-dimensional images of an object

Abstract

The present invention relates to an image processing apparatus for deriving multi-dimensional images of an object and an according system and method. The image processing apparatus comprises an interface configured to provide 3D image data of an object and to provide a sequence of images of the object. The image processing apparatus further comprises a processing unit configured to obtain a personalized 3D model of the object by applying a model-based segmentation to the 3D image data of the object and to adapt the personalized 3D model based on at least a part of the images of the sequence of images of the object.

Claims

1. An image processing apparatus for deriving multi-dimensional images of an object, the image processing apparatus comprising: an interface configured to: provide 3D image data of the object, and provide a sequence of images of the object; and a processor configured to: segment the object in the 3D image data into object parts by applying a model-based segmentation to the 3D image data, generate a personalized 3D model of the object from the object parts, segment the object in the images in the sequence of images into the object parts based on spatial information provided by the personalized 3D model, and adapt the object parts in the personalized 3D model to the object parts segmented in the sequence of images.

2. The image processing apparatus according to claim 1, further comprising: a display configured to display an overlay of the adapted personalized 3D model and the sequence of images of the object.

3. A system for deriving dynamic multi-dimensional images of an object, the system comprising: an image acquisition apparatus configured to: acquire the 3D image data of the object, and generate a plurality of images of the object; and the image processing apparatus according to claim 1.

4. The system according to claim 3, wherein the image acquisition apparatus is configured to: acquire the 3D image of the object and generate the sequence of images of the object.

5. The system according to claim 4, wherein the image acquisition apparatus is an ultrasound system.

6. The image processing apparatus according to claim 1, wherein the personalized 3D model is a deformable model.

7. A method for deriving dynamic multi-dimensional images of an object, the method comprising: providing 3D image data of the object; segmenting the object in the 3D image data into object parts by applying a model-based segmentation to the 3D image data; generating a personalized 3D model of the object from the object parts; providing a sequence of images of the object; segmenting the object in the images in the sequence of images into the object parts based on spatial information provided by the personalized 3D model; and adapting the object parts in the personalized 3D model to the object parts segmented in the sequence of images.

8. The method according to claim 7, wherein, during adapting the personalized 3D model, spatial information is provided by the personalized 3D model of the object.

9. The method according to claim 7, wherein the personalized 3D model is adapted to one or more images of the sequence of images by at least one of replicating and stacking the one or more images.

10. The method according to claim 7, wherein, for adapting the personalized 3D model to one or more images of the sequence of images of the object, landmarks are provided for registration of an individual object geometry with the personalized 3D model of the object.

11. The method according to claim 7, wherein the personalized 3D model is provided as a mesh of triangles and is adapted to minimize a model energy comprising an internal energy and an external energy.

12. The method according to claim 11, wherein the external energy is derived from target points close to the image planes corresponding to of one or more images of the sequence of images of the object.

13. The method according to claim 11, wherein the internal energy comprises penalization of deviations between a current state of the mesh and the mesh from the personalized 3D model.

14. The method according to claim 7, wherein the sequence of images of the object is generated at a rate of more than 10 Hz.

15. The method according to claim 7, wherein the personalized 3D model is a deformable model.

16. A non-transitory computer-readable storage medium having stored a computer program comprising instructions for controlling an image processing apparatus for deriving multi-dimensional images of an object, the instructions, which, when executed by a processor, cause the processor to: receive 3D image data of the object; receive a sequence of images of the object; segment the object in the 3D image data into object parts by applying a model-based segmentation to the 3D image data; generate a personalized 3D model of the object from the object parts; segment the object in the images in the sequence of images into the object parts based on spatial information provided by the personalized 3D model; and adapt the object parts in the personalized 3D model to the object parts segmented in the sequence of images.

17. The non-transitory computer-readable storage medium according to claim 16, wherein the personalized 3D model is a deformable model.

18. The non-transitory computer-readable storage medium according to claim 16, wherein the personalized 3D model is adapted to one or more images of the sequence of images by at least one of replicating the one or more images and stacking the one or more images.

19. The non-transitory computer-readable storage medium according to claim 16, wherein the personalized 3D model is provided as a mesh of triangles and is adapted to minimize a model energy comprising an internal energy and an external energy.

20. The non-transitory computer-readable storage medium according to claim 19, wherein the external energy is derived from target points close to image planes corresponding to one or more images of the sequence of images of the object.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) Exemplary embodiments of the invention will be described in the following with reference to the following drawings:

(2) FIG. 1: a system according to the invention;

(3) FIG. 2A: a schematic illustration according to the invention;

(4) FIG. 2B: the schematic illustration of FIG. 2A;

(5) FIG. 2C: a combination of the illustration of FIGS. 2A and 2B;

(6) FIG. 3: a flow chart according to the method of the invention.

DETAILED DESCRIPTION OF EMBODIMENTS

(7) Certain embodiments will now be described in greater details with reference to the accompanying drawings. In the following description, like drawing reference numerals are used for like elements, even in different drawings. The matters defined in the description, such as detailed construction and elements, are provided to assist in a comprehensive understanding of the exemplary embodiments. Also, well-known functions or constructions are not described in detail since they would obscure the embodiments with unnecessary detail. Moreover, expressions such as “at least one of”, when preceding a list of elements, modify the entire list of elements and do not modify the individual elements of the list.

(8) FIG. 1 shows a system 10 for deriving dynamic multi-dimensional images of an object. The system 10 comprises an image acquisition apparatus 14 and an image processing apparatus 16. The image acquisition apparatus 10 is configured to acquire 3D image data of an object 15, in this embodiment the heart of a patient 12.

(9) In this embodiment, the 3D image data is provided as 3D ultrasound scan. In other embodiments, the 3D image data is provided as a 3D scan acquired by computed tomography (CT) or magnetic resonance (MR). The 3D scan provides complete information of the object 15. The image acquisition apparatus 14 is also configured to generate a plurality of images of the object 15 as a sequence of images of the object. In this embodiment, the images are ultrasound images provided with a framerate of 10 Hz. However, a higher framerate up to 50 Hz or more is used in other embodiments. Also, in other embodiments other types of images which are suitable for tracking at such high framerate are used.

(10) The image processing apparatus 16 comprises an interface 18 and a processing unit 22. The interface 18 is configured to provide the 3D image data of the object 15 and the sequence of images. The processing unit 22 is configured to obtain a personalized 3D model of the object from the provided 3D image data. The 3D model is adapted based on at least a part of the image(s) of the sequence of images generated by the image acquisition apparatus 14.

(11) In this embodiment, the image processing apparatus 16 also comprises an external display unit 24 for displaying an overlay of the adapted personalized 3D model and the sequence of images. Thus, the dynamic multi-dimensional images of the object 15 are provided to a clinician to observe the fast changing dynamics of the heart 15 in this case. Optionally, the system 10 comprises an input device for rotating the multi-dimensional images of the object 15. The input device could also be used for operating the image acquisition and/or generating procedure.

(12) In other embodiments, the image acquisition apparatus 14 can comprise a first image acquisition unit for acquisition of the 3D image data and a second image acquisition unit for generating the sequence of images of the object. Such image acquisition apparatus 14 is an ultrasound system, for example.

(13) FIG. 2A shows a schematic illustration according to the invention. 3D image data 30 is provided as 3D ultrasound image data in this embodiment. The 3D image data 30 provides the upper heart chambers 40, 42 and the lower heart chambers 44, 46. This 3D scan is subsequently segmented using a model-based segmentation. A 3D model 32 of the object 15, the heart, is obtained providing a personalized model of the heart of the particular patient. This detailed, accurate and personalized 3D model 32 is then used for dynamically tracking and segmenting the heart in a fast 2D image acquisition as sequence of images 34.

(14) The sequence of images 34 of the object 15 is provided as a dynamic sequence 34 of ultrasound images. The upper heart chambers 40, 42 and the lower heart chambers 44, 46 are represented by the sequence 34 of images. Some images of the sequence might contain, in some embodiments, only a sub-region of the object 15, e.g. only the upper heart chamber 40. This ensures that a high framerate of higher than 10 Hz. The ultrasound images are 2D images, and can be provided as cross-plane images in other embodiments.

(15) By overlaying the adapted 3D model 32 and the images 34 of the object 15 the shape of the object, in this embodiment the heart, and its dynamic changes 36 over time t are made visible. In FIG. 2 the contraction and expansion of the heart chambers is clearly visible. As FIG. 2 can only provide an illustration on paper the changing dynamics are provided as a series of subsequent images.

(16) In those regions of the object 15 which are provided by the sequence of images 34, the personalized 3D model is adapted to the provided image. In the regions of the object not shown in the images 34, the shape of the object 15 is estimated by combining the information data from those regions with the dynamic information of those parts provided from the images 34 of the sequence.

(17) During adaptation of the personalized 3D model 32 spatial information is provided by the personalized 3D model of the object. In other words, missing spatial information of the sequence of images 34 is provided by the personalized 3D model 32 of the object. This can be achieved by replacing the generic mean shape or mesh usually used in model-based segmentation by the 3D model 32 of the object 15, which provides a personalized shape of the object 15.

(18) FIG. 2B shows the schematic illustration of FIG. 2A wherein the ultrasound images are replaced by line drawings for improved visibility.

(19) FIG. 2C shows a combination of the illustration of FIGS. 2A and 2B for improved understanding of the provided method and system for deriving dynamic multi-dimensional images of the object, in this case the heart.

(20) FIG. 3 shows a flow chart according to the method of the invention. 3D image data of an object is provided in step S1. A personalized 3D model 32 of the object is obtained by applying a model-based segmentation to the 3D image data in step S2. A sequence of images 34 of the object is provided at step S3 and the personalized 3D model is adapted based on at least a part of the images of the sequence in step S4. Thus, dynamic multi-dimensional images 36 are obtained to track the object 15 over time.

(21) In some embodiments, spatial information (S5) is provided by the personalized 3D model during adaptation of the 3D model 32. Also, landmarks (S6) are provided for registration of the individual object 15 geometry.

(22) The personalized 3D model 32 can be a deformable model provided as a mesh of triangles and is adapted to minimize a model energy (S7). The model energy comprises an internal and external energy. The external energy is derived from target points close to the image planes of the images 34. The internal energy comprises penalization of deviations between a current state of the mesh and the mesh from the personalized 3D model. Thus, full 3D segmentation is provided as fast and reliable 4D (3D+time) individual image of an object 15, such as the heart, in order to enhance navigation during catheter-based interventions, for example.

(23) The method, apparatus and system for deriving dynamic multi-dimensional images of an object as provided herewith can be used in multiple different scenarios. One scenario would be image guided interventions. Here, implementation in the Echo Navigator software is appropriate. For image guided interventions a CT image is often acquired before the actual procedure, e.g. for device planning Such 3D CT scan can be used and no additional recordings for image data are necessary. An additional scenario could be cardiac ultrasound with a real-time view of the segmented heart. The feasibility of a real-time, personalized segmentation, as provided by the invention, provides tremendous advantages over known solutions and applications.

(24) In another exemplary embodiment of the present invention, a computer program or a computer program element is provided that is characterized by being adapted to execute the method steps of the method according to one of the preceding embodiments, on an appropriate system.

(25) The computer program element might therefore be stored on a computer unit, which might also be part of an embodiment of the present invention. This computing unit may be adapted to perform or induce a performing of the steps of the method described above. Moreover, it may be adapted to operate the components of the above described apparatus. The computing unit can be adapted to operate automatically and/or to execute the orders of a user. A computer program may be loaded into a working memory of a data processor. The data processor may thus be equipped to carry out the method of the invention. This exemplary embodiment of the invention covers both, a computer program that right from the beginning uses the invention and a computer program that by means of an up-date turns an existing program into a program that uses the invention.

(26) Further on, the computer program element might be able to provide all necessary steps to fulfil the procedure of an exemplary embodiment of the method as described above.

(27) According to a further exemplary embodiment of the present invention, a computer readable medium, such as a CD-ROM, is presented wherein the computer readable medium has a computer program element stored on it which computer program element is described by the preceding section. A computer program may be stored and/or distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the internet or other wired or wireless telecommunication systems.

(28) However, the computer program may also be presented over a network like the World Wide Web and can be downloaded into the working memory of a data processor from such a network. According to a further exemplary embodiment of the present invention, a medium for making a computer program element available for downloading is provided, which computer program element is arranged to perform a method according to one of the previously described embodiments of the invention.

(29) As discussed above, the processing unit, for instance a controller implements the control method. The controller can be implemented in numerous ways, with software and/or hardware, to perform the various functions required. A processor is one example of a controller which employs one or more microprocessors that may be programmed using software (e.g., microcode) to perform the required functions. A controller may however be implemented with or without employing a processor, and also may be implemented as a combination of dedicated hardware to perform some functions and a processor (e.g., one or more programmed microprocessors and associated circuitry) to perform other functions.

(30) Examples of controller components that may be employed in various embodiments of the present disclosure include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs).

(31) In various implementations, a processor or controller may be associated with one or more storage media such as volatile and non-volatile computer memory such as RAM, PROM, EPROM, and EEPROM. The storage media may be encoded with one or more programs that, when executed on one or more processors and/or controllers, perform at the required functions. Various storage media may be fixed within a processor or controller or may be transportable, such that the one or more programs stored thereon can be loaded into a processor or controller.

(32) It has to be noted that embodiments of the invention are described with reference to different subject matters. In particular, some embodiments are described with reference to method type claims whereas other embodiments are described with reference to the device type claims. However, a person skilled in the art will gather from the above and the following description that, unless otherwise notified, in addition to any combination of features belonging to one type of subject matter also any combination between features relating to different subject matters is considered to be disclosed with this application. However, all features can be combined providing synergetic effects that are more than the simple summation of the features.

(33) While the invention has been illustrated, and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive. The invention is not limited to the disclosed embodiments. Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing a claimed invention, from a study of the drawings, the disclosure, and the dependent claims.

(34) In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. A single processor or other unit may fulfil the functions of several items re-cited in the claims. The mere fact that certain measures are re-cited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope.