Method and Apparatus for Ultrasound Imaging with Improved Beamforming

Abstract

Disclosed herein is a method of ultrasound imaging of an object using an ultrasound transducer which comprises an array of transducer elements capable of converting sound signals into electrical signals and vice versa, comprising the following steps: A) transmitting an ultrasound beam from said ultrasound transducer into the object, by activating a first subset of said transducer elements, B) detecting reflected signals in a time resolved manner by means of a second subset of said transducer elements, wherein timing information of a detected signal is associated with information regarding the depth where the detected signal was reflected within the object subjected to imaging, and wherein the reflected signals associated with said second subset of transducer elements resemble a set of two-dimensional ultrasound data, of which one dimension resembles the various transducer elements of said second subset and the other dimension resembles depth information, C) converting said two-dimensional ultrasound data into a scan object using a receive beamforming procedure which accounts for differences in distance of individual transducer elements from a given site of sound reflection within the object, repeating steps A) to C) for different choices regarding at least one of said first and second subsets and the timing of the activation of transducer elements within said first subset, thereby obtaining a plurality of scan objects, and a step of constructing a visual image from said plurality of scan objects, wherein said receive beamforming procedure employs a machine learning based receive beamforming model for mapping said two-dimensional ultrasound data to said scan object.

Claims

1. A method of ultrasound imaging of an object using an ultrasound transducer which comprises an array of transducer elements capable of converting sound signals into electrical signals and vice versa, comprising the following steps: A) transmitting an ultrasound beam from said ultrasound transducer into the object, by activating a first subset of said transducer elements, B) detecting reflected signals in a time resolved manner by means of a second subset of said transducer elements, wherein timing information of a detected signal is associated with information regarding the depth where the detected signal was reflected within the object subjected to imaging, and wherein the reflected signals associated with said second subset of transducer elements resemble a set of two-dimensional ultrasound data, of which one dimension represents the various transducer elements of said second subset and the other dimension represents depth information, C) converting said two-dimensional ultrasound data into a scan object using a receive beamforming procedure which accounts for differences in distance of individual transducer elements from a given site of sound reflection within the object, repeating steps A) to C) for different choices regarding at least one of said first and second subsets and the timing of the activation of transducer elements within said first subset, thereby obtaining a plurality of scan objects, and a step of constructing a visual image from said plurality of scan objects, wherein said receive beamforming procedure employs a machine learning based receive beamforming model for mapping said two-dimensional ultrasound data to said scan object.

2. The method of claim 1, wherein said machine learning based receive beamforming model employs one of a deep convolutional neural network or a recurrent neural network.

3. The method of claim 1, wherein said receive beamforming model is an end-to-end beamforming model receiving said two-dimensional ultrasound data as an input and directly converting it into said scan object.

4. The method of claim 1, wherein said receive beamforming model receives said two-dimensional ultrasound data and maps it onto a set of delay values and weight values for use in a delay-and-sum receive beamforming algorithm.

5. The method of claim 1, wherein said receive beamforming model is further configured to determine a spatial distribution of speed of sound within the object, wherein the method further comprises indicating speed of sound related information in the visual image.

6. The method of claim 1, wherein one or both of said first and second subsets of transducer elements corresponds to a number of transducer elements within a predefined aperture region centered at a given transducer element.

7. The method of claim 1, wherein said first and second subsets of transducer elements overlap with each other, wherein at least 50% of the transducer elements in one of said first and second subsets is also part of the other one of said first and second subsets.

8. The method of claim 1, wherein the first subset of transducer elements is larger than the second subset of transducer elements, and wherein the same first subset is combined with different second subsets, wherein the first subset corresponds to the entire array of transducer elements, while different second subsets are used as receive channels for receive beamforming.

9. (canceled)

10. The method of claim 1, wherein said scan object is a scan line, representing sound reflection at various depths along a line extending from said transducer into the object subjected to imaging.

11. The method of claim 1, wherein said step of constructing a visual image from said plurality of scan objects comprises one or more of a demodulation, a logarithmic compression and a scan conversion/re-interpolation.

12. The method of claim 1, wherein said machine learning based receive beamforming model has been trained using training data obtained with different conventional receive beamforming methods.

13. The method of claim 12, wherein said machine learning based receive beamforming model has been trained in a procedure, in which two or more receive beamforming procedures are carried out on the same two-dimensional ultrasound data but using different conventional receive beamforming methods, leading to a corresponding number of different scan objects, and wherein a resultant scan object is selected or derived from said plurality of different scan objects, and the training is carried out based on said resultant scan object.

14. The method of claim 1, wherein in step A), the activation of said first subset of transducer elements is controlled using a transmit beamforming procedure employing a machine learning based transmit beamforming model that has been trained in combination with said machine learning based receive beamforming model, and that receives, as at least part of its input, said two-dimensional ultrasound data or said scan objects, or using information regarding a spatial distribution of speed of sound within the object determined by means of said receive beamforming model.

15. An apparatus for ultrasound imaging of an object, said apparatus comprising an ultrasound transducer which comprises an array of transducer elements capable of converting sound signals into electrical signals and vice versa, and a control unit, wherein said control unit is configured for controlling the apparatus to carry out the following steps: A) transmitting an ultrasound beam from said ultrasound transducer into the object, by activating a first subset of said transducer elements, B) detecting reflected signals in a time resolved manner by means of a second subset of said transducer elements, wherein timing information of a detected signal is associated with information regarding the depth where the detected signal was reflected within the object subjected to imaging, and wherein the reflected signals associated with said second subset of transducer elements resemble a set of two-dimensional ultrasound data, of which one dimension represents the various transducer elements of said second subset and the other dimension represents depth information, C) converting said two-dimensional ultrasound data into a scan object using a receive beamforming procedure which accounts for differences in distance of individual transducer elements from a given site of sound reflection within the object, repeating steps A) to C) for different choices regarding at least one of said first and second subsets and the timing of the activation of transducer elements within said first subset, thereby obtaining a plurality of scan objects, and a step of constructing a visual image from said plurality of scan objects, wherein said receive beamforming procedure employs a machine learning based receive beamforming model for mapping said two-dimensional ultrasound data to said scan object.

16. The apparatus of claim 15, wherein said machine learning based receive beamforming model employs one of a deep convolutional neural network or a recurrent neural network.

17. The apparatus of claim 15, wherein said receive beamforming model is an end-to-end beamforming model receiving said two-dimensional ultrasound data as an input and directly converting it into said scan object.

18. The apparatus of claim 15, wherein said receive beamforming model is configured to receive said two-dimensional ultrasound data and to map it onto a set of delay values and weight values for use in a delay-and-sum receive beamforming algorithm.

19. The apparatus of claim 15, wherein said receive beamforming model is further configured to determine a spatial distribution of speed of sound within the object, wherein the control unit is preferably further configured for indicating speed of sound related information in the visual image.

20. (canceled)

21. (canceled)

22. (canceled)

23. (canceled)

24. (canceled)

25. (canceled)

26. The apparatus of claim 15, wherein said machine learning based receive beamforming model has been trained, or is obtainable by training using training data obtained with different conventional receive beamforming methods, and wherein said machine learning based receive beamforming model has been trained in a procedure or is obtainable by training in a procedure, in which two or more receive beamforming procedures are carried out on the same two-dimensional ultrasound data but using different conventional receive beamforming methods, leading to a corresponding number of different scan objects, and wherein a resultant scan object is selected or derived from said plurality of different scan objects, and the training is carried out based on said resultant scan object.

27. (canceled)

28. The apparatus of claim 15, wherein said control unit is configured to control in step A) the activation of said first subset of transducer elements, using a transmit beamforming procedure employing a machine learning based transmit beamforming model that has been trained in combination with said machine learning based receive beamforming model, and that receives, as at least part of its input, said two-dimensional ultrasound data or said scan objects, or using information regarding a spatial distribution of speed of sound within the object determined by means of said receive beamforming model.

Description

SHORT DESCRIPTION OF THE FIGURES

[0056] FIG. 1 schematically shows an ultrasound transducer and a beam formed thereby.

[0057] FIG. 2 schematically shows the beamforming workflow according to an embodiment of the invention.

[0058] FIG. 3 is a schematic representation of a data flow in an end-to-end beamforming procedure.

[0059] FIG. 4 schematically shows the architecture of a deep convolutional neural network for use in receive beamforming.

[0060] FIG. 5 schematically shows a residual block with batch normalization between convolutional layers of the neural network of FIG. 4.

[0061] FIG. 6 is a conceptual dataflow diagram showing the generation of optimized scan lines by applying different conventional beamforming methods to the same two-dimensional ultrasound data.

[0062] FIG. 7a is a schematic representation of a speed of sound deepformer.

[0063] FIG. 7b is a schematic representation of an alternative speed of sound deepformer.

[0064] FIG. 8 shows a comparison of different beamforming strategies based on the Structural Similarity Index Measure (SSIM).

[0065] FIG. 9 shows representative examples of the similarity between deepformed images and corresponding delay-and-sum and minimum variance ground truth images.

[0066] FIG. 10 shows examples of the point spread function obtained using a wire phantom.

[0067] FIG. 11 shows images demonstrating how modifications of the ultrasound scanner parameter settings affect the deepformed images.

[0068] FIG. 12 shows a Generator and a Discriminator of a Generative Adversarial Network used for receive beamforming.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0069] For the purposes of promoting an understanding of the principles of the invention, reference will now be made to a preferred embodiment illustrated in the drawings, and specific language will be used to describe the same. It will nevertheless be understood that no limitation of the scope of the invention is thereby intended, such alterations and further modifications in the illustrated apparatus and such further applications of the principles of the invention as illustrated therein being contemplated as would normally occur now or in the future to one skilled in the art to which the invention relates.

[0070] FIG. 1 schematically shows the working principle of an ultrasound transducer 10 for use in a method and system according to the invention. The exemplary ultrasound transducer 10 shown in FIG. 1 comprises a linear array of n.sub.E transducer elements 12 which are capable of converting sound signals into electrical signals and vice versa. In the embodiment shown, the transducer elements 12 are piezoelectric elements.

[0071] Further schematically shown in FIG. 1 is an ultrasound beam 14 having a beam axis 16, wherein said beam 14 is focused in a focal region indicated with reference sign 18. In order to generate such a focused ultrasound beam 14, a subset of transducer elements 12, referred to as the “first subset” herein, is activated by means of corresponding electrical pulses 20. The first subset of transducer elements 12 is also referred to as “aperture” or “aperture region” of the transducer array herein.

[0072] For generating the individual activation pulses 20 for individual transducer elements 12, a carrier signal 22 is combined with a shape signal 24 such as to generate a main pulse 26. The individual activation pulses 20 are derived from the main pulse 26. As is seen in the schematic representation of FIG. 1, individual pulses 20 close to the beam axis 16 are delayed with respect to pulses 20 further away from the beam axis 16 such that the partial soundwaves (not shown) emitted by the individual transducer elements 12 arrive approximately simultaneously in the focal region 18 on the beam axis 16. The activation of the individual transducer elements 12 within the aperture is referred to as “transmit beamforming”. In conventional beamforming, a universal speed of sound is assumed for the tissue subjected to imaging, and based on this assumption, suitable delays can be calculated in a straightforward manner based on Pythagoras' rule. However, a closer inspection reveals that the speed of sound actually varies throughout the tissue, and the inventors have noticed that by not accounting for this inhomogeneous speed of sound distribution, simple, data independent beamforming methods such as the delay-and-sum beamforming method only allow for limited ultrasound imaging quality.

[0073] The thus generated ultrasound pulse will be reflected at the tissue and reflected signals are detected in a time resolved manner by means of a second subset of said transducer elements 12, which may, but need not be the same as the first subset. Herein, the timing information of the detected signal is associated with information regarding the depth in which the detected signal was reflected within the object subjected to imaging. Accordingly, the reflected signals associated with the second subset of transducer elements resemble essentially a set of two-dimensional ultrasound data, of which one dimension represents the various transducer elements of said second subset and the other dimension represents depth information. The two-dimensional ultrasound data are then converted into a scan object. In the embodiment shown, the scan object is a scan line, associated with the beam axis 16, representing sound reflection at various depths along said beam axis 16. The conversion of the two-dimensional ultrasound data into the scan object is referred to as receive beamforming herein.

[0074] The beamforming workflow according to an embodiment of the invention is shown in more detail with reference to FIG. 2. As mentioned with reference to FIG. 1, the transducer 10 comprises an array of n.sub.E piezoelectric elements 12 that are able to transmit and receive signals. A transmission phase (TX) and a reception phase (RX) are repeated alternatively n.sub.E times, once for each element e∈{1, . . . , n.sub.E} of the transducer 10, by shifting the aperture window over the transducer array, thereby selecting the aforementioned “first subset” of transducer elements 12.

[0075] During transmission, an ultrasound pulse of a given frequency is propagated in the tissues through electrical excitation of n.sub.A aperture elements centered around the element e. The direction and focal depth of the propagated wave is controlled through transmit beamforming as explained above. After propagation and reflection in the tissue, the received signals impinging on the transducer 10 constitute the raw channel data, of size n.sub.A×d, which is an example of the “two-dimensional ultrasound data” referred to above.

[0076] During receiving, a receive beamforming operation is carried out for converting the two-dimensional ultrasound data into a scan object. The receive beamforming procedure effectively accounts for differences in distance of individual transducer elements 12 from a given site of sound reflection within the object. This way, a 1D scan line, or radio frequency signal (RF, size 1×y.sub.RF) is obtained. Finally, to create a visually interpretable B-mode (i.e. “brightness mode”) image, the RF signal undergoes demodulation, for example via envelope detection, a log compression, to improve the dynamic range, and a scan conversion, which may comprise a re-interpolation from x.sub.RF×y.sub.RF to x.sub.B×y.sub.B.

[0077] While the receive beamforming procedure accounts for differences in distance of individual transducer elements 12 from the reflection site, in the end-to-end beamforming version of the invention, this is not carried out by introducing explicit delays to the received signals or the like. Instead, the receive beamforming procedure employs a machine learning based receive beamforming model for mapping the two-dimensional ultrasound data to the scan line. Since this process combines beamforming with deep learning, the process is also referred to as “deepforming” herein, and the model is referred to as a “deepformer”. The deepformer may be trained to map the receiving, demodulation, and log compression steps of the ultrasound image formation pipeline as shown in FIG. 2. Given a raw scan line X∈R.sup.d×nA and a fully processed beamformed scan line Y∈R.sup.1×yRF, the deepforming pipeline BF is achieved by learning the transformation BF(X)->Y. This embodiment hence employs an end-to-end deepforming, in which the two-dimensional ultrasound data is received as input and it is directly converted into the scan line. A straightforward operation is then finally applied to concatenate all the deep formed arrays and reshape them in the dimension of the visually understandable final image. The specific data flow of the beamforming part is further illustrated in FIG. 3. The reason why this last operation is provided after the deepforming is twofold. The first reason is that the reshape operation is very simple, as essentially only a resize operation is conducted. The second reason is that it is more robust to apply the deepformer to each channel data and to obtain a single factor, rather than extracting several image columns from a unique initial channel data.

[0078] The machine learning based receive beamforming models may employ one of a convolutional neural network or a recurrent neural network. As was pointed out above, the inventors have noticed that both, recurrent neural networks and deep convolutional neural networks are particularly suitable for creating a receive beamforming model for the purposes of the invention. One of the reasons for this is that recurrent neural networks and deep convolutional networks have the ability to learn custom spatial and temporal filters of an arbitrary dimension over the ultrasound data in order to create a mapping to beamformed scan data. Furthermore, representations of ultrasound data which display inferred contextual and diagnostic information may also be generated. The use of recurrent and convolutional neural network based filters allows for the steps to be concurrently performed in an optimal manner in a learning context. In one embodiment, the neural network employed for the end-to-end deepformer is a deep convolutional neural network with residual skip connections and 4 bottlenecks. The network used in a currently operative version uses 50 total layers. In FIG. 4, a 34 layer exemplary architecture diagram is shown. The currently employed end-to-end deepformer implementation employs batch normalization between convolutional layers of the residual blocks as illustrated in FIG. 5.

[0079] The blocks each describe computations performed on their inputs as indicated by the directed connections (the “arrows”). If an operation is labeled as “/2”, the operation is applied to every second pixel, resulting in an output of half the size.

[0080] The “Cony” blocks apply filters of the specified size (e.g. 3×3 pixels), “Pool” and “Avg pool” accumulate their input in a small spatial window, thereby performing smoothing. Blocks called “Res” are so called residual blocks, their structure is shown in FIG. 5. The “BN” blocks perform a process called batch normalization, in which its inputs are scaled based on statistics observed during training time. This improves the numeric stability of the overall process. “ReLU” blocks introduce non-linearity to the network, by rectifying their inputs, in the example shown, they compute max(o, x). And finally, the block “FC” is a layer in the neural network that has d outputs. In this layer, all inputs are connected to each output with a learned weight, effectively computing d linear combinations of its input.

[0081] The input “ultrasound data” as depicted in FIG. 4 is raw ultrasound data of dimension [aperture-size×depth], i.e. [n.sub.A×d] (cf. FIG. 2). The output size of the fully connected layer is [1×depth scanline].

[0082] As was indicated above, the performance of receive deepforming is limited by the quality of the underlying training data. When training the deepformer using data generated with a single conventional receive beamforming method, the deepformer will eventually mimic its behavior both, with regard to its strengths, but also with regard to its deficiencies. In a sophisticated implementation of deepforming as an end-to-end receive deepformer, for obtaining optimum training data, the output of multiple advanced conventional beamforming can be associated with a quality metric that can mathematically dictate the properties of an output image, for example color map, contrast etc. This way, an optimized scanline can be constructed based on any given two-dimensional ultrasound input data, using the strengths of various existing advanced beamforming methods, and this optimized scanline, together with the two-dimensional ultrasound data, can be used for training the deepformer.

[0083] A conceptual dataflow diagram is shown in FIG. 6. As indicated therein, the same two-dimensional ultrasound input data are subjected to different forms of conventional beamforming, such as delay-and-sum, minimum variance, delay-multiply-and-sum, or the like. Each of the conventional beamformers generate a corresponding beamformed scanline. Each of the beamformed scan lines is subjected to some quality metric awarding a quality score thereto. The quality metrics may involve automatically retrieved quality metrics, such as ultrasound confidence maps, image statistics or the like. However, at least some contribution to the quality score could also be based on expert input based on finally reconstructed images. If the quality metric indicates a high quality standard, a high weight factor is associated with the beamformed scanline, and vice versa. Then, a weighted sum of the individual beam formed scanlines is computed, which resembles an example of the “resultant scan object” referred to above, and which is derived from the set of beamformed scanlines obtained with different conventional beamformers. In a simple embodiment, the beam formed scanline having the highest quality score can simply be selected, which would correspond to assigning a weight factor of 1 to the beamformed scanline with the highest quality score and a weight factor of 0 to the other scanlines. However, in general the resultant scanline will be the weighted sum of individual scanlines. The training is then carried out based on this resultant beamformed scanline, which means that the thus obtained machine learning based receive beamforming model combines the advantages and strengths of various different conventional beamforming methods in one model.

[0084] As was explained above, one underlying assumption for all currently practiced beamforming approaches is a constant and predefined speed of sound in human tissue, amounting to 1540 m/s in most medical systems. This assumption is not only used for transmit beamforming, but also in all prior art receive beamforming methods known to the inventors. By modelling the receive beamforming and wave propagation into tissue as a neural network, a specific speed of sound value can be employed as a characteristic value to automatically optimize receive delays in the deepformer. The idea is that a given parameterization of the speed of sound for a received set of signals will for example minimize the variance and thus maximize image quality.

[0085] FIG. 7a is a schematic representation of a corresponding type of deepformer which is referred to as a “speed of sound deepformer” herein. It represents a machine learning based receive beamforming model which receives the two-dimensional ultrasound data and maps it onto a set of delay values and weight values for use in a delay-and-sum receive beamforming algorithm. This deepformer is referred to as a “speed of sound deepformer”, because the optimum delays obtained by it implicitly reflect the precise distribution of speed of sound in the tissue, and in particular reflect any deviation from the typical assumption of a uniform speed of sound at 1540 m/s. However, in a further variant shown in FIG. 7b, the speed of sound deepformer may explicitly determine estimated speed of sound at each location within the tissue. In some embodiments, as e.g. shown in FIG. 7b, this local speed of sound information may be outputted. For example, in one embodiment, the speed of sound related information may be indicated in the visual image, or be used to augment the visual image. For example, since different types of tissues will typically have different speeds of sound, that speed of sound information can help to improve the contrast of the ultrasound images, and thereby may be used for augmenting the ultrasound images.

[0086] While in the embodiment shown in FIG. 2, the deepforming has been applied only for receive beamforming, in a preferred embodiment, transmit beamforming is likewise based on a machine learning based model. For this purpose, in step A), the activation of said first subset of transducer elements, and in particular a relative weight and/or a relative delay of their activation, is controlled using a transmit beamforming procedure employing a machine learning based transmit beamforming model that has been trained, or is obtainable by training in combination with said machine learning based receive beamforming model, and that receives, as at least part of its input, said two-dimensional ultrasound data or said scan objects. According to this embodiment, the machine learning is hence not only employed on the receive side, but also for transmit beamforming. This allows for a yet further significant increase in the imaging quality. In particular, using machine learning, the transmit beamforming model can implicitly learn from the two-dimensional ultrasound data or scan objects information related to the location dependent speed of sound, and use this implicit knowledge in an optimum transmit beamforming leading to optimal focusing and hence improved image quality, in combination with receive beamforming with which transmit beamforming has been trained.

[0087] In an alternative embodiment, the activation of the first subset of transducer elements, and in particular a relative weight and/or a relative delay of their activation is controlled using information regarding the spatial distribution of speed of sound within the object determined by means of receive beamforming, for example by means of a variant of the speed of sound beamformer described above in a variant that allows for providing explicit speed of sound information.

[0088] Next, results obtained with end-to-end beamforming of the type shown in FIGS. 2 and 3 are presented. The network used for this end-to-end beamforming was built on the residual network (ResNet) architecture and consisted of 50 layers comprised of 4 blocks, of 3 residual units each with intermediate normalization layers. Due to the strong correlation between the raw input signals and the fully processed beamformed output, the identity mapping connections in ResNet allow for the propagation of important features to distant layers, while convolution filters can readily approximate the temporal and spatial integration of beamforming. For further details of the residual network architecture, reference is made to He, K., Zhang, X., Ren, S., Sun, J.: Identity mappings in deep residual networks. In: European Conference on Computer Vision. (2016) 630-645.

[0089] The network was implemented in Python and Tensorflow v1.5. Training was performed using the Adam optimizer (cf. Kingma, Diederik P., and Jimmy Ba. “Adam: A method for stochastic optimization.” arXiv preprint arXiv:1412.6980 (2014)) with an initial learning rate of 0.001 and the mean squared error (MSE) loss function. The loss was computed between the output of the network and a fully processed scan line of the same dimension. Training was completed with early-stopping regularization. To enforce the consistency of the scan line data along the axial dimension, the input to the network was the raw data for a full scan-line, i.e. a 2D structure spanned by the depth and the aperture. To cope with the variance in the raw data caused by attenuation and time gain compensation, both input and output data were normalized depth-wise. This operation was conducted by calculating the means and standard deviations at equal depths of all the processed scan lines in the training set, followed by normalization to zero mean and unit standard deviation.

[0090] Two networks of the same architecture were trained on beamformed scan-line data, processed using conventional delay-and-sum beamforming (DS in the following) and conventional minimum variance beamforming (MV), respectively. Experimental inference was performed on a NVIDIA TITAN Xp GPU (Pascal architecture) with an average inference time of 0.9364±0.0055 s per image.

[0091] Data acquisition ultrasound imaging was performed using a publicly available beamforming software pipeline (SUPRA) with a cQuest Cicada scanner (Cephasonics, CA, USA) equipped with a 128 elements linear transducer (CPLA12875, 7 MHz). The imaging depth was set to 40 mm, pulse frequency of 7 MHz with a single focus at 20 mm, and the dynamic range was 50 dB. Ground truth B-mode images were systematically generated using the previously collected raw ultrasound data via SUPRA's DS and MV beamformers. The dimension of the resulting data was as follows: raw data, (d×n.sub.E×n.sub.A)=(2077×256×64); RF signal, (x.sub.RF×y.sub.RF)=(256×2000); B-mode image, (x.sub.B×y.sub.B)=(753×800), corresponding to 37.5×40.0 mm.sup.2.

[0092] In vivo ultrasound images were acquired on 22 healthy participants (age: 32±13 years, 14 males). All scans were performed for both sides (left, right) and both orientations (longitudinal, transverse), for a duration of at least 10 s (yielding a minimum of 19 frames per scan), while a sweep was conducted to introduce variability in the imaging plane. Human acquisitions were performed free-hand, and participant data was randomly assigned to one of three sets: [0093] The training set (n=14) was used to train the network using 4 anatomies (common carotid artery, thyroid gland, bicep muscle fibers, forearm). A random sub-sample of 38400 scan lines was generated (1.25% of the total available training data) and loaded for training, to comply with memory restrictions. [0094] The validation set (n=5) was used to evaluate the network with the same anatomies as the training set. [0095] The test set (n=3) was used to evaluate the network with three previously unseen and untrained anatomies (femoral artery, calf muscle, Achilles tendon). As phantom data for evaluation purposes, a set of 3 additional frames are acquired on a wire phantom and a LEGO model, both immersed in water, as well as a CIRS phantom (model 040GSE).

[0096] To investigate the influence of the ultrasound scanner configuration on the deepformer performances, three additional images were acquired on the CIRS phantom using different parameter settings: config 1, focus depth of 30 mm instead of 20 mm; config 2, central frequency of 5 MHz instead of 7 MHz; config 3, dynamic range of 80 instead of 50. A mechanical probe holder was used to fix the transducer in a position.

[0097] Evaluation was performed first with a focus on the suitability of deepforming for B-mode image generation. In this view, quantitative results of image similarity between fully beamformed and deepformed B-mode images are indicated for the Structural Similarity Index Measure (SSIM) in FIG. 8, and the overall errors are indicated in the table below. More precisely, FIG. 8 shows the Structural Similarity Index Measure (SSIM) for the participants of the validation set (v) and the testing set (t), comparing the different beamforming strategies, where DS: delay-and-sum, MV: Minimum Variance, DF: Deepforming, GT: Ground Truth (explicit beamforming). L and T stand for Longitudinal and Transverse, respectively. Evaluation of the beamforming quality was carried out qualitatively by comparing the deepformed images with the corresponding ground truth representations, and quantitatively using the SSIM, as described in Wang, Z., Bovik, A. C., Sheikh, H. R., Simoncelli, E. P.: Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13(4) (2004) 600-612, and the Mean Square Error (MSE) metric. The metric provides a strong discrimination of structural reconstruction quality, and thus lends itself well to the noisy nature of ultrasound imaging. This way, the difference in B-mode reconstructions for both MV and DS beamforming can be observed, yielding an average similarity of 0.607±0.262 over the whole test set.

[0098] Shown in the below table is a comparison of the deepformed (DF) and the ground truth (GT) images, for Delay and Sum (DS) as well as Minimum Variance (MV), in the validation (v) and testing (t) datasets, using the Structural Similarity Index Measure (SSIM) and the Mean Square Error (MSE).

TABLE-US-00001 SSIM (v) MSE (v) SSIM (t) MSE (t) DF-DS vs GT-DS 0.564 ± 25.4 ± 0.547 ± 27.2 ± 0.222 28.1 0.231 36.7 DF-MV vs GT-MV 0.497 ± 14.2 ± 0.488 ± 14.1 ± 0.248 18.2 0251 17.8 GT-MV vs GT-DS 0.607. ± 27.0 ± 0.589 ± 29.2 ± 0.262 33.6 0.271 42.4 DF-MV vs DF-DS 0.607 ± 17.5 ± 0.584 ± 17.6 ± 0.224 25.1 0.234 24.9

[0099] It can also be observed that while the overall similarity is lower for DF-MV vs GT-MV (MV) and DF-DS vs GT-DS (DS), they still show widely similar behavior across the test set, with overall similarities of 0.497±0.248 and 0.564±0.222 for MV and DS, respectively. This is also confirmed when comparing the results of the two beamforming methods qualitatively to their respective deepforming counterparts, as indicated in FIG. 9. FIG. 9 shows representative examples of the similarity between deepformed images and the corresponding DS and MV ground truth. Each frame is 37.5×40.0 mm.sup.2

[0100] It is important to note here that the validation set represents unseen data only, and that it also includes anatomies which were not even contained in the training set, thus indicating that deepforming effectively achieves a generalization of the beamforming transformation from raw to pre-scan-converted data.

[0101] Another interesting aspect of the proposed model can be seen for the results of the LEGO phantom (FIG. 9, last row), depicting B-mode images for a structure with vastly different imaging properties (high acoustic impedance and speed of sound) to the training and validation sets. These results are confirmed by the point spread function (PSF) for the four compared methods, created by scanning a previously unseen wire phantom, as shown in FIG. 10. The relatively similar PSF size between the deepformer and the ground truth suggests that image resolution is within the same range. Each image is 4×4 mm.sup.2.

[0102] While deepforming clearly expresses artifacts unrelated to the side-lobes, the PSFs show the ability to provide a generalized mapping of beamforming methods. The evaluation of different imaging settings, config 0, config 1, and config 2 as described above, for acquisitions on a general purpose ultrasound phantom is illustrated in FIG. 11. Images using the config 3 settings are not depicted because no significant change could be observed. Each frame in FIG. 11 is 37.5×40.0 mm.sup.2. Similar to the results above, it can be seen that the trained deepformer can partially cope with deteriorated and unseen raw data parametrizations, where a change in focus and transmit frequency severely impacts received channel data.

[0103] The appearance of deepformed images is determined by an objective function minimized during training. The most common objectives that are often selected for reconstruction tasks are the l.sub.1 and l.sub.2 or Mean Square Error (MSE) loss functions. More advanced reconstruction losses include the Structural Similarity Index (SSIM) and the Multi-Scale Structural Similarity Index (MS-SSIM) as described in Wang, A. Bovik, H. Sheikh, and E. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE TIP, 13(4): 600-612, 2004. MS-SSIM is a multi-scale version of the SSIM, calculated in a pyramid of M levels, and the corresponding loss function can be formulated as:

[00001]${ℒ}_{\mathrm{MS}-\mathrm{SSIM}}=1-l_M\left(p\right).Math.\underset{j=1}{\overset{M}{.Math.}}{\mathrm{cs}}_j\left(p\right).$

[0104] Hybrids of the aforementioned losses are commonly deployed, and a recent example is the composite loss:

.sub.1+MS-SSIM=α.sub.MS-SSIM+(1−α).sub.1,

where α was set to 0.84 (H. Zhao, O. Gallo, I. Frosio, and J. Kautz. Loss functions for neural networks for image processing, 2015. arXiv: 1511.08861). Experimentation with existing loss formulations led to the identification of the need for an improved objective function tailored to ultrasound applications. .sub.MS-SSIM with its property of preserving the contrast in high-frequency regions was found to be an advantageous ingredient. Even though .sub.1 preserves colors and luminance, it does not produce the same contrast as .sub.MS-SSIM, which is why a loss based on the Peak Signal-to-Noise-Ratio (PSNR) may be preferably employed. PSNR is commonly used to measure the quality of reconstruction of lossy compression codecs, for example in applications of image compression. A corresponding loss function may be defined as:

[00002]${ℒ}_{\mathrm{PSNR}}=1-\frac{10\times {\log }_{10}\left(\frac{1}{\mathrm{MSE}\left(x\left(p\right),y\left(p\right)\right)}\right)}{{\mathrm{PSNR}}_{\max }},$

where PSNR.sub.max is the maximum possible pixel value of the image. Since it has been observed that PSNR does not always correlate well with humans' perception of image quality, in preferred embodiments a higher weight is assigned to the .sub.MS-SSIM during training, and a Total Variation (TV) regularization term is incorporated to increase homogeneity in the reconstructions. TV simultaneously preserves the edges and smoothens the noise in flat image regions, even at low signal-to-noise ratios, as was shown by D. Strong and T. Chan. Edge-preserving and scale-dependent properties of total variation regularization. Inverse Problems, 19(6):S165, 2003). For a 2D image y it can be computed as:

[00003]${ℒ}_{\mathrm{TV}}=\underset{i,j}{.Math.}\sqrt{{.Math.y_{i+1,j}-y_{i,j}.Math.}^2+{.Math.y_{i,j+1}-y_{i,j}.Math.}^2}$

[0105] The final proposed objective function for preferred embodiments of the deepformer is:

.sub.DF=α.sub.MS-SSIM+(1−α).sub.PSNR+.sub.TV

where an α of 0.75 was selected as a preferable value after extensive experiments.

[0106] Besides beamforming, the inventors have noticed that anatomies can be classified effectively from raw ultrasound signals. Classifying anatomies has many important applications in medical imaging tasks, such as segmentation of images or assisting robotic navigation. While it would in principle be possible to subject the constructed visual image obtained by the beamformer to some classification algorithm, the present inventors have noticed that better results can be achieved if the anatomy classification is carried out based on so-called latent data in one of the aforementioned bottleneck layers, which are also referred to as the “latent space”. The latent space contains a compressed representation of the image. The bottleneck or latent space is much more compact in size (64×2077) than the input raw signals and the ground truth (2077×256×64).

[0107] In some practical implementations, the classification was carried out using the ResNet18 algorithm as known from K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition, IEEE CVPR, 2016, which was trained with cross-entropy loss. More precisely, the algorithm was trained using Stochastic Gradient Descent with learning rate initialized to 0.001 and momentum of 0.9. Simply put, cross-entropy indicates the distance between what the model believes the output distribution should be and what the original distribution really is. Gradient descent is best used when the parameters cannot be calculated analytically (e.g. using linear algebra) and must be searched for by an optimization algorithm. A learning rate parameter (alpha) must be specified that controls how much the coefficients can change on each update. However, Gradient descent can be slow to run on very large datasets. In situations of large amounts of data, one can use a variation of gradient descent called stochastic gradient descent. In this variation, the gradient descent procedure described above is run but the update to the coefficients is performed for each training instance, rather than at the end of the batch of instances. A momentum term helps accelerate the optimization process and prevents the algorithm from getting stuck in a local, suboptimal solutions.

[0108] The anatomy classification was carried out for various target anatomies, which included common carotid artery, thyroid gland, bicep muscle fibers and forearm, with scans collected from both sides (left, right), thereby providing a total of eight anatomy classes. For comparison purposes, the anatomy classification was not only carried out based on the bottleneck feature, but also based on the Minimum Variance B-mode image and the full deepformer reconstruction. The performance is summarized in the below table:

TABLE-US-00002 Minimum Deepformer Deepformer Variance Reconstruction Bottleneck Accuracy 0.6486 0.6891 0.7540 F1-Score 0.6432 0.6781 0.7402

[0109] As is seen therein, the classification from the bottleneck feature achieves the highest accuracy and F−1 Score, with a significant margin of 0.07. This demonstrates that the encoding learned in the low-dimensional bottleneck or latent space of the deepformer can capture relevant information from the raw data better than if the learning was based on the reconstructed images, and thus can be used for various auxiliary tasks. The B-mode images reconstructed by the deepformer achieve a 0.04 accuracy improvement over the minimum variance ground truth images. This is expected, since the speckle noise in the ground truth induces ambiguity to the network and high frequencies that can “distract” the network's attention from the features that are crucial for the identification of each anatomy.

[0110] In some embodiments, the deepformer can acquire and increase robustness against variation in the sensitivity or even failure of individual transducer elements (such as individual piezo elements) by means of suitable training. In these embodiments, the raw data is “augmented” by artificially “deactivating” or “attenuating” individual transducer elements in the raw data, by simply omitting or suppressing corresponding channels in the raw data by appropriately scaling the signals or their spectra. This augmentation can be implemented as a random or pseudorandom process during the training phase. However, the B-mode image used for training is an image corresponding to fully operative transducer elements. This way, the deepformer learns to establish correct B-mode images (as would be obtained with fully functional transducer elements) from raw data, where individual transducer elements may have failed or may be attenuated. The inventors have noticed that there is sufficient “overlap” in the data associated with individual transducers such that the deepformer—after being trained in the above-mentioned manner—can reconstruct images in which the failure of the individual transducer elements is corrected for. Such pre-training models with activating a deactivating certain elements can in turn be used to create more tailored or even optimal transmission sequences.

[0111] In preferred embodiments, one or both of the receive beamforming model and transmit beamforming model is trained in a Generative Adversarial Network (GAN) scenario. For general details of GANs, reference is made to Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David WardeFarley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27. 2014. In particular, the GAN approach allows for end-to-end denoising of US images directly from the raw signal received by the transducer. For this task the inventors devised a Generative Adversarial Network (GAN) with tailored architectural choices and objective functions that can successfully reconstruct de-speckled B-mode US images, as will be described next.

[0112] Conditional GANs

[0113] GANs consist of two different networks, a Generator G.sub.θ.sub.G and a Discriminator D.sub.θ.sub.D, where the goal is to solve the adversarial min-max problem:

θ.sub.G min θ.sub.D max E.sub.Y˜p(Y)[log D.sub.θ.sub.D(Y)]+E.sub.z˜p(G(z))[log(1−D.sub.θ.sub.D(G.sub.θ.sub.G(z))]  (1)

[0114] Y is the target image and z is random generated noise. For a cGAN, instead of using z as input to the Generator, in the present embodiment it is conditioned to start with labeled data x.fwdarw.G.sub.θ.sub.G(x)). The purpose of this architecture is for the Discriminator to learn and classify the label images and the reconstructed images as Real and Fake, respectively. Then the Generator has to modify the reconstructed images in order to “fool” the Discriminator into thinking they are real.

[0115] Adversarial Network: Generator

[0116] The generator employed in this embodiment is depicted in FIG. 12, and is a modified version of the U-net architecture (see Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. CaRR.). This generator can be used as the received beamforming model described above. Note that similar generators can also be employed for the transmit beamforming models, as part of a corresponding GAN. The network consists of 3 Dense blocks, 1 Max-pooling for downsampling and 1 UnPooling for upsampling. Each Dense block is built with two 3×3 convolutions of stride 1, followed by batch normalization and PReLu. To avoid information loss and to get better reconstruction detail, skip connections are concatenated on the feature dimension between each convolution within the Dense blocks and also between each corresponding layer. Note that in FIG. 12, “k3n64s1” stands for 3×3 kernel filters outputting 64 channels with stride 1.

[0117] Adversarial Network: Discriminator

[0118] Since in this embodiment the images are of size (256, 1600), the discriminator is constructed to have a receptive field of 255 pixels and make its decision based on the whole width of the image. A Fully Convolutional Neural Network with 7 layers is used, where each layer has a 3×3 convolution of stride 2 followed by Batch Normalization and LeakyReLu (α=0.2). Then for the classification layer, two fully connected layers of size 256 and 1 are used, and the output is followed by a sigmoid activation. The discriminator is likewise depicted in FIG. 12.

[0119] Content Loss

[0120] For the challenging task of end-to-end US denoising combinations of loss functions were deployed that are able to reconstruct B-mode images with all the significant structures, without maintaining the speckle noise. Pixel-wise Mean Squared Error (MSE) loss is defined as:

[00004]$\begin{array}{cc}{L}_{\mathrm{MSE}}\left(G\left(x\right),Y\right)=\frac{1}{\mathrm{HW}}\underset{i=1}{\overset{W}{.Math.}}\underset{j=1}{\overset{H}{.Math.}}{\left({G\left(x\right)}_{i,j}-Y_{i,j}\right)}^2& \left(2\right)\end{array}$

[0121] Herein, x is the raw US data and Y is the target image with size (W, H). Optimization on only MSE loss converges into an averaged image. While this is good for de-noising, is also desired to avoid blurring out important textures.

[0122] As shown by Ding Liu, Bihan Wen, Jianbo Jiao, Xianming Liu, Zhangyang Wang, and Thomas S. Huang. Connecting image denoising and high-level vision tasks via deep learning. CoRR.), using pre-trained high-level vision task networks is beneficial for denoising, since it contributes towards maintaining the high-frequency textures. This property is crucial to the preservation of high quality features, significant for applications like automatic segmentation. The VGG19 is utilized to extract two feature layers and minimize the VGG Loss as follows:

[00005]$\begin{array}{cc}{L}_{\mathrm{VGG}}\left(G\left(x\right),Y\right)=\frac{1}{H_{\varphi }{W}_{\varphi }}\underset{i=1}{\overset{W_{\varphi }}{.Math.}}\underset{j=1}{\overset{H_{\varphi }}{.Math.}}{\left({\varphi \left(G\left(x\right)\right)}_{i,j}-{\varphi \left(Y\right)}_{i,j}\right)}^2& \left(3\right)\end{array}$

ϕ corresponds to the feature map obtained by the chosen VGG19 layers (2nd and 5th) before the max-pooling step, with size (H.sub.ϕ; W.sub.ϕ;).

[0123] Adversarial Loss

[0124] In addition to maintaining high quality content, an adversarial loss was deployed to keep the reconstructed images as close as possible to realistic US images. The generative adversarial loss is defined as:

[00006]$\begin{array}{cc}{L}_{\mathrm{Adv}}=\underset{n=1}{\overset{N}{.Math.}}-\log D_{{\theta }_D}\left(G_{{\theta }_G}\left(x\right)\right)& \left(4\right)\end{array}$

where it is tried to maximize the probability of the discriminator to classify the generated image G.sub.θ.sub.G(x) as a real natural image. Finally, the complete loss function is:

L=L.sub.Adv+γL.sub.MSE+βL.sub.VGG  (5)

(see Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. CaRR.)

[0125] Although preferred exemplary embodiments are shown and specified in detail in the drawings and the preceding specification, these should be viewed as purely exemplary and not as limiting the invention. It is noted in this regard that only the preferred exemplary embodiments are shown and specified, and all variations and modifications should be protected that presently or in the future lie within the scope of protection of the invention as defined in the claims.

REFERENCE SIGNS

[0126] 10 ultrasound transducer [0127] 12 transducer element [0128] 14 ultrasound beam [0129] 16 beam axis [0130] 18 focal region of ultrasound beam 14 [0131] 20 electrical pulse [0132] 22 carrier signal [0133] 24 shape signal [0134] 26 main pulse