System and method of refinement of machine learning network parameters for improved performance
12541901 ยท 2026-02-03
Assignee
- University Of Iowa Research Foundation (Iowa City, IA)
- CANON MEDICAL SYSTEMS CORPORATION (Tochigi, JP)
Inventors
- Samir Dev Sharma (Vernon Hills, IL, US)
- Mathews JACOB (Coralville, IA, US)
- Aniket PRAMANIK (Iowa City, IA, US)
- Sampada BHAVE (Vernon Hills, IL, US)
Cpc classification
G01R33/5608
PHYSICS
G01R33/561
PHYSICS
International classification
Abstract
A method for machine learning includes learning, during a training stage, network parameter values of a neural network to obtain a trained neural network configured to perform reconstruction of medical images; refining, during a subsequent refinement stage, the learned network parameter values to generate refined network parameter values defining a refined neural network; and applying input medical image data to the refined neural network to generate a reconstructed medical image. The method retains benefits of machine learning image reconstruction to obtain a desired reconstructed image.
Claims
1. A method of machine learning, the method comprising: learning, via processing circuitry during a training stage, network parameter values of a neural network to obtain a trained neural network configured to perform reconstruction of medical images; refining, via the processing circuitry during a subsequent refinement stage, the learned network parameter values of the trained neural network to generate refined network parameter values defining a refined neural network; and applying, via the processing circuitry, input medical image data to the refined neural network to generate a reconstructed medical image, wherein the refining step further comprises training a set of refined neural networks using a corresponding set of fixed values of a regularization parameter; refining network weights for each refined neural network of the set of refined networks corresponding to the set of fixed values; determining in the refinement stage, a particular value of the regularization parameter; selecting a particular refined network of the set of refined neural networks corresponding to the determined particular value of the regularization parameter; and applying the input medical image data to the selected particular refined neural network having the refined network weights.
2. The method of claim 1, wherein the network parameter values include the regularization parameter, and the method further comprises: learning, in the training stage, a first value of the regularization parameter; refining, in the refinement stage, the first value to a second value of the regularization parameter; and applying the input medical image data to the refined neural network having the refined second value of the regularization parameter.
3. The method of claim 2, wherein the refining of the regularization parameter value further comprises refining the regularization parameter based on quantitative metrics, including a peak signal-to-noise ratio or a structural similarity index.
4. The method of claim 2, wherein the refining of the regularization parameter value further comprises refining the regularization parameter based on medical image characteristics, including a signal-to-noise ratio or a contrast.
5. The method of claim 1, where the refining step comprises refining the network parameters so that the reconstructed medical image is generated with less blur artefact.
6. The method of claim 1, where the refining step comprises refining the network parameters so that the reconstructed medical image is generated with less noise artefact.
7. The method of claim 1, where the refined neural network performs image reconstruction using an unrolled model that includes one or more unrolls, wherein each unroll contains both data consistency and network blocks, with relative weighting of the blocks set by the regularization parameter.
8. The method of claim 1, wherein the learning step further comprises learning network parameter values of the neural network to obtain the trained neural network, which is configured to perform reconstruction of magnetic resonance (MR) images.
9. A method of machine learning, comprising: learning, via processing circuitry during a training stage, network parameter values of a neural network to obtain a trained neural network configured to perform reconstruction of medical images; refining, via the processing circuitry during a subsequent refinement stage, the learned network parameter values of the trained neural network to generate refined network parameter values defining a refined neural network; and applying, via the processing circuitry, input medical image data to the refined neural network to generate a reconstructed medical image, wherein the network parameter values include a regularization parameter, the method further comprises learning, in the training stage, a first value of the regularization parameter; refining in the refinement stage, the first value to a second value of the regularization parameter; and applying the input medical image data to the refined neural network having the refined second value of the regularization parameter, and the method further comprises displaying, on a display, the reconstructed medical image that is output from the refined neural network, and refining the regularization parameter value based on a preferred regularization parameter input based on the displayed reconstructed medical image.
10. An apparatus, comprising: processing circuitry configured to learn, during a training stage, network parameter values of a neural network to obtain a trained neural network configured to perform reconstruction of medical images; refine, during a subsequent refinement stage, the learned network parameter values of the trained neural network to generate refined network parameter values defining a refined neural network; and apply input medical image data to the refined neural network to generate a reconstructed medical image, wherein the processing circuitry is further configured to train a set of refined neural networks using a corresponding set of fixed values of a regularization parameter; refine network weights for each refined neural network of the set of refined networks corresponding to the set of fixed values; determine, in the refinement stage, a particular value of the regularization parameter; select a particular refined network of the set of refined neural networks corresponding to the determined particular value of the regularization parameter; and apply the input medical image data to the selected particular refined neural network having the refined network weights.
11. The apparatus of claim 10, wherein the network parameter values include the regularization parameter, and the processing circuitry is further configured to learn, in the training stage, a first value of the regularization parameter; refine, in the refinement stage, the first value to a second value of the regularization parameter; and apply the input medical image data to the refined neural network having the refined second value of the regularization parameter.
12. The apparatus of claim 11, wherein the processing circuitry is further configured to refine the regularization parameter value based on quantitative metrics, including a peak signal-to-noise ratio or a structural simulation index.
13. The apparatus of claim 11, wherein the processing circuitry is further configured to refine the regularization parameter value based on medical image characteristics, including a signal-to-noise ratio or a contrast.
14. The apparatus of claim 10, wherein the processing circuitry is further configured to refine the network parameters so that the reconstructed medical image is generated with less blur artefact.
15. The apparatus of claim 10, wherein the processing circuitry is further configured to refine the network parameters so that the reconstructed medical image is generated with less noise artefact.
16. The apparatus of claim 10, wherein the refined neural network is configured to perform image reconstruction using an unrolled model that includes one or more unrolls, wherein each unroll contains both data consistency and network blocks, with relative weighting of the blocks set by the regularization parameter.
17. The apparatus of claim 10, wherein the processing circuitry is further configured to learn network parameter values of the neural network to obtain the trained neural network, which is configured to perform reconstruction of magnetic resonance (MR) images.
18. An apparatus, comprising: processing circuitry configured to learn, during a training stage, network parameter values of a neural network to obtain a trained neural network configured to perform reconstruction of medical images; refine, during a subsequent refinement stage, the learned network parameter values of the trained neural network to generate refined network parameter values defining a refined neural network; and apply input medical image data to the refined neural network to generate a reconstructed medical image, wherein the network parameter values include a regularization parameter, the processing circuitry is further configured to learn, in the training stage, a first value of the regularization parameter; refine, in the refinement stage, the first value to a second value of the regularization parameter; and apply the input medical image data to the refined neural network having the refined second value of the regularization parameter, the apparatus further comprises a display to display a reconstructed medical image that is output from the refined neural network, and the processing circuitry is further configured to refine the regularization parameter value based on a preferred regularization parameter input based on the displayed reconstructed medical image.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
(1) A more complete appreciation of this disclosure and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, wherein:
(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9)
(10)
(11)
(12)
(13)
DETAILED DESCRIPTION
(14) In the drawings, like reference numerals designate identical or corresponding parts throughout the several views. Further, as used herein, the words a, an and the like generally carry a meaning of one or more, unless stated otherwise.
(15) Furthermore, the terms approximately, approximate, about, and similar terms generally refer to ranges that include the identified value within a margin of 20%, 10%, or preferably 5%, and any values therebetween.
(16) As mentioned above, a conventional approach to machine-learning artefact reduction requires a second image for blending. The blending approach will not work in image reconstruction, as a second image with good image quality is not available. The Siemens machine-learning reconstruction method uses a two-step approach. The first step is to perform machine-learning reconstruction. The second step is to do CG-SENSE reconstruction using a machine-learning output as a regularizer. Although a regularization parameter can be varied in the second step, the regularization parameter is not learned. Also, the Siemens approach loses the benefit of fast machine-learning reconstruction. The Philips approach automatically chooses regularization during the inference. Philips does not disclose learning a regularization parameter.
(17) The present disclosed embodiments include a refinement stage in order to achieve desired image quality in MR image reconstruction on a consistent basis.
(18) To overcome the above challenges, the present disclosure uses a conditional unrolled neural network architecture. In one embodiment, the method alternates between data-consistency and machine learning network blocks, with relative weighting of the blocks set by the regularization parameter 1. The regularization parameter is learned during training, refined in the refinement stage, and then applied during inference. The regularization parameter can be refined to be equal for all the unrolls. The regularization parameter can be refined to be different for each of the unrolls. The network weights can be refined to be equal for all the unrolls. The network weights can be refined to be different for each of the unrolls. The regularization parameter value can be refined based on quantitative metrics such as PSNR or SSIM. The regularization parameter value can be refined based on human (e.g. radiologist) preference. The regularization parameter value can be refined based on image characteristics such as SNR or contrast.
(19)
(20) In the training stage, training data 102 is used to train a deep learning neural network, such as a convolutional neural network (CNN) or a U-Net for image reconstruction. Training performed in step S104 can include a form of supervised learning. When a training loss function reaches a predetermined training criteria, the resulting neural network is a trained neural network 106. As noted above, an image reconstructed using the trained neural network can be blurry and/or noisy.
(21) Subsequently, in the refinement stage, in S112, the learned machine learning network parameters of the trained neural network are refined. The network parameters are refined to generate a refined network 114 to achieve a desired image quality.
(22) In the inference stage, an inference image 122 is input. In step S124, the refined machine learning network outputs a reconstructed image 126 having desired image quality characteristics, while retaining the benefits, including speed, of machine-learning image reconstruction (MLR).
(23) For purposes of background, an imaging system can be formulated as an operator that acts on a continuous domain image x:
.sup.2.fwdarw.C to yield a vector of measurements
(x)=bC.sup.N. The goal of image reconstruction is to recover a discrete approximation, denoted by the vector x
.sup.p from b. Conventional model-based imaging schemes use a discrete approximation of
, that maps x to b. Conventional model-based algorithms make the assumption that
B=(x).
(24) For example, in the single-channel Cartesian MRI acquisition setting, =SF, where F is the 2-D discrete Fourier transform, while S is the fat sampling matrix that pick rows of the Fourier matrix.
(25) The recovery of x from b is ill-posed, especially when A is a rectangular matrix. The general practice in conventional model-based imaging is to pose the recovery as a regularized optimization scheme:
(26)
where, is a regularization parameter. The regularization prior R:C.sup.n.fwdarw..sub.>0 is engineered to restrict the solutions to the space of desirable images. For example, R(x) is a small scalar when x is a noise and artifact-free image, while its value is high for noisy images. Classical choices include norms of wavelet coefficients, total variation, as well as their combinations.
(27) Alternatively, conventional deep learning based algorithms recover the images as
X.sub.rec=T.sub.w(A.sup.Hb),
where T.sub.w is a learned CNN. The operator A.sup.H() transforms the measurement data to the image domain, since CNNs are designed to work in the image domain. This gives the relation
X.sub.rec=T.sub.w(A.sup.HAx),
Thus, the CNN network is learned to invert the normal operator A.sup.HA: i.e., T.sub.w(A.sup.HA).sup.1 for signals living in the image set.
(28) For many measurement operators (e.g., Fourier sampling, blurring, and projection imaging), A.sup.HA is a translation-invariant operator; the convolutional structure makes it possible for CNNs to solve such problems. However, the receptive field of the CNN has to be comparable to the support of the point-spread function corresponding to (A.sup.HA). In applications involving Fourier sampling or projection imaging, the receptive field of the CNNs has to be the same as that of the image; large networks such as UNET with several layers are required to obtain such a large receptive field. A challenge with such large network with many free parameters is the need for extensive training data to reliably train the parameters. Another challenge is that the CNN structure may not be well-suited for problems such as parallel MRI, where A.sup.HA is not translational-invariant.
(29) A framework, termed as Model-based reconstruction using Deep Learning priors (MoDL), merges the model-based reconstruction schemes with a deep learning-based algorithm. The overall framework involves a data-consistency layer that takes output from a learned CNN to capture image redundancy.
(30) In the present disclosure, the reconstruction of the image xC.sup.n is formulated as an optimization:
(31)
(32) where .sub.W is a learned CNN estimator of noise and alias patterns, which depends on the learned parameters w.
(33) .sub.w block 210. Each layer 202 consists of convolution 222 and a non-linear activation function ReLU 226 (rectified linear unit, (x)=max(0, x)). Following the residual learning strategy, the learned noise from
.sub.w block 210 is added with the input of
.sub.w block to obtain the reconstructed image. The output of
.sub.w block 210 is fetched into data consistency (DC) layer as shown in
(34) As shown in
(35) In the MoDL framework, the CNN parameters [.sub.w(x)] and a regularization parameter are learned in the training stage. However, it has been found in practice that the estimated image using MoDL tends to include blur artefacts and noise artefacts. Disclosed embodiments add a refinement stage before the inference stage in order to achieve a preferred image quality with less blur artefact and less noise artefact.
(36) The refinement stage is performed on the trained network. In some embodiments, the data used for the refinement stage can be the original training data. Also, for embodiments in which the network weights are refined via a retraining, the original training set can be used.
(37) In some embodiments, the regularization parameter is first learned in the training stage and then refined before the inference stage. In some embodiments, the regularization parameter is refined to be equal for all of the unrolls. In some embodiments, the regularization parameter is refined to be different for each of the unrolls.
(38)
(39) In step S402, the convolution kernel weights can be updated by a method in which a set of trained networks are further refined using a corresponding set of values.
(40) In step S404, for each value in the set, the network weights from the initial training stage are refined to minimize the training loss.
(41) Then, in step S406, a particular value is selected as the one producing a preferred image quality and the network weights from the refinement training stage corresponding to the selected particular value of are used during inferencing.
(42) In some embodiments, the convolution kernel weights can be updated in step S402 without the retraining step S404. In some embodiments, the convolution kernel weights are refined to be equal for all unrolls. In some embodiments, the convolution kernel weights are refined to be different for all unrolls. In one embodiment, the trained weight parameters can be adjusted by scaling, in which all, or a subset, of the weights are multiplied or divided by a scale factor. In one embodiment, a majority of the weights are modified. The scale factor can be an integer value, for example two, where all weights are divided by two. In some embodiments, the scale factor can be much larger, on the order of 200. In some embodiments, the scale factor is determined empirically.
(43) In some embodiments, trained weight parameters can be refined to produce a preferred image quality. The trained parameters can be refined using any of several methods and resulting values can be evaluated using image quality metrics. In one embodiment, in the refinement stage, the trained parameters can be adjusted to arbitrary new values and the resulting values can be evaluated using image quality metrics such as a structural similarity index (SSIM), peak signal-to-noise ratio (PSNR), or entropy. In one embodiment, an image resulting from the refinement method can be displayed to a radiologist for evaluation of image quality.
(44) In one embodiment, an imaging system includes an input interface and a display to display a reconstructed medical image that is output from the refined neural network. The regularization parameter value is then refined based on a preferred regularization parameter input via the input interface based on the displayed reconstructed medical image.
(45) In one embodiment, an optimal refined image can be selected based on characteristics of an inference dataset, e.g., signal-to-noise ratio or contrast.
(46)
(47)
(48)
(49)
(50)
(51)
(52)
(53)
(54) The scanning device 1162 is configured to acquire scan data by scanning a region (e.g., area, volume, slice) of an object (e.g., a patient). The scanning modality may be, for example, magnetic resonance imaging (MRI), computed tomography (CT), positron emission tomography (PET), X-ray radiography, and ultrasonography.
(55) The one or more image-generation devices 1164 obtain scan data from the scanning device 1162 and generate an image of the region of the object based on the scan data. To generate the image, for example during intermediate image generation or during final image reconstruction, the one or more image-generation devices 1164 may perform a reconstruction process on the scan data. Examples of reconstruction processes include GRAPPA, CG-SENSE, SENSE, ARC, SPIRIT, and LORAKS, and compressed sensing.
(56) In an embodiment, after the one or more image-generation devices 1164 generate the image, the one or more image-generation devices 1164 send the image to the display device 1164, which displays the image.
(57) In another embodiment, and further to the above, the one or more image-generation devices 1164 may generate two images from the same scan data. The one or more image-generation devices 1164 may use different reconstruction processes to generate the two images from the same scan data, and one image may have a lower resolution than the other image. Additionally, the one or more image-generation devices 1164 may generate an image.
(58) Referring now to
(59) One or more smaller array RF coils 1279 can be more closely coupled to the patient's head (referred to herein, for example, as scanned object or object) in imaging volume 1276. As those in the art will appreciate, compared to the WBC (whole-body coil), relatively small coils and/or arrays, such as surface coils or the like, are often customized for particular body parts (e.g., arms, shoulders, elbows, wrists, knees, legs, chest, spine, etc.). Such smaller RF coils are referred to herein as array coils (AC) or phased-array coils (PAC). These can include at least one coil configured to transmit RF signals into the imaging volume, and a plurality of receiver coils configured to receive RF signals from an object, such as the patient's head, in the imaging volume 1276.
(60) The MRI system 1270 includes a MRI system controller 1283 that has input/output ports connected to a display 1280, a keyboard 1281, and a printer 1282. As will be appreciated, the display 1280 can be of the touch-screen variety so that it provides control inputs as well. A mouse or other I/O device(s) can also be provided.
(61) The MRI system controller 1283 interfaces with a MRI sequence controller 1284, which, in turn, controls the Gx, Gy, and Gz gradient coil drivers 1285, as well as the RF transmitter 1286, and the transmit/receive switch 1287 (if the same RF coil is used for both transmission and reception). The MRI sequence controller 1284 includes suitable program code structure 1288 for implementing MRI imaging (also known as nuclear magnetic resonance, or NMR, imaging) techniques including parallel imaging. Moreover, the MRI sequence controller 1284 includes processing circuitry to execute the scan control process illustrated in
(62) The MRI system components 1272 include an RF receiver 1289 providing input to data processor 1290 so as to create processed image data, which is sent to display 1280. The MRI data processor 1290 is also configured to access previously generated MR data, images, and/or maps, such as, for example, coil sensitivity maps, parallel image unfolding maps, distortion maps and/or system configuration parameters 1291, and MRI image reconstruction program code structures 1292 and 1293.
(63) In one embodiment, the MRI data processor 1290 includes processing circuitry. The processing circuitry can include devices such as an application-specific integrated circuit (ASIC), configurable logic devices (e.g., simple programmable logic devices (SPLDs), complex programmable logic devices (CPLDs), and field programmable gate arrays (FPGAs), and other circuit components that are arranged to perform the functions recited in the present disclosure.
(64) The processor 1290 executes one or more sequences of one or more instructions, such as method 100 described herein, contained in the program code structures 1292 and 1293. Alternatively, the instructions can be read from another computer-readable medium, such as a hard disk or a removable media drive. One or more processors in a multi-processing arrangement can also be employed to execute the sequences of instructions contained in the program code structures 1292 and 1293. In alternative embodiments, hard-wired circuitry can be used in place of or in combination with software instructions. Thus, the disclosed embodiments are not limited to any specific combination of hardware circuitry and software.
(65) Additionally, the term computer-readable medium as used herein refers to any non-transitory medium that participates in providing instructions to the processor 1290 for execution. A computer-readable medium can take many forms, including, but not limited to, non-volatile media or volatile media. Non-volatile media includes, for example, optical, magnetic disks, and magneto-optical disks, or a removable media drive. Volatile media includes dynamic memory.
(66) Also illustrated in
(67) Additionally, the MRI system 1270 as depicted in
(68) Furthermore, not only does the physical state of the processing circuits (e.g., CPUs, registers, buffers, arithmetic units, etc.) progressively change from one clock cycle to another during the course of operation, the physical state of associated data storage media (e.g., bit storage sites in magnetic storage media) is transformed from one state to another during operation of such a system. For example, at the conclusion of an image reconstruction process and/or sometimes an image reconstruction map (e.g., coil sensitivity map, unfolding map, ghosting map, a distortion map etc.) generation process, an array of computer-readable accessible data value storage sites in physical storage media will be transformed from some prior state (e.g., all uniform zero values or all one values) to a new state wherein the physical states at the physical sites of such an array vary between minimum and maximum values to represent real world physical events and conditions (e.g., the internal physical structures of a patient over an imaging volume space). As those in the art will appreciate, such arrays of stored data values represent and also constitute a physical structure, as does a particular structure of computer control program codes that, when sequentially loaded into instruction registers and executed by one or more CPUs of the MRI system 1270, causes a particular sequence of operational states to occur and be transitioned through within the MRI system 1270.
(69) The above-described hardware description is a non-limiting example of corresponding structure for performing the functionality described herein.
(70) Numerous modifications and variations of the present disclosure are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims, the invention may be practiced otherwise than as specifically described herein.