ADAPTIVE IMAGE PROCESSING METHOD AND SYSTEM IN ASSISTED REPRODUCTIVE TECHNOLOGIES

Abstract

Adaptive image processing, image analysis, pattern recognition, and time-to-event prediction in various imaging modalities associated with assisted reproductive technology. The reference image may be processed according to one or more adaptive processing frameworks for de-speckling or noise processing of ultrasound images. The subject image is processed according to various computer vision techniques for object detection, recognition, annotation, segmentation, and classification of reproductive anatomy, such as follicles, ovaries and the uterus. An image processing framework may also analyze secondary data along with subject image data to analyze time-to-event progression of the subject image.

Claims

1. A system for digital image processing in assisted reproductive technologies, the system comprising: an imaging sensor configured to collect one or more digital images of a reproductive anatomy of a patient; a computing device communicably engaged with the imaging sensor to receive the one or more digital images of the reproductive anatomy of the patient; and at least one processor communicably engaged with the computing device and at least one non-transitory computer-readable medium having instructions stored thereon that, when executed, cause the at least one processor to perform one or more operations, the one or more operations comprising: receiving the one or more digital images of the reproductive anatomy of the patient; processing the one or more digital images of the reproductive anatomy of the patient to detect one or more reproductive anatomical structures and annotate one or more anatomical features of the one or more reproductive anatomical structures; analyzing the one or more anatomical features according to at least one machine learning framework to predict at least one time-to-event outcome, wherein at least one time-to-event comprises an ovulatory trigger date within an ovulation induction cycle for the patient; and generating at least one graphical user output corresponding to one or more clinical actions related to the patient, wherein the one or more clinical actions comprise a recommended timing for administration of at least one pharmaceutical agent to the patient, wherein the at least one pharmaceutical agent comprises an ovulatory trigger agent.

2. The system of claim 1 wherein the one or more clinical actions comprise a recommended timing for sperm delivery or intrauterine insemination corresponding to the ovulation induction cycle.

3. The system of claim 1 wherein the one or more operations of the processor further comprise analyzing a plurality of electronic health record data of the patient, together with the one or more anatomical features, to predict the at least one time-to-event outcome.

4. The system of claim 3 wherein the plurality of electronic health record data comprises one or more data set selected from the group consisting of diagnostic results, body fluid biomarkers, hormone markers, hormone levels, genomic biomarkers, proteomic biomarkers, therapeutic treatments, treatment schedule, follicle size and number, follicle growth rate, pregnancy rate, and ovulatory induction data.

5. The system of claim 1 wherein the one or more operations of the processor further comprise analyzing a plurality of anonymized historical data from one or more anonymized ovulation induction patients, together with the one or more anatomical features, to predict the at least one time-to-event outcome.

6. The system of claim 5 wherein the plurality of anonymized historical data comprises one or more data set selected from the group consisting of diagnostic results, body fluid biomarkers, hormone markers, hormone levels, genomic biomarkers, proteomic biomarkers, therapeutic treatments, treatment schedule, follicle size and number, follicle growth rate, pregnancy rate, and ovulatory induction data.

7. The system of claim 1 wherein the machine learning framework is selected from the group consisting of an artificial neural network, a regression model, a convolutional neural network, a recurrent neural network, a fully convolutional neural network, a dilated residual network, and a generative adversarial network.

8. The system of claim 1 wherein the one or more reproductive anatomical structures comprise one or more ovarian follicles and the one or more anatomical features comprise a quantity and size of the one or more ovarian follicles.

9. The system of claim 1 wherein the one or more operations of the processor further comprise receiving reproductive physiology data of the patient and analyzing the reproductive physiology data, together with the one or more anatomical features, to predict the at least one time-to-event outcome.

10. The system of claim 1 wherein the one or more operations of the processor further comprise analyzing the one or more anatomical features according to the at least one machine learning framework to assess a risk of multiple pregnancy for the patient.

11. A method for processing digital images in assisted reproductive technologies, the method comprising: obtaining, with an ultrasound device, one or more digital images of a reproductive anatomy of a patient; receiving, with at least one processor, the one or more digital images; processing, with the at least one processor, the one or more digital images to detect one or more reproductive anatomical structures of the reproductive anatomy of a patient; processing, with the at least one processor, the one or more digital images to annotate, segment, or classify one or more anatomical features of the one or more reproductive anatomical structures; analyzing, with the at least one processor, the one or more anatomical features according to at least one machine learning framework to predict at least one time-to-event outcome, wherein at least one time-to-event comprises an ovulatory trigger date within an ovulation induction cycle for the patient; and generating, with the at least one processor, at least one clinical recommendation comprising a recommended timing for administration of at least one pharmaceutical agent to the patient, wherein the at least one pharmaceutical agent comprises an ovulatory trigger agent.

12. The method of claim 11 wherein the at least one clinical recommendation comprises a recommended timing for sperm delivery or intrauterine insemination corresponding to the ovulation induction cycle.

13. The method of claim 11 wherein the one or more reproductive anatomical structures comprise one or more ovarian follicles and the one or more anatomical features comprise a quantity and size of the one or more ovarian follicles.

14. The method of claim 11 further comprising analyzing, with the at least one processor, the one or more anatomical features according to the at least one machine learning framework to assess a risk of multiple pregnancy for the patient.

15. The method of claim 13 further comprising analyzing, with the at least one processor, the one or more anatomical features according to the at least one machine learning framework to determine a maturity rate of the one or more ovarian follicles of the patient.

16. The method of claim 11 further comprising analyzing, with the at least one processor, a plurality of electronic health record data of the patient, together with the one or more anatomical features, to predict the at least one time-to-event outcome.

17. The method of claim 16 wherein the plurality of electronic health record data comprises one or more data set selected from the group consisting of diagnostic results, body fluid biomarkers, hormone markers, hormone levels, genomic biomarkers, proteomic biomarkers, therapeutic treatments, treatment schedule, follicle size and number, follicle growth rate, pregnancy rate, and ovulatory induction data.

18. The method of claim 11 further comprising analyzing, with the at least one processor, a plurality of anonymized historical data from one or more anonymized ovulatory induction patients, together with the one or more anatomical features, to predict the at least one time-to-event outcome.

19. The method of claim 18 wherein the plurality of anonymized historical data comprises one or more data set selected from the group consisting of diagnostic results, body fluid biomarkers, hormone markers, hormone levels, genomic biomarkers, proteomic biomarkers, therapeutic treatments, treatment schedule, follicle size and number, follicle growth rate, pregnancy rate, and ovulatory induction data.

20. A non-transitory computer-readable medium having computer-executable instructions stored thereon that, when executed, cause at least one processor to perform one or more operations of a method for digital image processing, the one or more operations comprising: receiving one or more digital images of a reproductive anatomy of a patient; processing the one or more digital images of the reproductive anatomy of the patient to detect one or more reproductive anatomical structures and annotate one or more anatomical features of the one or more reproductive anatomical structures; analyzing the one or more anatomical features according to at least one machine learning framework to predict at least one time-to-event outcome, wherein at least one time-to-event comprises an ovulatory trigger date within an ovulatory induction cycle for the patient; and generating at least one graphical user output corresponding to one or more clinical actions related to the patient, wherein the one or more clinical actions comprise a recommended timing for administration of at least one pharmaceutical agent to the patient, wherein the at least one pharmaceutical agent comprises an ovulatory trigger agent.

Description

BRIEF DESCRIPTION OF DRAWINGS

[0047] The above and other objects, features and advantages of the present disclosure will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0048] FIG. 1A is an architecture diagram of an exemplary computing apparatus and/or system in which one or more aspects of the present disclosure may be implemented and/or embodied;

[0049] FIG. 1 is an illustration of the elements of a Convolutional Neural Network architecture, in accordance with an embodiment of the present disclosure;

[0050] FIG. 2 is a schematic diagram of a deep learning (DL) pipeline development process, in accordance with an embodiment of the present disclosure;

[0051] FIG. 3 is a schematic diagram of the US-DRN architecture, in accordance with an embodiment of the present disclosure;

[0052] FIG. 4 is a flowchart of a deep learning network for the US image de-speckling process, in accordance with an embodiment of the present disclosure;

[0053] FIG. 5 is a schematic diagram of a computer-aided diagnosis (CAD) architecture and processing method, in accordance with an embodiment of the present disclosure;

[0054] FIG. 6 is a flowchart of a process for the detection of a recruitable follicle, in accordance with an embodiment of the present disclosure;

[0055] FIG. 7 is a schematic diagram of the CNN architectures for detection of recruitable follicles, in accordance with an embodiment of the present disclosure;

[0056] FIG. 7B is a schematic diagram of the multitask end-to-end deep learning framework for follicle segmentation, characterization, and tracking in accordance with an embodiment of the present disclosure;

[0057] FIG. 7C is a schematic diagram of the Mark R-CNN architecture for follicle instance segmentation, in accordance with an embodiment of the present disclosure;

[0058] FIG. 7D is a flow diagram for follicle tracking, in accordance with an embodiment of the present disclosure;

[0059] FIG. 8 is a flowchart of the follicle detection and localization framework, in accordance with an embodiment of the present disclosure;

[0060] FIG. 9 is a flowchart of the follicle tracking framework, in accordance with an embodiment of the present disclosure;

[0061] FIG. 10 is a diagram of an EMR data convolution network architecture, in accordance with an embodiment of the present disclosure;

[0062] FIG. 11 is a schematic diagram of a survival convolution neural network architecture, in accordance with an embodiment of the present disclosure;

[0063] FIG. 12 is a schematic diagram of a computer product architecture, in accordance with an embodiment of the present disclosure;

[0064] FIG. 13 is a process flow diagram of a method for determining a follicular maturity date of an ovarian follicle, in accordance with an embodiment of the present disclosure;

[0065] FIG. 14 is a process flow diagram of a method for generating a clinical recommendation, in accordance with an embodiment of the present disclosure;

[0066] FIG. 14B a process flow diagram of a process for generating a clinical recommendation associated with the OI procedure, in accordance with an embodiment of the present disclosure;

[0067] FIG. 15 is a process flow diagram of a method for generating a clinical recommendation associated with the OI procedure, in accordance with an embodiment of the present disclosure; and

[0068] FIG. 15B is a process flow diagram of a process for generating a clinical recommendation associated with the OI procedure, in accordance with an embodiment of the present disclosure.

DETAILED DESCRIPTION

[0069] It should be appreciated that all combinations of the concepts discussed in greater detail below (provided such concepts are not mutually inconsistent) are contemplated as being part of the inventive subject matter disclosed herein. It also should be appreciated that terminology explicitly employed herein that also may appear in any disclosure incorporated by reference should be accorded a meaning most consistent with the particular concepts disclosed herein.

[0070] Embodiments of the present disclosure provide for an ensemble of Deep Learning (DL) systems and methods in the provision of assisted reproductive technology (ART) for the diagnosis, treatment, and the clinical management of infertility. The ensemble comprises one or more artificial neural networks (ANNs) systems and methods for de-speckle or noise processing of ultrasound images, using computer vision techniques (e.g., Convolution Neural Networks) for object detection, recognition, annotation, segmentation, classification, counting, and tracking of reproductive anatomy, such as follicles and ovaries. ANNs systems and methods are also assembled to analyze electronic medical records to identify and phenotype characteristic time-to-event patient outcomes for predicting the timing of optimal follicle extraction and implantation in patients. The methods and systems are incorporated into a cloud-based computer program and mobile application that enable physician and patient access to clinical insights in the clinical and patient management of infertility.

[0071] It should be appreciated that various concepts introduced above and discussed in greater detail below may be implemented in any of numerous ways, as the disclosed concepts are not limited to any particular manner of implementation. Examples of specific implementations and applications are provided primarily for illustrative purposes. The present disclosure should in no way be limited to the exemplary implementation and techniques illustrated in the drawings and described below.

[0072] Where a range of values is provided, it is understood that each intervening value, to the tenth of the unit of the lower limit unless the context clearly dictates otherwise, between the upper and lower limit of that range and any other stated or intervening value in that stated range is encompassed by the invention. The upper and lower limits of these smaller ranges may independently be included in the smaller ranges, and are also encompassed by the invention, subject to any specifically excluded limit in a stated range. Where a stated range includes one or both of the endpoint limits, ranges excluding either or both of those included endpoints are also included in the scope of the invention.

[0073] As used herein, exemplary means serving as an example or illustration and does not necessarily denote ideal or best.

[0074] As used herein, the term includes means includes but is not limited to, the term including means including but not limited to. The term based on means based at least in part on.

[0075] Turning now descriptively to the drawings, in which similar reference characters denote similar elements throughout the several views, FIG. 1A depicts an exemplary computing system in which certain illustrated embodiments of the present invention may be implemented.

[0076] Referring now to FIG. 1A, a processor-implemented computing device in which one or more aspects of the present disclosure may be implemented is shown. According to an embodiment, a processing system 100a may generally comprise at least one processor 102a, or processing unit or plurality of processors, memory 104a, at least one input device 106a and at least one output device 108a, coupled together via a bus or group of buses 110a. In certain embodiments, input device 106a and output device 108a could be the same device. An interface 112a can also be provided for coupling the processing system 100a to one or more peripheral devices, for example interface 112a could be a PCI card or PC card. At least one storage device 114a which houses at least one database 116a can also be provided. The memory 104a can be any form of memory device, for example, volatile or non-volatile memory, solid state storage devices, magnetic devices, etc. The processor 102a could comprise more than one distinct processing device, for example to handle different functions within the processing system 100a. Input device 106a receives input data 118a and can comprise, for example, a keyboard, a pointer device such as a pen-like device or a mouse, audio receiving device for voice controlled activation such as a microphone, data receiver or antenna such as a modem or wireless data adaptor, data acquisition card, etc. Input data 118a could come from different sources, for example keyboard instructions in conjunction with data received via a network. Output device 108a produces or generates output data 120a and can comprise, for example, a display device or monitor in which case output data 120a is visual, a printer in which case output data 120a is printed, a port for example a USB port, a peripheral component adaptor, a data transmitter or antenna such as a modem or wireless network adaptor, etc. Output data 120a could be distinct and derived from different output devices, for example a visual display on a monitor in conjunction with data transmitted to a network. A user could view data output, or an interpretation of the data output, on, for example, a monitor or using a printer. The storage device 114a can be any form of data or information storage means, for example, volatile or non-volatile memory, solid state storage devices, magnetic devices, etc.

[0077] In use, the processing system 100a is adapted to allow data or information to be stored in and/or retrieved from, via wired or wireless communication means, at least one data storage structure (e.g. database) 116a. The interface 112a may allow wired and/or wireless communication between the processing unit 102a and peripheral components that may serve a specialized purpose. In general, the processor 102a can receive instructions as input data 118a via input device 106a and can display processed results or other output to a user by utilizing output device 108a. More than one input device 106a and/or output device 108a can be provided. It should be appreciated that the processing system 100a may be any form of terminal, server, specialized hardware, or the like.

[0078] It is to be appreciated that the processing system 100a may be a part of a networked communications system. Processing system 100a could connect to a network, for example the Internet or a WAN. Input data 118a and output data 120a could be communicated to other devices via the network. The transfer of information and/or data over the network can be achieved using wired communications means or wireless communications means. A server can facilitate the transfer of data between the network and one or more databases. A server and one or more databases provide an example of an information source.

[0079] Thus, the processing computing system environment 100a illustrated in FIG. 1A may operate in a networked environment using logical connections to one or more remote computers. The remote computer may be a personal computer, a server, a router, a network PC, a peer device, or other common network node, and typically includes many or all of the elements described above.

[0080] It is to be further appreciated that the logical connections depicted in FIG. 1A include a local area network (LAN) and a wide area network (WAN), but may also include other networks such as a personal area network (PAN). Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets, and the Internet. For instance, when used in a LAN networking environment, the computing system environment 100a is connected to the LAN through a network interface or adapter. When used in a WAN networking environment, the computing system environment typically includes a modem or other means for establishing communications over the WAN, such as the Internet. The modem, which may be internal or external, may be connected to a system bus via a user input interface, or via another appropriate mechanism. In a networked environment, program modules depicted relative to the computing system environment 100a, or portions thereof, may be stored in a remote memory storage device. It is to be appreciated that the illustrated network connections of FIG. 1A are exemplary and other means of establishing a communications link between multiple computers may be used.

[0081] FIG. 1A is intended to provide a brief, general description of an illustrative and/or suitable exemplary environment in which embodiments of the below described present invention may be implemented. FIG. 1A is an example of a suitable environment and is not intended to suggest any limitation as to the structure, scope of use, or functionality of an embodiment of the present invention. A particular environment should not be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in an exemplary operating environment. For example, in certain instances, one or more elements of an environment may be deemed not necessary and omitted. In other instances, one or more other elements may be deemed necessary and added.

[0082] In the description that follows, certain embodiments may be described with reference to acts and symbolic representations of operations that are performed by one or more computing devices, such as the computing system environment 100a of FIG. 1A. As such, it will be understood that such acts and operations, which are at times referred to as being computer-executed, include the manipulation by the processor of the computer of electrical signals representing data in a structured form. This manipulation transforms the data or maintains them at locations in the memory system of the computer, which reconfigures or otherwise alters the operation of the computer in a manner understood by those skilled in the art. The data structures in which data is maintained are physical locations of the memory that have particular properties defined by the format of the data. However, while an embodiment is being described in the foregoing context, it is not meant to be limiting as those of skill in the art will appreciate that the acts and operations described hereinafter may also be implemented in hardware.

[0083] Embodiments may be implemented with numerous other general-purpose or special-purpose computing devices and computing system environments or configurations. Examples of well-known computing systems, environments, and configurations that may be suitable for use with an embodiment include, but are not limited to, personal computers, handheld or laptop devices, personal digital assistants, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network, minicomputers, server computers, game server computers, web server computers, mainframe computers, and distributed computing environments that include any of the above systems or devices.

[0084] Embodiments may be described in a general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. An embodiment may also be practiced in a distributed computing environment where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

[0085] With the exemplary computing system environment 100a of FIG. 1A being generally shown and discussed above, description will now turn towards illustrated embodiments of the present disclosure.

[0086] Referring now to FIG. 1, an architecture diagram of a Convolutional Neural Network architecture 100 is shown. A convolutional neural network (CNN) is a special type of artificial neural network (ANN). The fundamental difference between a densely connected layer of an ANN and a convolution layer is that ANNs learn global patterns in their input feature space. In contrast, convolution layers learn local patterns that are usually small 2D windows, patches, filters, or kernels 102 of an input image (or input layer) 104. The patterns learned by CNNs are translational invariant, allowing global pattern recognition within an image or sequence. A CNN can also learn spatial hierarchies of patterns whereby a first convolution layer (or hidden layer) 106 can learn small local patterns such as edges and additional or subsequent layers will learn larger patterns comprising features of the previous or first layer.

[0087] A CNN learns highly non-linear mappings by interconnecting layers of artificial neurons arranged in many different layers with non-linear activation functions. A CNN architecture comprises one or more convolutional layers 106, 110 interspersed with one or more sub-sampling layers 108, 112 or non-linear layers, which are typically followed by one or more fully connected layers 114, 116. Each element of the CNN receives inputs from a set of features in the previous layer. The CNN learns concurrently because the neurons in the same feature map (or output image) 120 have identical weights or parameters. These local shared weights reduce the complexity of the network such that when multi-dimensional input data enters the network, the CNN reduces the complexity of data reconstruction in the feature extraction and regression or classification process.

[0088] In mathematics, a tensor is a geometric object that maps in a multi-linear manner geometric vectors, scalars, and other tensors to a resulting tensor. Convolutions operate over 3D tensors (e.g., vectors), called feature maps (e.g., 120), with two spatial axes (height and width) as well as a depth axis (also called the channels axis). In general computer vision, a CNN is typically designed to classify color images that contain three image channelsRed, Green and Blue (RGB). For an RGB image, the dimension of the depth axis is three (3), because the image has three color channels, Red, Green, and Blue. For a black-and-white picture, the depth is one (1) (i.e., levels of gray). The convolution operation extracts patches 122 from its input feature map and applies the same transformation to all of these patches, producing an output feature map 124. This output feature map is still a 3D tensor, having a width and a height. Its depth can be arbitrary, because the output depth is a parameter of the layer, and the different channels in that depth axis no longer stand for specific colors as in RGB input; rather, they stand for filters. Filters encode specific aspects of the input data at a height level. A single filter could be encoded with, for example, morphology, texture, or size of a follicle.

[0089] Convolutions are defined by two key parameters: [0090] (1) size of the patches extracted from the inputstypically 11, 33 or 55 and [0091] (2) depth of the output feature mapthe number of filters computed by the convolution. In general, these start with a depth of 32, continue to a depth of 64, and terminate with a depth of 128 or 256.

[0092] A convolution operates by sliding these windows of size 33 or 55 over a 2D or 3D input feature map, stopping at every location, and extracting a patch 122 of surrounding features [shape (window Height, window Width, input Depth)]. Each such patch is then transformed (via a tensor product with the same learned weight matrix, called the convolution kernel) into an ID vector of shape (output depth). All of these vectors are then spatially reassembled into, for example, a 3D output map of shape (Height, Width, output Depth). Every spatial location in the output feature map corresponds to the same location in the input feature map (for example, the lower-right corner of the output contains information about the lower-right corner of the input).

[0093] During training, a CNN is adjusted or trained so that the input data leads to a specific output estimate. The CNN is adjusted using back propagation based on a comparison of the output estimate and the ground truth (i.e., true label) until the output estimate progressively matches or approaches the ground truth. The CNN is trained by adjusting the weights (w) or parameters between the neurons based on the difference between the ground truth and the actual output. The weights between neurons are free parameters that capture the model's representation of the data and are learned from input/output samples. The goal of model training is to find parameters (w) that minimize an objective loss function L(w), which measures the fit between the predictions of the model parameterized by w and the actual observations or the true label of a sample. The most common objective loss functions are the cross-entropy for classification and mean-squared error for regression. In other implementations, the convolutional neural network uses different loss functions such as Euclidean loss and softmax loss.

[0094] Currently CNNs are trained with stochastic gradient descent (SGD) using mini-batches. SGD is an iterative method for optimizing a differentiable objective function (e.g., loss function), a stochastic approximation of gradient descent optimization. Many variants of SGD are used to accelerate learning. Some popular heuristics, such as AdaGrad, AdaDelta, and RMSprop tune a learning rate adaptively for each feature. AdaGrad, arguably the most popular, adapts the learning rate by caching the sum of squared gradients with respect to each parameter at each time step. The step size for each feature is multiplied by the inverse of the square root of this cached value. AdaGrad leads to fast convergence on convex error surfaces, but because the cached sum is monotonically increasing, the method has a monotonically decreasing learning rate, which may be undesirable on highly nonconvex loss surfaces. Momentum methods are another common SGD variant used to train neural networks. These methods add to each update a decaying sum of the previous updates. In other implementations, the gradient is calculated using only selected data pairs fed to a Nesterov's accelerated gradient and an adaptive gradient to inject computation efficiency. The major shortcoming of training using gradient descent, as well as its variants, is the need for large amounts of labeled data. One way to deal with this difficulty is to resort to the use of unsupervised learning. Data augmentation is essential to teach the network the desired invariance and robustness properties, when only few training samples are available.

[0095] The convolution layers (e.g., 106,110) of a CNN serve as feature extractors. Convolution layers act as adaptive feature extractors capable of learning and decomposing the input data into hierarchical features. In one implementation, the convolution layers take two images as input and produce a third image as output. In such an implementation, convolution operates on two images in two-dimension (2D), with one image being the input image 104 and the other image, the kernel (e.g., 102), applied as a filter on the input image 104, producing an output image. The convolution operation includes sliding the kernel 102 over the input image 104. For each position of the kernel 102, the overlapping values of the kernel and the input image 104 are multiplied and the results are added. The sum of products is the value of the output image 120 at the point in the input image 104 where the kernel 102 is centered. The resulting different outputs from many kernels are called feature maps (e.g., 120,124).

[0096] Once the convolutional layers (e.g., 106, 110) are trained, they are applied to perform recognition tasks on new inference data. Since the convolutional layers learn from the training data, they avoid explicit feature extraction and learn implicitly from the training data. Convolution layers use convolution filter kernel weights, which are determined and updated as part of the training process. The convolution layers extract different features of the input image 104, which are combined at higher layers (e.g., 108,110,112). A CNN uses a various number of convolution layers, each with different convolving parameters such as kernel size, strides, padding, number of feature maps, and weights.

[0097] Sub-sampling layers (e.g., 108, 112) reduce the resolution of the features extracted by the convolution layers to make the extracted features or feature maps (e.g., 120,124) robust against noise and distortion, reduce the computational complexity, to introduce invariance properties, and to reduce the chances of overfitting. It summarizes the statistics of a feature over a region in an image. In one implementation, sub-sampling layers (e.g., 108,112) employ two types of pooling operations: average pooling and max pooling. The pooling operations divide the input into non-overlapping two-dimensional spaces. For average pooling, the average of the four values in the region is calculated for pooling. The output of the pooling neuron is the average value of the input values that reside with the input neuron set. For max pooling, the maximum value of the four values is selected for pooling. Max pooling identifies the most predictive feature within a sampled region and reduces the resolution and memory requirements of the image.

[0098] In a CNN, a non-linear layer is implemented for neuron activation in conjunction with convolution. Non-linear layers use different non-linear trigger functions to signal distinct identification of likely features on each hidden layer (e.g., 106,110). Non-linear layers use a variety of specific functions to implement the non-linear triggering, including the Rectified Linear Unit (ReLU), Parametric Rectified Linear Unit (PreLU), hyperbolic tangent, absolute of hyperbolic tangent, and sigmoid and continuous trigger (non-linear) functions. In a preferred implementation, ReLUs are used for activation. The advantage of using the ReLU function is that the convolutional neural network is trained many times faster. ReLU is a non-continuous, non-saturating activation function that is linear with respect to the input if the input values are larger than zero and zero otherwise. In other implementations, the non-linear layer uses a power unit activation function.

[0099] A CNN can also implement a residual connection which comprises reinjecting previous representations into the downstream flow of data by adding a past output tensor to a later output tensor, which helps prevent information loss along the data-processing flow. Residual connections address two common problems that plague any large-scale deep-learning model: vanishing gradients and representational bottlenecks. A residual connection makes the output of an earlier layer available as input to a later layer, effectively creating a shortcut in a sequential network. Rather than being concatenated to the later activation, the earlier output is often summed with the later activation, which assumes that both activations are the same size. If they are of different sizes, a linear transformation can be used to reshape the earlier activation into the target shape.

[0100] Residual learning of a CNN was originally proposed to solve the performance degradation problem, where the training accuracy begins to degrade along with the increasing of network depth. By assuming that the residual mapping is much easier to be learned than the original unreferenced mapping, a residual network explicitly learns a residual mapping for a few stacked layers. A residual network stacks a number of residual units to alleviate the degradation of training accuracy. Residual blocks make use of special additive skip connections to address vanishing gradients in deep neural networks. At the beginning of a residual block, the data flow is separated into two streams. The first carries the unchanged input of the block, while the second applies weights and non-linearities. At the end of the block, the two streams are merged using an element-wise sum (or subtraction). The main advantage of such constructs is to allow the gradient to flow through the network more easily. Residual networks enable CNNs to be easily trained and improve accuracy for applications such as image classification and object detection.

[0101] A known problem in deep learning is the covariate shift where the distribution of network activations changes across layers due to the change in network parameters during training. The changing scale and distribution of inputs at each layer implies that the network has to significantly adapt its parameters at each layer and thereby training has to be slow (i.e., use of small learning rate) for the loss to keep decreasing during training (i.e., to avoid divergence during training). A common covariate shift problem is the difference in the distribution of the training and test set which can lead to suboptimal generalization performance.

[0102] In one implementation, Batch Normalization (BN) is proposed to alleviate the internal covariate shift by incorporating a normalization step, a scale step, or a shift step. BN is a method for accelerating deep network training by making data standardization an integral part of a network architecture. BN guarantees more regular distributions at all inputs. BN can adaptively normalize data even as a mean variance change over time during training. It internally maintains an exponential moving average of the batch-wise mean and variance data. The main effect is to aid with gradient propagation similar to residual connections. The BN layer can be used after a convolutional, densely, or fully connected layer but before the outputs are fed into an activation function. For convolutional layers, the different elements of the same feature mapi.e. the activations at different locationsare normalized in the same way in order to obey the convolutional property. Thus, all activations in a mini-batch are normalized over all locations, rather than per activation.

[0103] The one or more convolutional layers 106,110, interspersed with one or more sub-sampling layers 108, 112 are typically followed by one or more fully connected (FC) layers 114,116. FC layers are used to concatenate the multi-dimension feature maps (e.g., 120,124, etc.) and to make the feature map into a fixed-size category and generating a feature vector for a classification output layer 118. FC layers are typically the most parameter and connection intensive layers. In one implementation, global average pooling is used to reduce the number of parameters and optionally replace one or more FC layers for classification, by taking the spatial average of features in the last layer for scoring. This reduces the training load and bypasses the overfitting issues. The main idea of global average pooling is to generate the average value from each last layer feature map as the confidence factor for scoring, feeding directly into a softmax layer, which maps for example, 3D inputs into [0,1]. This allows for interpreting one or more output layers 118 as probabilities and selection of pixels (2D inputs) or voxels (3D inputs) with the highest probability.

[0104] In one implementation one or more autoencoders are used for dimensionality reduction. Autoencoders are neural networks that are trained to reconstruct the input data, and dimensionality reduction is achieved using a fewer number of neurons in the hidden layers (e.g., 106,110, etc.) than in the input layer 104. A deep autoencoder is obtained by stacking multiple layers of encoders with each layer trained independently (pretraining) using an unsupervised learning criterion. A classification layer can be added to the pretrained encoder and further trained with labeled data (fine-tuning).

[0105] FIG. 2 is a schematic 200 of a deep learning (DL) pipeline development process, according to various embodiments. A DL pipeline development comprises three phases: model selection (model selection and fitting on a training data set), model evaluation, and model distribution. In a preferred embodiment, an infrastructure is created to train, evaluate and distribute one or more ANN networks. In various embodiments one or more data set are correctly isolated to avoid biased evaluations, through a data set flow 202 and data partitioning step 204 into testing data set 206, validation data 208, and training data 210. In various embodiments, the one or more said dataset is processed through a data IO step 212 and then sampled 214, in different various ways depending on the phase of the pipeline. In applications with limited data, data sets are augmented 216 to compensate for small training data sets, wherein the training data set is too sparse to represent the variability in the distribution of images. Data augmentation artificially increases the variability of the training data set by introducing random perturbations during training, for example applying random spatial transformations or adding random image noise. In various embodiments, training and validation data samples are introduced (steps 218, 220) into a model selection 222. In various embodiments, the model selection process comprises a model fitting process 224 that includes configuring a network 226, selection of a loss function 228, and an optimization 230 process. In various embodiments, the output of the model selection 222 process comprises one or more hyper-parameter 232 and trained parameter 234. In various embodiments, the data of said parameters flow into a model inference 236 process whereby data from sampling 214 are included systematically to process the whole data set 202. The model is evaluated 238 to generate one or more result 240. In various embodiments, one or more validation model can be stored 242 in a model library 244 where stored models can be used to initialize step 245 model during model selection 222 or for trained model comparison 246 during model inference 236. In DL, it is common practice to adapt previous network architectures, trained or untrained, in part or in full for similar or different tasks. In various embodiments, model library 244 enables storing of models and parameters that are dependent on the application domain being addressed; for example, denoising, object detection, counting, and prediction. One or more network architecture can be constructed using a library (e.g., TensorFlow), which provides the tools for defining computational pipelines and executing them efficiently on hardware resources. One or more software application drivers may be used to define a common structure for one or more components of the pipeline.

[0106] Ultrasound images are affected by a strong multiplicative noise, the speckle, which generally impairs the performance of automated operations, like classification and segmentation, aimed at extracting valuable information for the end user. An object of the present disclosure is a DL approach, implemented preferably through one or more CNN. In various implementations, given a suitable set of images, a CNN is trained to learn an implicit model of the data, for example, noise to enable the effective de-speckling of new data of the same type. Noise in US images can vary in shape, size, and pattern, being nonlinear. The premise is that image speckle noise can be expressed more accurately through non-linear models. In various embodiments, the CNN architecture is assembled for learning a non-linear end-to-end mapping between noisy and clean US images with a dilated residual network (US-DRN). In various implementations, one or more skip connections together with residual learning are added to the denoising model to reduce the vanishing gradient problem. In various preferred embodiments, the model directly acquires and updates the network parameters from the training data and the corresponding labels in lieu of relying on a priori knowledge of a pre-determined image or a noise description model. Without being bound to theory, contextual information of an image can facilitate the recovery of degraded regions. In general, deep convolution networks can mainly enhance the contextual information through enlarging the receptive field by increasing the network depth or enlarging the filter (e.g. 102 of FIG. 1). However, as the network depth increases, the accuracy becomes saturated and then degrades rapidly. Enlarging the filter size can lead to more convolution parameters that greatly increase the computing power and training time. In various implementations, dilated convolutions are employed to both enlarge the receptive field while maintaining filter size. A common convolution receptive field has a linear correlation with the layer depth. In contrast, a dilated convolution receptive field has an exponential correlation with the layer depth. As an example, for a kernel size=33, the dilation factors of the 33 dilated convolutions of an architecture with 7 layers, are set 1, 2, 3, 4, 3, 2, 1, respectively. In one implementation, a lightweight model comprises 7 dilated convolutions.

[0107] Referring to FIG. 3 a schematic 300 of the US-DRN architecture is shown, according to various embodiments. In various implementations, the architecture comprises one or more image 302 that serve as an input to one or more convolution layer 304, 306, 308, 310, 312, 314, 316, each operating similarly to layer 106 of FIG. 1. In various embodiments, the architecture is a residual network comprising one or more skip connections 318, 320, element-wise sum 322, 324, passing one or more feature information from a previous layer to a posterior layer while maintaining the image details and avoiding or reducing the vanishing gradient problem. In various embodiments, the network learns the resulting estimated components of speckle image (or subtracting image) 326. An output denoised image 328 is produced by subtracting image 326 from input image 302 with elementwise subtractor 330 via skip connection 332. In various alternative embodiments, elementwise subtractor 330 can comprise a division-wise element. In this implementation, the input image 302 is divided by estimated speckle image 326. The result is then passed through a non-linear function layer, optionally a hyperbolic tangent layer, to produce a denoised image 328. Training becomes much more effective by setting a dual goal of reproducing the noise. This is important for the detection, recognition, segmentation, or classification of a reproductive anatomy for ART given the inherent scarcity of training data. The implementation of residual mapping can lead to more effective learning and rapidly reduces the loss function after passing through a multi-layer network. Without being bound to theory, most pixel values in a residual image are close to zero and the spatial distribution of the residual feature maps should be very sparse, leading the gradient descent process to a smoother hyper-surface of loss to filter parameters. The search for an allocation that is close to optimal becomes quicker and easier, enabling the addition of more layers to the network to improve performance.

[0108] FIG. 4 is a flowchart 400 of a deep learning network for the US image de-speckling process, according to various embodiments. The three main procedures involved in this method are data processing of a training dataset, CNN training, and image de-speckling. In various embodiments, data processing (step 402) comprises the processing of a clean image 404 and a speckled image 406 that can include normalization and division of an image into smaller patches. The divided patches provide inputs into a feedforward network for CNN training. One or more noisy patches (e.g., kernel 102 of FIG. 1) from speckled image 406 provide inputs and the clean patches of clean image 404 are target outputs. In one implementation, back-propagation SDG can reduce the error and increase accuracy. In various embodiments, a patch undergoes a feedforward process and then back-propagation to update one or more network parameter step 408. A patch is randomly selected for input and used only once in the training process as SGD is adopted for optimization. The network selects one input patch with a corresponding clean patch, calculates the error between the clean patch and network output, and then calculates the errors between different hidden layers (e.g., 204,206,208 of FIG. 2). In each iteration a new learned image 410 is created that is subsequently improved via further minimization of the loss function by calculation 412. The weights are updated in network parameter step 408 by adding the current values with the calculation results of the partial derivative of error. After updating, the error and loss between a patch x and patch y decrease is to be used to de-speckle new images from a learned network 414 with parameters obtained by minimal loss convergence. The results of the trained network are a collection of weights and thresholds. The learned network 414 enables a noisy image 416 to be de-speckled into a denoised image (or clean image) 418.

[0109] In various non-limiting embodiments, the training of said US-DRN comprises the use of 100 to 500 images, optionally obtained from a US scanner or device, that are further resized, (e.g., 256256). In various embodiments, one or more 2D channel can be assigned to a corresponding axial, coronal or sagittal slices in a Volume of Interest (VOI). In various embodiments, a 3D US dataset is resampled to extract one or more VOI at differing physical scales with a fixed number of voxels. Each VOI can be translated along a random vector in 3D space for N number of repetitions. Each VOI may also be translated around a randomly oriented vector for N number of repetitions by one or more random angles for expansion of the training dataset. In various embodiments, the size of a kernel or patch can be selectively set (e.g., 4040) as well as the stride (e.g., 1 to 10). In various embodiments, network training comprises the use of an optimization method (e.g., ADAM optimization) as the gradient descent method, mini-batches (e.g., 16) with a learning rate (e.g., 0.0002), over several epochs (e.g. 20, etc.). In various embodiments, the training regularization parameter is set equal to a chosen value (e.g., 0.002). In various embodiments, the denoiser model training platform comprises the optional use of Matlab R2014b (Mathworks company in Natick, Mass., USA), the CNN toolbox was MatConvnet (MatConvnet-1.0-beta24, Mathworks, Natick, Mass.), and the GPU platform Nvidia Titan X Quadro K6000 (NVIDIA Corporation, Santa Clara, Calif.). In various embodiments, an alternative CNN toolbox comprises a proprietary framework, or one or more open framework, including but not limited to, Caffe, Torch, GoogleNet, as well as alternative deep learning models including, but not limited to, VGG, LeNet, AlexNet, ResNet, U Net, the like, or combinations thereof. In various embodiments, the performance evaluation of the filter system and method comprises the use of, but not limited to, standard deviation (STD), peak signal-to-noise ratio (PNSR), equivalent looks (ENL), and edge preservation index (EPI), the structural similarity index measurement (SSIM), and an unassisted measure of the quality of the first-order and second-order descriptors of the denoised image ratio (UM). The higher the PSNR value is, the stronger the denoising ability of the algorithm. If the ENL value is bigger, the visual effect is better. The EPI value reflects the retentive ability of the boundary, and a bigger value is better. The SSIM indicates the similarity of the image structure after denoising, and it is as big as possible. The UM does not depend on the source image to assess the denoised imagewhen the value is smaller, the ability of the speckle suppression is stronger. In various embodiments, the method comprises the use of 3D convolution to extract more information compared to using multiple input channels to perform 2D convolution.

[0110] An object of the present disclosure is an ANN system and method for object detection, recognition, annotation, segmentation, or classification of at least one US image, preferably using a denoised image processed by the said US-DRN architecture, in the provision of ART for the diagnosis, treatment and clinical management of clinical infertility. FIG. 5 is a schematic 500 of a computer-aided diagnosis (CAD) architecture and processing method, according to various embodiments. In various embodiments, the CAD architecture comprises one or more modular components including an image pre-processing module 502, an image segmentation module 506, and feature Extraction and Selection module 508, and a Classification module 510. In various embodiments, image preprocessing module 502 executes one or more steps, including but not limited to, enhancement, smoothing or reduction of speckle, resulting for example, in a denoised image 418 of FIG. 4. In various embodiments, image segmentation module 506 executes one or more steps, including but not limited to, partitioning of the image into one or more non-overlapping regions, and one or more identified regions of interest (ROIs) or VOIs are separated from the background. In various embodiments, one or more ROI or VOI is used for feature extraction. In various embodiments, feature Extraction and Selection module 508 executes one or more step, including but not limited to, feature extraction or removal, selection of a subset of features to build an optimal set of features for accurate distinguishing of one or more relevant features of a reproductive anatomy. In various embodiments, a Classification module 510 executes one or more step, including but not limited to, applying classification techniques to classify one or more reproductive anatomy. In various embodiments, a chosen CNN architecture and method enable the detection, recognition, annotation, segmentation, or classification of a(n): ovary, cyst, cystic ovary, polycystic ovary, follicle, antral follicle, or the like.

[0111] An object of the present disclosure is a framework for object detection, recognition, annotation, segmentation, or classification of at least one US image, using one or more software Application drivers in the provision of ART for the diagnosis, treatment and clinical management of clinical infertility. In various embodiments one or more application driver defines a common structure for a particular module function of CAD system described in FIG. 5. The Application functions to instantiate data, application objects, and distribution of workload as well as combining results from various computational resources (e.g., multiple CPU or GPUs). The Application driver delegates application-specific functionality to separate Application classes. In various embodiments, an Application driver can be configured from a command line or programmatically using a human-readable configuration file, preferably containing the data set definitions and settings that can deviate from defaults. The Application classes encapsulate standard analysis by connecting, including but not limited to, a Reader driver to load data, a Sampler driver to generate data samples for processing, a Network driver, preferably a CNN, to process the inputs (e.g., 418 of FIG. 4.), and an Output handler (comprising drivers, for example, Loss function 228 driver and Optimization 230 driver of FIG. 2) and an Aggregator driver during inference and evaluation. In various embodiments, a driver further comprises one or more sub-components, for example, to perform data augmentation.

[0112] CAD systems for analysis encompass a number of tasks or applications of a clinical work flow: detection, registration, reconstruction, enhancement, model representation, segmentation, classification, etc. Different applications use different types of inputs and outputs, different networks, and different evaluation metrics. In a preferred embodiment, the framework platform is designed in a modular fashion to support the addition of any new Application type, by encapsulation of workflows in Application classes. The Application class defines the required data interface for the Network and Loss function, facilitates the instantiation of data sampler and output objects, connects them as required, and specifies the training regimen. In a non-limiting example, during training, a uniform Sampler driver enables the generation of small image patches and corresponding labels, processed by said CNN to generate segmentations, using a Loss function driver to compute the loss used for back-propagation using an Adam Optimizer function driver. During inference, a Grid Sampler driver can generate a set of non-overlapping patches to convert the image to segment, the said network to generate corresponding segmentations, and a Grid Sample Aggregator driver to aggregate the patches into a final segmentation.

[0113] A DL architecture comprises a complex composition of simple functions that can be simplified in by repeated reuse of conceptual blocks. In one implementation, the framework platform comprises conceptual blocks represented by encapsulated Layer classes, or inline using, for example, the TensorFlow's scoping system. In various embodiments, one or more composite layers are constructed as simple compositional layers and TensorFlow operations. In one implementation, visualization of the network graph is automatically supported as a hierarchy at different levels of detail using the TensorBoard visualizer. In various embodiments, Layer objects define one or more scope upon instantiation, enabling repeated reuse to allow complex weight-sharing without breaking encapsulation. In various embodiments, one or more Reader classes enable the loading of an image file from one or more medical file format for a specific data set and applying image-wide pre-processing. In various implementations, the framework platform uses nibabel to facilitate a wide range of data format. In a preferred embodiment, the framework platform incorporates flexibility in mapping from input dataset into packets of data to be processed and from the processed data into useful outputs. The former is encapsulated in one or more Sampler classes, and the latter is encapsulated in Output handlers. The instantiation of matching Samplers and Output handlers is delegated to the Application class. Samplers generate a sequence of packets of corresponding data for processing. Each packet contains all the data for one independent computation (e.g., one step of gradient descent during training), including images, labels, classifications, noise samples or other data needed for processing. During training, samples are taken randomly from the training data, while during inference and evaluation the samples are taken systematically to process the whole data set. During training, the Output handlers take the network output, compute a loss and the gradient of the loss with respect to the trainable variables, and use an Optimizer driver to iteratively train the model. During inference, the Output handlers generate useful outputs by aggregating one or more network outputs and performing any necessary post-processing (e.g. resizing the outputs to the original image size). In various embodiments, Data augmentation and Normalization within the platform are implemented as Layer classes applied in the Sampler. In a preferred embodiment, the framework platform enables supports for mean, variance and histogram intensity data normalization, and flip, rotation and scaling for spatial Data augmentation.

[0114] FIG. 6 is a schematic flow chart 600 of a process for the detection of a recruitable follicle. The detection process comprises a cascade CNN base method. Firstly, one or more ROI or VOI of at least one US image, denoised and preprocessed using, for example, image processing module 502 of FIG. 5, is coarsely delineated or outlined (step 602) by a skilled ultrasonographer or physician to serve as a training data set. Secondly, a CNN (e.g., n-convolution and n-pooling layers) is trained 604 to segment recruitable follicles 606 and generate corresponding segmentation probability maps 608. Thirdly, all the segmentation probability maps are split into different connected regions, using one or more operators, including but not limited to, binarization operator, erosion operator, or a dilation operator. Finally, a CNN 610 is employed to detect a recruitable follicle 612, generating an output 614, based on US images patches re-labeled by one or more split segmentation probability maps 608.

[0115] Referring to FIG. 7, a schematic of the CNN architectures for detection of recruitable follicles is shown, according to an embodiment. In this non-limiting implementation, an image (e.g., 418 of FIG. 4) is introduced as input into a first convolution layer (Cony) 704 of CNN 604 architecture of FIG. 6 that generates one or more feature maps 706 using one or more filter (e.g., 1313), a stride size of 2 pixel and a padding size of 66 pixels, subsequently subject max pooling. The following convolution layers 708 both generate 265 feature maps 710 of size 4545 through filters of size 55. A padding size of 2 pixels and a stride size of 2 pixels are used in the second convolution layer. Max pooling reduces the sizes of the feature maps further. A padding size of 2 pixels and stride size of 1 pixel are used in subsequent layers 712. The residual convolution layers comprise of padding size of 1 pixel and a stride size of 1 pixel. Moreover, each of the residual convolutional layers 712, 714, except the last two convolutional layers, comprises filters of size 33, generates 384 feature maps of size 2222. The second to last one convolutional layer 714, with filters of size 33, generates 256 feature maps of size 2222. The last convolutional layer 716, with filters of size 33, generates one feature map of size 4444. Two max-pooling layers 706, 710 with window size of 33 follow after the first convolutional layer 704 and the third convolutional layers 708, respectively. In one embodiment, a stride size of two pixels are used in these two pooling layers. A padding size of one pixel is only, optionally, used in the first pooling layer 706. In addition, the function parametric rectified linear unit (PReLu) is used as the activation function, whose parameters can be adaptively learned using one or more said learning methods of the present disclosure. Further, for follicle detection, the CNN 610 architecture comprises four convolutional layers 720, 724, 728, 732, four pooling layers 722, 726, 730, 734, and two fully connected layers 736, 738 with 64, 1 output, respectively. The first convolutional layer 720 generated from maps 718 (e.g., 608 of FIG. 6) feature maps of size 6464 through filters of size 55, with a stride size of one pixel, and a padding size of two pixels. The second convolutional layer 724 generates 64 feature maps of size 3232 through filters of size 55, with a padding size of two pixels and a stride size of one pixel. The third convolutional layer 728 generates 64 feature maps of size 1616 through filters of size 33, with a padding size of one pixel and a stride size of one pixel. The last convolutional layer 732 generates 384 feature maps of size 88 through filters of size 33, with a padding size of one pixel and a stride size of one pixel. In a preferred embodiment, the feature maps in current layers are connected to all of the feature maps in the previous layers. The first two convolutional layers 720,724 are both followed by the max-pooling layers 722, 726 with a padding of 1, a stride of 2, and window size of 3. The third convolutional layer 728 is followed by the max-pooling layer 730 with a stride of 2, and window size of 2. The fourth convolutional layer 732 is followed by the max-pooling layer 734 with a stride of 8, and window size of 8. The activation function is a rectified linear unit (ReLU), which is the point-wise nonlinearity applied to all hidden units. In various embodiments, a local response normalization scheme is also applied after each of the ReLU operations. After the output of the second fully-connected layer 738, a softmax layer 740 is used to generate a distribution over the class labels (e.g., non recruitable or recruitable follicle) by minimizing the cross-entropy loss between the predicted labels (i.e. recruitable follicle) and ground truth labels (i.e., a priori segmented recruitable follicle).

[0116] In one implementation, the training process for detecting recruitable follicle comprises three steps. Firstly, CNN 604 of FIG. 6 is trained with random initialized parameters using data containing patches extracted from the images of recruitable follicles. In one implementation, the image patches of size (e.g., 353353 702 of FIG. 7) cropped sampled randomly from these recruitable follicle images are inputs of the CNN 604 of FIG. 6. They are labeled by probability maps with pixel values in the interval [e.g., 0:1; 0:9], which are determined according to the relation between the image patches and their corresponding binary masks. The segmentation probability maps 608 of FIG. 6 about recruitable follicles are the outputs of the CNN 604 of FIG. 6. In various embodiments, improved performance comprises the use of a multi-view strategy to train the CNN 604 of FIG. 6. Secondly, one or more splitting method, consisting of continuous binarization operator, erosion operator and dilation operator, is to split connected regions of the segmentation probability maps generated by the CNN 604 of FIG. 6 into several isolated connected regions. For example, the continuous binarization operator, with the values from a step 0.01 interval [0:15; 0:7], can be used to binarize the segmentation probability maps 608 of FIG. 6. In various embodiments, one or more workflow applications of said framework platform described in FIG. 5 can be implemented to perform the one or more module function of the process for identification, detection segmentation, or classification of one or more recruitable follicle. In various embodiments, the model training platform comprises the optional use of Matlab R2014b (Mathworks company in Natick, Mass., USA), the CNN toolbox was MatConvnet (MatConvnet-1.0-beta24, Mathworks, Natick, Mass.), and the GPU platform Nvidia Titan X Quadro K6000 (NVIDIA Corporation, Santa Clara, Calif.). In various embodiments, an alternative CNN toolbox comprises a proprietary framework, or one or more open framework, including but not limited to, Caffe, Torch, GoogleNet, as well as alternative deep learning models including, but not limited to, VGG, LeNet, AlexNet, ResNet, U Net, the like, or combinations thereof. In various embodiments, the detection process enables detection, identification, segmentation, or classification of alternative reproductive anatomy(ies) and not limited to recruitable follicles, including but not limited to, a (n): ovary, cyst, cystic ovary, polycystic ovary, follicle, antral follicle, or the like. In various embodiments, one or more results are electronically recorded in at least one electronic health record database. In alternative embodiments, the said results are transmitted and stored within a database residing in a cloud-based server.

[0117] An object of the present disclosure is the said ANN system and method for an object detection framework in the provision of ART for the diagnosis, treatment and clinical management of clinical infertility. In various embodiments, the said system and method enables object detection (e.g., follicle detection), localization, counting, and tracking (e.g., follicle growth rate) over time of one or more reproductive anatomy from one or more US images. In various embodiments, one or more detectors or classifiers are trained in the pixel space, where the locations of one or more target reproductive anatomy(ies) are labeled (e.g., follicle). For follicle detection, the output space comprises one or more sparsely labeled pixel indicating follicle centers. In various embodiments, the output space is encoded to a compressed vector of fixed dimension, preferably shorter than the original sparse pixel space (i.e., compressed sensing). In various embodiments, a CNN regresses the said vector from the input pixels (e.g., US image). In various embodiments, follicle locations on the output pixel is recovered using normalization, including but not limited to, L.sub.1 normalization.

[0118] Without being bound to theory, the Nyquist-Shannon sampling theorem states that a certain minimal sampling rate is required for the reconstruction of a band-limited signal. Compressed sensing (CS) has the potential to reduce the sampling and computation requirements for sparse signals under a linear transformation. The premise of CS is that an unknown signal of interest is observed (sensed) through a limited number of linear observations. It has been proven that it is possible to obtain a stable reconstruction of the unknown signal from these observations, under the assumptions that the signal is sparse and matrix dis-coherence. The signal recovery technique generally relies on convex optimization methods with a penalty expressed by L1 normalization, for example orthogonal match pursuit or augmented Lagrangian method.

[0119] An object of the present disclosure is an ANN system and method for an object detection and characterization framework in the provision of ART and OI for the treatment and clinical management of clinical infertility. In various embodiments, the ANN comprises an architecture configured for instance segmentation to enable object detection (e.g., follicle detection), localization, morphology characterization, sizing, counting, and tracking (e.g., follicle growth rate) over time of one or more reproductive anatomy from one or more US images. In one embodiment, the architecture comprises an ensemble of machine learning methods for instance segmentation, edge detection and enhancement, quantification, size, and counting of one or more reproductive anatomy, including but not limited to a follicle. In various embodiments, the architecture comprises one or more ANN architecture, including but not limited to a Fast/Faster CNN, a Fully Convolutional Network (FCN), a Mask Regional Convolutional Neural Network (Mask R-CNN), or combinations thereof. In various embodiments, edge detection comprises the use of one or more methods including but not limited to Gradient, Laplacian, or the like. In one embodiment, the edge method includes but is not limited to, one or more filters, such as a Sobel filter. In various embodiments, the method comprises a multi-task, end-to-end, deep learning framework combined with image processing methods for morphology characterization, including but not limited to anatomical size, length, width, diameter, volume (e.g., follicle volume).

[0120] Referring to FIG. 7B, a schematic diagram of the multitask end-to-end deep learning framework for follicle segmentation, characterization, and tracking 700b is shown, according to an embodiment. In this non-limiting implementation, an image 702b (e.g., 418 of FIG. 4) is introduced as input into Mark R-CNN 704b architecture to perform instance segmentation. Instance segmentation requires the correct detection of all objects in an image while precisely segmenting each instance of a detected object. Instance segmentation combines the classification of individual objects and localizes each using a bound box with semantic segmentation which classifies each pixel into a fixed set of categories without differentiation. The output of the 704b network is processed by one or more edge filter 706b since the Mask R-CNN network provides a coarse segmentation of a region containing follicles. In various embodiments, edge detection is used to identify the areas with significant brightness gradient in TVUS image 702b. In one embodiment, the Sobel image gradient filters are selected to minimize computation overhead. The Sobel filters are 2D filters for edge detection of one or more follicle, describing a first order gradient operation, rotation-dependent, in a vertical and horizontal direction. An edge of a follicle in image 702b corresponds to high absolute response to the filter's direction. In one embodiment, once follicles are detected, counting can be performed using a follicle counter 708b comprising a Visual Geometry Group (VGG) CNN architecture. In various embodiments, the CNN for follicle counting is a VGG-11 network consisting of 11 layers of 2D convolutions with a filter of 33 pixels. The number of filters will vary from 64 to 512. Each convolution layer will be followed sequentially by a batch normalization layer, a leaky ReLU, and a max pooling layer. The VGG-11 is used for feature extraction. The counting prediction will use three fully connected layers (e.g., dimension: 1024,512,1), separated by a batch normalization layer, and a leaky ReLU, to prevent outputting a negative number. In another embodiment, the morphologies of detected follicles are characterized using morphology characterizer 710b for size, area, volume, the like, or combinations thereof. In one embodiment, the area outside of the follicle mask in the bounding box is removed and the grayscale image in the mask is binarized using Otsu's method. The follicle mask is distributed a little wider than the follicle so some noise may still remain in the mask after the binarization. In an embodiment, this noise can be removed by calculating the area and eccentricity through morphological analysis and deleting regions below a preset size and eccentricity. One or more measurement dimension (e.g., length, width, etc.) of a follicle can be calculated using skeletonization and edges of the follicle object by the following steps: (a) find two neighboring pixels in the skeleton; (b) calculate the orientation of the pixel using the two neighboring pixels; (c) draw a normal line of the pixel according to the calculated orientation; (d) find the intersecting pixels with both edges; and (e) calculate the pixel distance between the two intersection pixels. Skeletonization is used to convert pixel-wise follicles into single-pixel wide representations. In various embodiments, follicle tracking comprises the use of the outputs of follicle counter 708b and Characterizer 710b as input into follicle tracker 712b. In one embodiment, follicle tracker 712b comprises the use of a linear assignment problem-based approach to construct growth curves and track follicles from inter-day 2D TVUS images over a period of time to determine individual areas (dia) versus time growth curves. For example, given a set of detected follicles throughout a time-lapse image sequence, the algorithm first links the detected follicles between consecutive frames, and then links the track segments generated in the first step to simultaneously close gaps and capture follicle motion events. In certain embodiments, a histogram (# of follicles vs. diameter/day) is constructed by computing the number of follicles using the output of counter 708b and the change in follicle area/diameter using the output of characterizer 710b from each day image acquisition for each follicle. In one embodiment, a spatial map of follicle growth rates with single follicle resolution is constructed using the growth rate and instances of detected follicles.

[0121] Referring to FIG. 7C, a schematic diagram of the Mark R-CNN 704b architecture for follicle instance segmentation 700c is shown, according to an embodiment. The architecture comprises one or more feature pyramid network (FPN) 702c based on the ResNet architecture altered according to one or more TVUS image 702b of FIG. 7B, as the backbone for the Mask R-CNN. Mask R-CNN is a complement to the Faster R-CNN architecture which enables mask prediction. Mask R-CNN replaces the Region-of-Interest (RoI) pooling layer in Faster R-CNN network with an ROI Align 704c to introduce an interpolation process to solve the alignment problem caused by direct sampling through pooling. In various embodiments, one or more Fully Connected Network Layers (FCN) 702c, 706c,708c,710c, 712c is configured in sequence and or parallel and used to predict pixel-level instance masks of one or more follicles. In various embodiments, the Mask R-CNN 704b network of FIG. 7B is constructed with three stages for coarse follicle detection and localization: feature extraction, region proposal, and prediction. The Mask R-CNN uses the said FPNs to generate candidate region RoI, feature extraction by ResNet 714c, for example, non-limiting ResNet-101, and then to obtain the pyramid feature maps 716c of follicles through pixel-level information. This addition allows the network to make use of high-resolution feature maps in the lower layers for accurate localization. The extraction process is the same as that of Faster R-CNN using Region Proposal Network (RPN) 718c for bounding box proposal generation performing object/nonobject binary classification and bounding box regression. The feature maps 716c of each RoI region in image 702b of FIG. 7B and each RoI is corrected using ROI Align 704c. The classification and bounding box of each are then predicted using Fully Connected FC layer 720c after obtaining the feature map of each RoI region. Each RoI uses the designed Fully Connected Network FCN 722c framework to predict the category of each pixel in the RoI region. As an end result, the Mark R-CNN 704b produces one or more segmentation mask for one or more follicles.

[0122] Referring now to FIG. 7D, a process flow diagram for follicle tracking 700d is shown. According to an embodiment, follicle tracking performed by Follicle Tracker 712b of FIG. 7B comprises one or more steps that links the detected follicles between two or more consecutive frames of TVUS image 702b of FIG. 7B, and then links the track segments generated in the first step to simultaneously close gaps and capture follicle motions. In a first process flow step, one or more follicle track segments may be constructed by linking the detected follicles between consecutive frames, with the constraint that a follicle in one frame could link to at most one follicle in the previous or following frame. The tracks are constructed from one or more image sequence (702d) by detecting follicles (704d) in each frame to determine follicle positions per frame (706d). In a subsequent process flow step, the follicles are linked in between consecutive frames (708d) to create tracking segments (710d). In a subsequent process flow step, the gaps between images may be closed to capture merging and splitting events (712d) between the initial track segments to completion (714d). In various embodiments, one or more linear assignment problems (LAPs) are solved to formulate both the frame-to-frame follicle linking step and the gap closing, merging and splitting step. In the LAP framework, every potential assignment (follicle assignment in the first step, track segment assignment in the second step) may be characterized by a cost function and solved by a global or local cost minimization matrix. In certain embodiments, one or more follicles or tracks are assigned to one or more potential assignments in the frame-to-frame follicle linking step (708d). A follicle in the source frame t could link to a follicle in the target frame t+1 (cost function A). In an alternative embodiment, a follicle in the source frame could link to nothing, leading to a track segment end (cost function B), or a particle in the target frame could get linked by nothing, leading to a track segment start (cost function C). In various embodiments, in the gap closing, merging and splitting step (712d), six types of potential assignments for follicles can be in cost competition. The end of a track segment can be linked to the start of another track segment, thereby closing a gap (cost function D), the end of a track segment could link to a middle point of another track segment, leading to a merge (cost function E), or the start of a track segment can be linked by a middle point of another track segment, leading to a split (cost function F). In an alternative embodiment, the end of a track segment can be linked to null, leading to a track termination (cost function G), the start of a track segment can be linked by null, leading to a track initiation (cost function II) or the track segment middle points introduced for merging and splitting can be linked to null, refusing a merge or a split (cost functions D and B). In this step, all track segments throughout a whole sequence compete with each other. In various embodiments, the cost functions are tailored to the specific tracking application, for example, under one or more assumptions of follicle motion (e.g., isotropic random motion, Brownian motion, etc.).

[0123] Referring now to FIG. 8, a flow chart of the follicle detection and localization framework 800 is shown, according to various embodiments. The framework comprises a follicle localization encoding phase using one or more random projection, a CNN based regression model to capture one or more relationship between a US image and the encoded signal y, and a decoding phase for recovery and detection. In various embodiments, during training, the ground truth locations of follicles are indicated by a pixel-wise binary annotation map 804. In various embodiments, one or more encoding scheme 806 converts a follicle location from the pixel space representation of image 802 to a compressed signal y 808. Training pairs 810, each consisting of a US image 812 (e.g., 418 of FIG. 4) containing one or more follicles and compressed signal y 808, then train a CNN 814 to function as a multi-label regression model. In one implementation, a Euclidean loss is employed during training, given suitability for a performing regression. In various embodiments, data augmentation comprises one or more image rotation of the training sets for robustness to rotations. In various embodiments, during testing, the trained CNN 814 generates an output for an estimated signal y 816 for each test image 818 provided as input to the first convolution layer. Subsequently, a decoding scheme 820 is used to estimate the ground truth follicle location prediction 822 by performing L.sub.1 minimization recovery on the estimated signal y 816, with one or more sensing matrix 824, determined by one or more encoding or decoding schemes.

[0124] A number of encoding schemes may be employed by the said framework. In various embodiments, the framework employs one or more random projection-based encoding schemes. In various embodiments, the center of every follicle is attached with a dot mark, a cross mark, or a bounding box. In one embodiment, pixel-wise binary annotation map 804 comprises a size of w-by-h indicating the location of one or more follicles by labeling 1 at the pixel of the follicle centroids, otherwise label 0 at background pixels. In one embodiment, annotation map 804 is vectorized by the concatenation of every row of map 804 into a binary vectorflength wh. Therefore, a positive element in map 804 with {x,y} coordinates will be encoded to the [x+h(y-1)]-th position in the vector f A random projection is applied after the generation of vector f Vector f can be represented by one or more linear observation y, which is proportional to sensing a matrix 824 and vector f. Without being bound to theory, sensing matrix 824 preferably satisfies one or more conditions, including but not limited to, isometric property. In one implementation, matrix 824 is a random Gaussian matrix. In alternative embodiments, another encoding scheme 806 is employed, particular for processing of large images, to reduce computational burden. In various embodiments, the coordinates of every follicle centroid are projected onto multiple observation axes. A set of observation axes are created with an N total number of observations. In one implementation, the observation axes are uniformly distributed around image 802. For the n-observation axis oak, the location of follicles is encoded into a R-length sparse signal. Perpendicular signed distances (A) are calculated from follicles to the n-observation axis oak. Thus, f.sub.n contains signed distances as a measure of distance and location of which side of oak follicles. The encoding of follicle locations under oak is yk, obtained by a random projection. Similarly, y.sub.n is a proportional matrix 824 times f.sub.n, the signed distances. In various embodiments, the process is repeated for all the N observation axes to obtain each y.sub.n. The joint representation of follicle locations is derived from the encoding result y after concatenation of the total y.sub.n. Similarly, a decoding scheme may be employed by the said framework to recover the vector f. In various embodiments, accuracy recovery from the encoded signal y is obtained by solving an L.sub.1 normalization convex optimization problem. The recovery of f enables the localization of every true follicle, localized N time, with N predicted positions 822.

[0125] The follicle detection and localization framework comprise one or more CNN 814 for building at least one regression model between a US image 812 and its follicle location representation or compressed signal y 808. In one implementation, CNN 814 comprises a network consisting of, but not limited to, 5 convolution layers and 3 fully connected layers. In an alternative implementation, CNN 814 comprises a deep neural network, for example with a 100-layer model. In other implementations, CNN 814 comprises one or more CNN disclosed within the present disclosure. In various embodiments, one or more loss function may be employed, including but not limited to, Euclidean loss, or other said loss functions of the present disclosure. In various embodiments, the dimension of the output layer of said CNN may be modified to the length of compressed signal y 808. In various embodiments, one or more CNN 814 model may be further optimized using additional learning methods, including but not limited to Multi-Task Learning (MTL), for localization and follicle counting. In various embodiments, during training, one or more labels are provided to a CNN. In one implementation, an encoded vector y carrying pixel-level location information of follicles. In another implementation, a scalar or follicle count (c), representing the total number of follicles in training image patch, filter, or kernel. In various embodiments, two or more said labels may be concatenated into a final training label. One or more loss function is then applied on the combined label. Therefore, the supervision information for both follicle detection and counting can be jointly used for optimizing the CNN model parameters. A large number of square patches may be employed for training. Along with each training patch, a signal (i.e. the encoding result: y) may be employed to indicate the location of target follicles present in each patch. Data augmentation may be employed by performing patch rotation on the collection of training patches making the system rotation invariant. In various embodiments, one or more MTL framework may be employed to address the cases of touching and clustered follicles. In one implementation, one or more follicle appearance, including but not limited to, texture, morphology, borders, contour information are integrated into an MTL framework to form a deep-contour aware network, preferably the complementary appearance and contour information can further improve the discriminative capability of intermediate features, and hence more accurately separate the touching or clustered follicle into individual ones. In various embodiments, the CNNs are trained in an end-to-end manner to boost performance. In various embodiments, the model training platform comprises the optional use of Matlab R2014b (Mathworks company in Natick, Mass., USA), the CNN toolbox was MatConvnet (MatConvnet-1.0-beta24, Mathworks, Natick, Mass.), and the GPU platform Nvidia Titan X Quadro K6000 (NVIDIA Corporation, Santa Clara, Calif.). In various embodiments, alternative CNN toolbox comprises a proprietary framework, or one or more open framework, including but not limited to, Caffe, Torch, GoogleNet, as well as alternative deep learning models including, but not limited to, VGG, LeNet, AlexNet, ResNet, U Net, the like, or combinations thereof. In various embodiments, the said process enables detection, localization, and counting of alternative reproductive anatomy(ies) and not limited to follicles, including but not limited to, a (n): ovary, cyst, cystic ovary, polycystic ovary, or the like. In various embodiments, one or more results are electronically recorded in at least one electronic health record database. In alternative embodiments, the said results are transmitted and stored within a database residing in a cloud-based server.

[0126] Referring to FIG. 9, a flowchart 900 of the follicle tracking framework is shown, according to various embodiments. The follicle tracking framework comprises one or more CNN 902, 904 specifically tasked for distinguishing follicle size, volume, or quality and then using the network to generate tracking feature maps in real-time. In various embodiments, the tracking framework learns a correlation filter specifically for each tracked follicle, training on features extracted by the CNN 902, 904. In various implementations, deep semantics are combined with spatial resolution of early filters for combined accurate tracking. In a preferred embodiment, at least one said network is trained to perform multiple tasks, taking advantage of pre-computing feature maps from one or more CNN network segmentation processes, for example, 606,608 of FIG. 6. The follicle tracking system comprises a hierarchical tracker to perform correlation-filter tracking based on extracted features. In various embodiments, the tracker computes one or more circularized kernels in Fourier space to increase performance. At each tracked follicle, a search window is located over the follicle in the first frame of the input. This frame is the set of neural network features obtained from one or more segmentation map output (e.g., 606, 608 of FIG. 6). Once a search window is positioned, one or more correlation filters are learned by minimizing a loss function, optionally performed in the Fourier space domain. At each tracking time step, the correlation filters are matched features within a search window Z overlaid on the last known location of the target follicle, obtained from one or more feature map 804 or follicle position prediction 822 of FIG. 8. In various embodiments, at least one filter is trained on one or more layers of a selected CNN. In various embodiments, at each time step, one or more correlation filters are matched features within a search window Z overlaid on the last known location of a target follicle. One or more matches are computed, preferably in the Fourier space domain. In various embodiments, one or more deep filters are propagated to higher layer levels in a weighted manner. One or more estimated new location is discovered by taking an argmax f.sub.o of {m,n} and a new search is realigned to the new location. Referring again to FIG. 9, in various embodiments, FollicleTrack 906 takes as input one or more segmented images (e.g., 814, 822 of FIG. 8) indicating one or more location of a tracking target follicle, preferably the first frame of a time-series sequence, one or more raw images, one or more processed images, or feature maps output (e.g., 606,608 of FIG. 6), one or more convolution layers of CNN 902 and CNN 904. One or more follicles can be selected for tracking, or segmented follicles can be tracked across the time series. In various embodiments, the time-series (e.g., sec, min, hr, day, etc.) sequence comprises one or more sequence of, including but not limited to, real-time transvaginal US images, stored sequences of transvaginal US images, retrieved images from a US scanner/device, transmitted from a US scanner/device. In various embodiments, CNN 902 is a deep convolutional neural network designed to further segment follicles in terms of physical dimensions and changes in dimensions. One or more training sets are used to train the network enabling the segmentation of follicle by size, shape, volume, distribution, average, standard deviation, morphology, position, displacement, growth rate from US images (e.g., 2D, 3D, etc.). In various implementations, one or more output of FollicleTrack 906 is a list containing, including but not limited to, one or more learned filter for each follicle, history of follicle centroid position (s) at each timestep, history of follicle location, follicle distribution (size, volume, etc.), movement, displacement, growth rate (size, volume, etc.). In various embodiments, the output data enables the annotation of one or more segmented or unlabeled images, or to directly map follicle trajectories (e.g., movement, growth rate). In various embodiments, US images are obtained from a historical database containing curated information on recruitable follicles, non-recruitable follicles, differentiated by size or volume, the like or combinations thereof. In various embodiments, the US images are annotated by a skilled ultrasonographer or a physician with expertise in the recognition of reproductive anatomy, preferably differentiating features of an optimal follicle, size, follicle distribution, volume, growth rate, for extraction and implantation. In various embodiments, CNN 902 comprises, but is not limited to, five convolution layers 908, 910, 912, 914, 916, followed by a fully connected layer 918, feeding into a final FC 920. One or more output 922 includes but is not limited to follicle size/volume range, distribution, average, standard deviation, or growth rate (e.g., 1 mm per day) of one or more follicle. In various embodiments, CNN 902 produces one or more set of masks and feature maps, per at least one frame of US image input, obtained from at least one patient, from at least one visit for follicle tracking. In various embodiments, CNN 904 is a deep convolutional neural network designed to further segment follicles in terms of quality or changes in quality. In one implementation, CNN 904 comprises six 33 convolution layers 924, 926, 928, 930, 932, 934, with ReLu activation, maxpool layer 936 and a softmax layer 938 layer, and output 940. In various implementations, drop out is implemented at one or more levels of the network. In one implementation, the network is programmed to produce output feature maps from each convolution layer as well as label classification scores. In various embodiments, the said convolution layers produce one or more feature maps for inter-follicle discrimination. In various embodiments, the said hierarchical method is combined with one or more other networks of the present disclosure to improve tracking accuracy. In one implementation, a weighted correlation filter at each search window provides a cost to a linear assignment, augmenting its ability to track follicles. In various embodiments, the model training platform comprises the optional use of Matlab R2014b (Mathworks company in Natick, Mass., USA), the CNN toolbox MatConvnet (MatConvnet-1.0-beta24, Mathworks, Natick, Mass.), and the GPU platform Nvidia Titan X Quadro K6000 (NVIDIA Corporation, Santa Clara, Calif.). In various embodiments, an alternative CNN toolbox comprises a proprietary framework, or one or more open framework, including but not limited to, Caffe, Torch, GoogleNet, as well as alternative deep learning models including, but not limited to, VGG, LeNet, AlexNet, ResNet, U Net, the like, or combinations thereof. In various embodiments, the said process enables detection, localization, counting, and tracking of alternative reproductive anatomy (ies) and not limited to follicles, including but not limited to, a (n): oocyte, blastocyst, ovary, cyst, cystic ovary, polycystic ovary, endometrial thickness, or the like. In various embodiments, one or more results are electronically recorded in at least one electronic health record database. In alternative embodiments, the said results are transmitted and stored within a database residing in a cloud-based server.

[0127] An object of the present disclosure is the said ANN system and method for analyzing an electronic medical record in the provision of ART for the diagnosis, treatment and clinical management of clinical infertility. In various embodiments, said system and method enable feature extraction or phenotyping of one or more patients from at least one longitudinal patient electronic medical record (EMR), electronic health record (EHR), database, or the like. Electronic phenotyping refers to the problem of extracting effective phenotypes from longitudinal patient health records. The challenges of effective extraction of features from a patient's EMR or EHR are high-dimensionality due to the large quantity of distinct medical events, temporality in which EHRs evolve over time, and sparsity of data, irregularity, and systematic errors or bias. A temporal matrix representation of data is employed to address representation of patient medical records as temporal matrices with one dimension corresponding to time and the other dimension corresponding to medical events. In various embodiments, temporal EHR information or medical record is converted into one or more binary sparse matrix, comprising horizontal dimension (time) and a vertical dimension (medical event). In one implementation, the (i,j)-entry in the matrix of a specific patient is equal to 1 if the i-th event is recorded or observed at time stamp j in the patient medical record. Referring to FIG. 10, a diagram 1000 of an EMR data convolution network architecture is shown, according to various embodiments. The architecture comprises a first layer (or matrix) 1002 containing one or more patient EMR matrix and CNN 1004. In various embodiments, CNN 1004 further comprises one or more convolution layer 1004a, preferably a one-side convolution layer, a pooling layer 1004b, preferably for introducing sparsity on the learning features, and an FC layer 1004c. In various embodiments, convolution layer 1004a comprises a convolution operator on the time dimension of the patient EMR matrix 1002. In one implementation, each event matrix 1002 of length l is represented by vector X, with x.sub.i as a d-dimension event vector corresponding to the i-th event items. In various embodiments, x.sub.i: i+j represents the concatenation of items x.sub.i, x.sub.i+1, x.sub.i+j. A one-sided convolution filter operation comprises a filter applied to a window of n event features to produce a new feature. For example, a feature c.sub.i is generated from a window of events (e.g., x.sub.i: i+n1) using one or more non-linear activation function, preferably ReLU. The filter is applied to each possible window of features in one or more event matrix 1002 to produce a feature map 1004d. Mean pooling layer 1004b is applied over one or more feature map 1004d to obtain an average value of c. In a preferred embodiment, one or more important feature with the highest value for each feature map is captured for feature extraction. The FC layer 1004c is a fully connected layer linking with one or more softmax classifiers for classification or prediction using one or more single-frame.

[0128] EMR data vary widely in time and temporal connectivity is required for prediction. In various embodiments, temporal smoothness is incorporated into the learning process using one or more temporal fusion. In various embodiments, one or more data sample is processed as a collection of short, fixed-sized sub-frames, of a single frame, containing several contiguous intervals in time. In one implementation, a model fuses information across the temporal domain, performed early in the network 1004, by modifying convolution layer 1004a to extend in time. In one implementation, proximal fusion combines information across an entire time window immediately on the basic event feature level. One or more filters of the convolution layer 1004a are modified to extend operation on one or more sub-frames. In another implementation, a distal fusion model performs fusion on the fully connected layer 1004c. In one embodiment, one or more separate single-frame network or sub-frames are merged in the fully connected layer, whereby detecting patterns existing in one or more sub-frames. In another implementation, a balance between proximal and distal temporal fusion enables the slow fusing of information throughout the network. In various embodiments, the higher layers of the network receive progressively more global information in time. In one implementation, connectivity is extended to all convolution layers in time and the fully connected layer 1004c can compute global pattern characteristics by comparison of all output layers. The framework enables the production of insightful patient phenotypes by taking advantage of the higher order temporal event relationships. In various embodiments, one or more recording of neuron activity enables the observation of patterns indicative of a health or medical condition. In one implementation, one or more neurons outputs receive the highest weights, preferably normalized, in one or more top layer for positive or negative classification of a condition. One or more regions appearing in a training set highly activating one or more corresponding neurons can be identified using one or more sliding window cut (min,max window size) to obtain one or more top ranked regions or patterns. In another implementation, one or more weights of neurons are aggregated and assigned to a medical or health condition and are important features for patient phenotype extraction and predictive purposes. In various embodiments, the model training platform comprises the optional use of Matlab R2014b (Mathworks company in Natick, Mass., USA), the CNN toolbox MatConvnet (MatConvnet-1.0-beta24, Mathworks, Natick, Mass.), and the GPU platform Nvidia Titan X Quadro K6000 (NVIDIA Corporation, Santa Clara, Calif.). In various embodiments, an alternative CNN toolbox comprises a proprietary framework, or one or more open framework, including but not limited to, Caffe, Torch, GoogleNet, as well as alternative deep learning models including, but are not limited to, VGG, LeNet, AlexNet, ResNet, U Net, the like, or combinations thereof.

[0129] In various embodiments, the said medical record comprises one or more stored patient record, preferably records of patients undergoing infertility treatment, ultrasound image, images of said reproductive anatomy, physician notes, clinical notes, physician annotation, diagnostic results, body-fluid biomarkers, hormone markers, hormone level, neohormones, endocabinoids, genomic biomarkers, proteomic biomarkers, Anti-Mullerian hormone, progesterone, FSH, inhibins, renin, relaxin, VEGF, creatine kinase, hCG, fetoprotein, pregnancy-specific b-1-glycoprotein, pregnancy-associated plasma protein-A, placental protein-14, follistatin, IL-8, IL-6, vitellogenin, calbindin-D9k, therapeutic treatment, treatment schedule, implantation schedule, implantation rate, follicle size, follicle number, AFC, follicle growth rate, pregnancy rate, date and time of implantation (i.e., event), CPT code, HCPCS code, ICD code, or the like. In various embodiments, the one or more patient phenotypes include, but are not limited to, infertility, anovulation, oligo ovulation, endometriosis, male factor infertility, tubal factor infertility, decreased ovarian reserve, patient risk of ovulation, patient having the optimal characteristics for implantation, patient ready for implantation, patient having one or biomarkers indicative of ovulation, patient having US images indicative of being optimal for extraction, etc. In various embodiments, one or more identified patient phenotype or predictive result from an output layer of said one or more CNN are recorded in at least one electronic health record database. In alternative embodiments, the said results are transmitted and stored within a database residing in a cloud-based server.

[0130] An object of the present disclosure is the said ANN system and method for predictive planning in the provision of ART for the diagnosis, treatment and clinical management of clinical infertility. A unified predictive framework comprises one or more survival convolution neural networks (SCNNs) to provide one or more prediction of time-to-event outcomes from at least one US image and one or more patient phenotype obtained from a patient medical record. In various embodiments, the framework comprises one or more image sampling and risk filtering technique for predictive purposes. In one implementation, one or more ROIs or VOIs of at least one US image is used to train a deep CNN seamlessly integrated with a Cox proportional hazards model to predict patient outcomes, including but not limited to, induction-termination of hormone therapy, having recruitable follicle, having dominant follicle, having matured follicle, ready for follicle extraction, optimal endometrial thickness for implantation. Referring to FIG. 11, a schematic diagram 1100 of a survival convolution neural network architecture is shown, according to various embodiments. The architecture comprises an n-layer CNN architecture 1102 with a Cox proportional hazards model 1118 to predict time-to-event data from an image 1106. In one non-limiting implementation, image feature extraction is achieved by four groups of convolutional layers. The first group 1108 comprises two convolutional layers with 64 33 kernels interleaved with local normalization layers and then followed with a single maximum pooling layer. The second group 1110 comprises two convolutional layers (128 33 kernels) interleaved with two local normalization layers followed by a single maximum pooling layer. The third group 1112 interleaves four convolutional layers (256 33 kernels) with four local normalization layers followed by a single maximum pooling layer. The fourth group 1114 contains interleaves of eight convolutional (512 33 kernels) and eight local normalization layers, with an intermediate pooling layer and a terminal maximum pooling layer. These four groups are followed by a sequence of three fully connected layers 1116 containing 1,000, 1,000, and 256 nodes, respectively. The terminally fully connected layer outputs a risk (e.g., likelihood of being extracted for implantation) prediction associated with the input image 1106. The predicted risks are input into a Cox proportional hazards layer 1118 to calculate a negative partial log likelihood 1120, to provide an error signal for backpropagation within CNN 1102. In various embodiments, one or more optimization methods are employed to optimize model weights, biases, and convolution kernels, for example, Adagrad algorithm. In one implementation, the nonlimiting parameters of Adagra include an initial accumulator value (e.g., 0.1), initial learning rate (e.g., 0.001), and an exponential decay factor (e.g., 0.1). In one implementation, model weights are initialized using, for example, a variance scaling method, and a weighing decay (e.g., 4e-4) applied to the fully connected layers during training. In various embodiments, mini-batches (containing e.g., recruitable follicles) are used for training, preferably over multiple epochs (e.g., 100; 1 epoch is one complete cycle through all training samples). In various embodiments, each mini-batch produces a model update, resulting in multiple updates per epoch. In one implementation, Cox likelihood is calculated locally within each mini-batch to perform updates. In another implementation, randomization of one or more mini-batch assignments are used at the beginning of each epoch to improve robustness. In yet another implementation, regulation is applied during training, optionally using random dropout 5% of weights in the last fully connected layer of fully connected layers 1116 in each mini-batch during training to avoid over fitting. During training, one or more identified recruitable follicle field (e.g., pixel area or volume sufficient to differentiate recruitable/non-recruitable) are sampled from a region (e.g., ROI or VOI) and treated as semi-independent training samples. In various embodiments, each recruitable follicle identity/field is paired with a patient time-to-event outcome from a medical record database 1122. In various embodiments, patient outcome information includes, but is not limited to, days on hormone therapy, demographic, age, presence or absence of one or more said diagnostic biomarker, therapeutic treatment, chronic biomarker, clinical notes, physician observations, follicular growth rate, follicle size, history of pregnancy, initiation-termination ovarian stimulation, number of cycle day, follicle retrieval, follicle recruitment, oocyte retrieval, follicle stage, follicle maturity, egg maturity, fertilization rate, blastocyst embryo development, embryo fragmentation, embryo growth rate, embryo grade, embryo trophectoderm grade, embryo inner cell mass grade, embryo size, embryo growth rate, embryo cell count, embryo metabolic parameters, embryo metabolome, embryo maturity, blastocyst development rate, embryo euploidy, embryo aneuploidy, embryo mosaicism, embryology database, embryo quality, implantation, or the like. In various embodiments, duplicate outcomes can be paired with one or more regions containing multiple follicles. One or more regions can be sampled at the beginning of each training epoch to generate a new set of ROIs or VOIs. In various implementations, randomization via transformation (e.g., translation, rotation, contrast, brightness, etc.) can be applied to an acquired field to improve robustness to follicle orientation or image variations. In various embodiments, one or more fields are sampled from each ROI or VOI for calculating a risk prediction using said SCNN 1102. For example, when predicting the outcome of a patient, 10 fields are sampled from each ROI to generate a representative collection of fields, and a risk is predicted for each field. In one implementation, a median field risk is calculated in each region, sorted and filtered, and the second highest value is selected as the patient risk. The selection of the second highest risk introduces robustness to outliers or high risks that occur due to image quality or artifacts. In various embodiments, the filter procedure enables the selection of fields using a conservative prognosis to ensure the accurate selection of recruitable follicles for implantation. In various embodiments, one or more diagnostic data 1124 may be incorporated into the SCNN 1102 to improve prognostic accuracy. In one implementation, the SCNN 1102 learns from diagnostic biomarkers and US images simultaneously by incorporating biomarker variables to influence the patterns learned by the network via patient blood work information during a visit. In one embodiment, the diagnostic data 1124 is incorporated into the fully connected layers 1116. In various embodiments, one or more prediction models are trained optionally using TensorFlow (v0.12.0) on servers equipped with dual Intel Xeon CPU E5-2630L v2 @ 2.40 GHz CPUs, 128 GB RAM, and dual NVIDIA K80 graphics cards. In various embodiments, an alternative CNN toolbox comprises a proprietary framework, or one or more open framework, including but not limited to, Caffe, Torch, GoogleNet, as well as alternative deep learning models including, but not limited to, VGG, LeNet, AlexNet, ResNet, U Net, the like, or combinations thereof In various embodiments, one or more patient time-to-event outcome predictive result from an output layer of said one or more CNN are recorded in at least one electronic health record database. In alternative embodiments, the said results are transmitted and stored within a database residing in a cloud-based server.

[0131] An object of the present disclosure is a computer program product for use in the provision of ART for the diagnosis, treatment and clinical management of clinical infertility. Referring to FIG. 12, a schematic 1200 of a computer product architecture is disclosed, according to various embodiments. The architecture comprises an AI Engine 1202 that receives one or more inputs from one or more sources of data. In one embodiment, AI Engine 1202 receives data from a plethora of sources, preferably, but not limited to, physiology data 1204, US imaging data 1206, and environmental data 1208. In various embodiments, physiology data 1204 comprises one or more said diagnostic results, data, biomarkers, genomic marker, proteomic marker, body fluid analyte, chem panel, measured hormone levels, or the like. In various embodiments, US image data 1206 comprises one or more said US image, retrieved from a US scanner/device, US image stored external to an US scanner/device, US image processed by one or more said CNN of the present disclosure. In various embodiments, environmental data 1208 comprises one or more said longitudinal collected medical record data, number of cycle day, time of day, week, or monthly data. The AI Engine 1202 can also receive inputs from one or more EMR database 1210 and one or more outcomes database (or cloud-based server) 1212. In various embodiments, EMR database 1210 comprises one or more longitudinal patient medical records, said records of patient under infertility treatment or management. In a similar manner, outcomes database 1212 comprises one or more longitudinal infertility patient medical records relating to success of implantation, pregnancy rate, or the like. The said patient medical record database may be located within a dedicated facility or available from an external source (e.g., crowdsource, public database, etc.), accessible via a communication network. The AI Engine 1202 can also receive one or more inputs of one or more data or dataset generated by one or more CNN disclosed within the present disclosure, stored on cloud-based server 1214. The said cloud server and services are commonly referred to as cloud computing, on-demand computing, software as a service (SaaS), platform computing, network-accessible platform, cloud services, data centers, and the like. The term cloud can include a collection of hardware and software that forms a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, services, etc.), which can be suitably provisioned to provide on-demand self-service, network access, resource pooling, elasticity and measured service, among other features. The AI Engine 1202 comprises one or more computing architecture, hardware, software, known to one of ordinary skill in the art, to enable it to execute one or more instructions or algorithms as to process one or more external inputs. One or more processed dataset or analysis or prediction or clinical insights generated by AI Engine 1202 may be sent and stored on cloud-based server 1214 for distribution. In various embodiments, cloud-based server 1214 comprises one or one or more software application (or desktop client) 1216 to enable the development of one or more software application product (or mobile client) 1218 providing one or more functions, including but not limited to, data processing, data analysis, data presentation in graphical form, data annotation, or the like. In various embodiments, the application product comprises a system and methods for collecting, processing, and synthesizing clinical insights from at least one patient data, US image from a US scanner/device, retrieved US image, patient medical record from an electronic medical record database, outcomes data, patient record relating to fertility, patient endocrinology record, patient clinical notes, physician clinical notes, data from database residing on said cloud-based server, results from one or more output layer of one or more said ANNs, artificial intelligence engine. In various embodiments, at least one application enables the retrieval or distribution of one or more clinical insights. Clinical insights may include a variety of reproductive endocrinology insights such as expected patient egg retrieval volume, recommended timing of egg retrieval, quantity of eggs to anticipate per patient and per day, anticipated maturity date of eggs, anticipated embryo number, anticipated embryo quality, quantity and scheduling of embryo biopsies, patient care planning, reduction of error rates and improvement of pregnancy rates. In various embodiments, a user (e.g., physician) may access the clinical insights from a desktop client 1216. Similarly, a patient may access the clinical insights from a mobile client 1218, or vice versa. In one implementation, mobile client 1218 comprises one or mobile app product 1220 capable of communicating with cloud-server 1214 via a communication network 1222 to access said information. Similarly, desktop client 1216 can access cloud-based server 1214 via communication network 1222. In various embodiments, communication network 1222 comprises, but is not limited to, one or more; LAN, WAN, wireless network, cellular network, Internet, the like, or combinations thereof.

[0132] Referring to FIG. 13, a process flow diagram for determining a follicular maturity date of an ovarian follicle is shown, according to various embodiments. In step 1302, one or more digital images of a reproductive anatomy is obtained to detect one or more reproductive anatomical structures. In step 1304, the said digital images are processed to detect one or more reproductive anatomy structures. In step 1306, the one or more digital images are processed to annotate, segment, or classify one or more anatomical features of the one or more reproductive anatomical structures. In step 1308, the one or more anatomical features are analyzed according to at least one linear or non-linear framework. In step 1310, at least one time-to-event outcome of an assisted reproductive procedure is predicted according to the at least one linear or non-linear framework. In an alternative step 1312, the one or more digital images are processed to measure a volume of one or more ovarian follicles of the patient whereby the output provides an additional input to step 1304. In another alternative step 1314, an output of step 1306 provides an input for the comparison of a first digital image of the reproductive anatomy of the patient with a second digital image of the reproductive anatomy of the patient to provide an output. In step 1316, the output of step 1314 serves as input to determine a follicular maturity date of the one or more ovarian follicles of the patient.

[0133] Referring to FIG. 14, a process flow diagram of a process for generating a clinical recommendation associated with the assisted reproductive procedure is shown, according to various embodiments. In step 1402, one or more digital images of a patient's reproductive anatomy is received through one or more imaging modalities. In step 1404, the one or more digital images are processed to detect one or more reproductive anatomical structures. In step 1406, the one or more digital images are processed to annotate, segment, or classify one or more anatomical features of the one or more reproductive anatomical structures. In step 1408, the one or more anatomical structures are analyzed according to at least one linear or non-linear framework. In step 1410, at least one time-to-event outcome of an assisted reproductive procedure is predicted according to at least one linear or non-linear framework. In step 1412, clinical recommendations associated with the assistive reproductive procedure are generated by the process.

[0134] Referring to FIG. 14B, a process flow diagram of a process for generating a clinical recommendation associated with the OI procedure is shown, according to various embodiments. In step 1402b, one or more digital images of a patient's reproductive anatomy is received through one or more imaging modalities. In step 1404b, the one or more digital images are processed to detect one or more reproductive anatomical structures. In step 1406b, the one or more digital images are processed to annotate, segment, or classify one or more anatomical features of the one or more reproductive anatomical structures. In step 1408b, the one or more anatomical structures are analyzed according to at least one linear or non-linear framework. In step 1410b, at least one time-to-event outcome of an OI treatment is predicted according to at least one linear or non-linear framework. In step 1412b, clinical recommendations associated with the determination of the optimal timing for OI, incorporating time-to-event outcome, are generated by the process with the goal to maximize the pregnancy rate while simultaneously minimizing the risk of multiple pregnancies. In various embodiments, the clinical recommendations are made in conjunction with the assessment of one or more risk factors, including but not limited to, a patient's age, duration of infertility, number or prior treatment cycles, peak serum E2 concentration on the day of trigger, and the number of follicles. Risk factors for high-order multiple pregnancy includes 7 preovulatory follicles (10-12 mm), E2>1,000 pg per mL, early cycles of treatment, age <32, low BMI, and use of donor sperm. The recommendation may include limiting pregnancies from OI to singletons or twins when there are no more than one or two preovulatory follicles 10-12 mm. The recommendation may include the determination of critical size for follicles predictive of multiple pregnancy (i.e., between 12 to 15 mm). All the follicles must be accounted for before triggering ovulation and particularly those of intermediate size (between 11 and 15 mm) when evaluating the risk of multiple pregnancies.

[0135] Referring to FIG. 15, a process flow diagram of a method for generating a clinical recommendation associated with the assisted reproductive procedure is shown, according to various embodiments. In step 1502, ultrasound images of a subject's ovarian follicles are obtained using an ultrasound device. In step 1504, the ovarian ultrasound images are analyzed, according to at least one linear or non-linear framework, to annotate, segment, or classify one or more anatomical features of the subject's ovarian follicles. In step 1506, a time-to-event outcome is predicted and subsequently in step 1508, one or more clinical recommendations associated with the assisted reproductive procedure is generated by the process. In another embodiment, additional inputs 1510 are obtained through one or more processes. In step 1512, a subject's electronic medical record is obtained as an input. In step 1514, an anonymized third-party electronic medical record is obtained as an input. In step 1516, a reproductive physiology data of the subject is obtained as an input. In step 1518, environmental data related to the subject's reproductive cycle is obtained as an input. In step 1520, these additional inputs 1510 are analyzed, according to at least one linear or non-linear framework, together with the ovarian ultrasound images, to predict a time-to-event outcome in step 1506, and subsequently in step 1508, to generate one or more clinical recommendations associated with the assisted reproductive procedure.

[0136] Referring to FIG. 15B, a process flow diagram of a method for digital image processing associated with an ovulation induction cycle is shown, in accordance with various embodiments. In step 1502b, one or more digital images (e.g. ultrasound images) of a subject's ovarian follicles are obtained using an ultrasound device. In step 1504b, the ovarian ultrasound images are analyzed, according to at least one linear or non-linear framework (i.e. machine learning framework), to annotate, segment, or classify one or more anatomical features of the subject's ovarian follicles. In certain embodiments, the one or more anatomical features comprise a quantity and size of the one or more ovarian follicles. In step 1506b, the at least one processor may analyze the one or more anatomical features according to at least one machine learning framework to predict at least one time-to-event outcome. In certain embodiments, the at least one time-to-event comprises an ovulatory trigger date within an ovulation induction cycle for the subject. In step 1508b, the at least one processor may generate one or more clinical recommendations associated with the ovulation induction cycle for the subject. In certain embodiments, the one or more clinical recommendations may comprise a recommended timing for administration of at least one pharmaceutical agent to the subject. In some embodiments, the at least one pharmaceutical agent comprises an ovulatory trigger agent. In certain embodiments, the one or more clinical recommendations may comprise a recommended timing for sperm delivery or intrauterine insemination corresponding to the ovulation induction cycle. Sperm delivery may include one or more means for sperm delivery, including artificial means (e.g. sperm delivered in a clinical setting) and/or natural means (e.g. sperm delivered through intercourse).

[0137] In accordance with certain aspects of the present disclosure, additional inputs 1520b may be obtained through one or more data inputs and/or data transfer interface. In step 1512b, a subject's electronic medical record may be obtained as a data input. In accordance with certain embodiments, subject's electronic medical record may comprise one or more data sets selected from the group consisting of diagnostic results, body fluid biomarkers, hormone markers, hormone levels, genomic biomarkers, proteomic biomarkers, therapeutic treatments, treatment schedule, follicle size and number, follicle growth rate, pregnancy rate, and ovulatory induction data. In step 1514b, an anonymized third-party electronic medical record is obtained as a data input. In accordance with certain embodiments, anonymized third-party electronic medical record may comprise one or more data sets selected from the group consisting of diagnostic results, body fluid biomarkers, hormone markers, hormone levels, genomic biomarkers, proteomic biomarkers, therapeutic treatments, treatment schedule, follicle size and number, follicle growth rate, pregnancy rate, and ovulatory induction data. In step 1516b, a reproductive physiology data of the subject is obtained as an input. In accordance with certain embodiments, the reproductive physiology data of the patient may comprise diagnostic results, diagnostic biomarkers, genomic markers, proteomic marker, body fluid analytes, chemical panels, and measured hormone levels. In step 1518b, environmental data related to the subject's reproductive cycle is obtained as an input. In accordance with certain embodiments, the environmental data may comprise longitudinal medical data of the patient collected based on day of reproductive cycle, time of day, and time of week.

[0138] In various embodiments, the method provides for the assessment of one or more risk factors including but not limited to, a patient's age, duration of infertility, number or prior treatment cycles, peak serum E2 concentration on the day of trigger, and the number of follicles. Risk factors for high-order multiple pregnancy includes 7 preovulatory follicles (10-12 mm), E2>1,000 pg per mL, early cycles of treatment, age <32, low BMI, and use of donor sperm. The said process may include the determination of critical size for follicles predictive of multiple pregnancy (i.e., between 12 to 15 mm) and accounting for all the follicles before triggering ovulation and particularly those of intermediate size (between 11 and 15 mm) when evaluating the risk of multiple pregnancies. In step 1520b, these additional inputs 1520b are analyzed, according to at least one linear or non-linear framework (i.e. machine learning framework), together with the ovarian ultrasound images, to predict a time-to-event outcome in step 1506b; and subsequently in step 1508b to generate one or more clinical recommendations associated with the OI treatment. In accordance with certain embodiments, the one or more clinical recommendations may comprise recommendations to maximize the pregnancy rate associated with the OI treatment while simultaneously minimizing the risk of multiple pregnancies.

[0139] As will be appreciated by one of skill in the art, the present invention may be embodied as a method (including, for example, a computer-implemented process, a business process, and/or any other process), apparatus (including, for example, a system, machine, device, computer program product, and/or the like), or a combination of the foregoing. Accordingly, embodiments of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.), or an embodiment combining software and hardware aspects that may generally be referred to herein as a system. Furthermore, embodiments of the present invention may take the form of a computer program product on a computer-readable medium having computer-executable program code embodied in the medium.

[0140] Any suitable transitory or non-transitory computer readable medium may be utilized. The computer readable medium may be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device. More specific examples of the computer readable medium include, but are not limited to, the following: an electrical connection having one or more wires; a tangible storage medium such as a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a compact disc read-only memory (CD-ROM), or other optical or magnetic storage device.

[0141] In the context of this document, a computer readable medium may be any medium that can contain, store, communicate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer usable program code may be transmitted using any appropriate medium, including but not limited to the Internet, wireline, optical fiber cable, radio frequency (RF) signals, or other mediums.

[0142] Computer-executable program code for carrying out operations of embodiments of the present invention may be written in an object oriented, scripted or unscripted programming language such as Java, Perl, Smalltalk, C++, or the like. However, the computer program code for carrying out operations of embodiments of the present invention may also be written in conventional procedural programming languages, such as the C programming language or similar programming languages.

[0143] Embodiments of the present invention are described above with reference to flowchart illustrations and/or block diagrams of methods, apparatus(es), systems, and computer program products. It will be understood that each block of the flowchart illustrations and/or block diagrams, and/or combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-executable program code portions. These computer-executable program code portions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a particular machine, such that the code portions, which execute via the processor of the computer or other programmable data processing apparatus, create mechanisms for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

[0144] These computer-executable program code portions (i.e., computer-executable instructions) may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the code portions stored in the computer readable memory produce an article of manufacture including instruction mechanisms which implement the function/act specified in the flowchart and/or block diagram block(s). Computer-executable instructions may be in many forms, such as program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments.

[0145] The computer-executable program code may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational phases to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the code portions which execute on the computer or other programmable apparatus provide phases for implementing the functions/acts specified in the flowchart and/or block diagram block(s). Alternatively, computer program implemented phases or acts may be combined with operator or human implemented phases or acts in order to carry out an embodiment of the invention.

[0146] As the phrases are used herein, a processor may be operable to or configured to perform a certain function in a variety of ways, including, for example, by having one or more general-purpose circuits perform the function by executing particular computer-executable program code embodied in computer-readable medium, and/or by having one or more application-specific circuits perform the function.

[0147] The terms program or software are used herein in a generic sense to refer to any type of computer code or set of computer-executable instructions that can be employed to program a computer or other processor to implement various aspects of the present technology as discussed above. Additionally, it should be appreciated that according to one aspect of this embodiment, one or more computer programs that when executed perform methods of the present technology need not reside on a single computer or processor, but may be distributed in a modular fashion amongst a number of different computers or processors to implement various aspects of the present technology.

[0148] All definitions, as defined and used herein, should be understood to control over dictionary definitions, definitions in documents incorporated by reference, and/or ordinary meanings of the defined terms.

[0149] The indefinite articles a and an, as used herein in the specification and in the claims, unless clearly indicated to the contrary, should be understood to mean at least one. As used herein, the terms right, left, top, bottom, upper, lower, inner and outer designate directions in the drawings to which reference is made.

[0150] The phrase and/or, as used herein in the specification and in the claims, should be understood to mean either or both of the elements so conjoined, i.e., elements that are conjunctively present in some cases and disjunctively present in other cases. Multiple elements listed with and/or should be construed in the same fashion, i.e., one or more of the elements so conjoined. Other elements may optionally be present other than the elements specifically identified by the and/or clause, whether related or unrelated to those elements specifically identified. Thus, as a non-limiting example, a reference to A and/or B, when used in conjunction with open-ended language such as comprising can refer, in one embodiment, to A only (optionally including elements other than B); in another embodiment, to B only (optionally including elements other than A); in yet another embodiment, to both A and B (optionally including other elements); etc.

[0151] As used herein in the specification and in the claims, or should be understood to have the same meaning as and/or as defined above. For example, when separating items in a list, or or and/or shall be interpreted as being inclusive, i.e., the inclusion of at least one, but also including more than one, of a number or list of elements, and, optionally, additional unlisted items. Only terms clearly indicated to the contrary, such as only one of or exactly one of, or, when used in the claims, consisting of, will refer to the inclusion of exactly one element of a number or list of elements. In general, the term or as used herein shall only be interpreted as indicating exclusive alternatives (i.e. one or the other but not both) when preceded by terms of exclusivity, such as either, one of, only one of, or exactly one of Consisting essentially of, when used in the claims, shall have its ordinary meaning as used in the field of patent law.

[0152] As used herein in the specification and in the claims, the phrase at least one, in reference to a list of one or more elements, should be understood to mean at least one element selected from any one or more of the elements in the list of elements, but not necessarily including at least one of each and every element specifically listed within the list of elements and not excluding any combinations of elements in the list of elements. This definition also allows that elements may optionally be present other than the elements specifically identified within the list of elements to which the phrase at least one refers, whether related or unrelated to those elements specifically identified. Thus, as a non-limiting example, at least one of A and B (or, equivalently, at least one of A or B, or, equivalently at least one of A and/or B) can refer, in one embodiment, to at least one, optionally including more than one, A, with no B present (and optionally including elements other than B); in another embodiment, to at least one, optionally including more than one, B, with no A present (and optionally including elements other than A); in yet another embodiment, to at least one, optionally including more than one, A, and at least one, optionally including more than one, B (and optionally including other elements); etc.

[0153] In the claims, as well as in the specification above, all transitional phrases such as comprising, including, carrying, having, containing, involving, holding, composed of, and the like are to be understood to be open-ended, i.e., to mean including but not limited to. Only the transitional phrases consisting of and consisting essentially of shall be closed or semi-closed transitional phrases, respectively, as set forth in the United States Patent Office Manual of Patent Examining Procedures, Section 2111.03.

[0154] The present disclosure includes that contained in the appended claims as well as that of the foregoing description. Although this invention has been described in its exemplary forms with a certain degree of particularity, it is understood that the present disclosure has been made only by way of example and numerous changes in the details of construction and combination and arrangement of parts may be employed without departing from the spirit and scope of the invention.