COMPUTER VISION SYSTEM AND METHOD FOR ASSESSING ORTHOPEDIC SPINE CONDITION

Abstract

A computer vision system and method for an orthopedic assessment of the human spine condition. The system uses frontal and sagittal images of the human spine to detect the vertebrae of the spine. More specifically, four edges of each and every vertebra are detected, and the corresponding straight lines, which can be used for assessment, diagnosis and evaluation of various spinal disorders and diseases by orthopedic doctors, are exported. The system has two phases. In phase one, deep learning algorithm for object detection is applied to detect and localize each and every vertebra of frontal and sagittal images of the human spine. In phase two, the system extracts straight lines that correspond to each of the four edges. Using the straight lines, metrics for the spinal assessment, such as the curvature of the spine, the distance of consecutive vertebrae, and crucial angles such as the Cobb angle, can be determined.

Claims

1. A computer vision system using an edge computing architecture to process frontal and sagittal images of a human spine in order to detect a plurality of edges of each and every vertebra of the human spine, and extract straight lines that correspond to the detected edges, comprising: an edge device pre-installed with a method of using first and second phases to perform a task of vertebrae and edge detection on the input images, and extract the corresponding straight lines for each of the four edges from each and every vertebra of the human spine, wherein the edge device includes training of weights and checkpoints needed for processing the frontal and lateral images of the human spine.

2. The computer vision system of claim 1, wherein the frontal and sagittal images of the human spine, which comprise radiation radiographs, magnetic resonance imaging and fluoroscopic or dynamic images, are obtained through radiation and magnetic fields or waves.

3. The computer vision system of claim 2, wherein the first phase includes object detection and deep learning models that are trained on the frontal and sagittal images of the human spine for detection and localization of each and every vertebra of the human spine, wherein the second phase is an edge detection process that locates the four edges of each and every vertebra of the human spine, and extracts the straight lines for each of the four edges of the vertebra, in a way that each straight line corresponds to one edge of a vertebra, and wherein the straight lines extracted for each vertebrae includes a first line for an upper edge, a second line for a lower edge, a third line for a left edge, and a fourth line for a right edge.

4. The computer vision system of claim 3, wherein any type of frontal and sagittal image of the human spine is used as an input and the location of each and every vertebra is determined and outputted contained within rotated bounding boxes, which enclose one vertebra each and are extracted in the form of pixel coordinates of the input image and a rotation angle.

5. The computer vision system of claim 1, wherein weights of deep learning and object detection models for all different kinds of frontal and sagittal pictures of the human spine, i.e. images taken by radiation, or magnetic fields or waves, along with model checkpoints, labels and tuned parameters, that are used to train the system, and wherein the system is trained on a supercomputer, and weights and checkpoints of the supercomputer have been exported and installed in the edge device to enable the edge device to generate predictions and final results based on the training.

6. The computer vision system of claim 5, wherein the second phase is the edge detection process that uses the rotated bounding boxes from the object detection model in phase one as the input, then outputs straight lines that correspond to the edges of each vertebra, and extracts the four edges of each and every vertebra of the human spine, with the edges extracted in the form of straight lines having one straight line for each edge and a total four straight lines extracted that correspond respectively to the upper edge, lower edge, left edge, and right edge.

7. The computer vision system of claim 6, wherein the edge detection process includes the steps of: inputting all of the rotated bounding boxes detected in the first phase of the system, and processing each and every rotated bounding box; applying Gaussian Blur algorithm followed by Canny edge detection algorithm to generate a result; defining a Region of Interest as a mask, and determining all edges, including horizontal and vertical edges, of each vertebra; applying the mask to the result; applying Hough transform; and applying Linear Regression algorithm to extract at least one corresponding straight line for each of the four vertebra edges.

8. The computer vision system of claim 7, wherein models and algorithms are pre-installed in the edge device to perform processing independently, and to provide with a final clinical report that is specialized on a spine disease on a monitor screen connected to the edge device for the orthopedic doctor to evaluate and assess the condition of the spine without requiring communication with one or more external processing devices.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Additional features and advantage of the present disclosure will be made apparent from the following detailed description of one or more exemplary embodiments with reference to the accompanying figures, which are given for illustrative purpose only, and thus are not limitative of the present disclosure, wherein:

[0036] FIG. 1 is a flow chart showing the system conducting a spine assessment;

[0037] FIG. 2 is a posteroanterior (PA) X-ray radiograph image with its vertebrae detected from phase one of the system;

[0038] FIG. 3 is another flow chart showing phase two of the system;

[0039] FIG. 4 shows exemplary spine conditions, disorders and diseases that can be assessed by the system;

[0040] FIG. 5 shows a specific spine disorder relating to Adolescent Idiopathic Scoliosis; and

[0041] FIG. 6 shows a calculation of the Cobb angle in two spine locations as depicted in FIG. 5.

DETAILED DESCRIPTION OF THE EXEMPLARY EMBODIMENTS

[0042] One or more exemplary embodiments according to the present disclosure on the system and method that aim to assist orthopedic doctors and clinicians in the assessment of the condition of a human spine will be described below with references to the accompanying figures.

[0043] FIG. 1 is a flow chart showing the system conducting a spine (scoliosis) assessment. The edge-device machine first receives as input a posteroanterior, PA (frontal) patient's X-ray radiograph. In other words, the input for the edge-device machine is an X-ray radiograph of the human spine. Next, the system uses the architecture of the edge computing framework. In FIG. 1, the depicted edge-device machine processing is performed using pre-installed and pre-loaded models and weights for object detection, i.e. vertebra detection in phase one, and the algorithms for edge detection and straight line extraction in phase two.

[0044] More specifically, in the first phase or phase one, object detection is performed where each and every vertebra of the human spine is detected and located. An object detection deep learning algorithm, which detects and locates each and every vertebra of the human spine in the X-ray radiograph of the patient, is also initiated. Each detected vertebra is enclosed into a rotated bounding box, which is determined in the image by its coordinates and a rotation angle. In the second phase, edge detection is performed in six steps, which are further depicted in FIG. 3. The output of phase two is straight lines which correspond to the edges of each vertebra. Then, a specialized clinical report, which in the spine assessment is related to scoliosis diagnosis, is generated according to the spinal disease.

[0045] In the second phase or phase two, the location of the upper and lower edges and how much tilting they have for each and every vertebra by extracting straight lines with their slopes are specified for each edge.

[0046] FIG. 2 shows a result of the phase one detection. Specifically, the depicted result is generated from the object detection deep learning model that detects and localizes each and every vertebra of the human spine. Each detected vertebra is enclosed within a rotated bounded box and has a probability assigned to it. Each rotated bounding box is determined by its coordinates on the image and a rotation angle.

[0047] FIG. 3 illustrates a flow chart for the second phase of the system in a further detail. Among the 6 steps shown, step 1 is the section of picking a bounding box; step 2 is the application of Gaussian blurring method and Canny edge detection algorithm; step 3 is the determination of a region of interest where either the horizontal edges are kept, or the vertical, or all of them; in step 4, the mask of step 3 is applied to the result of step 2; in Step 5, the Hough transform algorithm is applied; and in step 6, the Linear Regression algorithm is applied for each of the detected edges in order to extract straight lines that correspond to each detected edge.

[0048] The particular order of transformations in the second phase is discussed below. The first step involves a selection of a bounding box from the result of the object detection algorithm in the first phase of the system. The second step relates to Gaussian Blur and Canny Edge Detection. In particular, the Canny edge detector is a popular and widely used edge detection operator that uses a multi-stage algorithm to detect a wide range of edges in images.

[0049] In the third step, a Region of Interest (ROI), a.k.a. a mask, is defined. For this particular spine disorder (i.e. the scoliosis), one would be interested in the horizontal edges of each vertebra only. Therefore, the vertical edges are ignored. Thus, in order to capture the upper and lower edges of each vertebra only, using two white rectangles, the areas in the image are determined where the upper and the lower edges are estimated. That way, after discarding the parts of the image that hide the mask, the rest parts of the image can be ascertained.

[0050] In the fourth step, the mask of the third step is applied to the result of the second step. In other words, only the Region of Interest from the image of step 2 is kept. The result of this step is an image that contains only the area where the two horizontal edges of the vertebra fall.

[0051] In the fifth step, the Hough Transform algorithm is applied. This algorithm extracts many small straight line segments out of each edge. The purpose of this operation is to approximate the shape of each edge with multiple small straight line segments and to extract the coordinates of the data points that fall on these line segments. The input of this algorithm is the part of the image that contains one of the edges. The output will be data-points that define segments of lines that approximate the input edge. This algorithm is executed twice: one for the upper edge and another for the lower edge. As a result, the final output of this step will be two sets of data-points, one for the upper edge and another for the lower edge, each of which determine line segments that approximate each edge. In this step, each edge is approximated by many small line segments that are defined each by a starting data point and ending data point.

[0052] In the sixth step, the Linear Regression algorithm is applied in order to receive one final straight line for each edge. In the Linear Regression algorithm, a set of data-points are received as input that determine line segments that approximate an edge (that were generated on the fifth step), and the output is one straight line that passes through the middle of all the data points. In particular, the Linear Regression algorithm is applied twice, i.e., once for the upper set of data points (upper edge), and another time for the lower set of data points (lower edge).

[0053] FIG. 4 shows spine conditions, disorders and diseases where the system can be utilized. In particular, the spine conditions, disorder and diseases that the system can be used include: (1) Scoliosis (a sideways curvature of the spine which may cause the spine to be in the shape of a C or S instead of being straight); (2) Scheuermann's Kyphosis where vertebra is developed with a wedge shape; (3) Compression Fracture where the front of the vertebral body collapses and the back does not; (4) Thinning or Denerative Disks where the disc starts to shrink, lose its shape, lose its flexibility, wear out or get very thin as measured by the distance between consecutive vertebrae; and (5) Spondylolisthesis where vertebra slips forwards.

[0054] In the cases of Scheuermann's Kyphosis and Compression Fracture as shown in FIG. 4, an X-ray radiograph of the lateral (sagittal) view of the human spine is inputted, with the rest performed similar to the case of the Scoliosis, except that in phase one, an object detection, i.e. vertebra detection on the sagittal view of the X-ray radiograph, is applied, and that in phase two, the upper and lower edges for each and every vertebra are applied to yield a mask with two horizontal area rectangles, i.e., one for the upper edge and another for the lower edge. At the end, the system determines the vertebrae with the most wedged shape and calculates the curvature of the spine and other medical information, in order to assist orthopedics in assessing the severity of the condition.

[0055] Additionally, in the case of Spondylolisthesis, an X-ray radiograph of the lateral (sagittal) view of the human spine is needed as an input to the system. Hence, in phase one, object detection, i.e. vertebra detection on a sagittal X-ray, is performed. However, in this case, the vertical edges of each and every vertebra of the human spine are needed. As such, step 3 in phase two of the system needs to be modified in that the Region of Interest need to include the vertical edges for each vertebra as to allow the mask to keep the vertical edges instead of the horizontal edges.

[0056] FIG. 5 shows a specific spine disorder the system can be applied. A human back with Adolescent Idiopathic Scoliosis (AIS) with the vertebrae drawn is shown, as well as two ways of estimating the Cobb angle, which yield the same result. For instance, one way is to calculate the Cobb angle directly, and the other way is to calculate the Cobb angle geometrically.

[0057] FIG. 6 is another figure similar to FIG. 5 that shows again the calculation of the Cobb angle of a spine in two places for the case of Adolescent Idiopathic Scoliosis (AIS). Specifically, Scoliosis can be diagnosed using X-ray radiographs that can include a standing x-ray of the entire spine looking from the front and back ends.

[0058] In the exemplary embodiment as shown in FIGS. 5 and 6, the Cobb angle is measured at the angle between the two vertebrae that are most tilted relative to the horizontal at upper and lower levels of each curve. More specifically, having a Cobb angle of more than or equal to 10 degrees is regarded as a minimum angulation to define Scoliosis; a Cobb angle of between 10 to 25 degrees is regarded as a mild curvature; a Cobb angle of between 25 to 40 degrees is regarded as a moderate curvature; and a Cobb angle more than 40 degrees is regarded as a severe curvature. The inclination of a line joining the mid-points of two sides of the vertebra is parallel to the superior and inferior end plates, and such defined as the inclination angle. The greatest inclination angles at the upper and lower parts of curvature are classified as the upper and lower end vertebrae, respectively.

[0059] The system detects the upper and lower edges of each and every vertebra of the human spine and extracts two straight lines. One straight line corresponds to the upper edge, and another straight line to the lower edge. As such, the Cobb angle that is described hereinabove can be calculated by using these extracted straight lines. Such way is fast and accurate, and does not suffer from human error and saves doctors' valuable time.

[0060] The system makes use of the architecture of edge computing system. The algorithms and models are pre-installed in the edge-device machine. The edge device machine is installed in the doctor's exam room or any other appropriate facility of a hospital or a clinic. The whole processing is performed on the edge-device and the results are shown on a monitor screen that is connected to it. No any further communication of the edge device with any outer processing device is required.

[0061] Although the present disclosure has been described with reference to specific embodiments, this description is not meant to be construed in a limiting sense. It should be understood that the scope of the present disclosure is not limited to the above-mentioned embodiments, but is limited by the accompanying claims. It is, therefore, contemplated that the appended claims will cover all modifications that fall within the true scope of the present disclosure. Without departing from the object and spirit of the present disclosure, various modifications to the embodiments are possible, but they remain within the scope of the present disclosure, will be apparent to persons skilled in the art.