Fault diagnosis device based on common information and special information of running video information for electric-arc furnace and method thereof

Abstract

A fault diagnosis method for an electrical fused magnesia furnace includes steps of: 1) arranging six cameras; 2) obtaining video information by the six cameras and sending the video information to a control center; then analyzing the video information by a chip of the control center; wherein a multi-view-based fault diagnosis method is used by the chip, comprising steps of: 2-1) comparing a difference between two consecutive frame histograms for shots segmentation; 2-2) computing a set of characteristic values for each shot obtained by the step 2-1), and then computing color, texture, and motion vector information; finally, evaluating shot importance via entropy; 2-3) clustering shots together by calculating similarity; 2-4) generating and optimizing a multi-view video summarization with a multi-objective optimization model; and 2-5) providing fault detection and diagnosis; and 3) displaying results of the fault detection and diagnosis on a host computer inter face of the control center.

Claims

1. A fault diagnosis method based on common information and special information of running video information for an EFMF (electrical fused magnesium furnace), comprising steps of: 1) arranging six cameras, wherein three of the six cameras are respectively arranged at relative positions of three electrodes above the EFMF and aim at the electrodes of the EFMF, so as to monitor a furnace eruption fault; rest of the six cameras are symmetrically arranged around a furnace body by a 120 degree difference and aim at the furnace body, so as to monitor occurrence of a furnace leaking fault; 2) obtaining video information by the six cameras and sending the video information to a control center; then analyzing the video information by a chip of the control center; wherein in order to simplify a difficulty of analysis and improve a real-time performance of video data analysis, multi-view video summarization technology is introduced, so that industrial process monitoring based on running video information is able to be realized; specifically, a multi-view-based fault diagnosis method is used by the chip, comprising steps of: 2-1) comparing a difference between two consecutive frame histograms for shots segmentation; 2-2) computing a set of characteristic values for each shot obtained by the step 2-1), and then computing color, texture, and motion vector information; finally, evaluating shot importance via entropy; 2-3) clustering shots together by calculating similarity, wherein calculation of the similarity of the shots comprises the similarity of the shots in a mono-view and correlation of the shots in different views; 2-4) generating and optimizing a multi-view video summarization with a multi-objective optimization model; wherein the shot in the shot cluster is either reserved or abandoned so as to obtain the multi-view video summarization with a less number and a shorter length of the shots but contains more fully video information; and 2-5) providing fault detection and diagnosis; and 3) displaying results of the fault detection and diagnosis on a host computer inter face of the control center.

2. The fault diagnosis method, as recited in claim 1, wherein the step 2-2) specifically comprises a step of computing the color information by a color histogram; wherein an HSV (hue, saturation and value) color space is used to obtain color histogram information, so as to describe color entropy, wherein: for a frame f with N color values, a probability of appearance of a i.sup.th color value in an image is P.sub.i, thus the color entropy is defined as: $\begin{array}{cc}{E}_{\mathrm{HSV}}\left(f\right)=\underset{i=1}{\overset{N}{.Math.}}{p}_i\log \left(1/p_i\right)& \left(1\right)\end{array}$ wherein .sub.i=1.sup.Np.sub.i=1 and p.sub.i0.

3. The fault diagnosis method, as recited in claim 2, wherein the step 2-2) specifically comprises a step of computing the texture information by an edge direction histogram; wherein texture features are extracted using edge direction histogram descriptor; a sobel operator is selected to calculate an edge direction of each pixel; an image space is separated by four lines: horizontal, vertical, 45, and 135, in such a manner that the image is divided into eight bins on a center point of the image; then an edge direction information is gathered and an edge direction histogram is obtained; information entropy E.sub.EDGE(f) is calculated based on the edge direction histogram of each frame.

4. The fault diagnosis method, as recited in claim 3, wherein the step 2-2) specifically comprises a step of computing the motion vector information by a motion-related feature vector; wherein V(t,k) is used to represent a k.sup.th bin grey value of the color histogram of a frame t, where 0k127; a motion-related feature vector is represented by a histogram difference between the frame t and a previous frame t1, which is determined as $\begin{array}{cc}V\left(t,k\right)=.Math.V\left(t,k\right)-V\left(t-1,k\right).Math.& \left(2\right)\\ E_{\mathrm{motion}}=\underset{k=1}{\overset{127}{.Math.}}p\left\{k|V\left(t,k\right)\right\}\log \frac{1}{\left\{k|V\left(t,k\right)\right\}}.& \left(3\right)\end{array}$

5. The fault diagnosis method, as recited in claim 4, wherein the step 2-2) specifically comprises a step of evaluating the shot importance via the entropy; wherein an entropy fusion model is applied to deal with the entropy, and different weights are chosen to merger of different types of the entropy:
E.sub.com(f)=.sub.1E.sub.HSV(f)+.sub.2E.sub.edge(f)+.sub.3E.sub.motion(f)(4) wherein .sub.i meets: .sub.1+.sub.2+.sub.3=1; thus an important frame set is obtained:
F.sub.imp(Video)={f.sub.i1,f.sub.i2, . . . ,f.sub.in}(5) then an entropy score of each frame is obtained; wherein the entropy with a high score is defined by a threshold to retain, which indicates important frames; a definition of the entropy score is as follows: $\begin{array}{cc}=\frac{E_i\left(f\right)-\min E_i\left(f\right)}{\max E_i\left(f\right)-\min E_i\left(f\right)}\mathrm{Threshold}& \left(6\right)\end{array}$ wherein is the entropy score and the threshold is customized according to user requirements, and a range value of the threshold is in an interval [0,1]; E.sub.i(f) represents the entropy of the i.sup.th frame, max E.sub.i(f) and min E.sub.i(f) represent a maximum value and a minimum value of all entropy separately; the frame whose values are greater than the threshold is retained; finally, important shots S.sub.imp are defined as follows:
S.sub.imp=Int(f.sub.i1,f.sub.i2, . . . , f.sub.i)(7) wherein Int(.Math.) is an integration operation to combine the important frames of a same shot, so as to obtain the important shots.

6. The fault diagnosis method, as recited in claim 5, wherein in the step 2-3), the similarity of the shots in the mono-view is measured by two indexes: a temporal adjacency and a visual similarity; specifically, the temporal adjacency refers that two shots are likely to reflect a same event, which is defined as:
d.sub.T(T.sub.i,T.sub.j)=.sub.1+.sub.2|T.sub.iT.sub.j|+.sub.3|T.sub.iT.sub.j|.sup.2(8) wherein T.sub.i and T.sub.j respectively denote a time of middle frames of the i.sup.th and j.sup.th shots along a time axis in a same view. .sub.1, .sub.2 and .sub.3 are control coefficients; wherein correlation of the shots are measured by the color histogram and the edge direction histogram; a further Euclidean distance is used to measure a difference between two color histograms and two edge direction histograms separately; if k is the k.sup.th bin of the histogram, then: $\begin{array}{cc}{d}_H\left(H_i,H_j\right)=\sqrt{\underset{k=0}{\overset{n}{.Math.}}{\left(H_i\left(k\right)-H_j\left(k\right)\right)}^2}& \left(9\right)\\ d_G\left(G_i,G_j\right)=\sqrt{\underset{k=0}{\overset{m}{.Math.}}{\left(G_i\left(k\right)-G_j\left(k\right)\right)}^2}& \left(10\right)\end{array}$ wherein H.sub.i(k) and H.sub.j(k), respectively, denote the color histogram of the k.sup.th bin in the i.sup.th and j.sup.th frame, H.sub.i(k)H.sub.j(k) denotes a difference of the color histogram between the two corresponding bins k; and G.sub.i(k), G.sub.j(k) denote the edge direction histogram of the k.sup.th bin in the i.sup.th and j.sup.th frame respectively, G.sub.i(k)G.sub.j(k) denote a difference of the edge direction histogram between the two corresponding bins k; finally, the temporal adjacency and the visual similarity are combined to obtain the similarity of the shots under the same view, namely: $\begin{array}{cc}\mathrm{Diff}\left(i,j\right)={}_1{d}_T\left(T_i,T_j\right)+{}_2{d}_H\left(H_i,H_j\right)+{}_3{d}_G\left(G_i,G_j\right)& \left(11\right)\\ \mathrm{MonoSim}\left(i,j\right)=e^{-\frac{{\mathrm{Diff}\left(i,j\right)}^2}{2{}^2}}& \left(12\right)\end{array}$ wherein, .sub.1, .sub.2, .sub.3 are regularization parameters.

7. The fault diagnosis method, as recited in claim 6, wherein in the step 2-3), the correlation of the shots in the different views is measured by a principle component analysis-scale invariant feature transform (PCA-SIFT) algorithm; wherein n frames are randomly selected from each of the similar shots in each view for PCA-SIFT detection; through the feature vector generated by key points in each frame, the descriptor of each frame is obtained, and then the Euclidean distance d of the feature vector is considered as a similarity determination measure between the two frames to obtain a correlation degree of the shots under the different views; assuming that V is a selected shot in one view and S is a shot to be compared in another view, thus the distance between the two shots is be measured by: $\begin{array}{cc}d\left(S,V\right)=\frac{1}{n}\underset{i=1}{\overset{n}{.Math.}}\underset{f_{i}V,gS}{\min }\left[d\left(f_i,g\right)\right]& \left(13\right)\end{array}$ wherein n is a number of frames in V; g and f.sub.i are frames from S and V respectively; the correlation of the shots in the different views is transformed into a problem to find a shot S with minimum Euclidean distance between the feature vectors: $\begin{array}{cc}\stackrel{\_}{S}=\arg \underset{S}{\min }\left[d\left(S,V\right)\right]& \left(14\right)\end{array}$ thus the correlation of shots in the different views is defined as follows: $\begin{array}{cc}\mathrm{MultiSim}\left(S,V\right)=e^{-\frac{{d\left(S,V\right)}^2}{2{}^2}}& \left(15\right)\end{array}$ finally, the similar shot are gathered to form a similar shot cluster, wherein the similar shot cluster is finished by a K-means clustering algorithm.

8. The fault diagnosis method, as recited in claim 7, wherein in the step 2-4), a given decision vector x=(x.sub.1, x.sub.2, . . . , x.sub.n) is provided, which meets: $\begin{array}{cc}{x}_i=\{\begin{array}{cc}1,& \mathrm{shot}i\mathrm{is}\mathrm{reserved}\\ 0,& \mathrm{shot}i\mathrm{is}\mathrm{not}\mathrm{reserved}\end{array}& \left(16\right)\end{array}$ a multi-objective optimization function is given by $\begin{array}{cc}\max \mathrm{UV}s.t.\{\begin{array}{c}\left(f_2\left(x\right)\right)D_{\max }\\ \left(f_3\left(x\right)\right)I_{\min }\end{array}& \left(17\right)\end{array}$ wherein U=[1, 1, 1, 1], V=diag(f.sub.1(x),f.sub.2(x), f.sub.3(x),f.sub.4(x)); f.sub.1(x)=x.sub.1+x.sub.2+ . . . +x.sub.n represents a sum of all shots reserved; f.sub.2(x)=D.sub.1x.sub.1+D.sub.2x.sub.2+ . . . +D.sub.nx.sub.n represents a sum of every shot duration time, and D.sub.i denotes a time length of the i.sup.th shot; f.sub.3(x)=I.sub.1x.sub.1+I.sub.2x.sub.2+ . . . +I.sub.nx.sub.n represents the shot importance, and I.sub.i denotes importance of the i.sup.th shot; a last component is donated as f.sub.4(x)=.sub.i,j=1,ij.sup.nSim(S.sub.i,S.sub.j).Math.x.sub.i.Math.x.sub.j; in a constraint, (.Math.) denotes normalization of a linear function; D.sub.max and I.sub.min denote a maximum length of a shot duration and a minimum significance of the shots respectively when the video summarization is generated; and are control coefficients that meet D.sub.i=D.sub.max, I.sub.i=I.sub.min; an objective function is given by
u(F(x))=max.sub.i.Math.UV.sub.1(18) wherein F(x)=((f.sub.1(x)),(f.sub.2(x)),(f.sub.3(x)),(f.sub.4(x))).sup.T; and .sub.i is a weight value control coefficient of the objective function and meets .sub.1+.sub.2+.sub.3+.sub.4=1 with non-negative .sub.i; then the multi-view video summarization is optimized by solving x*: $\begin{array}{cc}{x}^{*}=\arg \underset{x}{\max }u\left(F\left(x\right)\right)s.t.A.Math.F\left(\begin{array}{c}{D}_{\max }\\ -I_{\min }\\ -u\left(F\right)\end{array}\right)\mathrm{wherein}A=\left(\begin{array}{cccc}0& 1& 0& 0\\ 0& 0& -1& 0\\ -1& -1& -1& -1\end{array}\right).& \left(19\right)\end{array}$

9. The fault diagnosis method, as recited in claim 8, wherein the step 2-5) specifically comprises a step of building monitoring indexes COMI and SPEI by characteristic variables extracted from the multi-view video summarization; wherein in view of a monitoring video in the different views, the video summarization is taken as common information of a surveillance video by a global multi-object optimization under the single view and the different views; then, video sets are obtained by extracting the important shots from the single view respectively, which will be regarded as information source of special parts of the monitoring video; wherein the monitoring indexes are defined as:
COMI=c.sup.T.sub.c.sup.1c(20)
SPEI.sub.l=s.sub.l.sup.T.sub.l.sup.1s.sub.l, l=1,2,3(21) wherein c is a vector of common information feature variables, and s.sub.l is a vector of special information feature variables at a l.sup.th single view; .sub.c and .sub.l are variance of modeling data about multi-view common video information and special video information respectively.

10. The fault diagnosis device, as recited in claim 9, wherein the cameras are CCD (charge-coupled device) cameras.

11. The fault diagnosis device, as recited in claim 10, wherein the multi-view-based fault diagnosis method used for analyzing the video information is stored in the chip of the control center.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) FIG. 1 illustrates fault diagnosis research contents and applications description.

(2) FIG. 2 shows a multi-view monitoring illustration.

(3) FIG. 3 illustrates a host computer interface of a fault diagnosis device.

(4) FIG. 4 is a fault diagnosis device monitoring block diagram.

(5) FIG. 5 illustrates EFMF smelting surface and furnace wall area division.

(6) FIG. 6 shows important shot sets extracted by multi-view method.

(7) FIG. 7 is multi-view monitoring probability density fitting curves.

(8) FIG. 8 shows furnace eruption fault monitoring results.

(9) FIG. 9 is a furnace eruption fault monitoring contributions plot.

(10) FIG. 10 illustrates furnace leaking fault monitoring results.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

(11) Embodiment 1: a fault diagnosis device based on common information and special information of running video information for an EFMF (electrical fused magnesia furnace).

(12) For the smelting process, we know that the strong coupling between the multi-source interference and the conventional monitoring variables due to the complex field conditions often makes process monitoring results of EFMF difficult to be satisfactory. Therefore, video information as process variables will be intended to solve the above difficulties. In order to obtain the above process data, we designed the following process monitoring device, as shown in FIG. 2.

(13) According to FIG. 2, in order to use the video data information better, the multi-view method is introduced into the device so as to omni-directional acquisition of the video information. It is well known that the furnace eruption fault of EFMF generally occurs at the superposition area of electrode arc heat, and the furnace leaking fault usually occurs at the arc of the electrode. Therefore, it is necessary to accurately and effectively monitor the smelting operation state near the electrode area. The way to achieve multi-view approach is to place a CCD camera at the relative position of the three electrodes above the EFMF respectively as shown in the FIG. 2 to monitor the furnace eruption fault, so that the dead zones in each camera can be monitored complementarily. And the three CCD cameras are symmetrically arranged around the furnace body by a 120 degree difference to monitor the occurrence of furnace leaking fault. The real-time video surveillance image at the electrode and the surface of the furnace can be seen in FIG. 2. Eventually, the video information obtained by the six CCD cameras will be collected into the control center and further called in the process of online detection and fault diagnosis. The called data will be applied to the fault diagnosis method based on the multi-view monitoring approach in the control center so as to finish the tasks of the processing, analysis and judgment of the monitoring data. Finally, according to the corresponding monitoring indexes, we can determine the running state of the EFMF and then guide the production. In addition, the data information after the analysis will be displayed in FIG. 3 through the host computer interface of control center.

(14) In FIG. 3, we can visually see the information obtained by the algorithm processing video data embedded in the device. The multi-view-based fault diagnosis method applied is stored in the chip of the control center, and the chip is shown in FIG. 2. The specific fault diagnosis process is illustrated in FIG. 4, which can be just regarded as structure block diagram of fault diagnosis device using the video information. From FIG. 4, we can see that after real-time multi-channel parallel acquisition of multiple CCD camera video information, we need to preprocess the raw data according to the corresponding method and further extract interested data features. According to FIG. 4, the fault diagnosis device designed can analyze the collected data by common information (COMI) and special information (SPEI) statistics in the common characteristics and special information sections of the video information, which aims to achieve the running state detection of EFMF and fault diagnosis when the fault occurs.

(15) The CCD camera monitoring location has been roughly introduced hereinbefore, and the CCD camera monitoring location settings will be explained in accordance with the fused magnesia smelting crystal structure and smelting work area division in the following. In FIG. 5, according to the crystalline state of fused magnesia finished product and melting state of materials during the smelting process in the furnace, the EFMF is divided into the following fused magnesia melting crystalline structure and the smelting workspace on the basis of the horizontal direction and perpendicular to the horizontal direction respectively. In the horizontal direction, furnace surface is divided into four regions from the center to the circumference of a circle, namely, I Impurity area, II Equiaxed crystal area, III Columnar crystal area and IV Leather sand layer area. Since the area I is located in the magnetic field overlap region where the three electrode alternating currents form, the arc generated by the electrode under the action of the ampere force will be far away from this area. Therefore, it is difficult to ensure that the raw material is sufficiently melted. However, the area II is closer to the arc and the temperature is relatively higher. The separated impurities by smelted fully in this area will move to the region I under the influence of the gas or move to the center under the extrusion of the first solidified magnesia, and then the final impurities will be formed in the area I. In addition, due to the gas and other dust from the decomposition of magnesite moving to the center area during the smelting process, it is easy to make the internal pressure increase causing the furnace eruption fault when the gas and other dust are covered by the raw material without smelting fully. The areas II and III are the final product areas obtained after the end of the production. Since the temperature of the area III is relatively constant at the center of electrode smelting heat, the magnesia crystals obtained are ideal; The area II is in the high temperature region and during the cooling crystallization, due to the internal heat of melting magnesia, equiaxed crystal takes shape caused by the lack of the required crystallization temperature difference, which we would not like to see. The ratio of columnar crystals to equiaxed crystals can be changed by temperature control or by changing the radius of the furnace. On account of the lower temperature at the edge of furnace, the magnesium carbonate cannot be completely melted. So the leather sand is consisted of magnesium carbonate without melting and silica migrating outward produced during smelting process in the area IV. Therefore, in order to monitor the occurrence of the furnace eruption fault and real-time acquire fused magnesia smelting state, and then to get a higher proportion of columnar crystals, corresponding cameras are placed above the furnace and around the electrode. In the direction perpendicular to the horizontal plane, the furnace in the smelting process can be divided into {circle around (1)} Raw material layer, {circle around (2)} Smelting layer and {circle around (3)} Crystal cooling layer from top to bottom. In these three parts, the temperature of the smelting layer is the highest and the most prone to danger. Layers {circle around (1)} and {circle around (2)} keep rising with the electrode lifting during the smelting process. In the process of electrode lifting, it is likely that the electrode is from the center position to near the furnace wall bringing about furnace leaking fault. Therefore, for fault alarming on the furnace timely, monitoring the furnace body in the vertical direction is also needed.

(16) Fault diagnosis device based on multi-view method has carried on the detailed introduction hereinbefore. Then a description of multi-view-based fault diagnosis method needs giving in the following.

(17) Embodiment 2: a fault diagnosis method based on common information and special information of running video information for an EFMF.

(18) Multi-view technology is method which shoots the same scene from different viewpoints and then extracts the same shots or shots with more associated information. This can obtain all-round video image information in a plane so as to avoid blind spots. However, for continuous monitoring video, storage and analysis have been already a relatively troublesome problem. Then extraction and analysis of the monitoring video information from multiple cameras on the same scene are more complicated and difficult. In order to simplify the difficulty of analysis and improve the real-time performance of video data analysis, multi-view video summarization technology is introduced into the present invention, so that industrial process monitoring based on running video information can be realized. So, the multi-view video summarization technology will be explained in detail in this section.

(19) We first introduce the method of shots segmentation and then we demonstrate the entropy model to evaluate the importance of shots. Next, the similarity of shots in multi-view is detailed in order to facilitate the similar shots cluster. Later we implement the multi-objective optimization to select the most representative shots to generate the finial multi-view video summarization. Last but not least, we can build the monitoring indexes for fault detection and diagnosis.

(20) Shots Segmentation:

(21) The first step in the video summarization extraction process is shot segmentation. In the present invention, we use the method of grayscale histogram via comparing the difference between the two consecutive frame histograms.

(22) Entropy Model:

(23) The evaluation of the shot importance is an important part of multi-view video summarization because the importance of the shot often contains more useful information of the event. Generally speaking, static or low-activity shots can be defined as unimportant shots. In this part, we compute a set of characteristic values for each shot and then consider color, texture, and motion vector information. Finally, we represent them via entropy to evaluate the shot importance.

(24) Color Histogram: In this issue, it is in order to represent the importance of the shots that we decide to obtain the color histogram information using the HSV color space to describe the color entropy, because it is found to be more robust to small color changes of the multi-view camera.

(25) For a frame f with N color values, the probability of appearance of the i.sup.th color value in the image is P.sub.i, thus the color entropy is defined as:

(26) $\begin{array}{cc}{E}_{\mathrm{HSV}}\left(f\right)=\underset{i=1}{\overset{N}{.Math.}}{p}_i\log \left(1\text{/}{p}_i\right)\mathrm{Where}\underset{i=1}{\overset{N}{.Math.}}{p}_i=1\mathrm{and}{p}_{i}0.& \left(1\right)\end{array}$

(27) Edge Direction Histogram: These texture features are extracted using edge direction histogram descriptor. The sobel operator is selected to calculate the edge direction of each pixel. The image space is separated by four lines: horizontal, vertical, 45, and 135. As a result, the image is divided into eight bins on the center point of the image. Then the edge direction information is gathered and the edge direction histogram is obtained. Next, the information entropy E.sub.EDGE(f) can be calculated based on edge direction histogram of each frame.

(28) Motion-related Feature Vector: When the moving object changes in the same scene, it will lead to the pixels' changes. In this part, the motion entropy is defined based on the gray-scale histogram and some appropriate improvements on the gray-scale entropy of the image are carried on. Let V(t,k) represent the k.sup.th bin grey value of the color histogram of the frame t, where 0k127. Its motion-related feature vector can be represented by the histogram difference between frame t and its previous frame t1, which is determined as

(29) $\begin{array}{cc}V\left(t,k\right)=\left|V\left(t,k\right)-V\left(t-1,k\right)\right|& \left(2\right)\\ E_{\mathrm{motion}}=\underset{k=1}{\overset{127}{.Math.}}p\left\{k|V\left(t,k\right)\right\}\log \frac{1}{\left\{k|V\left(t,k\right)\right\}}& \left(3\right)\end{array}$

(30) According to different video contents and the user's needs, while emphasizing the impact of useful information, the entropy fusion model is applied to deal with the entropy. Taking into account the different entropy on the impact of the results, different weights are chosen to merger of different types of entropy:
E.sub.com(f)=.sub.1E.sub.HSV(f)+.sub.2E.sub.edge(f)+.sub.3E.sub.motion(f)(4)

(31) and .sub.i meets: .sub.1+.sub.2+.sub.3=1.

(32) Thus the important frame set can be obtained:
F.sub.imp(Video)={f.sub.i1,f.sub.i2, . . . , f.sub.in}(5)

(33) Then we can obtain the entropy score of each frame. Generally speaking, the higher the score is, the more information that the frame contains. The entropy with a high score can be defined by the threshold to retain, that is, the important frame. The definition of entropy score is as follow:

(34) $\begin{array}{cc}=\frac{E_i\left(f\right)-\min E_i\left(f\right)}{\max E_i\left(f\right)-\min E_i\left(f\right)}\mathrm{Threshold}& \left(6\right)\end{array}$

(35) Where is the entropy score. It can be customized according to user requirements, and the range value of the threshold is in the interval [0,1]. E.sub.i(f) represents the entropy of the i.sup.th frame, max E.sub.i(f) and min E.sub.i(f) represent the maximum and minimum values of all entropies separately. The frame whose values are greater than the threshold will be retained.

(36) Finally, the important shot S.sub.imp is defined as follows:
S.sub.imp=Int(f.sub.i1,f.sub.i2, . . . , f.sub.i)(7)

(37) where Int(.Math.) is the integration operation to combine the important frames which belong to the same shot. Thus we obtain the important shots.

(38) The similarity of shots in multi-view:

(39) A considerable part of shots above reflect similar events. Therefore, these shots should be clustered together by calculating their value of similarity. For the multi-view videos, each shot correlates closely with not only the temporally adjacent shots in its view but also the shots in other views. The calculation of the similarity of shots includes the similarity of shots in mono-view and the correlation of shots in different views.

(40) The similarity of the shots in mono-view can be measured by two indexes: temporal adjacency and visual similarity. In particular, temporal adjacency refers that two shots are likely to reflect the same event if they are temporally adjacent to each other and therefore they tend to have a high degree of similarity. Here we define:
d.sub.T(T.sub.i,T.sub.j)=.sub.1+.sub.2|T.sub.iT.sub.j|+.sub.3|T.sub.iT.sub.j|.sup.2(8)

(41) Where T.sub.i and T.sub.j respectively denote the time of the middle frames of the i.sup.th and j.sup.th shots along the time axis in the same view. .sub.1, .sub.2 and .sub.3 are the control coefficient.

(42) In terms of visual similarity, the color histogram and the edge direction histogram are applied to measure the correlation of the shots. Further Euclidean distance is used to measure the difference between two color histograms and two edge direction histograms separately. Suppose k be the k.sup.th bin of the histogram, then:

(43) $\begin{array}{cc}{d}_H\left(H_i,H_j\right)=\sqrt{\underset{k=0}{\overset{n}{.Math.}}{\left(H_i\left(k\right)-H_j\left(k\right)\right)}^2}& \left(9\right)\\ d_G\left(G_i,G_j\right)=\sqrt{\underset{k=0}{\overset{m}{.Math.}}{\left(G_i\left(k\right)-G_j\left(k\right)\right)}^2}& \left(10\right)\end{array}$

(44) Where H.sub.i(k) and H.sub.j(k), respectively, denote the color histogram of the k.sup.th bin in the i.sup.th and j.sup.th frame, H.sub.i(k)H.sub.j(k) denotes the difference of the color histogram between the two corresponding bins k. And G.sub.i(k), G.sub.j(k) denote the edge direction histogram of the k.sup.th bin in the i.sup.th and j.sup.th frame respectively, G.sub.i(k)G.sub.j(k) denote the difference of the edge direction histogram between the two corresponding bins k.

(45) Finally, combine the temporal adjacency and the visual similarity to obtain the similarity of the shots under the same view, namely:

(46) $\begin{array}{cc}\mathrm{Diff}\left(i,j\right)={}_1{d}_T\left(T_i,T_j\right)+{}_2{d}_H\left(H_i,H_j\right)+{}_3{d}_G\left(G_i,G_j\right)& \left(11\right)\\ \mathrm{MonoSim}\left(i,j\right)=e^{-\frac{{\mathrm{Diff}\left(i,j\right)}^2}{2{}^2}}& \left(12\right)\end{array}$

(47) Where, .sub.1, .sub.2, .sub.3 are the regularization parameter and then by setting the threshold, the similar shots can be obtained under the same view.

(48) Principle component analysis-scale invariant feature transform (PCA-SIFT) algorithm is used to measure the correlation of the shots in different views. It has been confirmed that SIFT features have the characteristic of scale invariance, even if the rotation angle, image brightness or shooting angle are changed, and it can still get a good detection effect. Therefore, it is suitable to find the similar shots in different views. The PCA-SIFT algorithm can effectively reduce the dimensionality while preserving the good characteristics of SIFT.

(49) We randomly select n frames from each of the similar shots in each view for PCA-SIFT detection. Through the feature vector generated by the key points in each frame, we get the descriptor of each frame, and then the Euclidean distance d of the feature vectors is considered as the similarity determination measure between the two frames to obtain the correlation degree of the shots under different views. Assuming that V is the selected shot in one view and S is the shot to be compared in another view, thus the distance between the two shots can be measured by:

(50) $\begin{array}{cc}d\left(S,V\right)=\frac{1}{n}\underset{i=1}{\overset{n}{.Math.}}\underset{f_{i}V,gS}{\min }\left[d\left(f_i,g\right)\right]& \left(13\right)\end{array}$

(51) where n is the number of frames in V.Math.g and f.sub.i are frames from S and V respectively.

(52) With this presentation, the correlation of the shots in different views can be transformed into the problem to find a shot S with minimum Euclidean distance between the feature vectors:

(53) $\begin{array}{cc}\stackrel{\_}{S}=\arg \underset{S}{\min }\left[d\left(S,V\right)\right]& \left(14\right)\end{array}$

(54) Thus the correlation of shots in different views can be defined as follow:

(55) $\begin{array}{cc}\mathrm{MultiSim}\left(S,V\right)=e^{-\frac{{d\left(S,V\right)}^2}{2{}^2}}& \left(15\right)\end{array}$

(56) Finally, the similar shots (including the same view and different views) are gathered to form similar shot cluster. Similar shot cluster can be finished by K-means clustering algorithm.

(57) The Multi-Objective Optimization:

(58) An ideal video summarization is expected to present as much video shot information as possible with the shortest summary duration and the minimum number of shots. At the same time, the most representative shot of the different views in the multi-view video is selected to present in the one video summarization. Therefore, we need to adopt the multi-objective optimization model to make the obtained multi-view video summarization as good as possible.

(59) Suppose a given decision vector x=(x.sub.1, x.sub.2, . . . , x.sub.n), and it meets:

(60) 0$\begin{array}{cc}{x}_i=\{\begin{array}{cc}1,& \mathrm{shot}i\mathrm{is}\mathrm{reserved}\\ 0,& \mathrm{shot}i\mathrm{is}\mathrm{not}\mathrm{reserved}\end{array}& \left(16\right)\end{array}$

(61) The multi-objective optimization function is given by

(62) $\begin{array}{cc}\max \mathrm{UV}s.t.\{\begin{array}{c}\left(f_2\left(x\right)\right)D_{\max }\\ \left(f_3\left(x\right)\right)I_{\min }\end{array}& \left(17\right)\end{array}$

(63) Where U=[1,1, 1, 1], V=diag(f.sub.1(x),f.sub.2(x), f.sub.3(x),f.sub.4(x)). f.sub.1(x)=x.sub.1+x.sub.2+ . . . +x.sub.n represents the sum of all shots reserved. f.sub.2(x)=D.sub.1x.sub.1+D.sub.2x.sub.2+ . . . +D.sub.nx.sub.n represents the sum of every shot duration time, and D.sub.i denotes the time length of the i.sup.th shot. f.sub.3(x)=I.sub.1x.sub.1+I.sub.2x.sub.2+ . . . +I.sub.nx.sub.n represents the importance of the shot, where the greater the value is, the more information the multi-view video summarization covers, and I.sub.i denotes the importance of the i.sup.th shot. And the last one can be donated as f.sub.4(x)=.sub.i,j=1,ij.sup.nSim(S.sub.i,S.sub.j).Math.x.sub.i.Math.x.sub.j.

(64) In the constraint, (.Math.) denotes normalization of the linear function. D.sub.max and I.sub.min denote the maximum length of shot duration and the minimum significance of the shots respectively when the video summarization is generated. and are the control coefficient that meet D.sub.i=D.sub.max, I.sub.i=I.sub.min. The objective function is given by
u(F(x))=max.sub.i.Math.UV.sub.1(18)

(65) Where F(x)=((f.sub.1(x)),(f.sub.2(x)),(f.sub.3(x)),(f.sub.4(x))).sup.T. And .sub.i is the weight value control coefficient of the objective function and meets .sub.1+.sub.2+.sub.3+.sub.4=1 with non-negative .sub.i.

(66) Then the multi-objective optimization above can be transformed into the 0-1 mixed integer programming problem and we can get the result of the optimization by solving the x*:

(67) $\begin{array}{cc}{x}^{*}=\arg \underset{x}{\max }u\left(F\left(x\right)\right)s.t.A.Math.F\left(\begin{array}{c}{D}_{\max }\\ -I_{\min }\\ -u\left(F\right)\end{array}\right)\mathrm{in}\mathrm{which}A=\left(\begin{array}{cccc}0& 1& 0& 0\\ 0& 0& -1& 0\\ -1& -1& -1& -1\end{array}\right).& \left(19\right)\end{array}$

(68) According to the results, the shot in the shot cluster can be either reserved or abandoned so as to obtain the multi-view video summarization with less number and shorter length of the shots but contains more fully video information in the result.

(69) Fault Detection and Diagnosis:

(70) In accordance with the ideal multi-view video summarization, we can build monitoring indexes COMI and SPEI by characteristic variable extracting from the video summarization.

(71) In view of the monitoring video in different views, the video summarization is taken as the common information of surveillance video by the global multi-object optimization under the single view and different views. Then, the video sets are obtained by extracting the important shots from the single view respectively, which will be regarded as information source of the special parts of monitoring video. Because there is a dead zone for each view in the monitoring process, it is significant to make use of the special information of single view to compensate the common information of the global video summarization. According to this idea, we construct the following monitoring indicators as:
COMI=c.sup.T.sub.c.sup.1c(20)
SPEI.sub.l=s.sub.l.sup.T.sub.l.sup.1s.sub.l, l=1,2,3(21)

(72) Where c is a vector of common information feature variables, and s.sub.l is a vector of special information feature variables at the l.sup.th single view. Moreover, .sub.c and .sub.l are the variance of modeling data about multi-view common video information and special video information respectively.

(73) Embodiment 3: experiment results.

(74) Magnesia is a kind of refractory material widely used in the field of metallurgical industry, glass industry, cement industry, household heater, chemical industry and so on. EFMF is one of the most widely used production device in the fused magnesia industry. In order to guarantee the normal operation of the EFMF, we must ensure its safety. If there is any fault during normal operation, the performance of control system could get severe damage, even lead to the breakdown of the entire system and enormous loss, when the fault will not be able to timely adjust or alarm. Therefore, the fault detection and diagnosis of EFMF is imperative.

(75) In the monitoring process, we select one of the most important shots belonging to time for a second video as a sample, and each shot consists of three frames of image. According to the monitoring indexes, we can obtain video summarization samples in single view and multi-view. As illustrated in FIG. 6, three single views of 1#, 2# and 3# CCD cameras are selected as the compensation information of multi-view video. And p, q denote the p.sup.th, q.sup.th monitoring sample, simultaneously seen as fault sample in FIG. 6.

(76) After obtaining the above four video sets, we extract eight texture features, seven Hu moment invariants, optical flow potential and acceleration potential from four video sets, totally seventeen variables. Since the three images within a shot change little, we compute the mean of three-frame image texture feature and Hu moment invariants feature as the first 15 variables in the shot features. In addition, we define the 2-norm of the optical flow field between the two images as the optical flow potential, and define the 2-norm of the acceleration field among the three images as the acceleration potential simultaneously. Similar to the above, we make the average of optical flow potential corresponding to image within the two shots and the average of acceleration potential corresponding to image within the three shots as the last two variables of shot characteristics. Therefore, in order to obtain the optical flow potential and the acceleration potential, it is necessary to obtain two pre-sampling shots before monitoring.

(77) First of all, we use the fault diagnosis device designed to detect the furnace eruption fault. In this process, 200 seconds of the normal running video are selected as the original modeling video data, 200 important shots are screened at single views and multi-view respectively, and then 200 sets of process variables are extracted as modeling samples corresponding to the four video sets. Next, 200 seconds of the original video are selected to extract 200 sets of process variables as the test data sets at single views and multi-view respectively, where furnace eruption fault occurs continuously. Furthermore, the modeling datasets extracted are standardized and calculated to obtain the collection of monitoring indexes according to the equations (20) and (21). And we carry out the probability statistics of the monitoring indexes of modeling data and carry on the kernel density fitting to the density function of the monitoring indexes. The probability density fitting curves of the monitoring indexes obtained from the multi-view method is shown in FIG. 7. According to FIG. 7, we can know that control limitations of 95% confidence bound are respectively 100, 118, 105 and 122 corresponding to statistics COMI and SPEI.sub.l (l=1, 2, 3).

(78) Based on the control limitations obtained from the probability density analysis of the modeling data, the detecting results of the test data are shown in FIG. 8. In FIG. 8, we find that the change trends in the detection results of (a), (b) and (d) are similar to each other in general. At the 66.sup.th sample point, the fault occurs continuously. And the system can be adjusted by itself to make the fault trend within controllable range among 66 seconds and 100 seconds. The furnace first appears more obvious trend of the furnace eruption fault at 108.sup.th second, after this, the state of the system exceeds the safety critical point, and the furnace eruption fault of more energy occurs more often. In FIG. 8, we find that the furnace eruption faults occurring always disappear soon, due to the strong surface tension of the molten magnesia and the decreasing in pressure after the release. In addition, the change of (c) is lagging behind the other three images obviously, indicating that the change at (c) is caused by (b) and (d), and then it can be judged that the furnace eruption fault occurs roughly between 1# and 3# CCD camera monitoring position. Therefore, through the analysis of FIG. 8, we find that whether the fault occurs or not can be judged according to the multi-view statistic COMI, and then the general location of the fault can be determined by the statistic SPEI.

(79) FIG. 8 has shown that the fault diagnosis device designed is effective for the discovery of the fault. Next, we analyze the fault contributions of the 131.sup.th sample point with the largest monitoring amplitude in FIG. 8. The variable fault contributions of the 131.sup.th sample point are shown in FIG. 9, and the variable values are obtained through normalizing according to the expectation and variance of the modeling data. In the figure, we find that the characteristic variables of the 3# special information are similar to the characteristic variables of common information roughly. Therefore, it can be considered that the monitoring range of 3# camera is the significantly main fault occurrence area. In addition, we conclude that it is bigger that the variable contributions of the optical flow potential and acceleration potential are about variables COMc and SPEs.sub.3, which indirectly describes the importance of constructing these two characteristic variables. And also illustrates that a large part of the fault severity changes can be reflected by the video flow field and the acceleration field.

(80) For the monitoring of the furnace leaking fault, the three cameras overlap mutual information are little in the furnace monitoring process, so we do not consider the common part of the video summarization information, and only use their respective single view to monitor the furnace leaking fault at the electrode. In the process of identifying fault, we assume that when the large bright incandescent yellow spot appears on the furnace wall, the furnace leaking fault occurs. As a result of the acquisition of the furnace leaking fault control limitation and the analysis of variable fault contribution plot consistent with the analysis of furnace eruption fault, the simulation detecting plot of the furnace leaking fault is only given here, shown in FIG. 10. In the experiment, we select 200 seconds of normal condition monitoring video to model, and then apply 1000 seconds video containing the furnace leaking fault to test. Same to the furnace eruption fault, the control limitations of 95% confidence bound about statistics SPEI.sub.l (l=1, 2, 3) are 405, 432 and 458 respectively. As shown in FIG. 10, 1# and 2# view points do not contain faults, but 3# point appears the fault at the 718.sup.th sampling point and has been maintained all the time. Besides, three monitoring indexes are in a substantial increase after the 600.sup.th second, indicating that all three electrodes have a tendency closing to the furnace wall or the currents continue to increase. Accordingly, the production site can check the corresponding electrode location and current output and then avoid the risk of furnace leaking fault under the tendency of monitoring indexes in a substantial increase.

(81) The above simulation experiments, furnace eruption fault and furnace leaking fault do not happen at the same time, however, they may also unfortunately occur at the same time in the actual production. Therefore, it is necessary to detect and anticipate the occurrence of faults in a timely and effective manner. The experimental results demonstrate that the fault diagnosis device designed has good performance for the fault detection, and the optical flow potential and the acceleration potential introduced need to consider the previous variable information, so it is helpful to predict the occurrence of the fault.

(82) Fault diagnosis device based on multi-view method has shown good performance in the process monitoring by using industrial video information. At the same time, the feature variables extracted by the multi-view video summarization method can effectively compress the raw video data and solve the problem that the video information is complicated and difficult to deal with. This solves the trouble of using the video information for online monitoring. Moreover, the constructed optical flow potential and the acceleration potential provide the possibility of predicting the occurrence of faults. All in all, the fault diagnosis device designed can solve the detection and diagnosis problems of furnace eruption fault and furnace leaking fault in the smelting process, effectively.

(83) One skilled in the art will understand that the embodiment of the present invention as shown in the drawings and described above is exemplary only and not intended to be limiting.

(84) It will thus be seen that the objects of the present invention have been fully and effectively accomplished. Its embodiments have been shown and described for the purposes of illustrating the functional and structural principles of the present invention and is subject to change without departure from such principles. Therefore, this invention includes all modifications encompassed within the spirit and scope of the following claims.