Methods for using feature vectors and machine learning algorithms to determine discriminant functions of minimum risk quadratic classification systems
10650287 ยท 2020-05-12
Inventors
Cpc classification
G06F18/254
PHYSICS
G06F17/18
PHYSICS
A61B5/7264
HUMAN NECESSITIES
G06F18/2415
PHYSICS
G06F17/16
PHYSICS
International classification
G06F17/11
PHYSICS
A61B5/00
HUMAN NECESSITIES
G06F17/18
PHYSICS
G06F17/16
PHYSICS
Abstract
Methods are provided for determining discriminant functions of minimum risk quadratic classification systems, wherein a discriminant function is represented by a geometric locus of a principal eigenaxis of a quadratic decision boundary. A geometric locus of a principal eigenaxis is determined by solving a system of fundamental locus equations of binary classification, subject to geometric and statistical conditions for a minimum risk quadratic classification system in statistical equilibrium. Feature vectors and machine learning algorithms are used to determine discriminant functions and ensembles of discriminant functions of minimum risk quadratic classification systems, wherein a discriminant function of a minimum risk quadratic classification system exhibits the minimum probability of error for classifying given collections of feature vectors and unknown feature vectors related to the collections.
Claims
1. A computer-implemented method of using feature vectors and machine learning algorithms to determine a discriminant function of a minimum risk quadratic classification system that classifies said feature vectors into two classes and using said discriminant function of said minimum risk quadratic classification system to classify unknown feature vectors related to said feature vectors, said method comprising: receiving an Nd data set of feature vectors within a computer system, wherein N is a number of feature vectors, d is a number of vector components in each feature vector, and each one of said N feature vectors is labeled with information that identifies which of two classes each one of said N feature vectors belongs to, and wherein each said feature vector is defined by a d-dimensional vector of numerical features, wherein said numerical features are extracted from digital signals; receiving within said computer system unknown feature vectors related to said data set; determining a kernel matrix using said data set, said determination of said kernel matrix being performed by using processors of said computer system to calculate a matrix of all possible inner products of signed reproducing kernels of said N feature vectors, wherein a reproducing kernel of a feature vector replaces said feature vector with a curve that contains first and second degree vector components, and wherein each one of said reproducing kernels of said N feature vectors has a sign of +1 or 1 that identifies which of said two classes each one of said N feature vectors belongs to, and using said processors of said computer system to calculate a regularized kernel matrix from said kernel matrix; determining scale factors of a geometric locus of signed and scaled reproducing kernels of extreme points using said regularized kernel matrix, wherein said extreme points are located within overlapping regions or near tail regions of distributions of said N feature vectors, said determination of said scale factors being performed by using said processors of said computer system to determine a solution of a dual optimization problem, wherein said scale factors and said geometric locus satisfy a system of fundamental locus equations of binary classification, subject to geometric and statistical conditions for a minimum risk quadratic classification system in statistical equilibrium, and wherein said scale factors determine conditional densities for said extreme points and also determine critical minimum eigenenergies exhibited by scaled extreme vectors on said geometric locus, wherein said critical minimum eigenenergies determine conditional probabilities of said extreme points and also determine corresponding counter risks and risks of a minimum risk quadratic classification system, wherein said counter risks are associated with right decisions and said risks are associated with wrong decisions of said minimum risk quadratic classification system, and wherein said geometric locus determines the principal eigenaxis of the decision boundary of said minimum risk quadratic classification system, wherein said principal eigenaxis exhibits symmetrical dimensions and density, wherein said conditional probabilities and said critical minimum eigenenergies exhibited by said minimum risk quadratic classification system are symmetrically concentrated within said principal eigenaxis, and wherein counteracting and opposing components of said critical minimum eigenenergies exhibited by corresponding components of said scaled extreme vectors on said geometric locus together with corresponding counter risks and risks exhibited by said minimum risk quadratic classification system are symmetrically balanced with each other about the geometric center of said principal eigenaxis, wherein the center of total allowed eigenenergy and minimum expected risk of said minimum risk quadratic classification system is located at the geometric center of said geometric locus, and wherein said geometric locus determines a primal representation of a dual locus of likelihood components and principal eigenaxis components, wherein said likelihood components and said principal eigenaxis components are symmetrically distributed over either side of the axis of said dual locus, wherein a statistical fulcrum is placed directly under the center of said dual locus, and wherein said likelihood components of said dual locus determine conditional likelihoods for said extreme points, and wherein said principal eigenaxis components of said dual locus determine an intrinsic coordinate system of geometric loci of a quadratic decision boundary and corresponding decision borders that jointly partition the decision space of said minimum risk quadratic classification system into symmetrical decision regions; determining said extreme vectors on said geometric locus using the vector of said scale factors, said determination of said extreme vectors being performed by using said processors of said computer system to identify said scale factors that exceed zero by a small threshold, and using said processors of said computer system to determine a sign vector of signs associated with said extreme vectors using said data set, and compute the average sign using said sign vector; determining a locus of risk for said minimum risk quadratic classification system using said reproducing kernels of said extreme vectors and said signed reproducing kernels of said N feature vectors and said vector of scale factors, said determination of said locus of risk being performed by using said processors of said computer system to calculate a matrix of inner products between said signed reproducing kernels of said N feature vectors and said reproducing kernels of said extreme vectors, and multiply said matrix by said vector of scale factors, and compute the average risk for said minimum risk quadratic classification system using said locus of risk; determining a discriminant locus for said minimum risk quadratic classification system using said geometric locus, said determination of said discriminant locus being performed by using said processors of said computer system to calculate a matrix of inner products between said signed reproducing kernels of said N feature vectors and said reproducing kernels of said unknown feature vectors, and multiply said matrix by said vector of scale factors; determining the discriminant function of said minimum risk quadratic classification system, using said average risk and said average sign and said discriminant locus, said determination of said discriminant function of said minimum risk quadratic classification system being performed by using said processors of said computer system to subtract said average risk from sum of said discriminant locus and said average sign, wherein said discriminant function of said minimum risk quadratic classification system satisfies said system of fundamental locus equations of binary classification, and wherein said discriminant function of said minimum risk quadratic classification system determines likely locations of said N feature vectors and also determines said geometric loci of said quadratic decision boundary and said corresponding decision borders that jointly partition said extreme points into said symmetrical decision regions, wherein said symmetrical decision regions span said overlapping regions or said tail regions of said distributions of said N feature vectors, and wherein said discriminant function of said minimum risk quadratic classification system satisfies said quadratic decision boundary in terms of a critical minimum eigenenergy and said minimum expected risk, wherein said counteracting and opposing components of said critical minimum eigenenergies exhibited by said corresponding components of said scaled extreme vectors on said geometric locus associated with said corresponding counter risks and risks exhibited by said minimum risk quadratic classification system are symmetrically distributed over said axis of said dual locus, on equal sides of said statistical fulcrum located at said geometric center of said dual locus, wherein said counteracting and opposing components of said critical minimum eigenenergies together with said corresponding counter risks and risks exhibited by said minimum risk quadratic classification system are symmetrically balanced with each other about said geometric center of said dual locus, and wherein said statistical fulcrum is located at said center of said total allowed eigenenergy and said minimum expected risk of said minimum risk quadratic classification system, wherein said minimum risk quadratic classification system satisfies a state of statistical equilibrium, wherein said total allowed eigenenergy and said expected risk of said minimum risk quadratic classification system are minimized, and wherein said minimum risk quadratic classification system exhibits the minimum probability of error for classifying said N feature vectors that belong to said two classes and said unknown feature vectors related to said data set; and determining which of said two classes said unknown feature vectors belong to using said discriminant function of said minimum risk quadratic classification system, said determination of said classes of said unknown feature vectors being performed by using said processors of said computer system to apply said discriminant function of said minimum risk quadratic classification system to said unknown feature vectors, wherein said discriminant function determines likely locations of said unknown feature vectors and identifies said decision regions related to said two classes that said unknown feature vectors are located within, wherein said discriminant function recognizes said classes of said unknown feature vectors, and wherein said minimum risk quadratic classification system decides which of said two classes said unknown feature belong to and thereby classifies said unknown feature vectors.
2. The method of claim 1, wherein the reproducing kernel is a Gaussian reproducing kernel: k.sub.x=exp(s=x.sup.2):0.010.1.
3. The method of claim 1, wherein the reproducing kernel is a second-order polynomial reproducing kernel: k.sub.x=(s.sup.Tx+1).sup.2.
4. A computer-implemented method of using feature vectors and machine learning algorithms to determine a fused discriminant function of a fused minimum risk quadratic classification system that classifies two types of said feature vectors into two classes, wherein said types of said feature vectors have different numbers of vector components, and using said fused discriminant function of said fused minimum risk quadratic classification system to classify unknown feature vectors related to said two types of said feature vectors, said method comprising: receiving an Nd data set of feature vectors within a computer system, wherein N is a number of feature vectors, d is a number of vector components in each feature vector, and each one of said N feature vectors is labeled with information that identifies which of two classes each one of said N feature vectors belongs to, and wherein each said feature vector is defined by a d-dimensional vector of numerical features, wherein said numerical features are extracted from digital signals; receiving an Np different data set of different feature vectors within said computer system, wherein N is a number of different feature vectors, p is a number of vector components in each different feature vector, and each one of said N different feature vectors is labeled with information that identifies which of said two classes each one of said N different feature vectors belongs to, and wherein each said different feature vector is defined by a p-dimensional vector of numerical features, wherein said numerical features are extracted from digital signals; receiving within said computer system unknown feature vectors related to said data set and unknown different feature vectors related to said different data set; determining a kernel matrix using said data set, said determination of said kernel matrix being performed by using processors of said computer system to calculate a matrix of all possible inner products of signed reproducing kernels of said N feature vectors, wherein a reproducing kernel of a feature vector replaces said feature vector with a curve that contains first and second degree vector components, and wherein each one of said reproducing kernels of said N feature vectors has a sign of +1 or 1 that identifies which of said two classes each one of said N feature vectors belongs to, and using said processors of said computer system to calculate a regularized kernel matrix from said kernel matrix; determining a different kernel matrix using said different data set, said determination of said different kernel matrix being performed by using saki processors of said computer system to calculate a matrix of all possible inner products of signed reproducing kernels of said N different feature vectors, wherein a reproducing kernel of a different feature vector replaces said different feature vector with a curve that contains first and second degree vector components, and wherein each one of said reproducing kernels of said N different feature vectors has a sign of +1 or 1 that identifies which of said two classes each one of said N different feature vectors belongs to, and using said processors of said computer system to calculate a regularized different kernel matrix from said different kernel matrix; determining a discriminant function of a minimum risk quadratic classification system using said regularized kernel matrix and said data set, said determination of said discriminant function of said minimum risk quadratic classification system comprising the steps of: determining scale factors of a geometric locus of signed and scaled reproducing kernels of extreme points using said regularized kernel matrix, wherein said extreme points are located within overlapping regions or near tail regions of distributions of said N feature vectors, said determination of said scale factors being performed by using said processors of said computer system to determine a solution of a dual optimization problem, wherein said scale factors and said geometric locus satisfy a system of fundamental locus equations of binary classification, subject to geometric and statistical conditions for a minimum risk quadratic classification system in statistical equilibrium, and wherein said scale factors determine conditional densities for said extreme points and also determine critical minimum eigenenergies exhibited by scaled extreme vectors on said geometric locus, wherein said critical minimum eigenenergies determine conditional probabilities of said extreme points and also determine corresponding counter risks and risks of a minimum risk quadratic classification system, wherein said counter risks are associated with right decisions and said risks are associated with wrong decisions of said minimum risk quadratic classification system, and wherein said geometric locus determines the principal eigenaxis of the decision boundary of said minimum risk quadratic classification system, wherein said principal eigenaxis exhibits symmetrical dimensions and density, wherein said conditional probabilities and said critical minimum eigenenergies exhibited by said minimum risk quadratic classification system are symmetrically concentrated within said principal eigenaxis, and wherein counteracting and opposing components of said critical minimum eigenenergies exhibited by corresponding components of said scaled extreme vectors on said geometric locus together with corresponding counter risks and risks exhibited by said minimum risk quadratic classification system are symmetrically balanced with each other about the geometric center of said principal eigenaxis, wherein the center of total allowed eigenenergy and minimum expected risk of said minimum risk quadratic classification system is located at the geometric center of said geometric locus, and wherein said geometric locus determines a primal representation of a dual locus of likelihood components and principal eigenaxis components, wherein said likelihood components and said principal eigenaxis components are symmetrically distributed over either side of the axis of said dual locus, wherein a statistical fulcrum is placed directly under the center of said dual locus, and wherein said likelihood components of said dual locus determine conditional likelihoods for said extreme points, and wherein said principal eigenaxis components of said dual locus determine an intrinsic coordinate system of geometric loci of a quadratic decision boundary and corresponding decision borders that jointly partition the decision space of said minimum risk quadratic classification system into symmetrical decision regions; determining said extreme vectors on said geometric locus using the vector of said scale factors, said determination of said extreme vectors being performed by using said processors of said computer system to identify said scale factors that exceed zero by a small threshold, and using said processors of said computer system to determine a sign vector of signs associated with said extreme vectors using said data set, and compute the average sign using said sign vector; determining a locus of risk for said minimum risk quadratic classification system using said reproducing kernels of said extreme vectors and said signed reproducing kernels of said N feature vectors and said vector of scale factors, said determination of said locus of risk being performed by using said processors of said computer system to calculate a matrix of inner products between said signed reproducing kernels of said N feature vectors and said reproducing kernels of said extreme vectors, and multiply said matrix by said vector of scale factors, and compute the average risk for said minimum risk quadratic classification system using said locus of risk; determining a discriminant locus for said minimum risk quadratic classification system using said geometric locus, said determination of said discriminant locus being performed by using said processors of said computer system to calculate a matrix of inner products between said signed reproducing kernels of said N feature vectors and said reproducing kernels of said unknown feature vectors, and multiply said matrix by said vector of scale factors; determining the discriminant function of said minimum risk quadratic classification system, using said average risk and said average sign and said discriminant locus, said determination of said discriminant function of said minimum risk quadratic classification system being performed by using said processors of said computer system to subtract said average risk from sum of said discriminant locus and said average sign, wherein said discriminant function of said minimum risk quadratic classification system satisfies said system of fundamental locus equations of binary classification, and wherein said discriminant function of said minimum risk quadratic classification system determines likely locations of said N feature vectors and also determines said geometric loci of said quadratic decision boundary and said corresponding decision borders that jointly partition said extreme points into said symmetrical decision regions, wherein said symmetrical decision regions span said overlapping regions or said tail regions of said distributions of said N feature vectors, and wherein said discriminant function of said minimum risk quadratic classification system satisfies said quadratic decision boundary in terms of a critical minimum eigenenergy and said minimum expected risk, wherein said counteracting and opposing components of said critical minimum eigenenergies exhibited by said corresponding components of said scaled extreme vectors on said geometric locus associated with said corresponding counter risks and risks exhibited by said minimum risk quadratic classification system are symmetrically distributed over said axis of said dual locus, on equal sides of said statistical fulcrum located at said geometric center of said dual locus, wherein said counteracting and opposing components of said critical minimum eigenenergies together with said corresponding counter risks and risks exhibited by said minimum risk quadratic classification system are symmetrically balanced with each other about said geometric center of said dual locus, and wherein said statistical fulcrum is located at said center of said total allowed eigenenergy and said minimum expected risk of said minimum risk quadratic classification system, wherein said minimum risk quadratic classification system satisfies a state of statistical equilibrium, wherein said total allowed eigenenergy and said expected risk of said minimum risk quadratic classification system are minimized, and wherein said minimum risk quadratic classification system exhibits the minimum probability of error for classifying said N feature vectors that belong to said two classes and said unknown feature vectors related to said data set; determining a different discriminant function of a different minimum risk quadratic classification system using said regularized different kernel matrix and said different data set, said determination of said different discriminant function of said different minimum risk quadratic classification system being performed by using said processors of said computer system to perform said steps of determining said discriminant function of said minimum risk quadratic classification system, wherein said different minimum risk quadratic classification system exhibits the minimum probability of error for classifying said N different feature vectors that belong to said two classes and said unknown different feature vectors related to said different data set; determining a fused discriminant function of a fused minimum risk quadratic classification system using said discriminant function of said minimum risk quadratic classification system and said different discriminant function of said different minimum risk quadratic classification system, said determination of said fused discriminant function of said fused minimum risk quadratic classification system being performed by using said processors of said computer system to sum said discriminant function of said minimum risk quadratic classification system and said different discriminant function of said different minimum risk quadratic classification system; and determining which of said two classes said unknown feature vectors and said unknown different feature vectors belong to using said fused discriminant function of said fused minimum risk quadratic classification system, said determination of said classes of said unknown feature vectors and said unknown different feature vectors being performed by using said processors of said computer system to apply said fused discriminant function of said fused minimum risk quadratic classification system to said unknown feature vectors and said unknown different feature vectors, wherein said fused discriminant function determines likely locations of said unknown feature vectors and said unknown different feature vectors and identifies said decision regions related to said two classes that said unknown feature vectors and said unknown different feature vectors are located within, wherein said fused discriminant function recognizes said classes of said unknown feature vectors and said unknown different feature vectors, and wherein said fused minimum risk quadratic classification system decides which of said two classes said unknown feature vectors and said unknown different feature vectors belong to and thereby classifies said unknown feature vectors and said unknown different feature vectors.
5. The method of claim 4, wherein the reproducing kernel is a Gaussian reproducing kernel: k.sub.x=exp(sx.sup.2):0.010.1.
6. The method of claim 4, wherein the reproducing kernel is a second-order polynomial reproducing kernel: k.sub.x=(s.sup.T x+1).sup.2.
7. A computer-implemented method of using feature vectors and machine learning algorithms to determine a discriminant function of an M-class minimum risk quadratic classification system that classifies said feature vectors into M classes and using said discriminant function of said M-class minimum risk quadratic classification system to classify unknown feature vectors related to said feature vectors, said method comprising: receiving M Nd data sets of feature vectors within a computer system, wherein M is a number of classes, N is a number of feature vectors in each one of said M data sets, d is a number of vector components in each feature vector, and each one of said N feature vectors in each one of said M data sets belongs to the same class and is labeled with information that identifies said class, and wherein each said feature vector is defined by a d-dimensional vector of numerical features, wherein said numerical features are extracted from digital signals; receiving within said computer system unknown feature vectors related to said M data sets; determining M ensembles of M1 discriminant functions of M1 minimum risk quadratic classification systems using said M data sets, wherein the determination of each one of said M ensembles comprises the steps of: determining M1 kernel matrices for a class of feature vectors using said M data sets, said determination of said M-1 kernel matrices being performed by using processors of said computer system to calculate M1 matrices, wherein each matrix contains all possible inner products of signed reproducing kernels of feature vectors that belong to said class and one of the other M-1 classes, wherein a reproducing kernel of a feature vector replaces said feature vector with a curve that contains first and second degree vector components, and wherein said N feature vectors that belong to said class have the sign +1, and said N feature vectors that belong to said other class have the sign 1, and wherein said M1 matrices account for all of the other said M1 classes, and calculating M1 regularized kernel matrices from said M1 kernel matrices; determining M1 discriminant functions of M1 minimum risk quadratic classification systems using said M1 regularized kernel matrices and said M data sets, wherein the determination of each one of said M1 discriminant functions of M1 minimum risk quadratic classification systems further comprises the steps of: determining scale factors of a geometric locus of signed and scaled reproducing kernels of extreme points using one of said regularized kernel matrices, wherein said extreme points are located within overlapping regions or near tail regions of distributions of feature vectors that belong to said class and one of the other said M1 classes, said determination of said scale factors being performed by using said processors of said computer system to determine a solution of a dual optimization problem, wherein said scale factors and said geometric locus satisfy a system of fundamental locus equations of binary classification, subject to geometric and statistical conditions for a minimum risk quadratic classification system in statistical equilibrium, and wherein said scale factors determine conditional densities for said extreme points and also determine critical minimum eigenenergies exhibited by scaled extreme vectors on said geometric locus, wherein said critical minimum eigenenergies determine conditional probabilities of said extreme points and also determine corresponding counter risks and risks of a minimum risk quadratic classification system, wherein said counter risks are associated with right decisions and said risks are associated with wrong decisions of said minimum risk quadratic classification system, and wherein said geometric locus determines the principal eigenaxis of the decision boundary of said minimum risk quadratic classification system, wherein said principal eigenaxis exhibits symmetrical dimensions and density, wherein said conditional probabilities and said critical minimum eigenenergies exhibited by said minimum risk quadratic classification system are symmetrically concentrated within said principal eigenaxis, and wherein counteracting and opposing components of said critical minimum eigenenergies exhibited by corresponding components of said scaled extreme vectors on said geometric locus together with corresponding counter risks and risks exhibited by said minimum risk quadratic classification system are symmetrically balanced with each other about the geometric center of said principal eigenaxis, wherein the center of total allowed eigenenergy and minimum expected risk of said minimum risk quadratic classification system is located at the geometric center of said geometric locus, and wherein said geometric locus determines a primal representation of a dual locus of likelihood components and principal eigenaxis components, wherein said likelihood components and said principal eigenaxis components are symmetrically distributed over either side of the axis of said dual locus, wherein a statistical fulcrum is placed directly under the center of said dual locus, and wherein said likelihood components of said dual locus determine conditional likelihoods for said extreme points, and wherein said principal eigenaxis components of said dual locus determine an intrinsic coordinate system of geometric loci of a quadratic decision boundary and corresponding decision borders that jointly partition the decision space of said minimum risk quadratic classification system into symmetrical decision regions; determining said extreme vectors on said geometric locus using the vector of said scale factors, said determination of said extreme vectors being performed by using said processors of said computer system to identify said scale factors that exceed zero by a small threshold, and using said processors of said computer system to determine a sign vector of signs associated with said extreme vectors using data set of said class and data set of said other class, and compute the average sign using said sign vector; determining a locus of risk for said minimum risk quadratic classification system using said reproducing kernels of said extreme vectors and said signed reproducing kernels of said N feature vectors and said vector of scale factors, said determination of said locus of risk being performed by using said processors of said computer system to calculate a matrix of inner products between said signed reproducing kernels of said N feature vectors and said reproducing kernels of said extreme vectors, and multiply said matrix by said vector of scale factors, and compute the average risk for said minimum risk quadratic classification system using said locus of risk; determining a discriminant locus for said minimum risk quadratic classification system using said geometric locus, said determination of said discriminant locus being performed by using said processors of said computer system to calculate a matrix of inner products between said signed reproducing kernels of said feature vectors that belong to said class and said other class and said reproducing kernels of said unknown feature vectors, and multiply said matrix by said vector of scale factors; determining the discriminant function of said minimum risk quadratic classification system, using said average risk and said average sign and said discriminant locus, said determination of said discriminant function of said minimum risk quadratic classification system being performed by using said processors of said computer system to subtract said average risk from sum of said discriminant locus and said average sign, wherein said discriminant function of said minimum risk quadratic classification system satisfies said system of fundamental locus equations of binary classification, and wherein said discriminant function of said minimum risk quadratic classification system determines likely locations of said N feature vectors from said class and said N feature vectors from said other class and also determines said geometric loci of said quadratic decision boundary and said corresponding decision borders that jointly partition said extreme points into said symmetrical decision regions, wherein said symmetrical decision regions span said overlapping regions or said tail regions of said distributions of said N feature vectors that belong to said class and said N feature vectors that belong to said other class, and wherein said discriminant function of said minimum risk quadratic classification system satisfies said quadratic decision boundary in terms of a critical minimum eigenenergy and said minimum expected risk, wherein said counteracting and opposing components of said critical minimum eigenenergies exhibited by said corresponding components of said scaled extreme vectors on said geometric locus associated with said corresponding counter risks and risks exhibited by said minimum risk quadratic classification system are symmetrically distributed over said axis of said dual locus, on equal sides of said statistical fulcrum located at said geometric center of said dual locus, wherein said counteracting and opposing components of said critical minimum eigenenergies together with said corresponding counter risks and risks exhibited by said minimum risk quadratic classification system are symmetrically balanced with each other about said geometric center of said dual locus, and wherein said statistical fulcrum is located at said center of said total allowed eigenenergy and said minimum expected risk of said minimum risk quadratic classification system, wherein said minimum risk quadratic classification system satisfies a state of statistical equilibrium, wherein said total allowed eigenenergy and said expected risk of said minimum risk quadratic classification system are minimized, and wherein said minimum risk quadratic classification system exhibits the minimum probability of error for classifying said N feature vectors that belong to said class and said N feature vectors that belong to said other class and said unknown feature vectors related to said data set of said class and said data set of said other class; determining a discriminant function of an M-class minimum risk quadratic classification system using said M ensembles of M1 discriminant functions of M1 minimum risk quadratic classification systems, said determination of said discriminant function of said M-class minimum risk quadratic classification system being performed by using said processors of said computer system to sum said M ensembles of M1 discriminant functions of M1 minimum risk quadratic classification systems; and determining which of said M classes said unknown feature vectors belong to using said discriminant function of said M-class minimum risk quadratic classification system, said determination of said classes of said unknown feature vectors being performed by using said processors of said computer system to apply said discriminant function of said M- class minimum risk quadratic classification system to said unknown feature vectors, wherein said discriminant function determines likely locations of said unknown feature vectors and identifies said decision regions related to said M classes that said unknown feature vectors are located within, wherein said discriminant function recognizes said classes of said unknown feature vectors, and wherein said M-class minimum risk quadratic classification system decides which of said M classes said unknown feature vectors belong to and thereby classifies said unknown feature vectors.
8. The method of claim 7, wherein the reproducing kernel is a Gaussian reproducing kernel: k.sub.x=exp (sx.sup.2):0.010.1.
9. The method of claim 7, wherein the reproducing kernel is a second-order polynomial reproducing kernel: k.sub.x=(s.sup.Tx+1).sup.2.
10. A computer-implemented method of using feature vectors and machine learning algorithms to determine a fused discriminant function of a fused M-class minimum risk quadratic classification system that classifies two types of said feature vectors into M classes, wherein said types of said feature vectors have different numbers of vector components, and using said fused discriminant function of said fused M-class minimum risk quadratic classification system to classify unknown feature vectors related to said two types of said feature vectors, said method comprising: receiving M Nd data sets of feature vectors within a computer system, wherein M is a number of classes, N is a number of feature vectors in each one of said M data sets, d is a number of vector components in each feature vector, and each one of said N feature vectors in each one of said M data sets belongs to the same class and is labeled with information that identifies said class, and wherein each said feature vector is defined by a d-dimensional vector of numerical features, wherein said numerical features are extracted from digital signals; receiving M Np different data sets of different feature vectors within said computer system, wherein M is said number of said classes, N is a number of different feature vectors in each one of said M different data sets, p is a number of vector components in each different feature vector, and each one of said N different feature vectors in each one of said M different data sets belongs to the same class and is labeled with information that identifies said class, and wherein each said different feature vector is defined by a p-dimensional vector of numerical features, wherein said numerical features are extracted from digital signals; receiving within said computer system unknown feature vectors related to said M data sets and unknown different feature vectors related to said M different data sets; determining M ensembles of M1 discriminant functions of M1 minimum risk quadratic classification systems using said M data sets, wherein the determination of each one of said M ensembles comprises the steps of: determining M1 kernel matrices for a class of feature vectors using said M data sets, said determination of said M1 kernel matrices being performed by using processors of said computer system to calculate M1 matrices, wherein each matrix contains all possible inner products of signed reproducing kernels of feature vectors that belong to said class and one of the other M1 classes, wherein a reproducing kernel of a feature vector replaces said feature vector with a curve that contains first and second degree vector components, and wherein said N feature vectors that belong to said class have the sign +1, and said N feature vectors that belong to said other class have the sign 1, and said M1 matrices account for all of the other said M1 classes, and calculating M1 regularized kernel matrices from said M1 kernel matrices; determining M1 discriminant functions of M1 minimum risk quadratic classification systems using said M1 regularized kernel matrices and said M data sets, wherein the determination of each one of said M1 discriminant functions of M1 minimum risk quadratic classification systems further comprises the steps of: determining scale factors of a geometric locus of signed and scaled reproducing kernels of extreme points using one of said regularized kernel matrices, wherein said extreme points are located within overlapping regions or near tail regions of distributions of feature vectors that belong to said class and one of the other said M1 classes, said determination of said scale factors being performed by using said processors of said computer system to determine a solution of a dual optimization problem, wherein said scale factors and said geometric locus satisfy a system of fundamental locus equations of binary classification, subject to geometric and statistical conditions for a minimum risk quadratic classification system in statistical equilibrium, and wherein said scale factors determine conditional densities for said extreme points and also determine critical minimum eigenenergies exhibited by scaled extreme vectors on said geometric locus, wherein said critical minimum eigenenergies determine conditional probabilities of said extreme points and also determine corresponding counter risks and risks of a minimum risk quadratic classification system, wherein said counter risks are associated with right decisions and said risks are associated with wrong decisions of said minimum risk quadratic classification system, and wherein said geometric locus determines the principal eigenaxis of the decision boundary of said minimum risk quadratic classification system, wherein said principal eigenaxis exhibits symmetrical dimensions and density, wherein said conditional probabilities and said critical minimum eigenenergies exhibited by said minimum risk quadratic classification system are symmetrically concentrated within said principal eigenaxis, and wherein counteracting and opposing components of said critical minimum eigenenergies exhibited by corresponding components of said scaled extreme vectors on said geometric locus together with corresponding counter risks and risks exhibited by said minimum risk quadratic classification system are symmetrically balanced with each other about the geometric center of said principal eigenaxis, wherein the center of total allowed eigenenergy and minimum expected risk of said minimum risk quadratic classification system is located at the geometric center of said geometric locus, and wherein said geometric locus determines a primal representation of a dual locus of likelihood components and principal eigenaxis components, wherein said likelihood components and said principal eigenaxis components are symmetrically distributed over either side of the axis of said dual locus, wherein a statistical fulcrum is placed directly under the center of said dual locus, and wherein said likelihood components of said dual locus determine conditional likelihoods for said extreme points, and wherein said principal eigenaxis components of said dual locus determine an intrinsic coordinate system of geometric loci of a quadratic decision boundary and corresponding decision borders that jointly partition the decision space of said minimum risk quadratic classification system into symmetrical decision regions; determining said extreme vectors on said geometric locus using the vector of said scale factors, said determination of said extreme vectors being performed by using said processors of said computer system to identify said scale factors that exceed zero by a small threshold, and using said processors of said computer system to determine a sign vector of signs associated with said extreme vectors using data set of said class and data set of said other class, and compute the average sign using said sign vector; determining a locus of risk for said minimum risk quadratic classification system using said reproducing kernels of said extreme vectors and said signed reproducing kernels of said N feature vectors and said vector of scale factors, said determination of said locus of risk being performed by using said processors of said computer system to calculate a matrix of inner products between said signed reproducing kernels of said N feature vectors and said reproducing kernels of said extreme vectors, and multiply said matrix by said vector of scale factors, and compute the average risk for said minimum risk quadratic classification system using said locus of risk; determining a discriminant locus for said minimum risk quadratic classification system using said geometric locus, said determination of said discriminant locus being performed by using said processors of said computer system to calculate a matrix of inner products between said signed reproducing kernels of said feature vectors that belong to said class and said other class and said reproducing kernels of said unknown feature vectors, and multiply said matrix by said vector of scale factors; determining the discriminant function of said minimum risk quadratic classification system, using said locus of average risk and said average sign and said discriminant locus, said determination of said discriminant function of said minimum risk quadratic classification system being performed by using said processors of said computer system to subtract said average risk from sum of said discriminant locus and said average sign, wherein said discriminant function of said minimum risk quadratic classification system satisfies said system of fundamental locus equations of binary classification, and wherein said discriminant function of said minimum risk quadratic classification system determines likely locations of said N feature vectors from said class and said N feature vectors from said other class and also determines said geometric loci of said quadratic decision boundary and said corresponding decision borders that jointly partition said extreme points into said symmetrical decision regions, wherein said symmetrical decision regions span said overlapping regions or said tail regions of said distributions of said N feature vectors that belong to said class and said N feature vectors that belong to said other class, and wherein said discriminant function of said minimum risk quadratic classification system satisfies said quadratic decision boundary in terms of a critical minimum eigenenergy and said minimum expected risk, wherein said counteracting and opposing components of said critical minimum eigenenergies exhibited by said corresponding components of said scaled extreme vectors on said geometric locus associated with said corresponding counter risks and risks exhibited by said minimum risk quadratic classification system are symmetrically distributed over said axis of said dual locus, on equal sides of said statistical fulcrum located at said geometric center of said dual locus, wherein said counteracting and opposing components of said critical minimum eigenenergies together with said corresponding counter risks and risks exhibited by said minimum risk quadratic classification system are symmetrically balanced with each other about said geometric center of said dual locus, and wherein said statistical fulcrum is located at said center of said total allowed eigenenergy and said minimum expected risk of said minimum risk quadratic classification system, wherein said minimum risk quadratic classification system satisfies a state of statistical equilibrium, wherein said total allowed eigenenergy and said expected risk of said minimum risk quadratic classification system are minimized, and wherein said minimum risk quadratic classification system exhibits the minimum probability of error for classifying said N feature vectors that belong to said class and said N feature vectors that belong to said other class and said unknown feature vectors related to said data set of said class and said data set of said other class; determining M different ensembles of M1 different discriminant functions of M1 different minimum risk quadratic classification systems using said M different data sets, said determination of said M different ensembles of M1 different discriminant functions of M1 different minimum risk quadratic classification systems being performed by performing said steps of determining M ensembles of M1 discriminant functions of M1 minimum risk quadratic classification systems; determining a fused discriminant function of a fused M-class minimum risk quadratic classification system using said M ensembles of M1 discriminant functions of M1 minimum risk quadratic classification systems and said M different ensembles of M1 different discriminant functions of M1 different minimum risk quadratic classification systems, said determination of said fused discriminant function of said fused M-class minimum risk quadratic classification system being performed by using said processors of said computer system to sum said M ensembles of M1 discriminant functions of M1 minimum risk quadratic classification systems and said M different ensembles of M1 different discriminant functions of M1 different minimum risk quadratic classification systems; and determining which of said M classes said unknown feature vectors and said unknown different feature vectors belong to using said fused discriminant function of said fused M-class minimum risk quadratic classification system, said determination of said classes of said unknown feature vectors and said unknown different feature vectors being performed by using said processors of said computer system to apply said fused discriminant function of said fused M-class minimum risk quadratic classification system to said unknown feature vectors and said unknown different feature vectors, wherein said fused discriminant function determines likely locations of said unknown feature vectors and said unknown different feature vectors and identifies said decision regions related to said M classes that said unknown feature vectors and said unknown different feature vectors are located within, wherein said fused discriminant function recognizes said classes of said unknown feature vectors and said unknown different feature vectors, and wherein said fused M-class minimum risk quadratic classification system decides which of said M classes said unknown feature vectors and said unknown different feature vectors belong to and thereby classifies said unknown feature vectors and said unknown different feature vectors.
11. The method of claim 10, wherein the reproducing kernel is a Gaussian reproducing kernel: k.sub.x=exp(sx.sup.2): 0.010.1.
12. The method of claim 10, wherein the reproducing kernel is a second-order polynomial reproducing kernel: k.sub.x=(s.sup.Tx+1).sup.2.
13. A computer-implemented method of using feature vectors and machine learning algorithms to determine a discriminant function of a minimum risk quadratic classification system that classifies said feature vectors into two classes and using said discriminant function of said minimum risk quadratic classification system to determine a classification error rate and a measure of overlap between distributions of said feature vectors, said method comprising: receiving an Nd data set of feature vectors within a computer system, wherein N is a number of feature vectors, d is a number of vector components in each feature vector, and each one of said N feature vectors is labeled with information that identifies which of two classes each one of said N feature vectors belongs to, and wherein each said feature vector is defined by a d-dimensional vector of numerical features, wherein said numerical features are extracted from digital signals; receiving an Nd test data set of test feature vectors related to said data set within said computer system, wherein N is a number of test feature vectors, d is a number of vector components in each test feature vector, and each one of said N test feature vectors is labeled with information that identifies which of said two classes each one of said N test feature vectors belongs to; determining a kernel matrix using said data set, said determination of said kernel matrix being performed by using processors of said computer system to calculate a matrix of all possible inner products of signed reproducing kernels of said N feature vectors, wherein a reproducing kernel of a feature vector replaces said feature vector with a curve that contains first and second degree vector components, and wherein each one of said reproducing kernels of said N feature vectors has a sign of +1 or 1 that identifies which of said two classes each one of said N feature vectors belongs to, and using said processors of said computer system to calculate a regularized kernel matrix from said kernel matrix; determining scale factors of a geometric locus of signed and scaled reproducing kernels of extreme points using said regularized kernel matrix, wherein said extreme points are located within overlapping regions or near tail regions of distributions of said N feature vectors, said determination of said scale factors being performed by using said processors of said computer system to determine a solution of a dual optimization problem, wherein said scale factors and said geometric locus satisfy a system of fundamental locus equations of binary classification, subject to geometric and statistical conditions for a minimum risk quadratic classification system in statistical equilibrium, and wherein said scale factors determine conditional densities for said extreme points and also determine critical minimum eigenenergies exhibited by scaled extreme vectors on said geometric locus, wherein said critical minimum eigenenergies determine conditional probabilities of said extreme points and also determine corresponding counter risks and risks of a minimum risk quadratic classification system, wherein said counter risks are associated with right decisions and said risks are associated with wrong decisions of said minimum risk quadratic classification system, and wherein said geometric locus determines the principal eigenaxis of the decision boundary of said minimum risk quadratic classification system, wherein said principal eigenaxis exhibits symmetrical dimensions and density, wherein said conditional probabilities and said critical minimum eigenenergies exhibited by said minimum risk quadratic classification system are symmetrically concentrated within said principal eigenaxis, and wherein counteracting and opposing components of said critical minimum eigenenergies exhibited by corresponding components of said scaled extreme vectors on said geometric locus together with said corresponding counter risks and risks exhibited by said minimum risk quadratic classification system are symmetrically balanced with each other about the geometric center of said principal eigenaxis, wherein the center of total allowed eigenenergy and minimum expected risk of said minimum risk quadratic classification system is located at the geometric center of said geometric locus, and wherein said geometric locus determines a primal representation of a dual locus of likelihood components and principal eigenaxis components, wherein said likelihood components and said principal eigenaxis components are symmetrically distributed over either side of the axis of said dual locus, wherein a statistical fulcrum is placed directly under the center of said dual locus, and wherein said likelihood components of said dual locus determine conditional likelihoods for said extreme points, and wherein said principal eigenaxis components of said dual locus determine an intrinsic coordinate system of geometric loci of a quadratic decision boundary and corresponding decision borders that jointly partition the decision space of said minimum risk quadratic classification system into symmetrical decision regions; determining said extreme vectors on said geometric locus using the vector of said scale factors, said determination of said extreme vectors being performed by using said processors of said computer system to identify said scale factors that exceed zero by a small threshold, and using said processors of said computer system to determine a sign vector of signs associated with said extreme vectors using said data set, and compute the average sign using said sign vector; determining a locus of risk for said minimum risk quadratic classification system using said reproducing kernels of said extreme vectors and said signed reproducing kernels of said N feature vectors and said vector of scale factors, said determination of said locus of risk being performed by using said processors of said computer system to calculate a matrix of inner products between said signed reproducing kernels of said N feature vectors and said reproducing kernels of said extreme vectors, and multiply said matrix by said vector of scale factors, and compute the average risk for said minimum risk quadratic classification system using said locus of risk; determining a discriminant locus for said minimum risk quadratic classification system using said geometric locus, said determination of said discriminant locus being performed by using said processors of said computer system to calculate a matrix of inner products between said signed reproducing kernels of said N feature vectors and said reproducing kernels of said N feature vectors and said N test feature vectors, and multiply said matrix by said vector of scale factors; determining the discriminant function of said minimum risk quadratic classification system, using said average risk and said average sign and said discriminant locus, said determination of said discriminant function of said minimum risk quadratic classification system being performed by using said processors of said computer system to subtract said average risk from sum of said discriminant locus and said average sign, wherein said discriminant function of said minimum risk quadratic classification system satisfies said system of fundamental locus equations of binary classification, and wherein said discriminant function of said minimum risk quadratic classification system determines likely locations of said N feature vectors and said N test feature vectors and also determines said geometric loci of said quadratic decision boundary and said corresponding decision borders that jointly partition said extreme points into said symmetrical decision regions, wherein said symmetrical decision regions span said overlapping regions or said tail regions of said distributions of said N feature vectors, and wherein said discriminant function of said minimum risk quadratic classification system satisfies said quadratic decision boundary in terms of a critical minimum eigenenergy and said minimum expected risk, wherein said counteracting and opposing components of said critical minimum eigenenergies exhibited by said corresponding components of said scaled extreme vectors on said geometric locus associated with said corresponding counter risks and risks exhibited by said minimum risk quadratic classification system are symmetrically distributed over said axis of said dual locus, on equal sides of said statistical fulcrum located at said geometric center of said dual locus, wherein said counteracting and opposing components of said critical minimum eigenenergies together with said corresponding counter risks and risks exhibited by said minimum risk quadratic classification system are symmetrically balanced with each other about said geometric center of said dual locus, and wherein said statistical fulcrum is located at said center of said total allowed eigenenergy and said minimum expected risk of said minimum risk quadratic classification system, wherein said minimum risk quadratic classification system satisfies a state of statistical equilibrium, wherein said total allowed eigenenergy and said expected risk of said minimum risk quadratic classification system are minimized, and wherein said minimum risk quadratic classification system exhibits the minimum probability of error for classifying said N feature vectors and said N test feature vectors related to said data set; determining which of said two classes said N feature vectors belong to using said discriminant function of said minimum risk quadratic classification system, said determination of said classes of said N feature vectors being performed by using said processors of said computer system to apply said discriminant function of said minimum risk quadratic classification system to said N feature vectors, wherein said discriminant function determines likely locations of said N feature vectors and identifies said decision regions related to said two classes that said N feature vectors are located within, wherein said discriminant function recognizes said classes of said N feature vectors, and wherein said minimum risk quadratic classification system decides which of said two classes said N feature vectors belong to belong to and thereby classifies said N feature vectors; determining an in-sample classification error rate for said two classes of feature vectors, said determination of said error rate being performed by using said processors of said computer system to calculate the average number of wrong decisions made by said minimum risk quadratic classification system for classifying said N features vectors; determining which of said two classes said N test feature vectors belong to using said discriminant function of said minimum risk quadratic classification system, said determination of said classes of said N test feature vectors being performed by using said processors of said computer system to apply said discriminant function of said minimum risk quadratic classification system to said N test feature vectors, wherein said discriminant function determines likely locations of said N test feature vectors and identifies said decision regions related to said two classes that said N test feature vectors are located within, wherein said discriminant function recognizes said classes of said N test feature vectors, and wherein said minimum risk quadratic classification system decides which of said two classes said N test feature vectors belong to and thereby classifies said N test feature vectors; determining an out-of-sample classification error rate for said two classes of feature vectors, said determination of said error rate being performed by using said processors of said computer system to calculate the average number of wrong decisions made by said minimum risk quadratic classification system for classifying said N test features vectors; determining a classification error rate for said two classes of feature vectors, said determination of said classification error rate being performed by using said processors of said computer system to average said in-sample classification error rate and said out-of-sample classification error rate; and determining a measure of overlap between distributions of feature vectors for said two classes of feature vectors using said N feature vectors and said extreme vectors, said determination of said measure of overlap being performed by using said processors of said computer system to calculate the ratio of the number of said extreme vectors to the number of said N feature vectors, wherein said ratio determines said measure of overlap.
14. The method of claim 13, wherein the reproducing kernel is a Gaussian reproducing kernel: k.sub.x=exp(sx.sup.2): 0.01y0.1.
15. The method of claim 13, wherein the reproducing kernel is a second-order polynomial reproducing kernel: k.sub.x=(s.sup.Tx+1).sup.2.
16. A computer-implemented method of using feature vectors and machine learning algorithms to determine a discriminant function of a minimum risk quadratic classification system that classifies collections of said feature vectors into two classes and using said discriminant function of said minimum risk quadratic classification system to determine if distributions of said collections of said feature vectors are homogenous distributions, said method comprising: receiving an Nd data set of feature vectors within a computer system, wherein N is a number of feature vectors, d is a number of vector components in each feature vector, and each one of said N feature vectors is labeled with information that identifies which of two collections each one of said N feature vectors belongs to, and wherein each said feature vector is defined by a d-dimensional vector of numerical features, wherein said numerical features are extracted from digital signals; determining a kernel matrix using said data set, said determination of said kernel matrix being performed by using processors of said computer system to calculate a matrix of all possible inner products of signed reproducing kernels of said N feature vectors, wherein a reproducing kernel of a feature vector replaces said feature vector with a curve that contains first and second degree vector components, and wherein each one of said reproducing kernels of said N feature vectors has a sign of +1 or 1 that identifies which of said two collections each one of said N feature vectors belongs to, and using said processors of said computer system to calculate a regularized kernel matrix from said kernel matrix; determining scale factors of a geometric locus of signed and scaled reproducing kernels of extreme points using said regularized kernel matrix, wherein said extreme points are located within overlapping regions or near tail regions of distributions of said N feature vectors, said determination of said scale factors being performed by using said processors of said computer system to determine a solution of a dual optimization problem, wherein said scale factors and said geometric locus satisfy a system of fundamental locus equations of binary classification, subject to geometric and statistical conditions for a minimum risk quadratic classification system in statistical equilibrium, and wherein said scale factors determine conditional densities for said extreme points and also determine critical minimum eigenenergies exhibited by scaled extreme vectors on said geometric locus, wherein said critical minimum eigenenergies determine conditional probabilities of said extreme points and also determine corresponding counter risks and risks of a minimum risk quadratic classification system, wherein said counter risks are associated with right decisions and said risks are associated with wrong decisions of said minimum risk quadratic classification system, and wherein said geometric locus determines the principal eigenaxis of the decision boundary of said minimum risk quadratic classification system, wherein said principal eigenaxis exhibits symmetrical dimensions and density, wherein said conditional probabilities and said critical minimum eigenenergies exhibited by said minimum risk quadratic classification system are symmetrically concentrated within said principal eigenaxis, and wherein counteracting and opposing components of said critical minimum eigenenergies exhibited by corresponding components of said scaled extreme vectors on said geometric locus together with corresponding counter risks and risks exhibited by said minimum risk quadratic classification system are symmetrically balanced with each other about the geometric center of said principal eigenaxis, wherein the center of total allowed eigenenergy and minimum expected risk of said minimum risk quadratic classification system is located at the geometric center of said geometric locus, and wherein said geometric locus determines a primal representation of a dual locus of likelihood components and principal eigenaxis components, wherein said likelihood components and said principal eigenaxis components are symmetrically distributed over either side of the axis of said dual locus, wherein a statistical fulcrum is placed directly under the center of said dual locus, and wherein said likelihood components of said dual locus determine conditional likelihoods for said extreme points, and wherein said principal eigenaxis components of said dual locus determine an intrinsic coordinate system of geometric loci of a quadratic decision boundary and corresponding decision borders that jointly partition the decision space of said minimum risk quadratic classification system into symmetrical decision regions; determining said extreme vectors on said geometric locus using the vector of said scale factors, said determination of said extreme vectors being performed by using said processors of said computer system to identify said scale factors that exceed zero by a small threshold, and using said processors of said computer system to determine a sign vector of signs associated with said extreme vectors using said data set, and compute the average sign using said sign vector; determining a locus of risk for said minimum risk quadratic classification system using said reproducing kernels of said extreme vectors and said signed reproducing kernels of said N feature vectors and said vector of scale factors, said determination of said locus of risk being performed by using said processors of said computer system to calculate a matrix of inner products between said signed reproducing kernels of said N feature vectors and said reproducing kernels of said extreme vectors, and multiply said matrix by said vector of scale factors, and compute the average risk for said minimum risk quadratic classification system using said locus of risk; determining a discriminant locus for said minimum risk quadratic classification system using said geometric locus, said determination of said discriminant locus being performed by using said processors of said computer system to calculate a matrix of inner products between said signed reproducing kernels of said N feature vectors and said reproducing kernels of said N feature vectors , and multiply said matrix by said vector of scale factors; determining the discriminant function of said minimum risk quadratic classification system, using said average risk and said average sign and said discriminant locus, said determination of said discriminant function of said minimum risk quadratic classification system being performed by using said processors of said computer system to subtract said average risk from sum of said discriminant locus and said average sign, wherein said discriminant function of said minimum risk quadratic classification system satisfies said system of fundamental locus equations of binary classification, and wherein said discriminant function of said minimum risk quadratic classification system determines likely locations of said N feature vectors and also determines said geometric loci of said quadratic decision boundary and said corresponding decision borders that jointly partition said extreme points into said symmetrical decision regions, wherein said symmetrical decision regions span said overlapping regions or said tail regions of said distributions of said N feature vectors, and wherein said discriminant function of said minimum risk quadratic classification system satisfies said quadratic decision boundary in terms of a critical minimum eigenenergy and said minimum expected risk, wherein said counteracting and opposing components of said critical minimum eigenenergies exhibited by said corresponding components of said scaled extreme vectors on said geometric locus associated with said corresponding counter risks and risks exhibited by said minimum risk quadratic classification system are symmetrically distributed over said axis of said dual locus, on equal sides of said statistical fulcrum located at said geometric center of said dual locus, wherein said counteracting and opposing components of said critical minimum eigenenergies together with said corresponding counter risks and risks exhibited by said minimum risk quadratic classification system are symmetrically balanced with each other about said geometric center of said dual locus, and wherein said statistical fulcrum is located at said center of said total allowed eigenenergy and said minimum expected risk of said minimum risk quadratic classification system, wherein said minimum risk quadratic classification system satisfies a state of statistical equilibrium, wherein said total allowed eigenenergy and said expected risk of said minimum risk quadratic classification system are minimized, and wherein said minimum risk quadratic classification system exhibits the minimum probability of error for classifying said N feature vectors that belong to said two collections of said feature vectors; determining which of said two collections said N feature vectors belong to using said discriminant function of said minimum risk quadratic classification system, said determination of said collections of said N feature vectors being performed by using said processors of said computer system to apply said discriminant function of said minimum risk quadratic classification system to said N feature vectors, wherein said discriminant function determines likely locations of said N feature vectors and identifies said decision regions related to said two collections that said N feature vectors are located within, wherein said discriminant function recognizes said collections of said N feature vectors, and wherein said minimum risk quadratic classification system decides which of said two collections said N feature vectors belong to belong to and thereby classifies said N feature vectors; determining an in-sample classification error rate for said two collections of feature vectors, said determination of said error rate being performed by using said processors of said computer system to calculate the average number of wrong decisions made by said minimum risk quadratic classification system for classifying said N features vectors; determining a measure of overlap between said distributions of said N feature vectors for said two collections of feature vectors using said N feature vectors and said extreme vectors, said determination of said measure of overlap being performed by using said processors of said computer system to calculate the ratio of the number of said extreme vectors to the number of said N feature vectors, wherein said ratio determines said measure of overlap; and determining if said distributions of said two collections of said N feature vectors are homogenous distributions using said in-sample classification error rate and said measure of overlap, wherein said distributions of said N feature vectors are homogenous distributions if said measure of overlap has an approximate value of one and said in-sample classification error rate has an approximate value of one half.
17. The method of claim 16, wherein the reproducing kernel is a Gaussian reproducing kernel: k.sub.x=exp(sx.sup.2): 0.01y0.1.
18. The method of claim 16, wherein the reproducing kernel is a second-order polynomial reproducing kernel: k.sub.x=(s.sup.Tx+1).sup.2.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
(1)
(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9)
(10)
(11)
(12)
DETAILED DESCRIPTION OF THE INVENTION
(13) Before describing illustrative embodiments of the invention, a detailed description of machine learning algorithms of the invention is presented along with a detailed description of the novel principal eigenaxis that determines a discriminant function of a minimum risk quadratic classification system.
(14) The method to determine a discriminant function of a minimum risk quadratic classification system that classifies feature vectors into two categories, designed in accordance with the invention, uses machine learning algorithms and labeled feature vectors to determine a geometric locus of signed and scaled reproducing kernels of extreme points for feature vectors x of dimension d belonging to either of two classes A or B, wherein the geometric locus satisfies a system of fundamental locus equations of binary classification, subject to geometric and statistical conditions for a minimum risk quadratic classification system in statistical equilibrium.
(15) The input to a machine learning algorithm of the invention is a collection of N feature vectors x.sub.i with labels y.sub.i
(x.sub.1,(x.sub.2,y.sub.2), . . . ,(x.sub.N,y.sub.N)
wherein y.sub.i=+1 if x.sub.iA and y.sub.i=1 if x.sub.iB, and wherein the N feature vectors are extracted from collections of digital signals.
(16) Denote a minimum risk quadratic classification system of the invention by
(17)
wherein A or B is the true category. The discriminant function D(s)=k.sub.s+.sub.D of the minimum risk quadratic classification system is represented by a novel principal eigenaxis that is expressed as a dual locus of likelihood components and principal eigenaxis components and is determined by a geometric locus of signed and scaled reproducing kernels of extreme points:
(18)
wherein k.sub.x.sub.
(19)
and wherein the scale factors .sub.1i* and .sub.2i* determine magnitudes .sub.1i*k.sub.x.sub.
(20)
wherein is a d1 geometric locus of signed and scaled reproducing kernels of extreme points that determines the principal eigenaxis of the decision boundary of a minimum risk quadratic classification system, wherein is expressed as a dual locus of likelihood components and principal eigenaxis components, and wherein k.sub.x.sub..sub.min (Z|.sub.min.sub.
(21)
wherein the system of N inequalities:
y.sub.i(k.sub.x.sub.
is satisfied in a suitable manner, and wherein the dual locus of satisfies a critical minimum eigenenergy constraint:
()=.sub.min.sub.
wherein the total allowed eigenenergy |Z|.sub.min.sub..sub.min (Z|.sub.min.sub.
(22)
wherein the objective function and its constraints are combined with each other, that is minimized with respect to the primal variables and .sub.0, and is maximized with respect to the dual variables .sub.i. The Lagrange multipliers method introduces a Wolfe dual geometric locus that is symmetrically and equivalently related to the primal geometric locus and finds extrema for the restriction of the primal geometric locus to a Wolfe dual principal eigenspace.
(23) The fundamental unknowns associated with the primal optimization problem in Eq. (1.1) are the scale factors .sub.i of the principal eigenaxis components
(24)
on the geometric locus of a Wolfe dual principal eigenaxis . Each active scale factor .sub.i determines a conditional density and a corresponding conditional likelihood for a reproducing kernel of an extreme point on a dual locus of likelihood components, and each active scale factor .sub.i determines the magnitude and the critical minimum eigenenergy exhibited by a scaled extreme vector on a dual locus of principal eigenaxis components.
(25) The Karush-Kuhn-Tucker (KKT) conditions on the Lagrangian function L.sub.() in Eq. (1.2)
(26)
determine a system of fundamental locus equations of binary classification, subject to geometric and statistical conditions for a minimum risk quadratic classification system in statistical equilibrium, that are jointly satisfied by the geometric locus of the principal eigenaxis and the geometric locus of the principal eigenaxis .
(27) Because the primal optimization problem in Eq. (1.1) is a convex optimization problem, the inequalities in Eqs (1.6) and (1.7) must only hold for certain values of the primal and the dual variables. The KKT conditions in Eqs (1.3)-(1.7) restrict the magnitudes and the eigenenergies of the principal eigenaxis components on both and , wherein the expected risk .sub.min (Z|.sub.min.sub.
|.sub.min.sub.
(28) Substituting the expressions for and in Eqs (1.3) and (1.4) into the Lagrangian functional L.sub.() of Eq. (1.2) and simplifying the resulting expression determines the Lagrangian dual problem:
(29)
wherein is subject to the constraints
(30)
and .sub.i0, and wherein .sub.ij is the Kronecker defined as unity for i=j and 0 otherwise.
(31) Equation (1.8) is a quadratic programming problem that can be written in vector notation by letting QI+{tilde over (X)}{tilde over (X)}.sup.T, wherein {tilde over (X)}
D.sub.yX, wherein D.sub.y is a NN diagonal matrix of training labels (class membership statistics) y.sub.i, and wherein the Nd matrix {tilde over (X)} is a matrix of labeled reproducing kernels of N feature vectors:
{tilde over (X)}=(y.sub.1k.sub.x.sub.
(32) The matrix version of the Lagrangian dual problem, which is also known as the Wolfe dual problem:
(33)
is subject to the constraints .sup.Ty=0 and .sub.i0, wherein the inequalities .sub.i0 only hold for certain values of .sub.i.
(34) Because Eq. (1.9) is a convex programming problem, the theorem for convex duality guarantees an equivalence and a corresponding symmetry between the dual loci of and . Accordingly, the geometric locus of the principal eigenaxis determines a dual locus of likelihood components and principal eigenaxis components, wherein the expected risk .sub.min (Z|.sub.min.sub.
.sub.min (Z|.sub.min.sub.
.sub.min (Z|.sub.min.sub.
.sub.min (Z|.sub.min.sub.
(35) The locations and the scale factors of the principal eigenaxis components on both and are considerably affected by the rank and the eigenspectrum of the kernel matrix Q, wherein a low rank kernel matrix Q determines an unbalanced principal eigenaxis and an irregular quadratic partition of a decision space. The kernel matrix Q has low rank, wherein d<N for a collection of N feature vectors of dimension d. These problems are solved by the following regularization method.
(36) The regularized form of Q, wherein <<1 and QI+{tilde over (X)}{tilde over (X)}.sup.T, ensures that Q has full rank and a complete eigenvector set, wherein Q has a complete eigenspectrum. The regularization constant C is related to the regularization parameter by 1/C. For N feature vectors of dimension d, wherein d<N, all of the regularization parameters {.sub.i}.sub.i=1.sup.N in Eq. (1.1) and all of its derivatives are set equal to a very small value: .sub.i=<<1, e.g. .sub.i==0.02. The regularization constant C is set equal to
(37)
(38) For N feature vectors of dimension d, wherein N<d, all of the regularization parameters {.sub.i}.sub.i=1.sup.N in Eq. (1.1) and all of its derivatives are set equal to zero: .sub.i==0.
(39) The regularization constant C is set equal to infinity: C=.
(40) The KKT conditions in Eqs (1.3) and (1.6) require that the geometric locus of the principal eigenaxis satisfy the vector expression:
(41)
wherein .sub.i0 and reproducing kernels k.sub.x of feature vectors x.sub.i correlated with Wolfe dual principal eigenaxis components
(42)
that have non-zero magnitudes .sub.i>0 are termed extreme vectors.
(43) Denote the scaled extreme vectors that belong to class A and class B by .sub.1i*k.sub.x.sub.
(44) Using Eq. (1.10), the class membership statistics and the assumptions outlined above, it follows that the geometric locus of the principal eigenaxis is determined by the vector difference between a pair of sides, i.e., a pair of directed line segments:
(45)
wherein .sub.1 and .sub.2 denote the sides of , wherein the side .sub.1 is determined by the vector expression
(46)
and the side of .sub.2 is determined by the vector expression
(47)
and wherein the geometric locus of the principal eigenaxis is determined by the vector difference of .sub.1 and .sub.2.
(48) All of the principal eigenaxis components .sub.1i*k.sub.x.sub.
(49)
determine an intrinsic coordinate system of geometric loci of a quadratic decision boundary and corresponding decision borders.
(50)
(51)
(52)
(53)
(54)
(55) The manner in which a discriminate function of the invention partitions the feature space Z=Z.sub.1+Z.sub.2 of a minimum risk quadratic classification system for a collection of N feature vectors is determined by the KKT condition in Eq. (1.7) and the KKT condition of complementary slackness.
(56) The KKT condition in Eq. (1.7) and the KKT condition of complementary slackness determine a discriminant function
D(s)=k.sub.s+.sub.0(1.12)
that satisfies the set of constraints: D(s)=0, D(s)=+1, and D(s)=1,
wherein D(s)=0 denotes a quadratic decision boundary that partitions the Z.sub.1 and Z.sub.2 decision regions of a minimum risk quadratic classification system
(57)
and wherein D(s)=+1 denotes the quadratic decision border for the Z.sub.1 decision region, and wherein D(s)=1 denotes the quadratic decision border for the Z.sub.2 decision region.
(58) The KKT condition in Eq. (1.7) and the KKT condition of complementary slackness also determines the following system of locus equations that are satisfied by .sub.0 and :
y.sub.i(k.sub.x.sub.
wherein .sub.0 satisfies the functional of in the following manner:
(59)
(60) Using Eqs (1.12) and (1.13), the discriminant function is rewritten as:
(61)
(62) Using Eq. (1.14) and letting D(s)=0, the discriminant function is rewritten as
(63)
wherein the constrained discriminant function D(s)=0 determines a quadratic decision boundary, and all of the points s on the quadratic decision boundary D(s)=0 exclusively reference the principal eigenaxis of .
(64) Using Eq. (1.14) and letting D(s)=+1, the discriminant function is rewritten as
(65)
wherein the constrained discriminant function D(s)=+1 determines a quadratic decision border, and all of the points s on the quadratic decision border D(s)=+1 exclusively reference the principal eigenaxis of .
(66) Using Eq. (1.14) and letting D(s)=1, the discriminant function is rewritten as
(67)
wherein the constrained discriminant function D(s)=1 determines a quadratic decision border, and all of the points s on the quadratic decision border D(s)=1 exclusively reference the principal eigenaxis of .
(68) Given Eqs (1.15)-(1.17), it follows that a constrained discriminant function of the invention
(69)
determines geometric loci of a quadratic decision boundary D(s)=0 and corresponding decision borders D(s)=+1 and D(s)=1 that jointly partition the decision space Z of a minimum risk quadratic classification system
(70)
into symmetrical decision regions Z.sub.1 and Z.sub.2:Z=Z.sub.1+Z.sub.2:Z.sub.1Z.sub.2wherein balanced portions of the extreme points x.sub.1i* and x.sub.2i* from class A and class Baccount for right and wrong decisions of the minimum risk quadratic classification system.
(71) Therefore, the geometric locus of the principal eigenaxis determines an eigenaxis of symmetry
(72)
for the decision space of a minimum risk quadratic classification system, wherein a constrained discriminant function delineates symmetrical decision regions Z.sub.1 and Z.sub.2:Z.sub.1Z.sub.2 for the minimum risk quadratic classification system
(73)
wherein the decision regions Z.sub.1 and Z.sub.2 are symmetrically partitioned by the quadratic decision boundary of Eq. (1.15), and wherein the span of the decision regions is regulated by the constraints on the corresponding decision borders of Eqs (1.16)-(1.17).
(74)
(75) Substitution of the vector expressions for and .sub.0 in Eqs (1.11) and (1.13) into the expression for the discriminant function in Eq. (1.12) determines an expression for a discriminant function of a minimum risk quadratic classification system that classifies feature vectors s into two classes A and B:
(76)
wherein feature vectors s belong to or are related to a collection of N feature vectors {k.sub.x.sub.
(77)
determines the average locus of the l extreme vectors {k.sub.x.sub.
(78)
accounts for class memberships of the principal eigenaxis components on .sub.1 and .sub.2. The average locus
(79)
determines the average risk for the decision space Z=Z.sub.1+Z.sub.2 of the minimum risk quadratic classification system
(80)
wherein the vector difference
(81)
determines the distance between a feature vector s and the locus of average risk Let s denote an unknown feature vector related to a collection of N feature vectors {x.sub.i}.sub.i=1.sup.N that are inputs to one of the machine learning algorithms of the invention, wherein each feature vector x.sub.i has a label y.sub.i wherein y.sub.i=+1 if x.sub.iA and y.sub.i=1 if x.sub.iB, and wherein a discriminant function of a minimum risk quadratic classification system has been determined. Now take any given unknown feature vector s.
(82) The discriminant function
(83)
of Eq. (1.18) determines the likely location of the unknown feature vector s, wherein the likely location of s is determined by the vector projection of
(84)
onto the dual locus of likelihood components and principal eigenaxis components .sub.1-.sub.2:
(85)
wherein the component of
(86)
along the dual locus of .sub.1.sub.2:
(87)
determines the signed magnitude
(88)
along the axis of .sub.1.sub.2, where is the angle between the transformed unknown feature vector
(89)
and .sub.1.sub.2, and wherein the decision region that the unknown feature vector s is located within is determined by the sign of the expression:
(90)
(91) Therefore, the likely location of the unknown feature vector s is determined by the scalar value of
(92)
along the axis of the dual locus .sub.i.sub.2, wherein the scalar value of the expression
(93)
indicates the decision region Z.sub.1 or Z.sub.2 that the unknown feature vector s is located withinalong with the corresponding class of s.
(94) Thus, if:
(95)
then the unknown feature vector s is located within region Z.sub.1 and sA, whereas if
(96)
then the unknown feature vectors s is located within region Z.sub.2 and sB.
(97) The minimum risk quadratic classification system of the invention decides which of the two classes A or B that the unknown feature vector s belongs to according to the sign of +1 or 1 that is output by the signum function:
(98)
and thereby classifies the unknown feature vector s.
(99) Thus, the discriminant function of the invention in Eq. (1.18) determines likely locations of each one of the feature vectors x.sub.i that belong to a collection of N feature vectors {x.sub.i}.sub.i=1.sup.N and any given unknown feature vectors s related to the collection, wherein the feature vectors are inputs to one of the machine learning algorithms of the invention and a discriminant function of a minimum risk quadratic classification system has been determined.
(100) Further, the discriminant function identifies the decision regions Z.sub.1 and Z.sub.2 related to the two classes A and B that each one of the N feature vectors x.sub.i and the unknown feature vectors s are located within, wherein the discriminant function recognizes the classes of each one of the N feature vectors x.sub.i and each one of the unknown feature vectors s, and the minimum risk quadratic classification system of the invention in Eq. (1.19) decides which of the two classes that each one of the N feature vectors x.sub.i and each one of the unknown feature vectors s belong to and thereby classifies the collection of N feature vectors{x.sub.t}N.sub.i and any given unknown feature vectors s.
(101) Therefore, discriminant functions of the invention exhibit a novel and useful property, wherein, for any given collection of feature vectors that belong to two classes and are inputs to a machine learning algorithm of the invention, the discriminant function that is determined by the machine learning algorithm determines likely locations of each one of the feature vectors that belong to the given collection of feature vectors and any given unknown feature vectors related to the collection, and identifies the decision regions related to the two classes that each one of the feature vectors and each one of the unknown feature vectors are located within, wherein the discriminant function recognizes the classes of the feature vectors and the unknown feature vectors according to the signs related to the two classes.
(102) The likelihood components and the corresponding principal eigenaxis components .sub.1i*k.sub.x.sub.
(103)
wherein the scale factors .sub.1i* and .sub.2i* of the geometric locus determine magnitudes .sub.1i*k.sub.x.sub.
(104) Scale factors are determined by finding a satisfactory solution for the Lagrangian dual optimization problem in Eq. (1.9), wherein finding a geometric locus of signed and scaled reproducing kernels of extreme points involves optimizing a vector-valued cost function with respect to constraints on the scaled extreme vectors on the dual loci of and , wherein the constraints are specified by the KKT conditions in Eqs (1.3)-(1.7).
(105) The Wolfe dual geometric locus of scaled extreme points on is determined by the largest eigenvector .sub.max of the kernel matrix Q associated with the quadratic form .sub.max.sup.TQ.sub.max in Eq. (1.9), wherein .sup.Ty=0, .sub.i*>0, and wherein .sub.max is the principal eigenaxis of an implicit quadratic decision boundaryassociated with the constrained quadratic form .sub.max.sup.TQ.sub.maxwithin the Wolfe dual principal eigenspace of , wherein the form of the inner product statistics contained within the kernel matrix Q determines an intrinsic coordinate system of the intrinsic quadratic decision boundary.
(106) Further, the intrinsic coordinate system of the intrinsic quadratic decision boundary of Eq. (1.9) is an inherent function of inner product statistics between feature vectors k.sub.x.sub.
(107) The theorem for convex duality indicates that the principal eigenaxis of satisfies a critical minimum eigenenergy constraint that is symmetrically and equivalently related to the critical minimum eigenenergy constraint on the principal eigenaxis of , within the Wolfe dual principal eigenspace of and : Z|.sub.min.sub.
max .sub.max.sup.TQ.sub.max=.sub.max.sub.
and the functional 1.sup.T.sup.TQ/2 in Eq. (1.9) is maximized by the largest eigenvector .sub.max of Q, wherein the constrained quadratic form .sup.TQ/2, wherein .sup.T.sub.maxy=0 and .sub.i*>0, reaches its smallest possible value. It follows that the principal eigenaxis components on satisfy minimum length constraints.
(108) The principal eigenaxis components on also satisfy an equilibrium constraint. The KKT condition in Eq. (1.4) requires that the magnitudes of the principal eigenaxis components on the dual locus of satisfy the locus equation:
(109)
wherein Eq. (1.20) determines the Wolf dual equilibrium point:
(110)
of a minimum risk quadratic classification system, wherein the critical minimum eigenenergies exhibited by the principal eigenaxis of are symmetrically concentrated.
(111) Given Eq. (1.21), it follows that the integrated lengths of the Wolfe dual principal eigenaxis components correlated with each class balance each other, wherein the principal eigenaxis of is in statistical equilibrium:
(112)
(113) Now, each scale factor .sub.1i* or .sub.2i* is correlated with a respective extreme vector k.sub.x.sub.
(114)
wherein .sub.1 and .sub.2 denote the sides of the dual locus of , wherein the side .sub.1 is determined by the vector expression
(115)
and wherein the side of .sub.2 is determined by the vector expression
(116)
(117) The system of locus equations in Eqs (1.20)-(1.23) demonstrates that the principal eigenaxis of is determined by a geometric locus of scaled, unit extreme vectors from class A and class B, wherein all of the scaled, unit extreme vectors on .sub.1 and .sub.2 are symmetrically distributed over either side of the geometric locus of the principal eigenaxis , wherein a statistical fulcrum is placed directly under the center of the principal eigenaxis of .
(118) Using Eq. (1.22) and Eq. (1.23), it follows that the length .sub.1, of .sub.1 is equal to the length .sub.2 of .sub.2: .sub.1=.sub.2. It also follows that the total allowed eigenenergies Z|.sub.1.sub.min.sub.
(119) The equilibrium constraint on the geometric locus of the principal eigenaxis in Eq. (1.20) ensures that the critical minimum eigenenergies exhibited by all of the principal eigenaxis components on .sub.1 and .sub.2 are symmetrically concentrated within the principal eigenaxis of :
(120)
(121) Using Eq. (1.24), it follows that the principal eigenaxis of .sub.lf satisfies a state of statistical equilibrium, wherein all of the principal eigenaxis components on are equal or in correct proportions, relative to the center of , wherein components of likelihood components and corresponding principal eigenaxis components of class Aalong the axis of .sub.1are symmetrically balanced with components of likelihood components and corresponding principal eigenaxis components of class Balong the axis of .sub.2.
(122) Therefore, the principal eigenaxis of determines a point at which the critical minimum eigenenergies exhibited by all of the scaled, unit extreme vectors from class A and class B are symmetrically concentrated, wherein the total allowed eigenenergy Z|.sub.min.sub.
(123) The scale factors are associated with the fundamental unknowns of the constrained optimization problem in Eq. (1.1). Now, the geometric locus of the principal eigenaxis can be written as
(124)
wherein each scale factor .sub.j is correlated with scalar projections
(125)
of a feature vector k.sub.x.sub.
(126) Further, given a kernel matrix of all possible inner products of reproducing kernels of a collection of N feature vectors {.sub.i}.sub.i=1.sup.N, the pointwise covariance statistic (k.sub.x.sub.
(127)
determines a unidirectional estimate of the joint variations between the random variables of each feature vector k.sub.x.sub.
(128)
and the feature vector k.sub.x.sub.
(129) Let i=1: l.sub.1, where each extreme vector k.sub.x.sub.
(130)
on .sub.1. Now take the extreme vector k.sub.x.sub.
(131)
Using Eqs (1.25) and (1.26), it follows that the geometric locus of the principal eigenaxis component
(132)
on .sub.1 is determined by the locus equation:
(133)
wherein components of likelihood components and principal eigenaxis components for class Aalong the axis of the extreme vector k.sub.x.sub.
(134)
wherein .sub.1i* determines a scale factor for the extreme vector
(135)
Accordingly, Eq. (1.27) determines a scale factor .sub.1i* for a correlated extreme vector k.sub.x.sub.
(136) Let i=1:l.sub.2, where each extreme vector k.sub.x.sub.
(137)
on .sub.2. Now take the extreme vector k.sub.x.sub.
(138)
Using Eqs (1.25) and (1.26), it follows that the geometric locus of the principal eigenaxis component
(139)
on .sub.2 is determined by the locus equation:
(140)
wherein components of likelihood components and principal eigenaxis components for class Balong the axis of the extreme vector k.sub.x.sub.
(141)
wherein .sub.2i* determines a scale factor for the extreme vector
(142)
Accordingly, Eq. (1.28) determines a scale factor .sub.2i* for a correlated extreme vector k.sub.x.sub.
(143) Given the pointwise covariance statistic in Eq. (1.26), it follows that Eq. (1.27) and Eq. (1.28) determine the manner in which the first and second order vector components of a set of l scaled extreme vectors {.sub.j*k.sub.x.sub.
(144)
or
(145)
and (3) a symmetrically balanced distribution of l scaled extreme vectors {.sub.k*k.sub.x.sub.
(146) Accordingly, the geometric locus of each principal eigenaxis component
(147)
or
(148)
on the geometric locus of the principal eigenaxis determines the manner in which the first and second order vector components of an extreme vector k.sub.x.sub.
(149)
wherein each scaled extreme vector has a sign on +1 or 1.
(150) It follows that the geometric locus of each principal eigenaxis component
(151)
or
(152)
on the geometric locus of the principal eigenaxis determines a conditional distribution of first and second degree coordinates for a correlated extreme point k.sub.x.sub.
(153)
determines a pointwise conditional density estimate
(154)
for the correlated extreme point k.sub.x.sub.
(155)
and wherein
(156)
determines a pointwise conditional density estimate
(157)
for the correlated extreme point k.sub.x.sub.
(158)
(159) Thus, each scale factor .sub.1i* or .sub.2i* determines a conditional density and a corresponding conditional likelihood for a correlated extreme point k.sub.x.sub.
(160) Therefore, conditional densities and corresponding conditional likelihoods .sub.1i*k.sub.x.sub.
(161)
wherein .sub.1i*k.sub.x.sub.
.sub.1=p(k.sub.x.sub.
wherein the area .sub.1i*k.sub.x.sub.
(162)
(163) Likewise, conditional densities and corresponding conditional likelihoods .sub.2i*k.sub.x.sub.
(164)
wherein .sub.2i*k.sub.x.sub.
.sub.2=p(k.sub.x.sub.
wherein the area .sub.2i*k.sub.x.sub.
(165)
(166) The integral of a conditional density function p (k.sub.x.sub.
(167)
over the decision space Z=Z.sub.i+Z.sub.2 of a minimum risk quadratic classification system, determines the conditional probability P (k.sub.x.sub.(Z.sub.1|.sub.1i*k.sub.x.sub.
.sub.min (Z.sub.1|.sub.1i*k.sub.x.sub.
(168)
for expected risks
(169)
of making decision errors.
(170) Accordingly, all of the scaled extreme vectors .sub.1i*k.sub.x.sub. for obtaining expected risks of making decision errors or costs C
for obtaining expected counter risks of making correct decisions.
(171) Therefore, the conditional probability function P(k.sub.1i*|.sub.1) for class A is given by the integral
P(k.sub.x.sub.
over the decision space Z=Z.sub.1+Z.sub.2 of a minimum risk quadratic classification system, wherein the integral of Eq. (1.29) has a solution in terms of the critical minimum eigenenergy Z|.sub.1.sub.min.sub.
(172) The integral of a conditional density function p(k.sub.x.sub.
(173)
over the decision space Z=Z.sub.i+Z.sub.2 of a minimum risk quadratic classification system, determines the conditional probability P(k.sub.x.sub.(Z.sub.1|.sub.2i*k.sub.x.sub.
.sub.min (Z.sub.1|.sub.2i*k.sub.x.sub.
(Z.sub.2|.sub.2i*k.sub.x.sub.
.sub.min (Z.sub.2|.sub.2i*k.sub.x.sub.
(174) Accordingly, all of the scaled extreme vectors .sub.2i*k.sub.x.sub. for obtaining expected risks of making decision errors or costs C
for obtaining expected counter risks of making correct decisions.
(175) Therefore, the conditional probability function P(k.sub.x.sub.
P(k.sub.x.sub.
over the decision space Z=Z.sub.1+Z.sub.2 of a minimum risk quadratic classification system, wherein the integral of Eq. (1.30) has a solution in terms of the critical minimum eigenenergy Z|.sub.2.sub.min.sub.
(176) Machine learning algorithms of the present invention find the right mix of principal eigenaxis components on the dual loci of and by accomplishing an elegant, statistical balancing feat within the Wolfe dual principal eigenspace of and . The scale factors {.sub.i*}.sub.i=1.sup.l of the principal eigenaxis components on play a fundamental role in the statistical balancing feat.
(177) Using Eq. (1.27), the integrated lengths
(178)
of the principal eigenaxis components on .sub.1 satisfy the identity:
(179)
and, using Eq. (1.28), the integrated lengths
(180)
of the principal eigenaxis components on .sub.2 satisfy the identity:
(181)
(182) Returning to Eq. (1.22), wherein the principal eigenaxis of is in statistical equilibrium, it follows that the RHS of Eq. (1.31) equals the RHS of Eq. (1.32):
(183)
wherein components of all of the extreme vectors and k.sub.x.sub.
(184)
wherein components of extreme vectors k.sub.x.sub.
(185) Using Eq. (1.33), it follows that components
(186)
of extreme vectors k.sub.x.sub.
(187)
of extreme vectors k.sub.x.sub.
(188)
of extreme vectors k.sub.x.sub.
(189)
of extreme vectors k.sub.x.sub.
(190)
wherein counteracting and opposing components of likelihoods of extreme vectors k.sub.x.sub.
(191) Now rewrite Eq. (1.33) as:
(192)
wherein components of all of the extreme vectors k.sub.x.sub.
(193)
wherein components of likelihoods of extreme vectors k.sub.x.sub.
(194)
that satisfy suitable length constraints, wherein the principal eigenaxis of and the principal eigenaxis of are both formed by symmetrical distributions of likelihoods of extreme vectors k.sub.x.sub.
(195) Given Eqs (1.33) and (1.34), it follows that the locus equation
(196)
determines the primal equilibrium point of a minimum risk quadratic classification systemwithin a Wolfe dual principal eigenspacewherein the form of Eq. (1.35) is determined by geometric and statistical conditions that are satisfied by the dual loci of and .
(197) A discriminant function of the invention satisfies the geometric locus of a quadratic decision boundary of a minimum risk quadratic classification system in terms of the critical minimum eigenenergy Z|.sub.min.sub..sub.min (Z|.sub.min.sub.
.sub.min (Z|.sub.min.sub.
.sub.min (Z|.sub.min.sub.
(198) The KKT condition in Eq. (1.7) on the Lagrangian function in Eq. (1.2) and the theorem of Karush, Kuhn, and Tucker determine the manner in which a discriminant function of the invention satisfies the geometric loci of the quadratic decision boundary in Eq. (1.15) and the quadratic decision borders in Eqs (1.16) and (1.17).
(199) Accordingly, given a Wolfe dual geometric locus of scaled unit extreme vectors
(200)
wherein {.sub.i*>0}.sub.i=1.sup.l and .sub.i=1.sup.l.sub.i*y.sub.i=0, it follows that the l likelihood components and corresponding principal eigenaxis components {.sub.i*k.sub.x.sub.
(201)
within the Wolfe dual principal eigenspace of the minimum risk quadratic classification system, wherein either .sub.i==0 or .sub.i=<<1, e.g. .sub.i==0.02.
(202) Take the set {.sub.1i*k.sub.x.sub.
(203)
wherein the constrained discriminant function k.sub.s+.sub.0=+1 satisfies the geometric locus of the quadratic decision border in Eq. (1.16) in terms of the critical minimum eigenenergy Z|.sub.1.sub.min.sub..sub.min (Z|.sub.1.sub.min.sub.
(204)
within the Wolfe dual principal eigenspace of the dual locus of .sub.1.sub.2, and wherein .sub.1 and .sub.1 are symmetrically and equivalently related to each other within the Wolfe dual-principal eigenspace.
(205) Take the set {.sub.2i*k.sub.x.sub.
(206)
wherein the constrained discriminant function k.sub.s+.sub.0=1 satisfies the geometric locus of the quadratic decision border in Eq. (1.17) in terms of the critical minimum eigenenergy Z|.sub.2.sub.min.sub..sub.min (Z|.sub.2.sub.min.sub.
(207)
within the Wolfe dual principal eigenspace of the dual locus of .sub.1.sub.2, and wherein .sub.2 and .sub.2 are symmetrically and equivalently related to each other within the Wolfe dual-principal eigenspace.
(208) Summation over the complete system of locus equations that are satisfied by .sub.1
(209)
and by .sub.2
(210)
and using the equilibrium constraint on the dual locus of in Eq. (1.22), wherein the principal eigenaxis of is in statistical equilibrium, produces the identity that determines the total allowed eigenenergy Z|.sub.min.sub..sub.min(Z|.sub.min.sub.
(211)
wherein the constrained discriminant function k.sub.s+.sub.0=0 satisfies the geometric locus of the quadratic decision boundary in Eq. (1.15) in terms of the critical minimum eigenenergy Z|.sub.1.sub.2.sub.min.sub..sub.min(Z|.sub.1.sub.2.sub.min.sub.
(212)
within the Wolfe dual principal eigenspace of the dual locus of .sub.1.sub.2, and wherein the dual loci of and are symmetrically and equivalently related to each other within the Wolfe dual-principal eigenspace.
(213) Given Eq. (1.39), it follows that the total allowed eigenenergy Z|.sub.1.sub.2.sub.min.sub..sub.min (Z|.sub.1.sub.2.sub.min.sub.
(214)
wherein regularization parameters .sub.i=<<1 determine negligible constraints on the minimum expected risk .sub.min (Z|.sub.1.sub.2.sub.min.sub.
(215) Now, take any given collection {x.sub.i}.sub.i=1.sup.N of feature vectors .sub.i that are inputs to one of the machine learning algorithm of the invention, wherein each feature vector .sub.i has a label y.sub.i wherein y.sub.i=+1 if .sub.iA and y.sub.i=1 if .sub.iB.
(216) The system of locus equations in Eqs (1.37)-(1.39) determines the manner in which a constrained discriminant function of the invention satisfies parametric, primary and secondary integral equations of binary classification over the decision space of a minimum risk quadratic classification system of the invention. The primary integral equation is devised first.
(217) Using Eq. (1.11), Eq. (1.13), Eq. (1.22) and Eqs (1.37)-(1.39), it follows that the constrained discriminant function
(218)
satisfies the locus equations
(219)
and
(220)
over the decision regions Z.sub.1 and Z.sub.2 of the decision space Z of the minimum risk quadratic classification system
(221)
wherein the parameters
(222)
and
(223)
are equalizer statistics.
(224) Using Eqs (1.40) and (1.41) along with the identity in Eq. (1.31)
(225)
and the identity in Eq. (1.32)
(226)
it follows that the constrained discriminant function satisfies the locus equation over the decision regions Z.sub.1 and Z.sub.2 of the decision space Z of the minimum risk quadratic classification system:
(227)
wherein both the left-hand side and the right-hand side of Eq. (1.42) satisfy half the total allowed eigenenergy Z|.sub.1.sub.2.sub.min.sub..sub.min (Z|.sub.1.sub.2.sub.min.sub.
(228)
(229) Returning to the integral in Eq. (1.29):
P(k.sub.x.sub.
wherein the above integral determines a conditional probability P (k.sub.x.sub.
P(k.sub.x.sub.
wherein the above integral determines a conditional probability P (k.sub.x.sub.
(230) Substituting the value for C.sub.1 into Eq. (1.29), and using Eq. (1.29) and Eq. (1.42), it follows that the conditional probability P(k.sub.x.sub.
(231)
over the decision space Z=Z.sub.1+Z.sub.2 of the minimum risk quadratic classification system, is determined by half the total allowed eigenenergy
(232)
and half the minimum expected risk
(233)
that is exhibited by the dual locus of =.sub.1.sub.2.
(234) Substituting the value for C.sub.2 into Eq. (1.30), and using Eq. (1.30) and Eq. (1.42), it follows that the conditional probability P (k.sub.x.sub.
(235)
over the decision space Z=Z.sub.1+Z.sub.2 of the minimum risk quadratic classification system, is determined by half the total allowed eigenenergy
(236)
and half the minimum expected risk
(237)
that is exhibited by the dual locus of =.sub.1.sub.2.
(238) Given Eqs (1.43) and (1.44), it follows that the integral of the conditional density function p(k.sub.x.sub.
(239)
and half the minimum expected risk
(240)
that is exhibited by the minimum risk quadratic classification system
(241)
(242) Therefore, the conditional probability P(k.sub.x.sub.
(243) Therefore, minimum risk quadratic classification systems of the invention exhibit a novel property of computer-implemented quadratic classification systems, wherein for any given collection of feature vectors {x.sub.i}.sub.i=1.sup.N that are inputs to one of the machine learning algorithms of the invention: (1) the conditional probability, (2) the minimum expected risk, and (3) the total allowed eigenenergy exhibited by a minimum risk quadratic classification system for class A is equal to (1) the conditional probability, (2) the minimum expected risk, and (3) the total allowed eigenenergy exhibited by the minimum risk quadratic classification system for class B.
(244) Using Eqs (1.43) and (1.44), it follows that the constrained discriminant function of the invention
(245)
is the solution of the parametric, fundamental integral equation of binary classification:
(246)
over the decision space Z=Z.sub.1+Z.sub.2 of the minimum risk quadratic classification system
(247)
of the invention, wherein the decision space Z is spanned by symmetrical decision regions Z.sub.1+Z.sub.2=Z:Z.sub.1Z.sub.2, and wherein the conditional probability P (Z.sub.1|.sub.1) and the counter risk .sub.min(Z.sub.1|.sub.1.sub.min.sub.
.sub.min(Z.sub.2|.sub.1.sub.min.sub.
.sub.min (Z.sub.1|.sub.2.sub.min.sub.
.sub.min(Z.sub.2|.sub.2.sub.min.sub.
.sub.min (Z|.sub.1.sub.2.sub.min.sub.
(248)
and the Wolfe dual equilibrium point:
(249)
of the integral equation f.sub.1 (D(s)).
(250) Further, the novel principal eigenaxis of the invention that determines discriminant functions of the invention along with minimum risk quadratic classification systems of the inventionsatisfies the law of cosines in the symmetrically balanced manner that is outlined below.
(251) Any given geometric locus of signed and scaled reproducing kernels of extreme points:
(252)
wherein the geometric locus of a principal eigenaxis determines a dual locus of likelihood components and principal eigenaxis components =.sub.1.sub.2 that represents a discriminant function D(s)=k.sub.s+.sub.0 of the invention, wherein principal eigenaxis components and corresponding likelihood components .sub.1i*k.sub.x.sub.
(253)
for the decision space Z.sub.1+Z.sub.2=Z: Z.sub.1Z.sub.2 of a minimum risk quadratic classification
(254)
of the invention, satisfies the law of cosines
(255)
in the symmetrically balanced manner:
(256)
wherein is the angle between is .sub.1 and .sub.2, and wherein the dual locus of likelihood components and principal eigenaxis components exhibits symmetrical dimensions and density, wherein the total allowed eigenenergy .sub.1.sub.min.sub.
.sub.1.sub.min.sub.
wherein the length of side .sub.1 equals the length of side .sub.2
.sub.1=.sub.2,
and wherein components of likelihood components and principal eigenaxis components of class Aalong the axis of .sub.1are symmetrically balanced with components of likelihood components and principal eigenaxis components of class Balong the axis of .sub.2:
(257)
wherein components of critical minimum eigenenergies exhibited by corresponding components of scaled extreme vectors from class A and corresponding counter risks and risks for class Aalong the axis of .sub.1, are symmetrically balanced with components of critical minimum eigenenergies exhibited by corresponding components of scaled extreme vectors from class B and corresponding counter risks and risks for class Balong the axis of .sub.2, and wherein the opposing component of .sub.2along the axis of .sub.1, is symmetrically balanced with the opposing component of .sub.1along the axis of .sub.2:
.sub.1[.sub.2cos .sub..sub.
wherein opposing components of likelihood components and principal eigenaxis components of class Balong the axis of .sub.1, are symmetrically balanced with opposing components of likelihood components and principal eigenaxis components of class Aalong the axis of .sub.2:
(258)
wherein opposing components of critical minimum eigenenergies exhibited by corresponding components of scaled extreme vectors from class B and corresponding counter risks and risks for class Balong the axis of is .sub.1, are symmetrically balanced with opposing components of critical minimum eigenenergies exhibited by corresponding components of scaled extreme vectors from class A and corresponding counter risks and risks for class Aalong the axis of .sub.2, and wherein opposing and counteracting random forces and influences of the minimum risk quadratic classification system of the invention are symmetrically balanced with each otherabout the geometric center of the dual locus :
(259)
(260) Accordingly, counteracting and opposing components of critical minimum eigenenergies exhibited by corresponding components of all of the scaled extreme vectors on the geometric locus of the principal eigenaxis =.sub.1.sub.2 of the invention, along the axis of the principal eigenaxis , and corresponding counter risks and risks exhibited by the minimum risk quadratic classification system
(261)
of the invention, are symmetrically balanced with each other about the geometric center of the dual locus , wherein the statistical fulcrum of is located.
(262) Now, take the previous collection {x.sub.i}.sub.i=1.sup.N of labeled feature vectors x.sub.i that are inputs to one of the machine learning algorithm of the invention, wherein each feature vector x.sub.i has a label y.sub.i wherein y.sub.i=+1 if x.sub.iA and y.sub.i=1 if .sub.iB.
(263) Given that a constrained discriminant function of the invention
(264)
is the solution of the parametric, fundamental integral equation of binary classification in Eq. (1.45), and given that the discriminant function is represented by a dual locus of likelihood components and principal eigenaxis components =.sub.1.sub.2 that satisfies the law of cosines in the symmetrically balanced manner outlined above, it follows that the constrained discriminant function satisfies the parametric, secondary integral equation of binary classification:
(265)
over the decision regions Z.sub.1 and Z.sub.2 of a minimum risk quadratic classification system, wherein opposing and counteracting random forces and influences of the minimum risk quadratic classification system are symmetrically balanced with each otherwithin the decision regions Z.sub.1 and Z.sub.2in the following manners: (1) the eigenenergy Z.sub.1|1.sub.min.sub..sub.min(Z.sub.1|.sub.1.sub.min.sub.
.sub.min (Z.sub.1|.sub.2.sub.min.sub.
.sub.min (Z.sub.2|.sub.2.sub.min.sub.
.sub.min(Z.sub.2|.sub.1.sub.min.sub.
.sub.min (Z.sub.1|.sub.1.sub.min.sub.
.sub.min (Z.sub.1|.sub.2.sub.min.sub.
.sub.min (Z.sub.2|.sub.2.sub.min.sub.
.sub.min (Z.sub.2|.sub.1.sub.min.sub.
.sub.min (Z|.sub.1.sub.2.sub.min.sub.
(266) Therefore, minimum risk quadratic classification systems of the invention exhibit a novel and useful property, wherein for any given collection of labeled feature vectors that are inputs to a machine learning algorithm of the invention, the minimum risk quadratic classification system determined by the machine learning algorithm satisfies a state of statistical equilibrium, wherein the expected risk and the total allowed eigenenergy exhibited by the minimum risk quadratic classification system are minimized, and the minimum risk quadratic classification system exhibits the minimum probability of error for classifying the collection of feature vectors and feature vectors related to the collection into two classes.
(267) Further, discriminant functions of minimum risk quadratic classification systems of the invention exhibit a novel and useful property, wherein a discriminant function D(s) of a minimum risk quadratic classification system is determined by a linear combination of a collection of extreme vectors k.sub.x.sub.
(268)
wherein the collection of extreme vectors {k.sub.x.sub.
(269)
wherein the output of the minimum risk quadratic classification system sign (D(s)) is related to the two classes, and wherein the minimum risk quadratic classification system sign (D(s)) exhibits the minimum probability of error for classifying feature vectors that belong to or are related to the collection of feature vectors used to determine the system sign (D(s)).
(270) Therefore, discriminant functions D(s) of a minimum risk quadratic classification system sign(D(s)) provide scalable modules that can be used to determine ensembles
(271)
of discriminant functions of minimum risk quadratic classification systems, wherein an ensemble of M1 discriminant functions of M1 minimum risk quadratic classification systems exhibits the minimum probability of error for classifying feature vectors that belong to or are related to M given collections of feature vectors.
(272) More specifically, discriminant functions of minimum risk quadratic classification systems provide scalable modules that are used to determine a discriminant function of an M-class minimum risk quadratic classification system that classifies feature vectors into M classes, wherein the total allowed eigenenergy and the minimum expected risk that is exhibited by the M-class minimum risk quadratic classification system is determined by the total allowed eigenenergy and the minimum expected risk that is exhibited by M ensembles of M1 discriminant functions of M1 minimum risk quadratic classification systems
(273)
wherein each minimum risk quadratic classification system sign(D.sub.ij(s)) of an ensemble
(274)
for a given class c.sub.i exhibits a total allowed eigenenergy and a minimum expected risk for a given collection of feature vectors, and wherein the total allowed eigenenergy and the expected risk that is exhibited by the ensemble E.sub.c.sub.
(275) It follows that discriminant functions of M-class minimum risk quadratic classification systems that are determined by machine learning algorithms of the invention exhibit the minimum probability of error for classifying feature vectors that belong to M collections of feature vectors and unknown feature vectors related to the M collections of feature vectors.
(276) It immediately follows that discriminant functions of minimum risk quadratic classification systems of the invention also provide scalable modules that are used to determine a fused discriminant function of a fused minimum quadratic classification system that classifies two types of feature vectors into two classes, wherein each type of feature vector has a different number of vector components. The total allowed eigenenergy and the minimum expected risk exhibited by the fused minimum risk quadratic classification system is determined by the total allowed eigenenergy and the minimum expected risk that is exhibited by an ensemble of a discriminant function of a minimum risk quadratic classification system sign (D(s)) and a different discriminant function of a different minimum risk quadratic classification system
(277)
wherein the total allowed eigenenergy and the expected risk exhibited by the fused minimum risk quadratic classification system is minimum for a given collection of feature vectors and a given collection of different feature vectors.
(278) Any given fused discriminant function of a fused minimum risk quadratic classification system
(279)
that is determined by a machine learning algorithm of the invention exhibits the minimum probability of error for classifying feature vectors that belong to or are related to a collection of feature vectors as well as different feature vectors that belong to or are related to a collection of different feature vectors.
(280) Discriminant functions of minimum risk quadratic classification systems of the invention also provide scalable modules that are used to determine a fused discriminant function of a fused M-class minimum risk quadratic classification system that classifies two types of feature vectors into M classes, wherein each type of feature vector has a different number of vector components, and wherein the total allowed eigenenergy and the minimum expected risk exhibited by the fused M-class minimum risk quadratic classification system is determined by the total allowed eigenenergy and the minimum expected risk that is exhibited by M ensembles of M1 discriminant functions of M1 minimum risk quadratic classification systems
(281)
and M different ensembles of M1 different discriminant functions of M1 different minimum risk quadratic classification systems
(282)
(283)
and wherein the total allowed eigenenergy and the expected risk exhibited by the fused M-class minimum risk quadratic classification system is minimum for M given collections of feature vectors and M given collections of different feature vectors.
(284) Therefore, fused discriminant functions of fused M-class minimum risk quadratic classification systems that are determined by machine learning algorithms of the invention exhibit the minimum probability of error for classifying feature vectors that belong to M collections of feature vectors and unknown feature vectors related to the M collections of feature vectors as well as different feature vectors that belong to M collections of different feature vectors and unknown different feature vectors related to the M collections of different feature vectors.
(285) Further, given that discriminant functions of the invention determine likely locations of feature vectors that belong to given collections of feature vectors and any given unknown feature vectors related to a given collection, wherein a given collection of feature vectors belong to two classes, and given that discriminant functions of the invention identify decision regions related to two classes that given collections of feature vectors and any given unknown feature vectors related to a given collection are located within, and given that discriminant functions of the invention recognize classes of feature vectors that belong to given collections of feature vectors and any given unknown feature vectors related to a given collection, wherein minimum risk quadratic classification systems of the invention decide which of two classes that given collections of feature vectors and any given unknown feature vectors related to a given collection belong to, and thereby classify given collections of feature vectors and any given unknown feature vectors related to a given collection, it follows that discriminant functions of minimum risk quadratic classification systems of the invention can be used to determine a classification error rate and a measure of overlap between distributions of feature vectors for two classes of feature vectors. Further, discriminant functions of minimum quadratic classification systems of the invention can be used to determine if distributions of two collections of feature vectors are homogenous distributions.
Embodiment 1
(286) The method to determine a discriminant function of a minimum risk quadratic classification system that classifies feature vectors into two classes, designed in accordance with the invention, is fully described within the detailed description of the invention.
(287) Receive an N d data set of feature vectors within a computer system wherein N is the number of feature vectors, d is the number of vector components in each feature vector, and each one of the N feature vectors is labeled with information that identifies which of the two classes each one of the N feature vectors belongs to.
(288) Receive unknown feature vectors related to the data set with the computer system.
(289) Choose a reproducing kernel and determine a kernel matrix using the data set by calculating a matrix of all possible inner products of signed reproducing kernels of the N feature vectors, wherein each one of the reproducing kernels of the N feature vectors has a sign of +1 or 1 that identifies which of the two classes each one of the N feature vectors belongs to, and calculate a regularized kernel matrix from the kernel matrix.
(290) Determine the scale factors of a geometric locus of signed and scaled reproducing kernels of extreme points by using the regularized kernel matrix to solve the dual optimization problem in Eq. (1.9).
(291) Determine the extreme vectors on the geometric locus by identifying scale factors in the vector of scale factors that exceed zero by a small threshold T e.g.: T=0.0050.
(292) Determine a sign vector of the signs associated with the extreme vectors using the data set, and compute the average sign using the sign vector.
(293) Determine a locus of risk and compute the average risk using the locus of risk.
(294) Determine a discriminant locus using the N feature vectors and feature vectors being classified to calculate a matrix of inner products between the signed reproducing kernels of the N feature vectors and the reproducing kernels of the feature vectors, and multiply the matrix by the vector of scale factors.
(295) Determine the discriminant function of the minimum risk quadratic classification system, wherein the minimum risk quadratic classification system is determined by computing the sign of the discriminant function, and classify any given unknown feature vectors.
Embodiment 2
(296)
(297) A discriminant function of an M-class minimum risk quadratic classification system that classifies feature vectors into M classes is determined by using a machine learning algorithm of the invention and M collections of N feature vectors, wherein each feature vector in a given collection belongs to the same class, to determine M ensembles of M1 discriminant functions of M1 minimum risk quadratic classification systems, wherein the determination of each one of the M ensembles involves using the machine algorithm to determine M1 discriminant functions of M1 minimum risk quadratic classification systems for a class c.sub.i of feature vectors, wherein the N feature vectors that belong to the class c.sub.i have the sign +1 and all of the N feature vectors that belong to all of the other M1 classes have the sign 1:
(298)
wherein the input of the machine learning algorithm for each discriminant function of a minimum risk quadratic classification system sign (D.sub.ij(s)) is the collection of N feature vectors that belongs to the class c.sub.i and a collection of N feature vectors that belongs to one of the other M1 classes, and wherein the ensemble E.sub.c.sub.
(299)
wherein the discriminant function D.sub.ij (s) discriminates between feature vectors that belong to class i and class j, and wherein the minimum risk quadratic classification system sign(D.sub.ij(s)) decides which of the two classes i or j that a feature vector s belongs to: according to the sign of +1 or 1 that is output by the signum function sign(D.sub.ij(s)), and wherein the output of the minimum risk quadratic classification system of the ensemble E.sub.c.sub.
(300)
(301) Therefore, the M ensembles of the M1 discriminant functions of the M1 minimum risk quadratic classification systems
(302)
determine the discriminant function of an M-class minimum risk quadratic classification system that classifies a feature vector s into the class c.sub.i associated with the ensemble E.sub.c.sub.
(303) The discriminant function of the M-class minimum risk quadratic classification system D.sub.E.sub.
(304)
exhibits the minimum probability of error for classifying feature vectors that belong to the M collections of N feature vectors and unknown feature vectors related to the M collections of N feature vectors, wherein the discriminant function of the M-class minimum risk quadratic classification system function determines likely locations of feature vectors that belong to and are related to the M collections of N feature vectors and identifies decision regions related to the M classes that the feature vectors are located within, wherein the discriminant function recognizes the classes of the feature vectors, and wherein the M-class minimum risk quadratic classification decides which of the M classes that the feature vectors belong to, and thereby classifies the feature vectors.
Embodiment 3
(305) A fused discriminant function of a fused minimum risk quadratic classification system that classifies two types of feature vectors into two classes, wherein the types of feature vectors have different numbers of vector components, is determined by using a machine learning algorithm of the invention and a collection of N feature vectors and a collection of N different feature vectors to determine an ensemble of a discriminant function of a minimum risk quadratic classification system
(306)
and a different discriminant function of a different minimum risk quadratic classification system
(307)
wherein the discriminant function and the different discriminant function are both determined by the process that is described in EMBODIMENT 1.
(308) The fused discriminant function of the fused minimum risk quadratic classification system
(309)
exhibits the minimum probability of error for classifying the feature vectors that belong to the collection of N feature vectors and unknown feature vectors related to the collection of N feature vectors as well as the different feature vectors that belong to the collection of N different feature vectors and unknown different feature vectors related to the collection of N different feature vectors, wherein the fused discriminant function determines likely locations of feature vectors that belong to and are related to the collection of N feature vectors as well as different feature vectors that belong to and are related to the collection of N different feature vectors and identifies decision regions related to the two classes that the feature vectors and the different feature vectors are located within, wherein the fused discriminant function recognizes the classes of the feature vectors and the different feature vectors, and wherein the fused minimum risk quadratic classification decides which of the two classes that the feature vectors and the different feature vectors belong to, and thereby classifies the feature vectors and the different feature vectors.
Embodiment 4
(310)
(311) A fused discriminant function of a fused M-class minimum risk quadratic classification system that classifies two types of feature vectors into M classes is determined by using a machine learning algorithm of the invention and M collections of N feature vectors to determine M ensembles of M1 discriminant functions of M1 minimum risk quadratic classification systems
(312)
as well as M collections of N different feature vectors to determine M different ensembles of M1 different discriminant functions of M1 different minimum risk quadratic classification systems
(313)
wherein the M ensembles and the M different ensembles are both determined by the process that is described in EMBODIMENT 2.
(314) The fused discriminant function of the fused M-class minimum risk quadratic classification system
(315)
exhibits the minimum probability of error for classifying feature vectors that belong to the M collections of N feature vectors and unknown feature vectors related to the M collections of N feature vectors as well as different feature vectors that belong to the M collections of N different feature vectors and unknown different feature vectors related to the M collections of N different feature vectors, wherein the fused discriminant function determines likely locations of feature vectors that belong to and are related to the M collections of N feature vectors as well as different feature vectors that belong to and are related to the M collections of N different feature vectors and identifies decision regions related to the M classes that the feature vectors and the different feature vectors are located within, wherein the fused discriminant function recognizes the classes of the feature vectors and the different feature vectors, and wherein the fused M-class minimum risk quadratic classification decides which of the M classes that the feature vectors and the different feature vectors belong to, and thereby classifies the feature vectors and the different feature vectors.
Embodiment 5
(316)
(317) The process of using a discriminant function of a minimum risk quadratic classification system to determine a classification error rate and a measure of overlap between distributions of feature vectors for two classes of feature vectors involves the following steps:
(318) Receive an N d data set of feature vectors within a computer system, wherein N is the number of feature vectors, d is the number of vector components in each feature vector, and each one of the N feature vectors is labeled with information that identifies which of the two classes each one of the N feature vectors belongs to.
(319) Receive an N d test data set of test feature vectors related to the data set within the computer system, wherein N is a number of test feature vectors, d is a number of vector components in each test feature vector, and each one of the N test feature vectors is labeled with information that identifies which of the two classes each one of the N test feature vectors belongs to.
(320) Determine the discriminant function of the minimum risk quadratic classification system by performing the steps outlined in EMBODIMENT 1.
(321) Use the minimum risk quadratic classification system to classify the N feature vectors.
(322) Determine an in-sample classification error rate for the two classes of feature vectors by calculating the average number of wrong decisions of the minimum risk quadratic classification system for classifying the N features vectors.
(323) Use the minimum risk quadratic classification system to classify the N test feature vectors.
(324) Determine an out-of-sample classification error rate for the two classes of test feature vectors by calculating the average number of wrong decisions of the minimum risk quadratic classification system for classifying the N test feature vectors.
(325) Determine the classification error rate for the two classes of feature vectors by averaging the in-sample classification error rate and the out-of-sample classification error rate.
(326) Determine a measure of overlap between distributions of feature vectors for the two categories of feature vectors using the N feature vectors and the extreme vectors that have been identified, by calculating the ratio of the number of the extreme vectors to the number of the N feature vectors, wherein the ratio determines the measure of overlap.
Embodiment 6
(327)
(328) Receive an N d data set of feature vectors within a computer system, wherein N is the number of feature vectors, d is the number of vector components in each feature vector, and each one of the N feature vectors is labeled with information that identifies which of the two collections each one of the N feature vectors belongs to.
(329) Determine the discriminant function of the minimum risk quadratic classification system by performing the steps outlined in EMBODIMENT 1.
(330) Use the minimum risk quadratic classification system to classify the N feature vectors.
(331) Determine an in-sample classification error rate for the two collections of feature vectors by calculating the average number of wrong decisions of the minimum risk quadratic classification system for classifying the N features vectors.
(332) Determine a measure of overlap between distributions of feature vectors for the two collections of feature vectors using the N feature vectors and the extreme vectors that have been identified, by calculating the ratio of the number of the extreme vectors to the number of the N feature vectors, wherein the ratio determines the measure of overlap.
(333) Determine if the distributions of the two collections of the N feature vectors are homogenous distributions by using the in-sample classification error rate and the measure of overlap, wherein the distributions of the two collections of the N feature vectors are homogenous distributions if the measure of overlap has an approximate value of one and the in-sample classification error rate has an approximate value of one half.
(334) Machine learning algorithms of the invention involve solving certain variants of the inequality constrained optimization problem that is used by support vector machines, wherein regularization parameters and reproducing kernels have been defined.
(335) Software for machine learning algorithms of the invention can be obtained by using any of the software packages that solve quadratic programming problems, or via LIBSVM (A Library for Support Vector Machines), SVMlight (an implementation of SVMs in C) or MATLAB SVM toolboxes.
(336) The machine learning methods of the invention disclosed herein may be readily utilized in a wide variety of applications, wherein feature vectors have been extracted from outputs of sensors that include, but are not limited to radar and hyperspectral or multispectral images, biometrics, digital communication signals, text, images, digital waveforms, etc.
(337) More specifically, the applications include, for example and without limitation, general pattern recognition (including image recognition, waveform recognition, object detection, spectrum identification, and speech and handwriting recognition, data classification, (including text, image, and waveform categorization), bioinformatics (including automated diagnosis systems, biological modeling, and bio imaging classification), etc.
(338) One skilled in the art will recognize that any suitable computer system may be used to execute the machine learning methods disclosed herein. The computer system may include, without limitation, a mainframe computer system, a workstation, a personal computer system, a personal digital assistant, or other device or apparatus having at least one processor that executes instructions from a memory medium.
(339) The computer system may further include a display device or monitor for displaying operations associated with the learning machine and one or more memory mediums on which computer programs or software components may be stored. In addition, the memory medium may be entirely or partially located in one or more associated computers or computer systems which connect to the computer system over a network, such as the Internet.
(340) The machine learning methods described herein may also be executed in hardware, a combination of software and hardware, or in other suitable executable implementations. The learning machine methods implemented in software may be executed by the processor of the computer system or the processor or processors of the one or more associated computer systems connected to the computer system.
(341) While the invention herein disclosed has been described by means of specific embodiments, numerous modifications and variations could be made by those skilled in the art without departing from the scope of the invention set forth in the claims.