Method for Near-Infrared Spectral Wavelength Selection Based on Improved Team Progress Algorithm
20220113250 · 2022-04-14
Inventors
Cpc classification
International classification
Abstract
The disclosure discloses a method for near-infrared spectral wavelength selection based on an improved team progress algorithm (iTPA), belonging to the field of near-infrared spectral detection. The method includes: equally dividing near-infrared spectral wavebands to be selected into elite groups, ordinary groups and garbage collection groups according to evaluation values from high to low; generating a new waveband in the elite group or the ordinary group, where a wavelength point of the new waveband is selected from a random waveband in the selected group, and the wavelength point of the new waveband inherits the selected wavelength point; and enabling the inherited new waveband to select a learning behavior or an exploration behavior according to a set probability to update the wavelength point of the new waveband to generate a candidate waveband; and selecting a waveband with a highest evaluation value in the elite group as the waveband to be selected. Under the condition of ensuring the model prediction accuracy, the disclosure greatly reduces the number of wavelength variables, reduces the complexity of the algorithm at the same time, and improves the non-destructive detection accuracy in crops.
Claims
1. A method for near-infrared spectral wavelength selection based on an improved team progress algorithm (iTPA), comprising: step 1: equally dividing near-infrared spectral wavebands to be selected into P wavebands, and determining an evaluation value of each waveband in the P wavebands, wherein each waveband comprises a plurality of wavelength points, and the P wavebands are all regarded as members in an iTPA model; step 2: dividing the P wavebands into N elite groups, M ordinary groups and L garbage collection groups according to the evaluation values from high to low, wherein N, M and L are integers respectively; step 3: generating a new waveband in the elite group or the ordinary group, wherein a wavelength point of the new waveband is inherited from a wavelength point selected in a random waveband in the group where the new waveband is generated; step 4: enabling the new waveband to select a learning behavior or an exploration behavior according to a preset probability to update the wavelength point of the new waveband to generate a candidate waveband, wherein the sum of probabilities of selecting the learning behavior and the exploration behavior by the new waveband is 100%; step 5: updating the candidate waveband: comparing an evaluation value of the candidate waveband with the waveband values in the elite group, the ordinary group and the garbage collection group respectively to determine whether the candidate waveband enters the elite group, the ordinary group or the garbage collection group; and step 6: customizing a number of iteration updates, and after the iteration, selecting a waveband with a highest evaluation value in the elite group as the waveband to be selected.
2. The method for near-infrared spectral wavelength selection based on an iTPA according to claim 1, wherein the iTPA model is:
3. The method for near-infrared spectral wavelength selection based on an iTPA according to claim 2, wherein the evaluation value f(x) is a function with a root mean square error of calibration (RMSEC) and a correlation coefficient(R) as a variables, where in the RMSEC is obtained by partial least squares (PLS) modeling with a waveband X and measured content physical and chemical values, and a calculation formula of the evaluation value is:
4. The method for near-infrared spectral wavelength selection based on an iTPA according to claim 3, wherein the learning behavior comprises: a new waveband x.sub.r is adjusted to a reference target direction, and the new waveband x.sub.r is generated in the ordinary group or the elite group; the reference target direction of the new waveband generated in the ordinary group is generated from a template value in the elite group; the reference target direction of the new waveband generated in the elite group is generated from a template value in the garbage collection group; the template value comprises: an elite group template e.sub.e and a garbage collection group template e.sub.l; the template value is taken from a mean value of the wavelengths of the wavebands in the group where the template value is located; an expression of generating a candidate waveband by the new waveband x.sub.r generated in the ordinary group after selecting the learning behavior is:
x.sub.c=(1−γ)x.sub.r+γe.sub.e (3); and an expression of generating a candidate waveband by the new waveband x.sub.r generated in the elite group after selecting the learning behavior is:
x.sub.c=(1+γ)x.sub.r−γe.sub.l (4), wherein in Formula, γ represents a random number in an interval [0,1], and x.sub.c represents a candidate waveband.
5. The method for near-infrared spectral wavelength selection based on an iTPA according to claim 4, wherein the new waveband generates a candidate waveband through an exploration behavior, comprising:
6. The method for near-infrared spectral wavelength selection based on an iTPA according to claim 5, wherein before step 1, the method further comprises: eliminating abnormal data in spectra by a Mahalanobis distance, and dividing the remaining sample points into calibration sets and prediction sets by a K-S method after eliminating the abnormal data in the spectra.
7. The method for near-infrared spectral wavelength selection based on an iTPA according to claim 6, wherein the update of the candidate waveband in step 5 comprises: if an evaluation value of the candidate waveband x.sub.c is higher than an evaluation value of the last x.sub.ewst in the elite group, x.sub.c enters the elite group, at the same time, evaluation values in the elite group are sorted again, and the last x.sub.ewst in the elite group after sorting again is directly eliminated without entering the ordinary group; the x.sub.ewst is a waveband with a lowest evaluation value in the elite group; if the evaluation value of the candidate waveband x.sub.c is lower than an evaluation value of the x.sub.ewst but higher than an evaluation value of the last x.sub.pwst in the ordinary group, whether the x.sub.c is obtained by exploration is checked, if the x.sub.c is obtained by exploration, the x.sub.c enters the ordinary group, at the same time, evaluation values in the ordinary group are sorted again, and the last x.sub.pwst in the ordinary group after sorting again is eliminated; the x.sub.pwst is a waveband with a lowest evaluation value in the ordinary group; if the x.sub.c is not obtained by exploration, the x.sub.c is directly discarded; if the evaluation value of the candidate waveband x.sub.c is lower than an evaluation value of a first x.sub.lbst in the garbage collection group, the x.sub.centers the garbage collection group, at the same time, evaluation values in the garbage collection group are sorted again, and a x.sub.lbst waveband in the garbage collection group after sorting again is eliminated to enable the evaluation values in the garbage collection group to be low all the time; the x.sub.lbst is a waveband with a highest evaluation value in the garbage collection group; and after the update of the waveband every time, the evaluation values in the three groups are sorted.
8. The method for near-infrared spectral wavelength selection based on an iTPA according to claim 7, wherein a shrinkage index of the new waveband inherited from the ordinary group is half of that inherited from the elite group.
9. The method for near-infrared spectral wavelength selection based on an iTPA according to claim 8, wherein a probability of selecting the learning behavior by a new member is 0.35, and a probability of selecting the exploration behavior by the new member is 0.65.
10. The method for near-infrared spectral wavelength selection based on an iTPA according to claim 9, wherein the method is applied to wavelength selection in non-destructive detection of crops.
Description
BRIEF DESCRIPTION OF FIGURES
[0038]
[0039]
[0040]
[0041]
[0042]
DETAILED DESCRIPTION
[0043] The disclosure will be further described in detail below with reference to the accompanying drawings and examples.
[0044] Example 1
[0045] As shown in
[0046] The data set includes 80 corn samples which are respectively measured by three spectrometers (m5, mp5, mp6). A wavelength range is 1100 nm to 2498 nm, an interval is 2 nm (700 variable), and the moisture, oil, protein and starch value of each sample are included. The data in the data set is originally collected at Cargill. The experimental data adopts the sample data collected by the device mp5 in the data set and the corresponding protein content value.
[0047] The method includes:
[0048] Step 1: Abnormal points are eliminated, and sample sets are divided.
[0049] Considering that errors will be generated when measuring spectral data by instruments and abnormal spectra will be obtained, which will affect the model performance, the Mahalanobis distance is first used to eliminate the abnormal data in the spectra.
[0050]
TABLE-US-00001 TABLE 1 Protein content values (unit: g, content in per 100 g of corn) in calibration set and prediction set Maximum Minimum Mean Standard value value value deviation Calibration set 9.5950 7.6540 8.5692 0.4852 Prediction set 9.7110 8.1120 8.8488 0.4980
[0051] Step 2: An iTPA model is set to imitate learning and exploration processes of three groups in a team, namely learning and exploration processes of an elite group, an ordinary group and a garbage collection group, and reasonable member updating rules are designed to gradually increase the evaluation values thereof to achieve the global optimum. The algorithm model is expressed as follows:
[0052] where in Formula (1), a vector X represents a waveband containing a plurality of wavelengths, x.sub.1-x.sub.n represent all wavelength points in the waveband, x.sub.i represents the ith wavelength point in the waveband, b, and a, respectively represent upper and lower boundary values of the wavelength point, and a function f(x) represents an evaluation value of the waveband. The evaluation value is a function with a root mean square error of calibration (RMSEC) and a correlation coefficient(R) as variables, wherein the RMSEC is obtained by partial least squares (PLS) modeling with the waveband X and measured content physical and chemical values. A calculation formula of the evaluation value is:
[0053] The evaluation value is gradually increased by updating the waveband to search for an optimal waveband.
[0054] Considering that in some practical application scenes, a waveband with a lowest evaluation value may be used as an optimal waveband, according to actual needs, the method of the application can also search for an optimal waveband by updating the waveband to gradually reduce the evaluation value. In this example, the evaluation value is gradually increased to search for an optimal waveband.
[0055] Step 3: An entire spectral waveband (that is, the calibration set after the division of the sample sets) is equally divided into P wavebands, and at the same time, an evaluation value of each waveband is determined, where each waveband is equivalent to the vector X in step 2.
[0056] The P wavebands are divided into N+M+L wavebands according to evaluation values from high to low, where N, M and L are integers respectively. The first N wavebands with highest evaluation values form the elite group, x.sub.e represents wavebands in the elite group, and there are N wavebands in total. The M wavebands with moderate evaluation values form the ordinary group, x.sub.p represents wavebands in the ordinary group, and there are M wavebands in total. The last L wavebands with lowest evaluation values form the garbage collection group, x.sub.l represents wavebands in the garbage collection group, and there are L wavebands in total.
[0057] The garbage collection group is a newly added group in the method of the application, and the evaluation values in this group are kept extremely low in an entire iterative process. The values of the N, M and L are generally taken according to experience, and the values of the N, M and L cannot be too different from each other.
[0058] In this example, the total number of members is set to 35, that is, P=35. PLS modeling is performed on each waveband and protein physical and chemical values to obtain an evaluation value corresponding to the waveband, where the number of elite groups is 10, the number of ordinary groups is 10, and the number of garbage collection groups is 15.
[0059] Step 4: A new waveband x.sub.r is generated.
[0060] A method for generating the new waveband is: the new waveband is generated from any one of the elite group and the ordinary group, and the wavelength point of the new waveband is inherited from the same wavelength point of a random waveband in the current group. That is, if the new waveband is generated from the elite group, and if the nth wavelength point of the new waveband is generated in the mth waveband in the elite group, this wavelength point needs to inherit the nth wavelength point in the mth waveband in the elite group.
[0061] Step 5: The new waveband x.sub.r selects a learning behavior or an exploration behavior according to a set probability/to update the wavelength point of the new waveband x.sub.r, so as to generate a candidate waveband x.sub.c. The sum of probabilities of selecting the learning behavior and the exploration behavior is 100%. In this example, a probability/of selecting the learning behavior by the new waveband x.sub.r is set to 0.35, and a probability of selecting the exploration behavior by the new waveband x.sub.r is set to 0.65.
[0062] Learning behavior:
[0063] If the new waveband x.sub.r performs the learning behavior, adjustment needs to be performed to a reference target direction. The reference target direction is respectively generated from template values of the elite group and the garbage collection group, which are respectively called an elite group template e.sub.e and a garbage collection group template e.sub.l; and the template value is taken from a mean value of the wavelengths of the wavebands in the group where the template value is located.
[0064] If the new waveband is generated in the ordinary group, adjustment is performed to the elite group template.
[0065] If the new waveband is generated in the elite group, adjustment is performed to an opposite direction of the garbage collection group template. The reason for the adjustment to the opposite direction of the garbage collection group template instead of adjustment to the opposite direction of the ordinary group template like the TPA is because an evaluation value of the ordinary group template increases in an iterative process, and a good adjustment direction cannot be provided later. However, an evaluation value of the garbage collection group template is low throughout the process, so that a good adjustment direction can be effectively provided for the new waveband x.sub.r generated in the elite group.
[0066] Specifically, when the new waveband is generated in the ordinary group, an expression of generating a candidate waveband by selecting the learning behavior to update the wavelength point of the new waveband is:
X.sub.c=(1−γ)x.sub.r+γe.sub.e (3)
[0067] When the new waveband is generated in the elite group, an expression of generating a candidate waveband by selecting the learning behavior to update the wavelength point of the new waveband is:
X.sub.c=(1+γ)X.sub.r−γe.sub.l (4).
[0068] In Formula (3) and Formula (4), y is a random number in an interval [0,1]. If a certain wavelength point of the x.sub.c crosses the boundary, a boundary value thereof is used instead. A near-infrared spectral wavelength range set in this example is 1100-2498 nm, that is, boundary values are 1100 nm and 2498 nm.
[0069] Exploration behavior:
[0070] If the new waveband x.sub.r performs the exploration behavior, each of the wavelength points x.sub.i(i=1,2, . . . , n) is randomly changed, and the exploration intensity is gradually reduced. An expression of generating the x.sub.c by two groups of new wavebands x.sub.r through the exploration behavior is:
[0071] In Formula (5), x.sub.ri represents a wavelength corresponding to the new waveband, that is, the ith wavelength of the waveband; K represents a maximum number of iterations of the algorithm, and k represents the current cumulative number of iterations; a shrinkage index a.sub.e,p represents: a.sub.e is selected when the new waveband is inherited from the elite group, and a.sub.p is selected when the new waveband is inherited from the ordinary group; and γ.sub.i represents a random number in an interval (0, 1), and m.sub.i takes a integer between 0 and 1 randomly. In this example, the shrinkage index of the elite group is set to 20, and the shrinkage index of the ordinary group is generally half of that of the elite group, taking 10.
[0072] Step 6: Member (candidate) wavebands are updated, including:
[0073] 6.1: If an evaluation value of the candidate waveband x.sub.c is higher than an evaluation value of the last x.sub.ewst in the elite group, x.sub.c enters the elite group, at the same time, evaluation values in the elite group are sorted again, and the last x.sub.ewst in the elite group after sorting again is directly eliminated without entering the ordinary group. The x.sub.ewst is a waveband with a lowest evaluation value in the elite group.
[0074] This is because the TPA sets the learning behavior which strengthens the directional search and local search abilities of the algorithm. If the eliminated last x.sub.ewst in the elite group enters the ordinary group, it is easy to cause the algorithm to fall into the local optimum.
[0075] 6.2: If the evaluation value of the candidate waveband x.sub.c is lower than an evaluation value of the x.sub.ewst but higher than an evaluation value of the last x.sub.pwst in the ordinary group, whether the x.sub.c is obtained by exploration is checked;
[0076] if the x.sub.c is obtained by exploration, the x.sub.c enters the ordinary group, at the same time, evaluation values in the ordinary group are sorted again, and the last x.sub.pwst in the ordinary group after sorting is eliminated; the x.sub.pwst is a waveband with a lowest evaluation value in the ordinary group; and
[0077] if the x.sub.c is not obtained by exploration, the x.sub.c is directly discarded. This is because the learning behavior is more likely to generate a candidate member with a high evaluation value and has strong convergence, which easily leads to the assimilation of wavebands in the ordinary group and reduces the global optimization ability.
[0078] 6.3: If the evaluation value of the candidate waveband x.sub.c is lower than an evaluation value of the first x.sub.lbst in the garbage collection group, the x.sub.c enters the garbage collection group, at the same time, evaluation values in the garbage collection group are sorted again, and the x.sub.lbst waveband in the garbage collection group after sorting is eliminated to enable the evaluation values in the garbage collection group to be low all the time, thereby providing a more correct update direction for the new waveband inherited from the elite group to perform the learning behavior. The x.sub.lbst is a waveband with a highest evaluation value in the garbage collection group.
[0079] Step 7: A number of iteration updates is customized. In this example, the number of iterations is set to 1000. After the iteration, a waveband with a highest evaluation value in the elite group is selected as the waveband object to be selected.
[0080] Example 2
[0081] In order to examine the effect of the variable selection algorithm proposed in Example 1 on modeling prediction, the settings are as follows:
[0082] The total number of members is set to 35, that is, P=35, where the number of elite groups is 10, the number of ordinary groups is 10, and the number of garbage collection groups is 15.
[0083] A probability of selecting the learning behavior by a new member is set to 0.35, that is, a probability of selecting the exploration behavior is 0.65.
[0084] The shrinkage index of the elite group is set to 20, and the shrinkage index of the ordinary group is generally 10 which is half of that of the elite group.
[0085] The number of iterations is set to 1000.
[0086]
[0087] Classical near-infrared spectral wavelength selection algorithms, such as a genetic algorithm (GA), principal component analysis (PCA) and a team progress algorithm (TPA), are compared with the improved team progress algorithm (iTPA) proposed in the application in selection variable modeling effect.
[0088] In the GA, an original spectrum is equally divided into a plurality of sub-intervals, and an optimal sub-interval combination corresponding to a maximum fitness value is obtained by evolution iteration. A PLS model is established based on the wavebands selected by genes, R and RMSEC values of the model are calculated, and a fitness function F is consistent with an evaluation value function f(x) in Formula (2) as follows:
[0089] The total 700 wavelength points of the original spectrum are divided into 35 equidistant intervals, that is, the length of a genetic code is 35, and each gene contains 20 wavelength points. The number of groups is set to 50, a crossover probability is set to 0.85, a mutation probability is set to 0.1, and a maximum number of iterations is set to 200 generations. In a population evolution process, optimal fitness individuals in the evolution process within a maximum number of iterations are found. Due to the randomness of the GA, the TPA and the iTPA, the above three algorithms are respectively operated for 50 times to obtain a mean value.
[0090]
[0091] The original spectrum is subjected to PCA dimensionality reduction to select wavelength variables, and the variables with the top 20 contribution rates are selected. Through calculation, the cumulative contribution rate of the top 20 variables reaches 0.999997. The 20 variables and protein content are subjected to PLS modeling, and relevant predicted values are obtained. Relevant evaluation parameters obtained by PLS modeling include: an RMSEP, a relevant coefficient of prediction (Rp), an RMSEC and a relevant coefficient of calibration (Rc), where the smaller the RMSEP and the RMSEC, the higher the prediction accuracy; the larger the Rp and the Rc, the higher the prediction accuracy; and in the same case of PLS modeling, the higher the prediction accuracy, the more information of the prediction objects contained in the wavelengths selected, and the better the wavelength selection method.
[0092] Analytical data: it can be seen from Table 2 that through the wavelength selection method iTPA-PLS proposed in the application, the RMSEP is 0.1177, the Rp is 0.9704, the RMSEC is 0.1019, and the Rc is 0.9759. Through the full-spectrum PLS (that is, a method of using full-spectrum data for prediction without wavelength selection), the RMSEP is 0.1789. Through the existing TPA-PLS, the RMSEP is 0.1193. The GA-PLS has the best overall prediction effect, but the mean number of wavelength points selected by the GA-PLS is as high as 346, which is far greater than the mean number of wavelength points (19.60) selected by the wavelength selection method proposed in the application.
[0093] By performing PCA on the original spectrum, 20 wavelength points (the contribution rate reaches 0.999997) are selected to compare with the wavelength selection method proposed in the application, which selects nearly the same number of wavelengths. The model RMSEP obtained after wavelength selection by PCA is 0.2306, which is far greater than the RMSEP value obtained by the iTPA. Therefore, the wavelength selection method proposed in the application can greatly reduce the number of wavelength points on the premise of maintaining the predictive ability, and effectively reduce the calculation amount of modeling.
TABLE-US-00002 TABLE 2 Algorithm comparison Number of wavelengths Rc RMSEC Rp RMSEP F-PLS 700 0.9517 0.1474 0.9307 0.1789 GA-PLS 346 0.9791 0.0978 0.9735 0.1114 TPA-PLS 19.64 0.9755 0.1056 0.9696 0.1193 iTPA-PLS 19.60 0.9759 0.1019 0.9704 0.1177 PCA 20 0.9569 0.1417 0.8916 0.2306
[0094] The protection scope of the disclosure is not limited to the above examples. Any modification, equivalent replacement and improvement that can be made by the persons skilled in the art within the spirit and principle of the concept of the disclosure shall be included in the protection scope of the disclosure.