MACHINE LEARNING VARIABLE SELECTION AND ROOT CAUSE DISCOVERY BY CUMULATIVE PREDICTION

Abstract

A sequence of models accumulates r-squared values for an increasing number of variables in order to quantify the importance of each variable to the prediction of a targeted yield or parametric response.

Claims

1. A method, comprising: configuring a machine learning model to predict a target feature based on a plurality of process parameters provided as inputs to the machine learning model; selecting a first one of the process parameters and determining a first r-squared value for predicting the target feature based on the first selected process parameter; selecting a second one of the process parameters and determining a second r-squared value for predicting the target feature based on the second selected process parameter; selecting additional ones of the process parameters and determining corresponding additional r-squared values for predicting the target feature based on the additional selected process parameters until an accumulation of the first, second and additional r-squared values increases by less than a threshold value; and ranking from high to low the process parameters on the basis of r-squared values; and identifying as key inputs to the machine learning model a set of high ranking process parameters.

Description

DESCRIPTION OF DRAWINGS

[0008] FIG. 1 is a table illustrating incremental and cumulative contribution to yield prediction due to selected process parameters.

[0009] FIG. 2 is a graphical plot illustrating the wafer-level correlation between predicted yield and actual yield for a plurality of wafer samples based on the process parameters identifies in FIG. 1.

[0010] FIG. 3 is a bar graph illustrating the relative impact on yield of selected process parameters for two of the wafer samples from FIG. 2.

[0011] FIG. 4 is a graphical plot illustrating the wafer level correlation between actual yield predicted yield from a model built using only the most important input variable.

[0012] FIG. 5 is a bar graph illustrating the average of relative impact on yield of selected process parameters for two of the wafer samples from FIG. 2.

[0013] FIG. 6 are heat maps illustrating predicted spatial patterns and actual spatial patterns for two of the wafer samples from FIG. 2 after observation based on a single parameter.

[0014] FIG. 7 are heat maps illustrating predicted spatial patterns and actual spatial patterns for two of the wafer samples from FIG. 2 after observation based on two parameters.

[0015] FIG. 8 is a flow chart illustrating a method for identifying cumulative impact of process parameters.

DETAILED DESCRIPTION

[0016] Machine learning (ML) is playing a more active role in modeling semiconductor processes, especially for prediction and root cause analysis. Some of the principals of stepwise regression can be used to optimize the input parameters for any complex ML approach (e.g., neural networks, extreme gradient boosting, etc.), and in particular, an ML model based on a cross-validated r-squared approach to remove variables that do not improve prediction on the training data and only keep those variables that are important to prediction.

[0017] However, that concept can be taken further by first building the best one-variable model possible. Next, build the best two-variable model possible, which by definition must include the variable from the best one-variable model; and then build the best three-variable model possible, which by definition must include the variables from the best two-variable model. This process is repeated until all variables are rank-ordered in terms of most important to least important or until the best n-variable model predicts a desired or acceptable percentage of the cross-validated r-squared values for all variables selected by the stepwise variable selection process.

[0018] Thus, a cumulative sequence of models can be used to quantify the importance of each variable to the overall prediction of a targeted yield (or continuous parametric) response based on an improvement in cross-validated r-squared values.

[0019] For example, FIG. 1 shows a table 100 with several process parameters listed entries in order to demonstrate how a cumulative improvement in yield performance results by using r-squared values as the measure. In table 100, rows 111, 112, 113 list selected wafer parameters, with the wafer group identifier listed in column 121, the source of the wafer parameter listed in column 122, the cumulative cross-validated r-squared percentage value listed in column 123, the incremental cross-validated r-squared percentage value of that parameter listed in column 124, and the parameter number, e.g., parameter ranking by incremental impact, listed in column 125.

[0020] The first parameter listed in row 111 is PCM-32, its source (column 122) is a continuous parametric measurement of a first physical test structure on the wafer, and its incremental contribution (column 124) to an identified variance or excursion is 15.4% based on the r-squared value from a statistical analysis. Adding the second parameter PCM-1 (row 112), a continuous parametric measurement of a second, different test structure on the same wafer, has an incremental impact of 12.4% and increases the explained cumulative variance to 27.8%, while adding the third parameter PE-1 (row 113), a process tool identified in the wafer equipment history (WEH), with an incremental impact of 9.9%, raises the cumulative variance total to 37.7%. Based on this cross-validated r-squared statistical analysis, these are the only three variables important enough, e.g., approximately 10% of more incremental contribution to the variance in this example, to include in the machine learning model.

[0021] Understanding how important each parameter is to the overall model is valuable, but may not be as valuable as understanding how important each parameter is to a particular observation. For example, several low yielding wafers might be best predicted by contact resistance changes, giving the user information that allows a more rapid diagnosis of the root cause. Consider how this works for a simple example data set.

[0022] FIG. 2 shows a correlation plot 200 of several hundred wafers predicted based on a machine learning model of actual yield versus the predicted yield for the parameters considered in FIG. 1, namely, PCM-1, PCM-2 and PE-1. Two randomly selected wafers 210, 220 from different lots (Lot134_W06 and Lot143_W09) are both well predicted in the final model, but the question is which parameters are important to each wafer.

[0023] First, the measures of accuracy are defined. Since this particular model was focused on predicting die and wafer yield, the two measures of accuracy are (i) the difference between predicted wafer yield and actual wafer yield, and (ii) the r-squared value in the die level correlation of predicted yield and actual yield across the wafer. For the die level correlation, the data is smoothed by averaging each die with the immediately surrounding die, as represented by line 230, to get a smoother representation of yield and yield prediction. The accuracy of wafer prediction is considered for each of the three cumulative predictions identified in FIG. 1.

[0024] The wafer level prediction error for each cumulative model is shown in bar graph 300 in FIG. 3. The predicted error for wafer 210 (Lot134_W06) is shown on the left-hand bar graph 310 and the predicted error for wafer 220 (Lot143_W09) is shown on the right-hand bar graph 320. Bar graph 310 shows the wafer 210 has a very large error 312 when the prediction depends only on the single parameter PCM-2. However, the prediction error 314 is much better (i.e., lower) once parameter PCM-1 is included in the prediction. This implies that yield for wafer 210 (Lot134_W06) is being more driven by PCM-1 than other parameters since the prediction error does not change significantly (316, 318) when other parameters are considered. This can be confirmed by looking at the wafer level prediction plot of the first cumulative model versus the actual yield.

[0025] Similarly, the predicted error for wafer 220 (Lot143_W09) is driven primarily by PCM-2 since the predicted error 322 does not change much when other parameters (324, 326, 328) are also considered.

[0026] FIG. 4 is a wafer level plot 400 of yield versus the average value for PCM-2 and confirms that the yield loss for wafer 220 (Lot143_W09) is not predicted from the model using only PCM-2 since that point is so far off the average line. However, this model does a good job of predicting wafer 210 (Lot134_W06) since the point is very close to the average line which indicates that this wafer yield is primarily driven by PCM-2. This can be further confirmed by looking at the spatial correlation for the first model using only PCM-2 and the second cumulative model using both PCM-2 and PCM-1.

[0027] FIG. 5 is a bar graph 500 that shows the spatial r-squared values for each of the cumulative models for the two example wafers, wafer 210 (Lot134_W06) on the left-hand graph 510 and wafer 220 (Lot143_W06) on the right-hand graph 520. From graph 510, the wafer yield prediction shows that wafer 210 (Lot134_W06) was primarily predicted by the parameter PCM-1 since the plot of spatial r-squared is much better once PCM-1 is added to the model at plot 512, which is consistent with the wafer yield prediction. From graph 520, the wafer yield prediction shows that wafer 220 (Lot143_W09) was primarily predicted by the parameter PCM-2, as indicated by the first plot 521 and very similar spatial r-squared value for all four cumulative predictions.

[0028] As shown in FIG. 6, these same spatial pattern matches can easily be observed from the wafer maps of actual yield and cumulative model predictions. For example, window 600 shows the predicted map spatial pattern for wafer 210 (Lot134_W06) in the left-hand panel 610 and the predicted map spatial pattern for wafer 220 (Lot143_W09) in the right-hand panel 620 based on the single parameter PCM-2 Window 601 shows the actual map spatial pattern for wafer 210 (Lot134_W06) in the left-hand panel 611 and the actual map spatial pattern for wafer 220 (Lot143_W09) in the right-hand panel 621.

[0029] For wafer 220 (Lot143_W09), the first cumulative predicted model shown in panel 620 clearly matches fairly well with the actual map shown in panel 621. However, the predicted pattern for wafer 210 (Lot134_W06) in panel 610 does not match well with the actual pattern shown in panel 611

[0030] FIG. 7 further confirms the r-squared-based observations above, namely, that the second cumulative model does not significantly impact the prediction of wafer 220 (Lot143_W09), but now the prediction for wafer 210 (Lot134_W06) in panel 710 matches fairly well with the actual map shown in panel 711, thereby indicating that parameter PCM-1 is the culprit for this wafer (and no significant impact on wafer 220 as seen from similar maps in panel 720 and 721).

[0031] This approach can be used in this example for die level yield prediction, as described herein, or for any complex multi-variate machine learning problem, provided that there are separable identifiable root causes.

[0032] A flowchart is presented in FIG. 8 to describe the general method for a cumulative determination regarding key variables. In step 802, a machine learning model is configured to predict a target feature. In step 804, a process parameter is selected, and in step 806, the r-squared values for the prediction of the target feature on the basis of the selected process parameter are determined. The r-squared values are accumulated, and if the cumulative total continues to increase more than a threshold amount in step 808, then another process parameter is chosen in step 810, and the r-squared values for prediction of the target feature on the basis of the next selected process parameter are determined.

[0033] Once the cumulative total is no longer increasing significantly in step 808, i.e., the increase is less than the threshold, then the process parameters will be ranked in step 812 by r-squared values. Finally, in step 814, the highest-ranking process parameters, i.e., those parameters that have a demonstrated impact on the target features by virtual of the cross-validated r-square values approach, will be identifies as key variables to the ML model.