HIDDEN MARKOV MODEL-BASED GESTURE RECOGNITION WITH FMCW RADAR

Abstract

A gesture recognition system is shown using a 77 GHz FMCW radar system. The signature of a gesturing hand is measured to construct an energy distribution in velocity space over time. A gesturing hand is fundamentally a dynamical system with unobservable “state” (i.e. the type of the gesture) which determines the sequence of associated observable velocity-energy distributions, therefore a Hidden Markov Model is used to for gesture recognition. A method for reducing the length of the feature vectors by a factor of 12 is also shown, by re-parameterizing the feature vectors in terms of a sum of Gaussians without decreasing the recognition performance.

Claims

1. A method of gesture recognition comprising the steps of: collecting Frequency Modulated Continuous Wave (FMCW) radar reflectance data reflected from a gesturing object, processing said data by a one dimensional FFT to determine range to the reflecting object, gating said range data to select a predetermined range of interest, processing said FMCW data with a one dimensional FFT to determine velocity of the reflecting object, gating said velocity data to select a predetermined velocity range of interest, calculating the velocity energy of said velocity data, identifying the gesture from the velocity energy signature.

2. The method of claim 1, wherein: the step of velocity energy calculation is followed by a Gaussian parameterization of the velocity energy data.

3. The method of claim 1, wherein: the step of identifying the gesture is performed by a plurality of Hidden Markov Models.

4. An apparatus for confidence estimation for optical flow comprising of: A processor operable to collect Frequency Modulated Continuous Wave (FMCW) radar reflectance data reflected from a gesturing object, process said data to determine range to the reflecting object, gate said range data to select a predetermined range of interest, process said FMCW data to determine velocity of the reflecting object, gate said velocity data to select a predetermined velocity range of interest, calculate the velocity energy of said velocity data, identify the gesture from the velocity energy signature.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0007] These and other aspects of this invention are illustrated in the drawings, in which:

[0008] FIG. 1 is an implementation of the Frequency Modulated Continuous Wave radar hardware;

[0009] FIG. 2 is a flow chart showing the primary steps of the invention;

[0010] FIG. 3 is a time-velocity plot illustrating the evolution of the velocity energy during four distinct gestures. The specific gestures are described in table 1; and

[0011] FIG. 4 is a continuation of the time-velocity plot of FIG. 3.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

[0012] Measurements were performed using the FMCW system described earlier. Each frame consisted of 1022 chirps, with an additional frame spacing on the order of 200 ms due to hardware limitations. Processing was broken into two components performed separately: first, the radar data was collected and processed to extract the desired feature vectors; second, the feature vectors were used to train a recognition system based on Hidden Markov Models using the publicly available HTK software package. Recognition was performed using both high dimension and reduced dimension feature vectors.

[0013] In FMCW radar, both range and velocity information can be extracted and processed as part of the recognition task. This is in contrast with using a single tone Continuous Wave system only measuring the micro-Doppler spectrum, and thus is sensitive to any motion within the field of view. The main advantage is that range gating may be performed prior to the construction of the velocity-energy vectors used to create equivalent “time-velocity diagrams”. This eliminates undesirable interference from other objects that may be present in the scene and allows for segmentation based on other criteria, such as a known region-of-interest where gestures may be performed. In principle, gestures could also be resolved for multiple areas simultaneously. The algorithm flow can be described as a sequence of simple steps, shown in FIG. 2.

[0014] For range processing the raw IQ data in 201 obtained from the A/D converter is processed by a simple one-dimensional FFT in 202 to obtain the range profile for each chirp individually.

[0015] The range FFT results are gated in 203 for a region of interest. In this implementation, the region is from 0.7 to 1.4 meters. The range-gated data is stored sequentially in a “frame buffer” until a complete frame has been acquired (1022 chirps).

[0016] Each range in the frame buffer is individually processed by a second FFT in 204 to obtain the velocity information for that range bin. The velocity FFT output is gated in 205 to velocities of ±1 m/s and stored in the “velocity buffer.”

[0017] The velocity buffer's magnitude squared is integrated along the range axis for each velocity bin in 206 to obtain the final velocity-energy signature for a particular frame.

[0018] These velocity-energy vectors are either processed directly or passed to Gaussian parameterization step 207 before being processed in 208. They can also be visualized over time as a time-velocity diagram, which illustrates the temporal structure of various gestures. In FIGS. 3 and 4, the time-velocity diagram for six gestures is shown and annotated according to the gesture definitions in table 1.

TABLE-US-00001 TABLE 1 Description of gestures used in experiment Gesture ID Gesture Description A Waving the hand towards and away from the 301 radar B Rotating the hand as if grasping a doorknob, 302 in the radial direction C Twiddling the index and middle fingers back 303 and forth rapidly D Rotating the entire palm 90 degrees in the 304 radial direction E A grasping or “zoom”-like motion extending 401 and closing all fingers F Waving the hand perpendicular to the radar 402 boresight

[0019] Initially, all six gestures in table 1 were intended to be used for classification. However, during initial data collection we restricted the data to only four gestures, A, B, C, and E. Gestures D and F were chosen to be eliminated because of their similarity to B and A respectively. For each of the remaining four gestures, approximately 100 training sets consisting of 30 frames of continuous demonstration of the gesture were collected. Two people performed the gestures while the radar was collecting data for an extended period of time (generally several minutes at a time) and then a software program was used to automatically segment the data into 30-frame segments. The start and stop times of each segment were not aligned with any markers and the segmentation was performed automatically and without examining the data. This means it is possible that only a partial gesture was present in a very small number of frames, but the classification results suggest that this was not the case.

[0020] Each individual frame consisted of a 128-point velocity energy vector. The data was collected in an open lab space without isolation (i.e. not in an anechoic chamber). Furthermore, the radar is capable of detecting targets at a distance of up to 21 meters; this creates the potential for undesirable reflections to be captured from the ceiling and other test equipment present in the lab. However, no such out-of-range interference is present due to the range and velocity gating performed, restricting the processing only to the region-of-interest.

[0021] With the data labeled by the type of gesture being performed, it was used to train four Hidden Markov Models (HMMs) using the publicly available Hidden Markov Model Toolkit (HTK). Each HMM had five emitting states plus two non-emitting states required by HTK. After training the HMM for each gesture to convergence, the training data was re-used for the recognition task. The confusion matrix for recognition of the training set is shown in table 2 for the 30 frame configuration. Ideally, a separate test set would be used to test the recognition accuracy; however, due to the limited amount of data collected a simple test was performed to determine whether identifying characteristics for each gesture could be extracted by an HMM at all.

TABLE-US-00002 TABLE 2 Confusion matrix for recognition of the training set with 30 frames per gesture Recognized True A B C E A 101 0 0 0 B 0 95 2 2 C 0 9 88 1 E 0 21 35 42

[0022] Table 2 illustrates that the waving gesture A, which can be seen in table 3 to be dramatically different from all of the other gestures, is the easiest to identify, is never misidentified, and is never incorrectly supplied as the label for a different gesture. On the other hand, the zoom gesture E can be seen as presenting significant similarity to gestures B and C, resulting in many misclassifications. As a consequence, the aggregate accuracy of 82.3% may be misleading. A, B, and C are generally well recognized (far above the average) while E is very poorly recognized (far below the average). The summary accuracy calculations are listed in table 3.

TABLE-US-00003 TABLE 3 Recognition accuracy for 30, 15, and 10 frames per gesture Frame count Gesture 30 15 10 A  100% 99.5% 98.8% B 96.0% 88.0% 82.8% C 89.8% 86.4% 84.0% E 42.9% 52.1% 49.8% Total 82.3% 81.5% 79.0%

[0023] After testing the performance using 30 frames per gesture, the training and recognition tasks were repeated using 15 and 10 frames per gesture, creating new segmentations of the original training data and producing up to 300 examples for each gesture in the shortest case. The recognition accuracy decreased as expected but remained sufficient for basic recognition tasks, with a 79% overall accuracy in the 10 frame per gesture case. These results are also summarized in table 3. Individual gesture accuracies follow the basic trend established in table 2 for the 30 frame case where zoom performs the worst and waving performs the best.

[0024] As mentioned in earlier, radar returns from a human hand at short range do not appear as a discrete set of points; instead, they appear as a distributed target that produces varying returns in multiple range and velocity bins. After the transformation to a velocity-energy vector, this distribution might still remain. The effects of processing (such as windowing in our FFT) or observation (such as the point-spread function of our radar) further distribute energy from a single conceptual source into multiple observation bins. The observed velocity-energy vector may be composed of a weighted sum of L Gaussians plus some unmodeled data and noise η,

[00003]$f\left(\upsilon \right)=\underset{l=1}{\overset{L}{.Math.}}.Math..Math.w_l.Math.\exp \left(\frac{-{\left(\upsilon -{\mu }_l\right)}^2}{{\sigma }_l^2}\right)..Math.E\left(\upsilon \right)=\eta \left(\upsilon \right)+f\left(\upsilon \right)$

Thus for a fixed small L, such as L=5, by finding w.sub.1, μ.sub.1, σ.sub.1 such that

e.sup.2=(E−f).sup.T(E−f)

is minimized, we may replace the 128-point feature vector by the 15-point Gaussian parameterization vector. In the final version, the μ.sub.1 parameters were not used for classification due to extremely low sample variance, resulting in a compression ratio larger than 12 to 1.

[0025] The optimization step shown is quite sensitive to the initial guess and refinement criteria. The initial guess is provided by the peak location, magnitude, and approximate width based on neighboring local minima. This is refined using Newton refinement so long as each iteration reduces the residual energy and the weight remains positive. If either condition is not met, the previous iteration is used for that parameter triple, a new residual is calculated, and the process is repeated until L parameter triples have been extracted. This re-parameterization proved to be extremely effective for the gesture recognition task. For 30- and 15-frame gestures, the recognition accuracy improved to 83.3% and 82.4% respectively. For 10-frame gestures, the recognition accuracy decreased to 75.85%. These results suggest that the hypothesis regarding the underlying structure of the velocity-energy vector is accurate and provides a means for reducing the dimension of the feature vector by a factor of 10 or more while preserving the information required for gesture recognition.

[0026] This invention demonstrates the feasibility of short range gesture recognition using a 77-GHz FMCW radar system, using a Hidden Markov Model for classification. The radar system is capable of operating in real time and performing gesture classification for targets located within a specific area of the range-Doppler plane, eliminating energy contributed by other targets that might disrupt the gesture recognition task. A re-parameterization of the velocity-energy calculations in terms of an underlying Gaussian structure preserves most of the information required for gesture recognition while offering data size savings of more than 12×.