Iterative process for calibrating a direct neural interface
11497429 · 2022-11-15
Assignee
Inventors
Cpc classification
G06F3/015
PHYSICS
G06F3/038
PHYSICS
A61B5/374
HUMAN NECESSITIES
A61B2560/0223
HUMAN NECESSITIES
A61B5/24
HUMAN NECESSITIES
A61H2230/105
HUMAN NECESSITIES
A61H3/00
HUMAN NECESSITIES
International classification
A61H3/00
HUMAN NECESSITIES
A61B5/24
HUMAN NECESSITIES
A61B5/245
HUMAN NECESSITIES
Abstract
The subject of the invention is a method for calibrating a direct neural interface. The calibration is performed by considering a so-called input calibration tensor, formed on the basis of measured electrophysiological signals and so-called output calibration tensor, formed on the basis of measured output signals. The method comprises the application of a least squares multivariate regression implemented by considering a covariance tensor and a cross-covariance tensor which are established on the basis of input and output calibration tensors corresponding to a current calibration period. The method takes into account covariance and cross-covariance tensors established during an earlier calibration period prior to the current calibration period, these tensors being weighted by a forget factor.
Claims
1. A method for calibrating a direct neural interface, the interface being configured to acquire electrophysiological signals, using sensors, and to form an input tensor therefrom, so as to establish, using a predictive model, an output tensor, the latter being intended to generate a control signal of an effector, the predictive model resulting from a calibration, the calibration comprising calibration periods, each calibration period comprising: a) acquiring input electrophysiological calibration signals and forming an input calibration tensor therefrom, the electrophysiological calibration signals being produced by the cortex of an individual and measured by the sensors, the electrophysiological calibration signals being representative of neural activity during the calibration period; b) acquiring output calibration signals and forming an output calibration tensor therefrom, each output calibration signal being representative of a control signal for the effector or of a state of the effector during the calibration; c) calculating a covariance tensor, representing a covariance of the input calibration tensor; d) calculating a cross-covariance tensor representing a covariance of the input calibration tensor and of the output calibration tensor; e) applying a partial least squares multivariate regression on the basis of the covariance tensor obtained in c) and of the cross-covariance tensor obtained during d) so as to obtain a predictive model, as well as calibration parameters; a) to e) being implemented during each calibration period; wherein, after a first calibration period, c) comprises taking into account of a covariance tensor resulting from an earlier calibration period, and weighted by a forget factor; d) comprises taking into account of a cross-covariance tensor calculated resulting from the earlier calibration period and weighted by the forget factor; e) comprises taking into account of calibration parameters resulting from the earlier calibration period, so that after the first calibration period, each calibration period provides an updated predictive model.
2. The method of claim 1, wherein the forget factor is a positive real number lying strictly between 0 and 1.
3. The method of claim 1, wherein during e), the calibration parameters comprise projection vectors, so that after the first calibration period, e) comprises a taking into account of projection vectors resulting from an earlier calibration.
4. The method of claim 1, wherein during e) the partial least squares multivariate regression is performed according to several iterations, to each iteration there corresponding an iteration rank, each iteration generating a predictive model, the predictive model being configured to provide an estimated output tensor, on the basis of an input tensor; and wherein: the input tensor is formed using the electrophysiological signals acquired at various instants within a calibration period; the estimated output tensor comprises data of outputs intended to form control signals to control the effector.
5. The method of claim 4, further comprising determining of an optimal iteration rank, the determination of the optimal iteration rank comprising: taking into account an input calibration tensor and of an output calibration tensor corresponding to a current calibration period; taking into account of the predictive models respectively associated with various iteration ranks of the partial least squares multivariate regression performed during an earlier calibration period; applying each predictive model to the input calibration tensor of the current calibration period, so as to obtain an estimated the output calibration tensor of the current calibration period, each estimated the output calibration tensor being associated with the rank of the iteration corresponding to the predictive model on the basis of which the estimation is performed; comparing, for each iteration rank, each estimated the output calibration tensor, with the output calibration tensor of the current calibration period; and determining of the optimal, iteration rank from each comparison.
6. The method of claim 1, wherein each sensor is one of: a cortical electrode; or a magnetoencephalography sensor; or an electroencephalography electrode.
7. The method of claim 1, wherein the effector is an actuator configured to perform an action under the effect of the control signal.
8. The method of claim 7, wherein the effector is an exoskeleton or a robot or a computer.
9. A method of controlling an effector through a direct neural interface, comprising: i) acquiring, using sensors, electrophysiological signals produced in the cortex of an individual, the electrophysiological signals being representative of a cortical activity; ii) processing, using a processing unit, the electrophysiological signals, thereby to form input data, assembled according to an input tensor; iii) applying a predictive model to the input tensor so as to estimate an output tensor iv) forming a control signal for the effector on the basis of the output tensor; wherein during iii) the predictive model is determined by a calibration carried out according to a calibration method according to claim 1.
10. The method of claim 9, wherein during iii), the predictive model corresponding to an optimal iteration rank is applied, the optimal iteration rank being determined according taking into account an input calibration tensor and of an output calibration tensor; taking into account of the predictive models respectively associated with various iteration ranks of the partial least squares multivariate regression performed during an earlier calibration period; applying each predictive model to the input calibration tensor of the current calibration period, so as to obtain an estimated the output calibration tensor of the current calibration period, each estimated the output calibration tensor being associated with the rank of the iteration corresponding to the predictive model on the basis of which the estimation is performed; comparing, for each iteration rank, each estimated the output calibration tensor, with the output calibration tensor of the current calibration period; and determining of the optimal, iteration rank from each comparison.
11. A direct neural interface, comprising: sensors, configured to acquire electrophysiological signals representative of a cortical activity; a processor, for processing the electrophysiological signals; an effector, configured to be actuated by a control signal generated by the processor; wherein the processor is configured to implement ii) to iv) of the method of claim 9, on the basis of the electrophysiological signals acquired by the sensors.
Description
FIGURES
(1)
(2)
(3)
(4)
(5)
(6)
DISCLOSURE OF PARTICULAR EMBODIMENTS
(7)
(8) The effector 5 is an actuator able to perform an action under the effect of the control signal which is addressed to it. It may entail an exoskeleton, a robot, or a computer. The effector is then controlled by the electrophysiological signals acquired by the cortical electrodes 2, via an algorithm coded in the memory 4 and implemented by the processor 3. The device 1 is a direct neural interface, in the sense that it allows the control of the effector 5 by the electrophysiological signals measured by the sensors 2. The electrophysiological signals can be acquired by cortical electrodes placed in contact with the cortex, magnetoencephalography sensors or electroencephalograph electrodes.
(9) As indicated in conjunction with the prior art, the control signal is formed by the processor 3 on the basis of the electrophysiological signals, grouped together, after preprocessing, according to a tensor, the so-called input tensor, by implementing a predictive model. The predictive model generates an output tensor, grouping together data, so-called output data, allowing the control of the effector 5. The predictive model forms the subject of learning, or calibration, during which the control signal, the so-called output signal, is trained.
(10) In this example, the electrophysiological signals S.sub.1 . . . S.sub.I.sub.
(11) Each electrophysiological signal acquired by an electrode forms for example the subject of a frequency analysis according to I.sub.2 frequency bands. For example, I.sub.2=15. The frequency bands lie for example between 10 kHz and 150 kHz. The frequency analysis may be for example a wavelet transform.
(12) Each electrophysiological signal can also form the subject of a decimation, so as to extend over I.sub.1 time increments. For example, I.sub.1=10.
(13) The electrophysiological signals, after preprocessing (frequency analysis and decimation) are grouped into blocks of input data. Each block of input data x(n), corresponding to an instant n, is three-dimensional, and of dimension I.sub.1×I.sub.2×I.sub.3. The blocks of input data x(n) can be grouped into a tensor X, the so-called input tensor, grouping together a number N consecutive blocks of input data x(n). The input tensor is of dimension N×I.sub.1×I.sub.2×I.sub.3.
(14) To each input data block x(n) there corresponds a control signal y(n), representing the control to be addressed to effectors at the instant n. As indicated previously, a predictive model makes it possible, on the basis of the input tensor X, to obtain a so-called output tensor Y. The output tensor groups together N consecutive blocks of output data y(n), in such a way that each block of input data x(n) is associated with a block of data output y(n), the block of output data allows the formation of the control signal for one or more effectors. The output tensor can correspond to a state of the effector, for example a position of the effector, such as desired by the individual generating the electrophysiological signals. The control signals are then adapted to configure the effector according to this state.
(15) The dimension of each block of output data can vary according to typical cases. In the example considered, each output block drives three articulations of an arm, in this instance the wrist, the elbow and the shoulder, with three degrees of freedom per articulations. Each output block y(n) is therefore a 3×3 matrix, corresponding to the state of the three articulations.
(16) The establishment of a predictive model according to the invention is now described in conjunction with
(17) The output calibration signals are either measured, or imposed, but they are known, in contradistinction to the phase of use of the direct neural interface, in the course of which the output signals are estimated on the basis of the input electrophysiological signals. The output calibration signals are representative of a control signal for the effector, or of a state of the effector, during the calibration. For example, a trajectory is measured or imposed on the effector, in which case the output calibration data represent the various positions of the effector during the trajectory, each position corresponding to a state of the effector. It is also possible to observe the behavior of the individual and the intentional actuation, by the latter, of the effector, in which case the output calibration data correspond to the control signal for the effector or to the state of the effector (actuated or non-actuated). For example, as described in patent U.S. Pat. No. 9,480,583, the effector is a pedal and the output calibration signal represents the actuation of the pedal by the individual.
(18) The input calibration data constitute a so-called calibration input tensor X.sup.(t) while the output calibration data form an output calibration tensor Y.sup.(t). t designates a time period, the so-called calibration period, at which the calibration input and output data are acquired, the input and output calibration data preferably being acquired simultaneously.
(19) Each input calibration block x(n) has dimension I.sub.1× . . . ×I.sub.k: each of its coordinates is denoted x.sub.n,I.sub.
(20) Each output calibration block y(n) has dimension J.sub.1× . . . ×J.sub.m: each of its coordinates is denoted y.sub.n,J.sub.
(21) Step 100: constitution of the calibration tensors X.sup.(t) and Y.sup.(t). The input tensor can comprise a number N of input calibration and of calibration output data blocks x(n) and y(n).
(22) The dimension of the input calibration tensor X.sup.(t) is N×I.sub.1× . . . ×I.sub.k.
(23) The dimension of the output calibration tensor Y.sup.(t) is ×J.sub.1× . . . ×J.sub.m.
(24) One of the particularities of the calibration is that after a first calibration period, t>1, it takes into account the earlier calibration tensors X.sup.(t−1) and Y.sup.(t−1), in the form of covariance and cross-covariance tensors. Stated otherwise, during a first calibration, corresponding to t=1, the calibration algorithm is implemented on the basis of the first calibration tensors X.sup.(t=1) and Y.sup.(t=1). During the following calibrations, account is taken of the, so-called current, calibration tensors acquired at a period t, the so-called current period, during which each calibration is performed. These calibrations take account of the covariance tensors established on the basis of the earlier calibration tensors X.sup.(t−1), Y.sup.(t−1) obtained at an earlier calibration period prior to the current period. The contribution of the earlier calibration tensors is however weighted by a forget factor λ, as described in conjunction with step 140.
(25) Step 110: normalization: the calibration tensors X.sup.(t) and Y.sup.(t) are preferentially normalized, in such a way that each of their terms is centered and reduced with respect to an average value and to a standard deviation. Accordingly, a so-called effective number of observations is determined: N.sup.(t)=λN.sup.(t−1)+N.sup.(t) and a weighted sum
(26)
and a weighted quadratic sum
(27)
are calculated respectively according to the expressions:
(28)
where x.sub.n,i.sub.
(29) During the first calibration (t=1), the first term of each expression is zero.
(30) It is then possible to establish an average value
(31)
and a standard deviation
(32)
of the terms x.sub.n,i.sub.
(33)
(34) In an identical manner, it is possible to establish an average and a standard deviation of the terms y.sub.n,j.sub.
(35)
(36) The normalization consists in centering and reducing each term x.sub.n,i.sub.
(37)
(38) Step 120: calculation of the covariance tensor and of the cross-covariance tensor.
(39) The covariance tensor is:
C.sub.XX.sup.(t)=X.sup.(t)×.sub.1X.sup.(t) (10)
C.sub.XY.sup.(t)=X.sup.(t)×.sub.1Y.sup.(t) (11)
×.sub.1 designates the tensor product according to mode 1, described in the literature, in particular in the thesis mentioned in the prior art.
(40) Step 130: taking into account of a previous calibration. The covariance and cross-covariance tensors are modified, by taking into account covariance and cross-covariance tensors relating to an earlier calibration period, this earlier period being denoted t−1.
C.sub.XX.sup.(t)=λC.sub.XX.sup.(t−1)+X.sup.(t)×.sub.1X.sup.(t) (12)
C.sub.XY.sup.(t)=λC.sub.XY.sup.(t−1)+X.sup.(t)×.sub.1Y.sup.(t) (13)
(41) This step is a key step of the method, since it allows account to be taken of data used during a previous calibration phase, the taking into account of this history being weighted by a forget factor λ. The forget factor λ is a real lying strictly between 0 and 1.
(42) Thus, when a calibration is carried out at a current calibration period t, it takes account of a previous calibration, carried out at an earlier calibration period t−1. The method thus makes it possible to perform an updating of calibrations, each calibration taking into account a calibration performed previously. The taking into account of the history is weighted by the forget factor λ. The smaller it is, the less the history is taken into account.
(43) During the first calibration period, (t=1), C.sub.XX.sup.(t=1)=X.sup.(t=1)×.sub.1X.sup.(t=1) and C.sub.XY.sup.(t=1)=X.sup.(t=1)×.sub.1Y.sup.(t=1).
(44) Step 140: updating of the predictive model
(45) In the course of this step, the predictive model is updated by applying an algorithm known by the acronym NPLS (N-Partial Least Squares), signifying partial least squares multivariate regression, to the cross-covariance and covariance tensors resulting from step 130. The algorithm followed is an iterative algorithm, according to the principles described in the publication Dayal B. “Improved PLS Algorithms”, Journal of Chemometrics, Vol. 11, 73-85 (1997), and more precisely the modified algorithm number 2 appended to said publication. This iterative algorithm makes it possible to obtain, at each iteration f: a set of projection vectors, w.sub.1.sup.f, . . . w.sub.k.sup.f. The vector w.sub.1.sup.f is of dimension I.sub.1, the vector w.sub.2.sup.f is of dimension I.sub.2 . . . the vector the vector w.sub.k.sup.f is of dimension I.sub.k. The set of iterations makes it possible to form sets of projection vectors {w.sub.1.sup.f, . . . w.sub.k.sup.f}.sub.f=1.sup.F.sup.
(46) f is the rank of the iteration. It entails an integer lying between 1 and a maximum number of iteration F, the latter being for example equal to 200.
(47) The detail of this step, and the obtaining of the magnitudes listed hereinabove, are described subsequently, in conjunction with
(48) Subsequent to this step, one has a predictive model allowing the estimation Ŷ.sup.f of an output tensor Y on the basis of an input tensor X according to the expression:
Ŷ.sup.f={tilde over (B)}.sup.fX+Y.sub.0.sup.f (14)
(49) The notation Ŷ.sup.f designates the fact that the tensor Ŷ is estimated according to the predictive model, in this instance the matrix {tilde over (B)}.sup.f and the bias tensor Y.sub.0.sup.f that are defined in the course of iteration f.
(50) It is noted that in contradistinction to the prior art, the input and output tensors allowing the implementation of the NPLS algorithm of are not input tensors X and Y obtained respectively with measured input and output calibration data blocks, as is described in patent U.S. Pat. No. 9,480,583. The invention implements: the tensor C.sub.XX.sup.(t), which expresses a covariance of the input calibration tensor X.sup.(t), of dimension I.sub.1× . . . ×I.sub.k×I.sub.1× . . . ×I.sub.k; the tensor C.sub.XY.sup.(t), which expresses a cross-covariance of the input calibration tensor X.sup.(t) and of the output calibration tensor Y.sup.(t), of dimension I.sub.1× . . . ×I.sub.k×J.sub.1× . . . ×J.sub.m.
(51) The dimension of the tensors on the basis of which the iterative calibration is carried out is therefore reduced with respect to the prior art, in which the input and output calibration tensors are respectively X.sup.(t) and Y.sup.(t), whose respective dimensions are N×I.sub.1× . . . ×I.sub.k and N×J.sub.1× . . . ×J.sub.m.
(52) Step 140 is carried out by incrementing the iteration rank f until a maximum iteration rank F is reached.
(53) Step 150: end of algorithm. One then has a predictive model corresponding to each iteration rank f, calibrated on the basis of the calibration tensors X.sup.(t) and Y.sup.(t) the calibration being carried out with the aid of an earlier calibration tensor X.sup.(t−1) and Y.sup.(t−1), and more precisely with the aid of the covariance and cross-covariance tensors resulting from the earlier calibration.
(54) The predictive calibration model can be updated on the basis of new calibration tensors X.sup.(t+1) and Y.sup.(t+1), by repeating steps 100 to 150, while incrementing the calibration period t.
(55) The updating frequency is variable. The inventors have found that it is preferable to carry out, initially, various successive calibrations, by considering so-called initial calibration tensors X.sup.ini and Y.sup.ini of large dimension, with N=7000. Such calibration tensors are acquired according to a long initial calibration duration, that may for example last more than 10 minutes. Indeed, because of the decimation, each time increment is spaced 100 ms apart. An acquisition duration N=7000 corresponds to 7000 data blocks x(n) and y(n), acquired according to a sliding acquisition duration of 1 s, each block being offset from the previous or following block by 100 ms. In the course of the initial calibration, each tensor X.sup.ini and Y.sup.ini is segmented into 70 segments X.sup.(t=1) . . . X.sup.(t=70) and Y.sup.(t=1) . . . Y.sup.(t=70), or buffer, each segment comprising N=100 input and output calibration data blocks of x(n) and y(n). Each segment X.sup.(t) and Y.sup.(t) constitutes respectively an input and output calibration tensor. Thus, steps 100 to 150 hereinbelow are repeated successively, by considering each calibration tensor X.sup.(t) and Y.sup.(t) with 1≤t≤70.
(56) On completion of these 70 successive iterative calibrations, one has a robust prediction model, that can be updated regularly for example daily or several times a day, by a short calibration sequence, requiring few data blocks x(n) and y(n), for example N=100, which, in view of the duration of each block (1 s) and of the offset between each block (100 ms), corresponds to a duration of 10 seconds. During the calibration, the user performs predetermined movements, so as to train the state of the output effector.
(57) Between each calibration, the predictive model is used so as to estimate a state of the output tensor Y corresponding to an input tensor X by implementing the prediction expression (14). The main steps, described in
(58) Step 200: acquisition of electrophysiological signals, preprocessing, obtaining of blocks of input data x(n) to form input tensor X
(59) Step 210: normalization of the input tensor, as described in conjunction with step 110.
(60) Step 220: estimation of the output tensor, Ŷ.sup.f by applying the expression (14). The use of the expression (14) requires the choice of an iteration rank f. This choice may be arbitrary, by being defined, generally once and for all, on the basis of preliminary trials. The output tensor thus estimated makes it possible to obtain control signals for controlling the effector 5. For example, the output tensor comprises a state of the effector desired by the user, and the control signals formed make it possible to attain this state.
(61) The inventors have found that during the use of the predictive model, that is to say during the estimation of an output tensor Ŷ.sup.f on the basis of a measured tensor X, it is possible to determine an optimal iteration rank f, this optimal number being denoted F*
(62) The main steps of the determination of the optimal number F* are described hereinafter in conjunction with
(63) Step 300: initialization: f=1
(64) Step 310: estimation Ŷ.sup.f(t) of Y.sup.(t) by using a prediction matrix {tilde over (B)}.sup.f(t−1) obtained during the previous calibration: Ŷ.sup.(t)=X.sup.(t){tilde over (B)}.sup.f(t−1)+Y.sub.0.sup.f
(65) Step 320: calculation of an error corresponding to iteration rank f, according to the expression:
e.sub.f.sup.t=γe.sub.f.sup.t−1+Error(Ŷ.sup.f(t),Y.sup.(t))
Error (Ŷ.sup.f(t),Y(t)) represents an error of estimation of Y.sup.(t), for example a quadratic error.
(66) When t=1 (subsequent to the first calibration), e.sub.f.sup.t=1=Error(Ŷ.sup.f(t=1), Y.sup.(t=1)). γ is a forget factor, independent of the forget factor γ considered during steps 100 to 150, lying strictly between 0 and 1.
(67) Step 330: incrementation of f until f=F.
(68) Step 340: determination of F*, such that F*=arg min.sub.fe.sub.f.sup.t.
(69) According to this algorithm, the optimal iteration rank F* is adjusted after each updating of the predictive model, that is to say after each calibration period. This algorithm uses the input and output calibration tensors corresponding to a current calibration period t. It consists in obtaining an estimation of the output calibration tensor Y.sup.(t), at the current period, by applying a predictive model, ({{tilde over (B)}.sup.f}.sub.f=1.sup.F.sup.
Ŷ.sup.f=F*={tilde over (B)}.sup.f=F*X+Y.sub.0.sup.f=F* (15)
(70) The inventors have experimentally tested the algorithm for determining the optimal iteration rank F* after 70 calibrations, each calibration being carried out on a buffer of 100 data blocks (N=100). Accordingly, use has been made of calibration signals measured on a monkey, by a matrix of 64 cortical electrodes. The calibration signals measured by the electrodes are processed, so as to form the input calibration tensor X.sup.(t) such as described previously. The movements of the 3 articulations of an arm of the monkey were recorded and form the output calibration tensor Y.sup.(t).
(71)
(72)
(73) The input data are the input calibration X.sup.(t) and output calibration Y.sup.(t) tensors, as well as projection vectors {w.sub.1.sup.f . . . w.sub.k.sup.f}.sub.f=1.sup.F.sup.
(74) Sub-step 141: Initialization: the internal matrices P, Q and R, defined hereinafter, are initialized to 0. P∈.sup.(I.sup.
.sup.(J.sup.
.sup.(I.sup.
.sup.(I.sup.
.sup.(I.sup.
(75) Sub-step 142: Calculation of eigenvectors. If m>1 (m is the order of the output tensor), we determine the eigenvector {tilde over (q)} corresponding to the highest eigenvalue of a matrix consisting of the product of the matrices C.sub.XX.sup.(t).sup.
(76) Sub-step 143: Formation of a tensor C on the basis of a matrix C formed by a matrix product C.sub.XY.sup.(t)q.: C=C.sub.XY.sup.(t){tilde over (q)}. C∈.sup.(I.sup.
(77) Sub-step 144: Decomposition by parallel factorial analysis of the tensor C formed during step 143 by considering the projection vectors {w.sub.1.sup.f, . . . w.sub.n .sup.f}.sup.t−1 of an earlier calibration, or the projection vectors initialized to 1 during the first calibration.
(78) Sub-step 145: formation of a tensor w=w.sub.1.sup.f∘ . . . ∘w.sub.n.sup.f and formation of a vector w∈.sup.(I.sup.
(79) Sub-step 146: formation of the regression vector. We put r=w and we update the regression vector r by r=r−[P(:,u).sup.TwR(:,u)] by varying u between 1 and f−1.
(80) Sub-step 147: determination of internal vectors p and q, and updating of internal matrices P and Q and R.
(81)
P(:,f)=p; Q(:,f)=q; R(:,f)=r
(82) Sub-step 148: Deflation.
C.sub.XY.sup.(t)=C.sub.XY.sup.(t)−(r.sup.TC.sub.XX.sup.(t)r)pq.sup.T
(83) Sub-step 149: Updating of the matrices formation of the prediction matrix B.sup.f: B.sup.f=R(:,1:f) Q(:,1:f).sup.T
(84) The prediction matrix can form the subject of a normalization. Each term b.sub.i.sub.
(85)
(86)
and
(87)
being defined in equations (6) and (4) respectively.
(88) Each coefficient {tilde over (b)}.sub.i.sub.
(89) Moreover, the algorithm generates the bias tensor Y.sub.0.sup.f, each term
(90)
of which is defined by the expression:
(91)
the quantities
(92)
and
(93)
being defined in expressions (4) and (3).
(94) The output data of the algorithm are those described in conjunction with step 140.