METHOD AND APPARATUS FOR DETERMINATION OF A TRAJECTORY AND IMAGING DEVICE

Abstract

A method of determination of a trajectory of a living tissue, in which subsequent frames representing images of living tissue are acquired with at least a first imaging device and at least a first segment of a trajectory of the living tissue is determined with a use of relative displacements of at least first subset of subsequent frames and recalculation thereof to coordinates corresponding to time of frame acquisition to form a vector Tm elements tm wherein element tm is at least one coordinate of living tissue, element tm of vector Tm is determined with a use of relative displacements p.sub.m,k of at least two preceding frames.

Claims

1-10. (canceled)

10. A method for determining a trajectory of a living tissue, the method comprising: providing a first imaging apparatus having a field of view of at least 1° in X plane and Y plane; acquiring a first frame representing images of the living tissue at a frame acquisition rate of at least 500 FPS and a frame size of at least 1000 pixels; acquiring subsequent frames representing images of the living tissue with the first imaging apparatus; and determining at least a first segment of the trajectory of the living tissue using relative displacements of at least a first subset of subsequent frames and recalculation thereof to coordinates corresponding to time of frame acquisition to form a vector Tm of elements tm wherein element tm is at least one coordinate of the living tissue and wherein element tm of vector Tm is determined with a use of relative displacements p.sub.m,a, p.sub.m,b of at least two preceding frames and relative displacements p.sub.m,a, p.sub.m,b are determined using predetermined criterion.

11. The method according to claim 10, further comprising: using at least a second subset of frames to calculate alignment frames corresponding to the at least first segment and at least a second segment; and minimizing cumulative error in determining of the trajectory of the living tissue by recalculation of relative displacement between elements of trajectory in the at least first segment and the at least second segment.

12. The method according to the claim 11, wherein the alignment frames are acquired with the first imaging apparatus.

13. The method according to claim 12, further comprising: detecting a loop in the trajectory wherein at least two non-adjacent segments are in proximity with each other; and identifying alignment frames contained in the at least two non-adjacent segments.

14. Method according to claim 11, wherein the alignment frames are obtained with a second imaging apparatus.

15. A device for determining a trajectory of a movement of a living tissue, comprising: a first imaging apparatus having a field of view of at least 1° in X plane and Y plane having a frame acquisition rate of at least 500 FPS and frame size of at least 1000 pixels; and configured to acquire subsequent frames representing images of the living tissue with the first imaging apparatus; and a processing unit, configured to determine at least a first segment of the trajectory of the living tissue using relative displacements of at least a first subset of subsequent frames and recalculation thereof to coordinates corresponding to time of frame acquisition to form a vector Tm of elements tm wherein element tm is at least one coordinate of the living tissue and wherein element tm of vector Tm is determined with a use of relative displacements p.sub.m,a, p.sub.m,b of at least two preceding frames and relative displacements p.sub.m,a, p.sub.m,b are determined using predetermined criterion.

16. The device of claim 15, wherein the device is configured: to use at least a second subset of frames to calculate alignment frames corresponding to the at least first segment and at least a second segment; and to minimize cumulative error in determining of the trajectory of the living tissue by recalculation of relative displacement between elements of trajectory in the at least first segment and the at least second segment.

17. The device of claim 16, wherein the alignment frames are acquired with the first imaging apparatus.

18. The device of claim 15, further comprising a second imaging apparatus configured to reposition images acquired with the second imaging apparatus to compensate for movement of the living tissue.

19. The device of claim 18, wherein the processing unit is configured to receive frames from the second imaging device to use as alignment frames.

20. An article comprising a non-transitory computer readable medium having instructions stored thereon which, when executed by a processor coupled to at least one memory device, are operable to: provide a first imaging apparatus having a field of view of at least 1° in X plane and Y plane; acquire a first frame representing images of a living tissue at a frame acquisition rate of at least 500 FPS and a frame size of at least 1000 pixels; acquire subsequent frames representing images of the living tissue with the first imaging apparatus; and determine at least a first segment of the trajectory of the living tissue using relative displacements of at least a first subset of subsequent frames and recalculation thereof to coordinates corresponding to time of frame acquisition to form a vector Tm of elements tm wherein element tm is at least one coordinate of the living tissue and wherein element tm of vector Tm is determined with a use of relative displacements p.sub.m,a, p.sub.m,b of at least two preceding frames and relative displacements p.sub.m,a, p.sub.m,b are determined using predetermined criterion.

21. The article of claim 20, further operable to: use at least a second subset of frames to calculate alignment frames corresponding to the at least first segment and at least a second segment; and minimize cumulative error in determining of the trajectory of the living tissue by recalculation of relative displacement between elements of trajectory in the at least first segment and the at least second segment.

Description

[0016] The invention has been described below in detail, with reference to the following figures wherein

[0017] FIG. 1 shows schematically frames and areas along the trajectory;

[0018] FIG. 2a presents a flowchart of an embodiment of the method according to the invention;

[0019] FIG. 2b illustrates displacements between a frame and previous frames;

[0020] FIG. 3 presents a flowchart of an another embodiment to the method according to the invention;

[0021] FIG. 4a presents a block diagram of imaging device applicable for use in the system according to the invention and a method according to the invention;

[0022] FIG. 4b illustrates Lissajous scanning pattern and

[0023] FIG. 4c shows reconstructed rectangular frame from the scanned points;

[0024] FIG. 4d shows exemplary frames acquired with the first imaging device. FIG. 5a-d show various loops of trajectory.

EMBODIMENTS OF THE INVENTION

[0025] FIG. 1 shows schematically an eye and frames acquired with imaging device FET. Subsequent frames are in different locations due to eye movements. Trajectory of the frames in eye coordinate system is exactly complementary to trajectory of an eye in the imaging device FET coordinate system. Therefore determination of frame is the same operation as determination the trajectory of eye living tissue in imaging device coordinate system—those two trajectories differ only in sign.

[0026] A flow chart of an example of the method according to the invention is shown in FIG. 2a. The method of determination of a trajectory T of a living tissue O, is in this example applied to an eye. Movements of the eye ball are among the fastest in the body, and therefore, the method of determination of trajectory is applicable to determination of trajectories of other organs. Eyeball in the setup discussed below can move 1000°/s—that contributes to 300 mm/s for an average eye.

[0027] Frames of the eye are acquired with the first imaging device FET with frame acquisition rate of 1240 FPS. The field of view was set to 3.37°×3.24°. The imaging beam diameter at the cornea was equal to 1.96 mm. The theoretical on-axis beam size on the retina in this case was 8.5 μm. The confocality of the system was deliberately sacrificed to gain sensitivity by using a 100-μm pinhole. The airy disc diameter at the detection plane was 36.4 μm. The confocality described in terms of times-diffraction-limited spot size was equal to 2.75 and frame size was accordingly 4432 pixels (140×140 pixels).

[0028] Further studies have shown that reliable trajectories can be obtained with frame rate as low as 500 FPS and frame size as low as 1000 px. Naturally higher frame rates and larger frames allow more accurate determination of trajectory.

[0029] First imaging device delivered subsequent frames including F1, Fa, Fb, Fm. Frames represent images of small parts of eye. The relation between frame rate and frame size contributes to a result where every frame is a part of subset of subsequent frames.

[0030] According to the method positions of the frames in the said subset of frames relative to the position of at least first area is determined—like in a state of the art methods with that difference that reference frame is not selected.

[0031] Said positions are thereafter recalculated to sets of coordinates (points) forming at least the first segment of trajectory T corresponding to time of acquisition. Sets of coordinates form a vector Tm elements tm wherein element tm is a point of trajectory represented as a set of two coordinates:

t.sub.m={x.sub.m,y.sub.m}  (1)

and corresponding to a single frame f.sub.m.

[0032] In the present example the trajectory reconstruction algorithm operates in two stages. In the first stage, the retinal trajectory is estimated by the so-called N-back algorithm, in which each new frame is aligned against n previous frames as shown in FIG. 2b and FIG. 2c. In the second stage, cumulative errors in the trajectory reconstructed in the first stage are corrected.

[0033] After start 200 of the process a first frame f1 is retrieved in acquisition step 200. It involves scanning and recalculation of pattern to the obtain rectangular frame as described in more detail below.

[0034] Further frames are repeatedly retrieved in acquisition step 211.

[0035] A new trajectory point of the retina was estimated by estimating 250 the displacements between the frame incoming in acquisition step 211 and a subset of the ones acquired earlier. These displacements are added to the trajectory points found for the latter frames. In the simplest case, the displacement from only one previous frame was considered, which is noted as follows:

t.sub.m=t.sub.m−1+p.sub.m,m−1  (2)

where t.sub.m={x.sub.m,y.sub.m} is the trajectory point in Euclidean space, m is the iterator of frames/trajectory points and p.sub.m,m−1 denotes the displacement between frames m and m−1. Displacements are shown in FIG. 2c. Formally, for frames a, b p.sub.a,b is estimated by minimizing a criterion function that measures the alignment of frames f.sub.a to f.sub.b. Various criteria could be employed interchangeably based on the hardware requirements and computation latency needed. In this example, the enhanced correlation coefficient (ECC) disclosed in Evangelidis, Georgios D., and Emmanouil Z. Psarakis. “Parametric image alignment using enhanced correlation coefficient maximization.” IEEE Transactions on Pattern Analysis and Machine Intelligence 30.10 (2008): 1858-1865 was used, as it can provide sub-pixel precision and gradient descent-based computation can be run on a CPU. Naturally person skilled in the art can easily suggest and implement other applicable criterions for estimation of frame distance.

[0036] The accuracy and resistance to noise of the simple trajectory building algorithm, is increased in the N-back version of the algorithm, by calculating the displacement of the new frame from N previously acquired frames and averaging them:

[00001]$\begin{array}{cc}{t}_m=\frac{1}{N}{.Math.}_{n\in B}\left(t_{m-n}+p_{m,m-n}\right)& \left(3\right)\end{array}$

[0037] where n represents indices of frames calculated in reference to the newly acquired frame (i.e. n=1 denotes the frame that directly precedes the new frame). Using trial-and-error experiments, a set B∈{1,2,4,8,16,32,128,256,512} of the frame indices was chosen. In this way, the method compares not only consecutive frames but also tries to align frames that were acquired almost half a second earlier. If the criterion D(f.sub.a,f.sub.b) cannot be calculated due to low SNR or a displacement exceeding the size of the frames, the corresponding index n, is removed from set B. N represents the total number of frames in the set B. After calculations frame fm is added 202 to the subset of frames used to determine following points of trajectory. Averaging allows reduction of error. A person skilled in the art may find a different composition of frames indices B that will maximize the efficiency of the method in terms of computation speed and required accuracy.

[0038] The alignment of frames described above is prone to small errors in p.sub.a,b that will propagate in t.sub.m over time. Due to the recursive nature of the method, the trajectory estimation error can be modeled as a random walk process, which leads to the conclusion that the ideally calibrated system has non-stationary zero mean error, with the error's variance growing linearly with time—meaning that errors have tendency to cumulate.

[0039] In order to correct this non-stationary error, an additional technique can be applied. It exploits the fact that the eye returns from time to time to the same location and the error accumulated by Eqs. 2-3 can be corrected by a new displacement calculation. The technique introduces alignment (or key) frames KF, which can be intuitively understood as a subset of all frames in the acquired dataset with translations p.sub.a,b.sup.KF that can be calculated, for frames f.sub.a and f.sub.b in close spatial locations on the retina, independent of time separation. Practically alignment frames can be detected 202 by detection of loops or intersections of the trajectory T in real time or in post processing. Alignment frame allow to minimize cumulative error by correction of the trajectory 251. Examples of loops with intersections are given in FIG. 5a and FIG. 5c and examples of loops without intersections are given in FIG. 5b and FIG. 5d. As long as two nonadjacent segments of trajectory T can be observed in the same frame this frame can be used as alignment frame KF.

[0040] Formally, these corrections for KF can be performed using the multidimensional scaling (MDS) mathematical framework disclosed in I. Borg, and P. Groenen. “Modern multidimensional scaling: Theory and applications.” Journal of Educational Measurement 40.3 (2003): 277-280, in which distances between KF are used to correct all the trajectory points from the first stage. In other words, a matrix P.sup.KF is constructed with all translation distances, |p.sub.a,b.sup.KF|, between the KF that can be calculated. A trajectory T.sup.KF is found by minimizing a stress function, σ, with respect to the KF positions, {circumflex over (T)}.sup.KF=[{circumflex over (t)}.sub.1.sup.KF, {circumflex over (t)}.sub.2.sup.KF, {circumflex over (t)}.sub.3.sup.KF, . . . ].

σ({circumflex over (T)}.sup.KF)=Σ.sub.a<bw.sub.a,b(P.sub.a,b.sup.KF−|{circumflex over (t)}.sub.a.sup.KF−{circumflex over (t)}.sub.b.sup.KF|)  (4)

where P.sub.a,b.sup.KF=|p.sub.a,b.sup.KF| are distance computed between an a-th and b-th alignment frame and w.sub.a,b is a weight that indices missing or imputed values. Naturally person skilled in the art can suggest alternative scheme for cumulative error minimization using alignment frames—possibly suboptimal but still operational.

[0041] The alignment frame trajectory, T.sup.KF, has an inherently low sampling rate, but has theoretically zero non-stationary errors. The final retinal trajectory, T.sup.FET, is estimated by casting the N-back trajectory, T, on the alignment frame trajectory, T.sup.KF, exploiting the interpolation for trajectory points in between the alignment frames.

[0042] Accordingly if alignment frame is detected in step 202 trajectory is corrected in step 251.

[0043] After end of tracking 203 corrected trajectory is output in step 204 and algorithm is terminated 205.

[0044] The method is easily implemented in computer program product run on a processing unit fed with frames in real time or offline processing. The former has an advantage of backward recalculation of the whole trajectory in step 252 after end of tracking 203.

[0045] Alignment frames can alternatively by obtained from the second imaging device offering slower frame rate and higher resolution. Such device can provide alignment frames corresponding to whole trajectory or at least its larger parts. It is a specially advantageous when method of determination of trajectory together with first imaging device FET is to be integrated with other imaging system such as WFSLO.

[0046] A block diagram of an example of such embodiment of the invention is shown in FIG. 3. After start 300 a frame from first imaging device is acquired 301 then within a loop the WFSLO is used to obtain an image of the living tissue in a strip-by-strip manner. In-between the strips of WFSLO subsequent frames are acquired with the first imaging device in acquisition step 310.

[0047] Displacements p.sub.a,b are calculated in the same manner as in the previous examples and frames already used are added to the pool for subsequent frames in step 301.

[0048] The difference consists in that in step 302 it is determined weather part of image already acquired with WFSLO can be used as alignment frame and if it is the case trajectory is corrected in step 351. After end of tracking 303 whole trajectory may be optionally recalculated 352 and output 304. Afterwards the process finishes 305.

[0049] It should be stressed that determination of trajectory without minimization of cumulative error is still useful in some applications especially the ones where it is a speed of human tissue that is crucial—eg. for online image stabilization to prevent motion artifacts in the obtained images that would cause image artifacts impossible to correct in post-processing with optical angiography being one of the possible examples.

[0050] A block diagram of a device FET applicable to use in the method according to the invention is shown in FIG. 4a. LD represents laser diode; CL represents a collimating lens; BS represents pellicle beamsplitter; L1-L9 represent achromatic doublets; MEMS 2D represents two-axis resonant MEMS scanning mirror; FET PS represents positioning galvanometric mirrors; D represents regulated iris aperture; PH represents pinhole; APD represents avalanche photodiode; HIL represents halogenic illumination lamp, BST represents Badal stage, AES represents artificial eye scanners set, AEO represents artificial eye objective, AET represents artificial eye test target. Stars and circles denote conjugated planes. Inset A: Targets used for in vivo experiments. Diameter of FT1 subtends 1.5° on the retina and was used for the experiment with static fixation. FT2 was used for the experiments with saccades; the diameters subtend 0.6° and the distance between the centers ranged from 1-8°. Inset B: artificial eye attachable to the system for testing.

[0051] The illumination source for this scheme is a 785-nm laser diode LD coupled with a single-mode fiber and collimated to a beam diameter of 0.7 mm by an aspherical lens CL. A pellicle beamsplitter BS reflects the beam and directs it onto a scanning mirror 2D MEMS with 1-mm active aperture, based on the microelectromechanical system (MEMS) technology. After reflecting off the scanning mirror, the beam passes through a telescope system composed of lenses L4 and L3.

[0052] The MEMS 2D scanning mirror was set to its maximum frequency, which resulted in a Lissajous scanning pattern, as shown in FIG. 4a. Therefore, to obtain frames in the first step, the raw images acquired by the detector APD needed to be resampled, to form the uniformly sampled rectangular images shown in FIG. 3b. This step was performed with a sparse matrix by dense vector multiplication f=M.Math.f.sup.L. f.sup.L=[f.sub.1.sup.L, f.sub.2.sup.L, . . . , f.sub.K.sup.L] represents intensity data vector acquired directly by APD. f.sub.i.sup.L is single APD reading. Resulting frame f is a N×N rectangular image organized in a single row-by-row vector of size N.sup.2. Resampling matrix M is a sparse matrix of size N.sup.2×K, constructed from the Lissajous coordinates in a a way that each row of M has a non-zero elements at indices that correspond to the s-closest distance between the Lissajous coordinates and the coordinates in the resulting frame f.

[0053] Person skilled in the art is able to offer a number of alternative constructions of the FET imaging device once learning the teaching of the present description that it has to provide frame acquisition rate of at least 500 FPS and frame size N×N of at least 1000 pixels and field of view of at least 1 deg.

[0054] FET device was tested in a system's including tracking performance with a use of a simplified artificial eye composed of X-Y galvanometric scanners set GVSC, imaging lens AEO, and test target AET was installed in the setup as shown in FIG. 4a. The imaging lens was chosen to be the same as L1 and arranged in a 4f system, in order to maintain the same scanning angles. The scanners were positioned in the eye pupil plane, and a voltage-to-optical-angle calibration was performed using the USAF 1951 test target. For tracking validation, a printed fundus image was used as the test target. The image was prepared in a way that allowed to obtain images, which resemble the ones from a preliminary study done in vivo with human eyes.

[0055] FIG. 4c shows frame reconstructed after Lissajous scanning. FIG. 4d shows frames acquired with field of vie 3.27°×3.24°. Frames 1-2 show part of an optic nerve, frames 3-4 show fovea; frames 5-15 show retinal vasculature with different sizes. Other experiments showed that tracking works for field of view as low as 1 degree in both planes.

[0056] Addition of apparatus according to the invention to the scanning laser ophthalmoscope known in the art allows simultaneous imaging and tracking trajectory of living human eye. Knowledge of the trajectory of the eye enables compensation of said movement in resulting images and consequently stabilization. It can either be done in real time or in post-processing.

[0057] It is stressed that the above-mentioned embodiments illustrate rather than limit the inventions, and that person skilled in the art will be able to design many alternative embodiments.

[0058] As one the inventions claimed is also a computer program product for running on a processing unit, at least including code portions for performing steps of a method according to the invention. The computer program may consist of or include or more of: a subroutine, a function, a procedure, a class, an object method, an object implementation, an executable application, an applet, a servlet, a source code, an object code, a shared library/dynamic load library and/or other sequence of instructions designed for execution on a programmable apparatus. The computer program may be provided on a data carrier, such as a CD-type optical disc, a DVD-type optical disc, a hard disk, or diskette, stored with data loadable in a memory of a computer system, the data representing the computer program. The data carrier may thus be a tangible data carrier. The data carrier may be a data connection, such as a telephone cable or a network cable. The data carrier may further be a non-tangible data carrier such as a wireless connection.

[0059] In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. Use of the verb “comprise” and its conjugations does not exclude the presence of elements or steps other than those stated in a claim. The article “a” or “an” preceding an element does not exclude the presence of a plurality of such elements. Use of numerals “first”, “second” and further by the name of elements does not introduce particular order but merely provides a way of distinction. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the device claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0060] It is a particular advantage of the method according to the invention that no reference frame is used but instead trajectory with respect to a number of frames is observed. That approach allows better and more reliable analysis of eye trajectory and velocity. Analysis of eye movement is in turn applicable in diagnosis of some neurodegenerative diseases including multiple sclerosis, Parkinson disease, Alzheimer disease, ataxias, strokes, palsies and others. It is also useful in diagnosis of diabetes, hypertension and others.

[0061] Knowledge of exact position and prediction of movement of living tissue is crucial in surgical operations especially with use of manipulators and robotics. Therefore although eyes were much emphasized in the description above the invention is applicable for determining trajectory of numerous human organs including muscles, especially heart but also bladder, liver, kidney ovaries, pancreas, thymus, trachea and others.

[0062] Specific patterns in movements of tissue is also applicable in biometry.