Method of volumetric imaging of a sample
11443464 · 2022-09-13
Assignee
Inventors
- Brendan Kennedy (Crawley, AU)
- Brooke Krajancich (Crawley, AU)
- Qi Fang (Crawley, AU)
- Andrea Curatolo (Crawley, AU)
Cpc classification
A61B1/267
HUMAN NECESSITIES
G06T11/008
PHYSICS
A61B5/0073
HUMAN NECESSITIES
G06T2207/10101
PHYSICS
International classification
A61B5/00
HUMAN NECESSITIES
Abstract
The present disclosure provides a method of volumetric imaging of a sample. The method comprises providing a plurality of depth images of a region of interest of the sample using a volumetric imaging system. The region of interest is below a surface area of interest of the sample. Each depth image is associated with a layer or slice of the region of interest and the plurality of depth images together forming a volumetric image of the region of interest. The method further comprises providing a surface image of the surface area of interest of the sample and identifying a surface image property of a surface feature of the surface area of interest. The method also comprises processing the plurality of depth images of the region of interest using the surface image property of the surface feature to improve a property of the depth images of the region of interest.
Claims
1. A method of volumetric imaging of biological tissue, the method comprising the steps of: providing a plurality of depth images of a region of interest of the biological tissue using a volumetric imaging system, the region of interest being below a surface area of interest of the biological tissue, each depth image being associated with a layer or slice of the region of interest and the plurality of depth images together forming a volumetric image of the region of interest; providing a surface image of the surface area of interest of the biological tissue; identifying a visible property of a surface portion of the surface area of interest using the surface image; identifying a portion of at least some of the depth images of the region of interest, the identified portion belonging to the same portion of the biological tissue as the surface portion having a tissue property that is different from a tissue property of surrounding tissue; associating the identified portion of at least some of the depth images with the visible property of a surface portion; and processing each of the plurality of depth images of the region of interest using the- visible property of a surface portion.
2. The method of claim 1, wherein the visible property is a colour of the surface portion.
3. The method of claim 1, wherein the surface image is acquired using a different imaging system to the volumetric imaging system.
4. The method of claim 1, comprising displaying the volumetric image with the same volume type having the surface image property.
5. The method of claim 1, wherein the imaging system is an OCT imaging system.
6. The method of claim 1, comprising determining a change in attenuation of an optical signal as a function of depth to determine a cross-sectional shape or depth extension of an identified portion.
7. The method of claim 1, comprising providing a series of surface images of the surface area of interest during movement of the volumetric imaging system and the surface area of interest relative to each other.
8. The method of claim 7, wherein the surface image property of the surface feature is a spatial position.
9. The method of claim 8, comprising identifying a change in the spatial position of the surface feature of each surface image and processing the plurality of depth images of the region of interest using the identified change of the spatial position to obtain a plurality of depth images of the region that are at least largely corrected for an impact of the movement of the volumetric imaging system and the surface area of the region of interest relative to each other.
10. The method of claim 1 comprising: moving at least a portion of the volumetric imaging system and the biological tissue relative to each other; providing a plurality of surface images during movement of the volumetric imaging system and the biological tissue relative to each other; identifying changes in spatial position of the surface features of the surface images and processing the plurality of depth images using the identified changes of the spatial positions to identify corrected spatial positions of the depth images; and generating at least one volumetric image using the depth images having the corrected spatial positions.
11. The method of claim 10 wherein the volumetric imaging system has a scanning range A and wherein the biological tissue and the volumetric scanning system are moved relative to each other such that a region B of the biological tissue is scanned and the region B has a surface extension that is larger than the scanning range A and wherein the at least one volumetric image is an image corresponding to the region B.
12. A method of volumetric imaging of a sample, the method comprising the steps of: providing a plurality of depth images using a volumetric imaging system during movement of a volumetric imaging system and the sample relative to each other, the volumetric imaging system being arranged for mechanical correction of the relative motion between the imaging system and the sample, each depth image being associated with a layer or slice of the region of interest and the plurality of depth images together forming a volumetric image of the region of interest; providing a plurality of surface images during movement of the volumetric imaging system and the sample relative to each other; identifying changes in spatial position of surface features of the surface images to obtain positional information; and controlling the volumetric imaging system to correct for an impact of the movement of the volumetric imaging system and the sample relative to each other using the obtained positional information; wherein the volumetric imaging system comprises a layer having known mechanical properties and incorporating a fiducial marker, the method further comprising: positioning the layer on the surface area of interest; applying a load to the region of interest and through the layer; detecting a deformation of the layer in response to the applied load; and using the fiducial marker for correcting for motion artefacts.
13. The method of claim 12 wherein the volumetric imaging system is arranged such that the movement correction can be performed mechanically using a suitable control system with a feedback loop.
14. The method of claim 12 wherein the layer comprises a layered structure within which the fiducial marker is positioned.
15. The method of claim 14 wherein the layered structure comprises layers having differing stiffness and wherein the fiducial marker is embedding in a layer having a stiffness that is greater than that of another layer of the layered structure.
16. The method of claim 15 wherein the layer in which the fiducial marker is positioned is in use located between the other layer having the lower stiffness and the region of interest and wherein the layered structure comprises a contact layer that has a stiffness that is lower than that of the layer in which the fiducial marker is embedded and wherein in use the layer in which the fiducial marker is embedded is positioned between two layers having the lower stiffness.
17. The method of claim 12 wherein the step of providing a plurality of depth images of a region of interest comprises OCT imaging and wherein the step of detecting a deformation of the layer in response to an applied load comprises OCT imaging of interfaces of the layer.
Description
BRIEF DESCRIPTION OF DRAWINGS
(1)
(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9)
(10)
(11)
(12)
(13)
(14)
(15)
DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS
(16) In general terms, embodiments of a method according to the invention involve improving, correcting or otherwise modifying a volumetric image of a sample using information derived from a surface image of a surface of the sample.
(17) The volumetric image is obtained using a volumetric imaging system, which acquires a plurality of depth images. In particular, each depth image is associated with a layer or slice of a region of interest of the sample, and the plurality of depth images together form the volumetric image of the region of interest.
(18) The surface image of the sample is preferably obtained using a different imaging technique. For example, the surface image may be a photograph of the sample acquired using a camera, or a micrograph of a sample taken using a microscope, while the depth image may be an OCT image of the sample acquired using an OCT imaging system.
(19) The method further involves identifying a surface image property of a feature or features of the surface image that has been obtained. The surface image property may for example be a visible property of the surface feature(s), such as an RGB colour. Alternatively, the surface image property may be a spatial position of the surface feature(s). The method then comprises processing the plurality of depth images using the surface image property to improve a property of the depth images of the region of interest.
(20)
(21) The method 100 involves step 102 of acquiring the surface image and the volumetric image of the sample.
(22) The volumetric image 300 and the surface image 200 of the sample are then correlated 104 with each other. This involves associating a surface portion of each depth image, i.e. each image slice, of the volumetric image 300, with a corresponding surface portion of the surface image 200.
(23) The method 100 then involves identifying 106 a visible property of one or more surface features of the surface image and corresponding material types, such as tissue types. In this example, the visible property is an RGB colour of the surface features. In particular, identifying 106 the RGB colour involves inspecting the surface image 200 to identify differently coloured portions of the image 200, such as the portions 202, 204 and 206. This can be done by visual inspection or using a suitable computer program if the image 200 is a digital image. Differently coloured portions of the sample are usually indicative of different biological tissue types, such as a muscular tissue, fatty tissue, a tumour or blood vessel in the case of a biological sample. The differently coloured portions can thus be associated with a particular tissue type.
(24) The RGB values of the differently coloured portions and the corresponding tissue type are then recorded 108 in a database, such as a data storage medium of a computer device. It will be appreciated that the database may already have existing RGB values and corresponding tissue types recorded, for example, from previous surface images. Entries in the database are preferably acquired under the same controlled imaging conditions, such as exposure time, lighting conditions etc.
(25) The method 100 also involves identifying 110 portions of the volumetric image 300 that correspond and/or belong to particular volume types. In this example, the volume type is a tissue type. This can be done by measuring an attenuation of the OCT signal with depth. In particular, the OCT image 300 in
(26) For example,
(27) In each of
(28) The identified tissue types and corresponding signal attenuation values associated with the OCT image 300 are then recorded 112 in a database, such as a data storage medium of a computer device. It will be appreciated that the database may already have existing RGB values and corresponding tissue types recorded, for example, from previous surface images.
(29) Then, RGB values from the surface image are assigned 114 to a surface portion of the volumetric images. In particular, surface portions (or other portions of a reference plane) of each voxel at the surface (or reference plane) of the volumetric image 300 are each associated with a corresponding RGB value derived from the surface image 200. Specifically, a corresponding RGB colour can be applied to the OCT image 300 such that the surface level of the OCT image 300 is coloured with substantially realistic colourings. Hence, RGB values corresponding to the structure at the tissue surface are used to colour-code the OCT image 300.
(30) However, it will be appreciated that since the OCT image 300 is constructed by depth images or 2D cross-sectional slices, such colour-coding can also be applied at different depths of the OCT image 200. Therefore, the method 100 involves further processing 116 the plurality of depth images using the RGB colours to improve a property, such as image contrast, of the depth images.
(31) In particular, according to this embodiment, for each voxel below the OCT image surface or reference plane, if the signal attenuation is maintained (i.e. still constant) compared to that of surface or reference plane, the RGB value assigned to the surface or reference plane will be applied 118 to that voxel.
(32) If not, another RGB value will be assigned 120 to that voxel, which may be derived from the RGB database and signal attenuation database referred to in steps 108 and 112. For example, if the signal attenuation for that voxel is determined, the signal attenuation database may have a record of what tissue type the signal attenuation corresponds to. The identified tissue type may then have a corresponding RGB value in the RGB database.
(33) As a result of the method 100, coloured OCT images 500 at different depths can be obtained, as shown in
(34) It will be appreciated that embodiments of the method 100 described above may provide the advantage of obtaining a more visually-appealing volumetric image, for example to clinicians, to facilitate rapid interpretation of images.
(35) It will also be appreciated that not only has the OCT image 300 been transformed into a colour image, the contrast of the OCT image 300 has been improved such that there is now improved delineation of different tissue types in the OCT image 300.
(36) An algorithm or appropriate computer programming software can be used to perform the method 600. Also, according to an alternative embodiment, if the surface image is a grayscale image, the surface image property may be a grayscale value of the surface image.
(37)
(38) To provide context, it is noted that surface images such as digital photographs are acquired in the plane orthogonal to the acquisition of 2D cross-sectional scans, such as scans that make up an OCT image. As a result, photographs provide a snapshot of the surface of a sample. In contrast, volumetric imaging techniques such as OCT or micro-elastography can only provide a view of the tissue surface once a 3D scan (i.e. a series of depth images, or 2D cross-sectional scans across the sample surface) has been acquired. However, a position of the sample may drift or change during the 2D cross-sectional scans, which may introduce motion artefact and result in a skewed or distorted image. This may occur for example when the sample is a portion of a patient's body, and the patient moves during the scan or in a handheld probe if the user's hand moves during the scan.
(39) To potentially address the issue of motion artefact described above, the method 600 comprises providing 602 a series of surface images of areas of interest of the sample during movement of the volumetric imaging system relative to the sample. For the avoidance of doubt, such movement can be movement of the imaging system, with respect to the sample, or movement of the sample with respect to the imaging system. Therefore, according to this embodiment, when the imaging system takes a linear scan of the sample to obtain a 2D cross-section, at least one surface image of the sample is also acquired, which will be associated with the particular line scan taken. However, it will be appreciated that in alternative embodiments, greater than or less than one surface image of the sample per B-scan line can be taken. For example, one surface image every two or more B-scan lines can be taken, and for the B-scan lines that do not have a corresponding surface image, a measure of shift (such as the transformation matrix described below) for those B-scan lines can be interpolated.
(40) In this example, the volumetric imaging system comprises a hand-held scanning probe capable of acquiring at least OCT images, such as a probe 12 shown in
(41) Further, in this example, the surface images provided are photographs of the sample. Preferably, the probe 12 is equipped with dual image acquisition capabilities so that the photographs of the sample can be obtained by the probe 12 simultaneously while the probe 12 scans the sample to obtain the volumetric images.
(42) The method 600 then comprises identifying 604 a change in the spatial position of the surface feature of each surface image. In this example, a particular en face plane (as opposed to the plane of a cross-sectional scan) of the OCT image is obtained. For each en face plane, a notional line of points referred to as a ‘B-scan line’ can be applied, as shown in
(43) The set of coordinates for each en face plane can then be transformed to a motion-corrected set of coordinates. This can be done, for example, by mapping each B-scan line according to the formula:
[i.sub.MCj.sub.MC1]=[ij1]*P.sup.−1*T.sub.jP
(44) In the formula above, T.sub.j is a transformation matrix derived from the surface image associated with the j.sup.th B-scan and the surface image taken associated with the first B-scan (or other reference B-scan). Matrices P.sup.−1 and P are used to convert [i j 1] to and from a form the photograph or surface image pixel space to the OCT image surface voxel coordinate space.
(45)
(46) Thus, in this example, the step of identifying 604 a change in the spatial position of the surface feature may comprise: identifying a position of a B-scan with respect to the surface image of the sample; identifying a corresponding position of the B-scan with respect to the reference surface image of the sample; and deriving a transformation matrix to transform the B-scan values (i, j) to corrected B-scan values (i.sub.MC, j.sub.MC).
(47) The plurality of depth images obtained by the probe 12 can then be processed 606 using the identified change of the spatial position, in order to obtain substantially corrected volumetric image. For example,
(48) However, depending on the nature of the relative movement between the scanner 12 and sample, the corrected image may have an irregular shape and/or there may be gaps in the image representing areas not scanned. Thus, to present the corrected image in a similar format to that usually obtained by OCT scans, missing portions of the corrected image may be interpolated. An example of a corrected OCT image with plot interpolations is shown in
(49) Assuming that pitch, yaw and z motion of the hand-held scanner 12 is negligible, the same motion correction transformations can be applied in the same way to every en face OCT plane in the acquired volumetric scan. It is envisaged that the transformation could however be extended to incorporate pitch, yaw and z motion.
(50) It will be appreciated that an algorithm or appropriate computer programming software can be used to perform the method 600.
(51)
(52) The method 1200 then comprises providing a series of surface images during movement of the imaging probe 12 across the sample (step 1206). The method further comprises identifying a change in spatial position of surface features of each surface image (step 1208) and processing the plurality of depth images using the identified change in the spatial position to identify a corrected position of each depth image (step 1210) such that the relative movement is at least largely compensated using this processing step. In this case at least one surface image is provided for each depth image during movement of the imaging probe across the sample. The method then comprises generating at least one volumetric image for the expanded region (such as the region below a surface area of 2A). A person skilled in the art will understand that the above-described method may be used to generate volumetric images of any extension beyond a scanning extension of the imaging probe.
(53) It will be understood to persons skilled in the art of the invention that many modifications may be made without departing from the spirit and scope of the invention.
(54) For example, the method may be conducted using a volumetric imaging system other than OCT, such as an ultra-sound imaging system. The method may be conducted in-vivo or ex-vivo. In the case of the former, the method may be performed through an endoscope, such as a bronchoscope incorporated with an OCT imaging system and a digital camera.
(55) As another example, instead of using a histology image 200, the surface image may be a photograph of the sample taken in vivo. An example of a colour photograph of a biological sample (breast cancer tissue) taken in vivo is shown in
(56) In an alternative embodiment a fiducial marker is provided in a different form. In this embodiment the fiducial marker is incorporated into a layered structure, such as a bi-layered structure having known mechanical properties. The bi-layered structure is used for indirect stress measurement of the tissue, which involves detecting a deformation of the layer positioned on the tissue in response to an applied load and then deducting mechanical or dynamic properties (such as stress) of the underlying tissue from a detected change in thickness of the layer in response to the applied load. Detecting a change in thickness of the bi-layered structure comprises in this example OCT imaging of interfaces of the bi-layered structure (for further details reference is being made to the applicant's co-pending U.S. patent application Ser. No. 15/023,663). In this embodiment the bi-layered structure enables the dual functions of stress measurement and motion correction. This is achieved by embedding the fiducial marker in a stiff layer that contacts the tissue and curing a second soft layer on top of the stiff layer. The soft layer has a stiffness similar to the tissue and is used for measuring stress. The reason for embedding the fiducial marker in a stiffer (which may comprise silicone) is to ensure that it is not deformed by the application of load. This would result in erroneous motion artefact correction. In a further variation of this embodiment the layer is a tri-layered structure, with the stiff layer incorporating the fiducial marker being sandwiched between two softer layers. One of the softer layer is positioned for contacting the tissue and is very thin (in the order of 100 um) and performs the function of allowing the tri-layered structure to adhere to the surface curvature of the tissue, ensuring a more even distribution of load within the tissue. This embodiment has the advantage that during a measurement that load can be applied evenly disturbing the application of the load, which may be caused by the marker shown 140 shown in
(57) As another example, the volumetric imaging system may be arranged for mechanical correction of the relative motion between the imaging system and the sample. Such mechanical correction may be substantially automatic and dynamic (i.e. substantially in real time, or with only minimal time delays), using a suitable control system with a feedback loop. In this embodiment, the step of processing the plurality of depth images of the region of interest using the surface image property may comprise controlling the volumetric imaging system to correct for an impact of the movement of the volumetric imaging system and the surface area of interest relative to each other. For example, during scanning of a sample using the probe 12 of the volumetric imaging system, a surface image may be taken (a “current surface image”). That surface image may be dynamically compared to a reference surface image taken of the sample prior to scanning. Then, using a shift/motion estimation algorithm, optical components of the probe 12 (such as mirrors) can be adjusted to counteract the image shift, such that the probe 12 captures a surface image as if there was less shift or no shift. The shift/motion algorithm could for example be based on an affine co-registration routine applied to the current and reference surface images. The same process can then be applied after such mechanical adjustment to counteract the image shift further, thus applying a feedback loop.
(58) As another example, the step of processing the plurality of depth images may comprise correcting for deformation in the surface plane. For instance, if a motion artefact causes a shear force to deform surface features relative to each other, this could be detected during the processing step and corrected in the OCT image. In particular, if it is assumed that the deformation is linear elastic, an affine transformation could be applied to the deformed image and the resultant image compared to the original image. This comparison could be achieved using cross-correlation. The deformed image could be digitally “un-deformed” using the affine transformation until the deformed and original image match. The coordinates of the affine transformation that lead to this matching between the images could then be applied to the image volume to correct for the induced deformation.
(59) In the claims which follow and in the preceding description of the invention, except where the context requires otherwise due to express language or necessary implication, the word “comprise” or variations such as “comprises” or “comprising” is used in an inclusive sense, i.e. to specify the presence of the stated features but not to preclude the presence or addition of further features in various embodiments of the invention.
(60) It is to be understood that, if any prior art publication is referred to herein, such reference does not constitute an admission that the publication forms a part of the common general knowledge in the art, in Australia or any other country.