OPTICAL IMAGING DEVICE FOR A MICROSCOPE

Abstract

An optical imaging device for a microscope comprises a first optical system configured to form a first optical image corresponding to a first region of a sample in accordance with a first imaging mode, a second optical system configured to form a second optical image corresponding to a second region of said sample, wherein said first and second regions spatially coincide in a target region of said sample and said first and second imaging modes are different from each other, a memory storing first distortion correction data suitable for correcting a first optical distortion caused by said first optical system in said first optical image, second distortion correction data suitable for correcting a second optical distortion caused by said second optical system in said second optical image, and transformation data suitable for correcting positional misalignment between said first and second optical images, and a processor which is configured to process first image data representing said first optical image based on said first distortion correction data for generating first distortion corrected image data, to process second image data representing said second optical image based on said second distortion correction data for generating second distortion corrected image data; and to combine said first and second distortion corrected image data based on said transformation data for generating combined image data representing a combined image which corresponds to said target region of said object.

Claims

1. An optical imaging device (102) for a microscope (100), comprising: a first optical system (108) configured to form a first optical image corresponding to a first region (210) of a sample (104) in accordance with a first imaging mode; a second optical system (112) configured to form a second optical image corresponding to a second region (214) of said sample (104), wherein said first and second regions (210, 214) spatially coincide in a target region (242) of said sample (104) and said first and second imaging modes are different from each other; a memory (134) storing first distortion correction data suitable for correcting a first optical distortion caused by said first optical system (108) in said first optical image, second distortion correction data suitable for correcting a second optical distortion caused by said second optical system (112) in said second optical image, and transformation data suitable for correcting positional misalignment between said first and second optical images, and a processor (124) which is configured to: process first image data representing said first optical image based on said first distortion correction data for generating first distortion corrected image data, process second image data representing said second optical image based on said second distortion correction data for generating second distortion corrected image data; and combine said first and second distortion corrected image data based on said transformation data for generating combined image data representing a combined image which corresponds to said target region (242) of said sample (104).

2. The optical imaging device (102) according to claim 1, wherein said first imaging mode is wide-field imaging mode and said second imaging mode is a confocal imaging mode.

3. The optical imaging device (102) according to claim 1, wherein said transformation data represents positional misalignment between a first optical reference image formed by said first optical system (108) in accordance with said first imaging mode and a second optical reference image formed by said second optical system (112) in accordance with said second imaging mode.

4. The optical imaging device (102) according to claim 1, wherein a calibration mode is provided in which said processor (124) is configured to generate said transformation data and to store said transformation data in said memory (134) prior to forming said first and second optical images.

5. The optical imaging device (102) according to claim 3, wherein said processor (124) is configured to determine correlation data representing a correlation between said first and second optical reference images and to generate said transformation data based on said correlation data.

6. The optical imaging device (102) according to claim 1, wherein the positional misalignment represented by the transformation data comprises translation, rotation, scaling, shearing, mirroring, and/or distortion.

7. The optical imaging device (102) according to claim 1, wherein said processor (124) is configured to update said transformation data and to store said updated transformation data in said memory (134).

8. The optical imaging device (102) according to claim 7, wherein said processor (124) is configured to cause said first optical system (108) to generate a sequence of first optical images in accordance with said first imaging mode and to cause said second optical system (112) to generate a sequence of second optical images in accordance with said second imaging mode, wherein said processor (124) is configured to determine a first tracking mark (452) within one of said first optical images and to determine a second tracking mark within one of said second optical images, and wherein said processor (124) is configured to perform tracking of said first tracking mark and said second tracking mark (452) and to update said transformation data based on said tracking.

9. The optical imaging device (102) according to claim 1, wherein said processor (124) is configured to combine said first and second distortion corrected image data based on said transformation data such that one of said first and second optical images is mapped to a reference system (KS1, KS2) defined by the other of said first and second optical images or such that both optical images are mapped to a common reference system (K53).

10. The optical imaging device (102) according to claim 1, wherein said first and second optical systems (108, 112) are installed in a fixed positional relationship to each other.

11. A method for imaging a sample (104) using a microscope (100), comprising the following steps: forming a first optical image corresponding to a first region (210) of a sample (104) in accordance with a first imaging mode using a first optical system (108); forming a second optical image corresponding to a second region (214) of said sample (104) in accordance with a second imaging mode using a second optical system (112), wherein said first and second regions (210, 214) spatially coincide in a target region (242) of said sample (104) and said first and second imaging modes are different from each other; obtaining first distortion correction data suitable for correcting a first optical distortion caused by said first optical system (108) in said first optical image; obtaining second distortion correction data suitable for correcting a second optical distortion caused by said second optical system (112) in said second optical image; obtaining transformation data suitable for correcting positional misalignment between said first and second optical images; processing first image data representing said first optical image based on said first distortion correction data for generating first distortion corrected image data; processing second image data representing said second optical image based on said second distortion correction data for generating second distortion corrected image data; and combining said first and second distortion corrected image data based on said transformation data for generating combined image data representing a combined image which corresponds to said target region (242) of said sample (104).

12. A method for calibrating an optical imaging device (102) for a microscope (100), said optical imaging device (102) comprising: a first optical system (108) configured to form a first optical image corresponding to a first region (210) of a sample (104) in accordance with a first imaging mode, a second optical system (112) configured to form a second optical image corresponding to a second region (214) of said sample (104), wherein said first and second regions (214) spatially coincide in a target region (242) of said sample (104) and said first and second imaging modes are different from each other, a processor (124), and a memory (134), wherein said method includes following calibration steps: obtaining first distortion correction data suitable for correcting a first optical distortion caused by said first optical system (108) in said first optical image; obtaining second distortion correction data suitable for correcting a second optical distortion caused by said second optical system (112) in said second optical image; obtaining transformation data suitable for correcting positional misalignment between said first and second optical images; and storing said first distortion correction data, said second distortion correction data and said transformation data in said memory (134) to be accessible by said processor (124).

13. The method according to claim 12, wherein a first optical reference image of a reference object (350) is formed using said first optical system (108) in accordance with said first imaging mode, a second optical reference image of said reference object is formed using said second optical system (112) in accordance with said second imaging mode, and said transformation data is determined based on positional misalignment of said first and second optical reference images.

14. The method according to claim 13, wherein said reference object comprises a grid (350) formed by a plurality of spots (352), said grid (350) being adapted to said first and second optical systems (112) such that each of the first and second optical systems (108, 112) is capable of imaging at least two of said plurality of spots (352) over a range of available magnifications.

15. A non-transitory computer-readable medium storing a computer program comprising instructions which, when the instructions are executed by a processor, cause the processor to perform the method according to claim 11.

Description

BRIEF DESCRIPTION OF THE DRAWING VIEWS

[0029] Hereinafter, specific embodiments are described referring to the drawings, wherein:

[0030] FIG. 1 is a schematic diagram showing an optical imaging device for a microscope according to an embodiment;

[0031] FIG. 2 is a diagram illustrating examples for a positional relationship between first and second regions of a sample to be imaged by the optical imaging device;

[0032] FIG. 3 is a diagram showing a reference object used for calibration according to an embodiment;

[0033] FIG. 4 is a diagram illustrating a tracking process for updating transformation data;

[0034] FIG. 5 is a flow diagram showing a method for calibrating the optical imaging device according to an embodiment;

[0035] FIG. 6 is a flow diagram showing a method for imaging a sample using the microscope according to an embodiment; and

[0036] FIG. 7 is a diagram illustrating a transformation of coordinate systems assigned to first and second optical images, respectively, into a common reference coordinate system.

DETAILED DESCRIPTION

[0037] FIG. 1 shows a microscope 100 comprising an optical imaging device 102 which is configured to image a sample 104 on a microscope stage 106 in accordance with different imaging modes. For instance, the optical imaging device 102 may serve to image the sample 104 in a wide-field imaging mode and a confocal imaging mode. Needless to say that these imaging modes are to be understood only as examples. Any other modes may be applied as long as these modes are adapted to provide image information which can be sensibly combined for generating a synthesized image.

[0038] The optical imaging device 102 comprises a first optical system 108 which is configured to form a first optical image corresponding to a first region 210 (see

[0039] FIG. 2) of the sample 104 in accordance with a first imaging mode being the wide-field imaging mode in the present embodiment. Likewise, the optical imaging device 102 comprises a second optical system 112 which is configured to form a second optical image corresponding to a second region 214 (see FIG. 2) of the sample 104 in accordance with a second imaging mode being the confocal imaging mode in the present embodiment. As schematically illustrated in FIG. 1, the first and second optical systems 108, 112 use different optical paths 116 and 118, respectively, along which detection light emanating from the sample 104 propagates through the optical system 108, 112 towards image sensors 120, 122 which are coupled to the optical systems 108, 112, respectively. The first image sensor 120 which is assigned to the first optical system 108 may be formed by a camera suitable for wide-field imaging. The second image sensor 122 which is assigned to the second optical system 112 may be formed by a sensor suitable for confocal imaging, e.g. a point detector.

[0040] The optical imaging device 102 further comprises a processor 124 which may serve to control the overall operation of the optical imaging device 102. In particular, the processor 124 is configured to process first image data representing a first optical image and second image data representing a second optical image, wherein the first and second optical images are generated on the image sensors 120, 122 by means of the first and second optical systems 108, 112, respectively. For this, the processor 124 is connected to the optical systems 108, 112 and the image sensors 120, 122 via control lines 126, 128, 130, 132.

[0041] The optical imaging device further comprises a memory 134 connected to the processor 124 via a control line 136. Further, a display unit 138 may be provided which is connected to the processor 124 via a control line 140.

[0042] Furthermore, in order to provide the different optical paths 116, 118 towards the first and second optical systems 108, 112, a beam splitter or any other suitable light deflector 142 may be included in the optical imaging device 102.

[0043] As mentioned above, the optical imaging device 102 is able to operate in the wide-field imaging mode and the confocal imaging mode in order to image the first region 210 and the second region 214 of the sample 104. The first and second regions 210, 214 spatially coincide in a target region of the sample 104 which is illustrated by a hatched area 242 in FIG. 2. The spatial coincidence may be realized in different ways. For example, the second region 214 assigned to the confocal imaging mode may be completely included in the first region as shown in FIG. 2(a). As the target region 242 is formed by an overlap of the first and second regions 210, 214, the target region 242 is identical to the second region 214 in the example illustrated in FIG. 2(a). Further, FIG. 2(b) shows an example in which the first and second regions 210, 214 are identical so that the target region 242 is identical to the respective regions likewise. In the example shown in FIG. 2(c), the first and second regions 210, 214 partially overlap so that the target region 242 is formed by an overlapping area which is common to both regions 210, 214. Needless to say that the spatial coincidence between the first and second regions 210, 214 is not limited to the examples shown in FIG. 2.

[0044] In the embodiment shown in FIG. 1, the memory 134 is provided for storing first distortion correction data and second distortion correction data. The first distortion correction data is suitable for correcting a first optical distortion, e.g. an optical aberration, which is caused by the first optical system 108 when generating the first optical image of the first region 210 of the sample 104 in accordance with the wide-field imaging mode. Likewise, the second distortion correction data serves to correct a second optical distortion which is caused by the second optical system 112 when generating the second optical image of the second region 214 of the sample 104 in accordance with the confocal imaging mode. As the first and second optical systems 108, 112 use the different optical paths 116, 118 for imaging the respective region 210, 214, the optical distortions induced by the first and second optical systems 108, 112 are independent of each other. Hence, the first and second distortion correction data can be independently determined and stored in the memory 134.

[0045] Further, the memory 134 stores transformation data which can be used for correcting positional misalignment between the first and second optical images created in the wide-field imaging mode and the confocal imaging mode, respectively. Whereas the first and second distortion correction data can be independently assigned to each optical system 108, 112, the transformation data stored in the memory 134 reflects the positional relationship between the optical systems 108, 112.

[0046] The processor 124 utilizes the first and second distortion correction data as well as the transformation data stored in the memory 134 for providing a combined image which corresponds to the target region 242 of the object 104. This combined image provides image information derived from both wide-field imaging and the confocal imaging. In order to create the combined image, the processor generates first distortion corrected image data by processing first image data representing the first optical image based on the first distortion correction data. Likewise, the processor 124 generates second distortion corrected image data by processing second image data representing the second optical image based on the second distortion correction data. Then, based on the transformation data, the processor combines the first and second distortion correction image data in order to create combined image data which represent the combined image to be displayed on the display unit 134.

[0047] The optical imaging device 102 may provide a calibration mode in which the processor 124 generates the transformation data and stores this data in the memory 134. This calibration mode is preferably applied in the process of manufacturing and assembling the optical imaging device 102 so that it can be automatically used at a later stage when a user operates the microscope 100 for imaging the sample 104. The first and second distortion correction data independently assigned to the respective optical systems 108, 112 may be generated by the processor 124 in the calibration mode likewise.

[0048] For calibrating the optical imaging device 102 a reference object may be used. Just as an example, such a reference object may be formed by a grid 350 as illustrated in FIG. 3.

[0049] The grid 350 comprises a plurality of spots 352 which are provided in a rectangular array. The grid 350 is adapted to the first and second optical systems 108, 112 such that both optical systems 108, 112 are capable to image at least two of the spots 352 over an available magnification range of the microscope 100 despite the fact that the optical systems 108, 122 apply different imaging modes.

[0050] For the purpose of calibration, the first optical system 108 generates a first optical reference image of the grid 350 in the wide-field imaging mode. Correspondingly, the second optical system 112 generates a second optical reference image of the grid 350 in the confocal imaging mode. Subsequently, the processor 124 generates the first and second distortion correction data and stores the data in the memory 134. Furthermore, the processor determines a positional misalignment between the first and second reference images representing the grid 350. Based on this misalignment, the processor 124 generates the transformation data and stores the data in the memory 134.

[0051] Using a reference object as shown in FIG. 3 for calibrating the optical imaging device 102 is to be understood merely as an example. Thus, the transformation data may also be generated based on live images which are created when the user operates the microscope 100 for imaging the sample 104. Further, live images may also be used in order to update the initial transformation data stored in the memory 134 in order to compensate for drift, structural modification, dynamic processes in the sample, etc. occurring during an experiment. For instance, the processor 124 may cause each of the optical systems 108, 112 to generate a sequence of optical images in accordance with the respective imaging mode. For each sequence, the processor determines a tracking mark 452 as illustrated in FIG. 4 within an initial image of this sequence and tracks the tracking mark 452 over time, i.e. over a plurality of images which are generated subsequent the initial image. Using the tracking mark 452, the processor 124 is enabled to generate updated transformation data for recalibrating the transformation without having to use a reference object as shown in FIG. 3.

[0052] In order to generate the transformation data, the processor 124 may be configured to determine correlation data which represents a correlation between the reference images. As explained above, images representing a reference object or live images may be used.

[0053] The flow diagram of FIG. 5 illustrates a method for calibrating the optical imaging device 102 according to an embodiment.

[0054] In step S2, the first optical system 108 images a reference object as illustrated in FIG. 3 for forming a first reference image. Image data representing the first reference image may be stored in an image memory (not shown in the Figures). In step S4, the processor 124 determines the first distortion correction data e.g. by comparing digital data representing the first reference image with nominal data which have been defined in advance to correspond to an ideal reference image, i.e. to an image which would be generated without any optical distortion. In step S6, the processor 124 stores the first distortion correction data in the memory 134.

[0055] In step S8, the second optical system 112 forms a second reference image, and the processor 124 stores corresponding image data in the image memory. In step S10, the processor 124 determines the second distortion correction data. In step S12, the processor 124 stores the second distortion correction data in the memory 134. The steps S8, S10, and S12 for generating and storing the second distortion correction data are performed in the same manner as steps S2, S4, and S6 with respect to the first distortion correction data.

[0056] In step S14, the processor 124 determines the transformation data based on the first and second reference images as explained above. Finally, in step S16, the processor 124 stores the transformation data in the memory 134.

[0057] The flow diagram shown in FIG. 6 illustrates a method for imaging the sample 104 according to an embodiment.

[0058] In step S12, the first optical system 108 forms the first optical image of the first target region 210 in accordance with the wide-field imaging mode. Correspondingly, in step 14, the second optical system 112 forms the second optical image of the second region 214 in accordance with the confocal imaging mode. As explained above with reference to FIG. 2, the first and second regions of the sample 104 spatially coincide in the target region 242. First and second image data representing the first and second optical images, respectively, are stored in the image memory in Steps 12 and 14.

[0059] After the first and second images have been generated, the processor 124 reads the first distortion correction data from the memory 134 in step S16. Likewise, in step S18, the processor 124 reads the second distortion correction data from the memory 134. In step S20, the processor 124 reads the transformation data from the memory 134.

[0060] In step S22, the processor 124 processes the first image data representing the first optical image based on the first distortion correction data in order to create first distortion corrected image data. Correspondingly, in step 24, the processor 124 processes the second image data representing the second optical image based on the second distortion correction data in order to create second distortion corrected image data. The first and second distortion corrected image data are stored in the image memory.

[0061] Finally, in step S26, the processor 124 combines the first and second distortion corrected image data using the transformation data read from the memory 134. Thus, combined image data are created which represent a combined image. The combined image corresponding to the target region 242 of the sample 104 may be displayed on the display unit 138. Further, the combined data may be stored in the image memory not shown in the Figures.

[0062] A specific transformation performed by the processor 124 for combining the first and second optical images is schematically illustrated in FIGS. 7. According to the example of FIG. 7, the processor 124 uses two mapping operation T′ and T″ for transforming a first coordinate system KS1 assigned to the first image and a second coordinate system KS2 assigned to the second image into a third coordinate system KS3. The third coordinate system KS3 represents a common reference system which is assigned to the combined image formed from the first and second images.

[0063] Needless to say the transformation shown in FIG. 7 is merely an example. Thus, the processor 124 may combine the first and second distortion corrected image data based on the transformation data such that the first coordinate system KS1 is mapped into the second coordinate system KS2 or vice versa.

[0064] Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Analogously, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus. Some or all of the method steps may be executed by (or using) a hardware apparatus, like for example, a processor, a microprocessor, a programmable computer or an electronic circuit. In some embodiments, some one or more of the most important method steps may be executed by such an apparatus.

[0065] Depending on certain implementation requirements, embodiments of the disclosure can be implemented in hardware or in software. The implementation can be performed using a non-transitory storage medium such as a digital storage medium, for example a floppy disc, a DVD, a Blu-Ray, a CD, a ROM, a PROM, and EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed. Therefore, the digital storage medium may be computer readable.

[0066] Some embodiments according to the disclosure comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0067] Generally, embodiments of the present disclosure can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may, for example, be stored on a machine readable carrier.

[0068] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0069] In other words, an embodiment of the present disclosure is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0070] A further embodiment of the present disclosure is, therefore, a storage medium (or a data carrier, or a computer-readable medium) comprising, stored thereon, the computer program for performing one of the methods described herein when it is performed by a processor. The data carrier, the digital storage medium or the recorded medium are typically tangible and/or non-transitionary. A further embodiment of the present disclosure is an apparatus as described herein comprising a processor and the storage medium.

[0071] A further embodiment of the disclosure is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may, for example, be configured to be transferred via a data communication connection, for example, via the internet.

[0072] A further embodiment comprises a processing means, for example, a computer or a programmable logic device, configured to, or adapted to, perform one of the methods described herein.

[0073] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0074] A further embodiment according to the disclosure comprises an apparatus or a system configured to transfer (for example, electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may, for example, be a computer, a mobile device, a memory device or the like. The apparatus or system may, for example, comprise a file server for transferring the computer program to the receiver.

[0075] In some embodiments, a programmable logic device (for example, a field programmable gate array) may be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware apparatus.

LIST OF REFERENCE SIGNS

[0076] 100 microscope

[0077] 102 optical imaging device

[0078] 104 sample

[0079] 106 microscope stage

[0080] 108 first optical system

[0081] 112 second optical system

[0082] 116 optical path

[0083] 118 optical path

[0084] 120 image sensor

[0085] 122 image sensor

[0086] 124 processor

[0087] 126 control line

[0088] 128 control line

[0089] 130 control line

[0090] 132 control line

[0091] 134 memory

[0092] 136 control line

[0093] 138 display unit

[0094] 140 control line

[0095] 142 light deflector

[0096] 210 first region

[0097] 214 second region

[0098] 242 target region

[0099] 350 grid

[0100] 352 spots

[0101] 452 tracking mark

[0102] KS1 first coordinate system

[0103] KS2 second coordinate system

[0104] KS3 third coordinate system

[0105] T transformation

[0106] T′ transformation