METHOD FOR MONITORING AN IMMERSION FLUID IN A MICROSCOPE

Abstract

Provision is made for a method of monitoring an immersion fluid in a microscope having a lens which images a sample located on a sample carrier. In a step a), a camera is positioned which has an image field which is oriented in such a way that it captures the sample carrier and a space between the sample carrier and the lens and adjoining the sample carrier towards the lens, which space is used to receive the immersion fluid. In a step b), the immersion fluid is applied into the space between the sample carrier and the lens. In step c), an image with the immersion fluid being in the space between the sample carrier and the lens is recorded, and in a step d), the position, the area and/or the contour of the immersion fluid on the sample carrier from the image recorded in step d) are/is determined.

Claims

1. A method for monitoring an immersion fluid in a microscope having a lens which images a sample located on a sample carrier, comprising the steps of: a) positioning a camera which has an image field which is oriented in such a way that it captures the sample carrier and a space between the sample carrier and the lens adjoining the sample carrier towards the lens, which space is used to receive the immersion fluid, b) applying the immersion fluid into the space between the sample carrier and the lens, c) recording an image of the space between the sample carrier and the lens, with the immersion fluid being in said space, d) determining the position, area and/or contour of the immersion fluid on the sample carrier from the image.

2. The method according to claim 1, wherein the camera is calibrated after step a).

3. The method according to claim 2, wherein the calibration is carried out in that a calibration pattern is positioned in a plane of the sample carrier and in the image field of the camera, and a calibration image is recorded.

4. The method according to claim 1, wherein an image distortion caused by the position of the camera in the space between the sample carrier and the lens is determined and the determination of the position, area and/or contour of the immersion fluid on the sample carrier is carried out from the image recorded in step c) using said image distortion determined.

5. The method according to claim 1, wherein, after step a), a background image of the space between the sample carrier and the lens is recorded, and the determination of the position, area and/or contour of the immersion fluid on the sample carrier is carried out from the image using said background image.

6. The method according to claim 1, wherein, in step d), the position, area and/or the contour of the immersion fluid on the sample carrier is determined by searching for a structure in the image which is located within a certain distance from a center of the lens, cannot be assigned to the lens and/or has an approximately annular shape.

7. The method according to claim 1, wherein, in a step e), the contour and/or the area of the immersion fluid on the sample carrier is assessed and/or the volume of the immersion fluid is estimated and/or a remaining duration of the immersion fluid on the sample carrier due to evaporation is determined.

8. The method according to claim 7, wherein, in the event of excessive deformation, insufficient volume and/or insufficient remaining duration of the immersion fluid, a slower movement of the sample, a haptic feedback, a warning to a user and/or an automatic immersion is initiated.

9. The method according to claim 1, wherein a light source for illuminating the image field is used, which light source is attached to the camera.

10. The method according to claim 1, wherein a light source present on the microscope is used to illuminate the image field

11. The method according to claim 9, wherein the image is recorded by the camera in the infrared spectral range.

12. The method according to claim 1, wherein the image is recorded using a polarization filter.

13. The method according to claim 9, wherein, in step c), several differently illuminated images are recorded, from the entirety of which the contour, the area and/or the position of the immersion fluid is analyzed.

14. The method according to claim 1, wherein, in step d), contaminants in the immersion fluid are detected.

15. The method according to claim 14, wherein the detected contaminants are filtered out of the image.

16. A device for monitoring an immersion fluid in a microscope having a lens which images a sample located on a sample carrier, wherein a camera having an image field is positioned such that the image field is oriented in such a way that it captures the sample carrier and a space between the sample carrier and the lens adjoining the sample carrier towards the lens, which space is used to receive the immersion fluid, processing means which is connected to the camera via an electric line, and said processing means is configured in such a way that it determines the position, area and/or contour of the immersion fluid on the sample carrier by using an image of the space between the sample carrier and the lens with immersion fluid in the space recorded by the camera.

17. The device according to claim 16 having an application means which is connected to the processing unit via an electric line and applies the immersion fluid into the space between the sample carrier and the lens.

18. The device according to claim 16, wherein the processing means determines an image distortion caused by the position of the camera in the space between the sample carrier and the lens with the aid of a calibration image provided by the camera which calibration image shows the space between the sample carrier and the lens with a calibration pattern in a plane of the sample carrier and in the image field of the camera.

19. The device according to claim 16, wherein the processing means uses a background image of the space between the sample carrier and the lens recorded by the camera for the determination of the position, area and/or contour of the immersion fluid on the sample carrier.

20. The device according to claim 16, wherein the processing means is configured in such a way that it determines the position, area and/or contour of the immersion fluid on the sample carrier by searching for a structure in the image which is located within a certain distance from a center of the lens, cannot be assigned to the lens and/or has an approximately annular shape.

21. The device according to claim 16, wherein the processing means is configured in such a way that it assesses the contour of the immersion fluid on the sample carrier and/or estimates the volume of the immersion fluid and/or determines a remaining duration of the immersion fluid on the sample carrier due to evaporation.

22. The device according to claim 21, wherein the processing means is configured in such a way that, in the event of excessive deformation, insufficient volume and/or insufficient remaining duration of the immersion fluid, it initiates a slower movement of the sample, a haptic feedback, a warning to a user and/or an automatic immersion.

23. The device according to claim 16, wherein a light source for illuminating the image field is attached to the camera.

24. The device according to claim 16, wherein a light source present on the microscope illuminates the image field.

25. The device according to claim 16, wherein the camera records the image in the infrared spectral range.

26. The device according to claim 16, wherein a polarization filter is arranged on the camera in such a way that the camera records the image using said polarization filter.

27. The device according to claim 23, wherein the processing means is configured in such a way that, in step c), several differently illuminated images are recorded, from the entirety of which the contour, the area and/or the position of the immersion fluid is analyzed.

28. The device according to claim 16, wherein the processing means is configured in such a way that it detects contaminants in the immersion fluid.

29. The device according to claim 28 filtering out the detected contaminants from the image.

Description

BRIEF DESCRIPTION OF DRAWINGS

[0043] FIG. 1 shows a device for monitoring an immersion fluid in a microscope,

[0044] FIG. 2 shows a flow chart of the method for monitoring an immersion fluid in a microscope,

[0045] FIG. 3 shows a flow chart of a calibration process,

[0046] FIGS. 4a and b show a safety distance as a function of a numerical aperture of a lens,

[0047] FIGS. 5a and b show a contour of the immersion fluid on the lens, and

[0048] FIG. 6 shows an evaporation of the immersion fluid on a sample carrier over time.

DETAILED DESCRIPTION OF THE INVENTION

[0049] FIG. 1 illustrates a device for monitoring an immersion fluid 6 in a microscope 1. A microscope with a microscope lens (not shown separately) captures a sample 4 located on a sample carrier 2 along an optical axis OA. Immersion fluid 6 is applied to sample carrier 2 and spans sample 4. An application means 8 is provided for applying immersion fluid 6 into a gap between the microscope lens and sample 4. A camera 12 spanning an image field 10 is oriented towards sample 4 and immersion fluid 6. Both application means 8 and camera 12 are connected to processing means 16 via electric lines 14. The processing means is in turn connected to a display device 18 via an electric line 14.

[0050] The user is basically free to position camera 12. As a consequence, camera 12 can in principle be placed anywhere as long as sample carrier 2 with sample 4 located thereon and immersion fluid 6 are clearly visible in image field 10. In FIG. 1, the attachment of camera 12 is not shown; camera 12 can be attached, for example, to a transmitted light arm of microscope 1 or to a microscope stand. Due to the possibility of free placement of camera 12, a plane in which the sample is located is generally distorted in perspective in an image that is recorded by camera 12. After the perspectively distorted image has been recorded by camera 12, it is transmitted to processing means 16 via lines 14 or by radio.

[0051] In order to be able to find and assesse immersion fluid 6, the perspectively distorted image, which was transmitted from camera 12 to processing means 16 via lines 14, must be rectified by a transformation so that the sample plane is represented in a transformed image as if camera 12 had recorded the sample plane exactly from above. For this purpose, the device according to FIG. 1 remains in place, preferably only sample 4 is removed and a calibration pattern with a known structure is introduced into a plane of sample carrier 2. Camera 12 remains in the same position. If the calibration pattern is positioned parallel to the plane of sample carrier 2 (or ideally in its place), a calibration image is recorded by camera 12 and transmitted to processing means 16. Processing means 16 then determines a necessary transformation, a so-called homography H, on the basis of the perspectively distorted image and the calibration image, and identifies therefrom an image distortion in the space between sample carrier 2 and the lens. For this purpose, processing means 16 compares the structure of the calibration pattern with its image according to the calibration image.

[0052] If the distortion is known, camera 12 records a further image, a background image 22. In a preferred embodiment, background image 22 represents the space between sample carrier 2 and microscope lens without immersion fluid 6. Background image 22 consequently shows the appearance of the microscope lens in the image without interfering influences. Background image 22 can preferably also be generated by recording a set of typical images 20 with typical environmental variations, such as, for example, different illuminations. In modifications, background image 22 is recorded with immersion fluid 6 introduced into the space between sample carrier 2 and microscope lens. After background image 22 has been recorded by camera 12, it is transmitted to processing means 16 via lines 14. In the case of recording a plurality of images 20 with environmental variations, processing means 16 compiles background image 22 from individual images 20. In the event that background image 22 was recorded with immersion means 6 in the space between sample support 2 and the microscope lens, processing means 16 uses background image 22 itself to estimate how the microscope lens looks without immersion fluid 6.

[0053] Application means 8 is used to introduce immersion fluid 6 into the space between sample carrier 2 and the lens. For this purpose, application means 8 is controlled by processing means 16.

[0054] If immersion fluid 6 is in the space between sample carrier 2 and the lens, camera 12 records an image with immersion fluid 6 being in the space between sample carrier 2 and lens and transmits it to processing means 16. The position of camera 12 is the same as when background image 22 and the calibration image were recorded.

[0055] Processing means 16 corrects the image with regard to the image distortion and then determines the position and contour 34 of immersion fluid 6 on sample carrier 2. To this end, processing means 16 searches in the image for a structure that is located within a certain distance from a center of the microscope lens, cannot be assigned to the microscope lens and has an approximately annular shape.

[0056] Processing means 16 is further connected to imaging device 18 via lines 14. Imaging device 18 is, for example, a monitor on which processing means 16 displays the results.

[0057] Processing unit 16 preferably assesses contour 34 of immersion fluid 6 on sample carrier 2 and/or estimates the volume of immersion fluid 6 and/or determines a remaining duration of immersion fluid 6 on sample carrier 2 due to evaporation.

[0058] In modifications, a light source is provided in the device according to FIG. 1, which light source generates an illumination field which illuminates image field 10. The light source can be attached directly to camera 12, for example; a light source already present on microscope 1 can also be used to illuminate image field 10. When image field 10 is actively illuminated with the light source, camera 12 and the light source preferably operate in the infrared range of the electromagnetic spectrum. This allows immersion fluid 6 to be monitored in parallel with fluorescence microscopy.

[0059] In modifications, processing means 16 detects contaminants of immersion fluid 6 and filters them out of the image. To this end, processing means 16 searches for areas in the image which are within contour 34 of immersion fluid 6 and have a strong visual deviation from the image. The contaminants can be, for example, air bubbles or dust. Processing means 16 is able to combine the detected contaminants with background image 22 and thereby remove them from the image. This can be done using standard algorithms from the field of image processing and/or machine learning.

[0060] FIG. 2 shows a flow chart for the method for monitoring immersion fluid 6 in a microscope 1 having a microscope lens which images a sample 4 located on a sample carrier 2.

[0061] In a step S1, camera 12 is oriented in such a way that image field 10 captures sample carrier 2 and the space between sample carrier 2 and microscope lens adjoining sample carrier 2 towards the microscope lens, which space is used to receive immersion fluid 6.

[0062] The camera is calibrated in a step S2. For this purpose, the calibration pattern is positioned parallel to the plane of sample carrier 2, ideally instead of sample carrier 2, and in image field 10, and the calibration image is recorded by camera 12.

[0063] Subsequently, in a step S3, the image distortion in the space between sample carrier 2 and microscope lens, which is a function of the position of camera 12, is determined. Homography H, which maps the plane of camera 12 onto the plane of sample carrier 2, is also determined or estimated. As a result, the images distorted due to the placement of camera 12, which are recorded in the further steps of the method, can be rectified based on the calibration image.

[0064] In a step S4, background image 22 of the space between sample carrier 2 and the lens is recorded. Background image 22 can be generated, for example, by a single recording of the space between the microscope lens and sample carrier 2 without interfering influences such as immersion fluid 6. Background image 22 can also be generated by recording a plurality of images 20 with typical environmental variations, such as, for example, different illuminations. Background image 22 is then, as illustrated in FIG. 5, composed of a plurality of images 20 by processing means 16. The appearance of the microscope lens without immersion fluid 6 can also be estimated by processing means 16 from a background image 22 with immersion fluid 6 being in the space between the lens and sample carrier 2.

[0065] In a step S5, application means 8 is used to introduce immersion fluid 6 in the space between microscope lens and sample carrier 2, before in a step S6, the image with immersion fluid 6 being in the space between the microscope lens and sample carrier 2 is recorded.

[0066] Then, in a step S7, the position and contour 34 of immersion fluid 6 on sample carrier 2 are determined from the image recorded in step S6 and the image distortion determined in step S3. Contour 34 of the immersion fluid on the sample carrier is preferably determined by searching for a structure in the image which is located within a certain distance from a center of the microscope lens, cannot be assigned to the microscope lens and has an approximately annular shape.

[0067] In modifications, following step S7, contour 34 of immersion fluid 6 on sample carrier 2 is assessed and/or the volume of immersion fluid 6 is estimated and/or the remaining duration of immersion fluid 6 on sample carrier 2 due to evaporation is determined.

[0068] A plane E1 of the camera image, a plane E2 of sample carrier 2 and a plane E3 of the calibration pattern are preferably used for calibration and determination of the image distortion. As already explained, an image distortion based on a calibration process is determined from these three planes in steps S2 and S3. An overview of the determination of the image distortion is depicted in FIG. 3.

[0069] FIG. 3 illustrates calibration process K in conjunction with the determination of the image distortion of processing means 16. Image coordinates C1 of plane E1 are linked with calibration coordinates C3 and sample carrier coordinates C2 via homography H. A transformation T takes place between sample carrier coordinates C2 and calibration coordinates C3.

[0070] Image coordinates C1 in plane E1 of the camera image represent sample carrier plane E2 as if camera 12 had recorded the image exactly from above. To calculate the distortion, sample carrier coordinates C2 and calibration coordinates C3 are first calculated in a transformation T and scaled. Coordinates C2 and C3 are then linked to image coordinates C1 via homography H. Homography H is used to map plane E1 onto plane E2. This makes it possible to rectify the images. Of course, the camera position and orientation can no longer be changed. In addition, with the aid of the calibration K, distortions that arise from the camera optics (e.g., barrel-shaped curvature) can be subtracted from the image.

[0071] In a modification, the point of intersection of the optical axis OA of the camera lens with sample carrier plane E2 in the image is localized in calibration K. In the event that camera 12 is oriented centrally from above or below towards sample carrier 2, this point of intersection is at the center point of the camera lens. The position of the camera lens in the image is normally known from the camera structure, but it can alternatively also be determined using a detection algorithm, possibly also manually.

[0072] FIG. 4A shows a front lens 36 having a diameter D1. A cover slip 40 is also shown. Immersion fluid 6 is located in the form of drops between cover slip 40 and front lens 36. A safety area 42 and an opening angle 44 of microscope lens 38 are defined. FIG. 4B shows a front lens 36 of a microscope lens with a comparatively higher aperture. Its diameter is D2. Immersion fluid 6 is located in the form of drops between cover slip 40 and front lens 36. A safety area 48 and an opening angle 50 of microscope lens 46 are defined.

[0073] The quality of the drop of immersion fluid 6 can be assessed by the extent to which front lens 36 of microscope lens 38, 46 is covered with immersion fluid 6. Microscope lenses 38, 46 with different apertures differ in the size of front lens 36. The size of front lens 36 can be taken from a lens database, for example. The numerical aperture describes the ability of a microscope lens 38, 46 to focus light. A higher aperture angle 44, 50 is therefore also associated with a higher aperture. For this reason, lens 38 has a smaller opening angle 44 than lens 46 with its opening angle 50.

[0074] For this reason, as depicted in FIGS. 4A and 4B, safety area 42, 48, which is dependent on the distance between sample carrier 2 and front lens 36, is defined around front lens 36 and which must be covered with immersion fluid 6 in order to ensure optimal image quality. This ensures that the front lens 36 and safety area 42, 48 are always adequately covered with immersion fluid 6, and appropriate countermeasures, such as warning the user or the automated introduction of immersion fluid 6 via application means 8, can be initiated in good time. This ensures that the image quality remains constant over the entire duration of the experiment.

[0075] FIGS. 5A and 5B each show a resulting image 32 with a contour 34 of immersion fluid 6.

[0076] An ideal drop of immersion fluid 6 completely wets front lens 36 of microscope lens 38, 46 and has a circular shape on sample carrier 2, which is shown in the image as a circular contour 34 (FIG. 7A). Due to external influences such as the movement of the sample table or if the volume of immersion fluid 6 is too small/large, it can happen that front lens 36 is not completely covered and/or the shape deviates significantly from that of a circle (FIG. 7B).

[0077] The quality of the drop of immersion fluid 6 is determined in embodiments in regards to the extent that the drop resembles a circle by calculating suitable features, such as, e.g., the eccentricity of an ellipse, which optimally approximates contour 34, of the region defined by contour 34 in resulting image 32 and comparing them with manually defined data or training data. Contour 34 of immersion fluid 6 is classified from “sufficiently circular” to “too strongly deformed”. In the event of excessive deformation, countermeasures, such as moving the table more slowly, haptic feedback, a warning to the user, etc., are carried out depending on the microscopy application.

[0078] In addition, a contact area of immersion fluid 6 on the sample carrier can easily be determined on the basis of the contour. In order to additionally be able to estimate the volume of the immersion fluid, calibration measurements are provided for each microscope lens 38, 46. With a known distance between microscope lens 38, 46 and sample carrier 2, several drops of immersion fluid 6 with different, known volumes are applied and their contact area with sample carrier 2 is then automatically determined from resulting image 32. This results in a 1:1 mapping between the volume of the immersion drop and its contact area for each microscope lens 38, 46. The volume of the drop of immersion fluid 6 is thus estimated from the contact area given a known distance between sample carrier 2 and microscope lens 38, 46. Optionally, it is also estimated how much volume still needs to be applied in order to achieve a desired target volume. Furthermore, by determining the volume of immersion fluid 6 and knowing the geometry of microscope lens 38, 46, it is possible to estimate whether the amount of immersion is sufficient for a predetermined recording in order to ensure a desired image quality at every position. Conversely, if the volume of immersion fluid 6 is known, it is also possible to determine the distance between sample carrier 2 and microscope lens 38, 46 from the contact area, which can be used, for example, as collision protection.

[0079] In FIG. 6, the monitoring of a drop of immersion fluid 6, in this case water, is depicted in resulting image 32a, b, c over a certain period of time. Evaporation E is depicted. The expected area of the immersion fluid is depicted on the y-axis, and the time course is depicted on the x-axis.

[0080] As a result of the evaporation of immersion fluid 6 on sample carrier 2, the contact area of sample carrier 2 with the immersion fluid becomes continuously smaller. Regression can be used to predict how the drop will change over time. The development over time of the amount of immersion fluid 6 on sample carrier 2 is monitored and predicted by processing means 16 on the basis of the images from camera 12. This ensures a timely warning to the user in the event of a critical reduction in immersion fluid 6 on sample carrier 2.

LIST OF REFERENCE NUMERALS

[0081] 1 microscope [0082] 2 sample carrier [0083] 4 sample [0084] 6 immersion fluid [0085] 8 application means [0086] 10 field of view [0087] 12 camera [0088] 14 electric lines [0089] 16 processing means [0090] 18 imaging device [0091] 20 typical image [0092] 22 background image [0093] 24 image [0094] 26 difference image [0095] 28 difference image in polar coordinates [0096] 30 path [0097] 32 resulting image [0098] 34 contour [0099] 36 front lens [0100] 38 low aperture lens [0101] 40 cover slip [0102] 42 security area [0103] 44 opening angle [0104] 46 high aperture lens [0105] 48 security area [0106] 50 opening angle [0107] C1 image coordinates [0108] C2 sample carrier coordinates [0109] C3 calibration coordinates [0110] D1 diameter of front lens [0111] D2 diameter of front lens [0112] E1 plane of the camera [0113] E2 sample carrier plane [0114] E3 calibration plane [0115] H homography [0116] K calibration [0117] OA optical axis [0118] S1 step 1 [0119] S2 step 2 [0120] S3 step 3 [0121] S4 step 4 [0122] S5 step 5 [0123] S6 step 6 [0124] S7 step 7 [0125] T transformation