Microscopy System and Method for Editing Overview Images

Abstract

In a computer-implemented method for editing overview images of a microscope, a raw overview image is obtained in a first step. The raw overview image is transformed by means of calibration data into an overview image in which image directions coincide with given directions. At least one geometric property of text of at least one text region in the overview image is determined. A transformed text region is calculated taking into account the determined geometric property so that the transformed text region exhibits a desired geometric property of the text.

Claims

1. A computer-implemented method for editing overview images of a microscope, comprising: obtaining a raw overview image of a sample carrier with an overview camera of a microscope; transforming the raw overview image with calibration data in order to calculate an overview image in which image directions coincide with given directions; determining at least one geometric property of text of at least one text region in the overview image; calculating a transformed text region taking into account the determined geometric property so that a desired geometric property of the text occurs in the transformed text region.

2. The method according to claim 1, wherein the calibration data define or help to determine how an oblique view is converted into a top view through the transforming of the raw overview image and whether an image mirroring is required for calculating the overview image.

3. The method according to claim 1, wherein it is determined as a geometric property of the text whether the text is mirror-reversed, wherein, in cases where the text is mirror-reversed, the calculating of the transformed text region comprises a mirroring of the text region so that a mirror-reversed text orientation does not occur in the transformed text region.

4. The method according to claim 1, wherein a rotational orientation is determined as a geometric property of the text, wherein the calculating of the transformed text region comprises a rotation of the text region relative to adjacent image content so as to provide a certain rotational orientation in the transformed text region.

5. The method according to claim 1, wherein a perspective distortion is determined as a geometric property of the text, wherein the calculating of the transformed text region comprises a perspective rectification of the text region.

6. The method according to claim 1, wherein one or more of the following processes are carried out: checking whether the text satisfies a minimum size, wherein the text is enlarged relative to surrounding image content in cases where the minimum size is not satisfied; checking whether a text sharpness, a brightness, a contrast or a color tone in the text region meets predetermined criteria, wherein the text sharpness, brightness, contrast or color tone in the text region is modified in cases where the criteria are not met; checking whether there is a partial concealment of text, in which case characters are added to the text with a machine-trained model; checking whether there is smearing of text, in which case an image processing for removal of smearing occurs.

7. The method according to claim 1, wherein calibration data is used in at least one of the determining of the geometric property and the calculating of the transformed text region.

8. The method according to claim 1, wherein characters of the text in the text region are identified using optical character recognition (OCR) and wherein the transformed text region is generated by replacing the text with newly generated characters corresponding to the characters identified with OCR but differing in their arrangement so as to satisfy the desired geometric property.

9. The method according to claim 8, wherein the newly generated characters generated by OCR are saved in the overview image as an additional text layer in order to enable further machine-readable text processing.

10. The method according to claim 1, wherein the transformed text region is saved or displayed on a screen in addition to the overview image containing the text region, or wherein the text region in the overview image is replaced by the transformed text region.

11. The method according to claim 1, wherein, in cases where the transformed text region has a different size or shape than the text region so that a gap occurs when the text region is replaced by the transformed text region, the gap is filled with a model trained for image reconstruction.

12. The method according to claim 1, wherein a transition is generated for a boundary between the transformed text region and an adjacent image content of the overview image by inputting the overview image with the transformed text region into a model trained for image reconstruction, which modifies an image area around the boundary.

13. The method according to claim 1, Wherein the text in the overview image is removed using a model trained for image reconstruction and the transformed text region is inserted thereafter.

14. The method according to claim 1, wherein the transforming of the raw overview image is based on a homography estimation or involves a perspective distortion of the raw overview image.

15. The method according to claim 1, wherein, using the calibration data, the transforming of the raw overview image for calculating the overview image occurs so that image directions in the overview image coincide with an orientation of selectable directional commands for an adjustable sample stage, wherein the directional commands can be entered via software or a control element on the microscope, or image directions in sample images captured by a sample camera different from the overview camera or image directions visible using a microscope eyepiece.

16. A microscopy system including a microscope comprising an overview camera and a sample camera; and a computing device configured to execute the method according to claim 1.

17. A computer program, comprising commands stored on a non-transitory computer-readable medium which, when the program is executed by a computer, cause the computer to execute the method according to claim 1.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0047] A better understanding of the invention and various other features and advantages of the present invention will become readily apparent by the following description in connection with the schematic drawings, which are shown by way of example only, and not limitation, wherein like reference numerals may refer to alike or substantially alike components:

[0048] FIG. 1 schematically shows an example embodiment of a microscopy system of the invention;

[0049] FIG. 2 schematically shows processes of an example embodiment of a method according to the invention;

[0050] FIG. 3 schematically shows processes of a further example embodiment of a method according to the invention;

[0051] FIG. 4 shows a flowchart of an example embodiment of a method according to the invention.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

[0052] Different example embodiments are described in the following with reference to the figures.

FIG. 1

[0053] FIG. 1 shows an example embodiment of a microscopy system 100 according to the invention. The microscopy system 100 comprises a computing device 10 and a microscope 1, which is a light microscope in the illustrated example, but which in principle can be any type of microscope. The computing device 10 comprises a computer program 11 stored on a data carrier for processing a microscope image according to a method according to the invention. The microscope 1 comprises a stand 2 via which further microscope components are supported. The latter can in particular include: an illumination device 5; an objective changer/revolver 3, on which an objective 4 is mounted in the illustrated example; a sample stage 6 with a holding frame for holding a sample carrier 7; and a sample camera 8. When the objective 4 is pivoted into the light path of the microscope, the sample camera 8 receives detection light from a sample area in which a sample can be located in order to capture a sample image. A sample can be any object, fluid or structure. In addition to or instead of the sample camera 8, it is also possible to use an eyepiece 12 for the generation of a sample image. The microscope 1 comprises an additional overview camera 9 for capturing an overview image, i.e. a raw overview image, of a sample environment. A field of view 9A of the overview camera 9 is larger than a field of view when a sample image is captured. In the illustrated example, the overview camera 9 views the sample carrier 7 via a mirror 9B. The mirror 9B is arranged on the objective revolver 3 and can be selected instead of the objective 4. In variants of this embodiment, the mirror is omitted or a different arrangement of the mirror or some other deflecting element is provided.

[0054] In the example shown, the overview camera 9 views the sample stage 6 from above and thus views a top side of a sample carrier 7 arranged there. Alternatively, the overview camera 9 can also be arranged so as to view the sample stage 6 from below and thus view an underside of a sample carrier 7. In the example shown, the sample camera 8 views the sample carrier 7 from above, although it is alternatively also possible for it to view the sample carrier 7 from below in an inverted arrangement. The sample camera 8 and the overview camera 9 can in particular also be pointed at the sample carrier 7 from different sides. Depending on the arrangement of the overview camera 9, it can be preferable to mirror captured raw overview images so that image directions coincide with fixed directions, e.g., so that, for a user in front of the microscope, the directions left and right coincide with the image directions left and right and are not reversed. Although image mirroring can be advantageous for the orientation of a user, it can in particular create issues for the legibility of text on the sample carrier 7, as described in greater detail with reference to the following figures.

FIG. 2

[0055] FIG. 2 schematically shows processes of a method according to the invention. A schematically illustrated raw overview image 20 has been captured by means of an overview camera. The raw overview image 20 is an oblique view of a sample carrier 7, which is a microtiter plate in the illustrated example. Image directions X′ and Y′ of the raw overview image 20, e.g. its horizontal and vertical directions, ideally coincide with given directions X1 and Y1, which is not the case here. The directions X1 and Y1 can relate to, e.g., directions of the sample stage as seen by a user in front of the microscope. Additionally or alternatively, an overview image ideally exhibits a top view instead of an oblique view. A top view denotes a viewing direction perpendicular to the sample stage while in an oblique view a viewing direction forms an angle with the perpendicular to the sample stage.

[0056] In a step S2, the raw overview image 20 is transformed, whereby an overview image 30 is obtained. The transformation can take the form of, e.g., a homography estimation by means of which it is calculated or estimated how a plane in space is seen from a different angle. The sample stage surface or a plane parallel to it can be assumed as the plane. Calibration data K are used for the transformation. The calibration data K depend on a position and orientation of the overview camera and any optical elements located in the optical path of the overview camera, e.g. mirrors. The transformation for calculating the overview image 30 can also occur by means of other mathematical operations, which can in particular comprise a distortion (different modifications of image width and image height), a rotation and/or a mirroring.

[0057] In the illustrated example, the calculated overview image 30 corresponds to a top view of the sample carrier 7. Image directions X, Y of the overview image 30 coincide with the directions X1, Y1. The directions X1, Y1 can also be called microscope directions or fixed/given directions. In order to form the overview image 30, the raw overview image 20 was, inter alia, mirrored. Mirroring can in particular be appropriate when the overview camera and the sample camera view the sample or sample carrier 7 from different sides so that an image direction to the right in the overview image approximately coincides with an image direction to the right (and not to the left) in an image captured by the sample camera.

[0058] As a result of the mirroring, however, text 45 on the sample carrier 7 is depicted mirror-reversed in the overview image 30. A mirror-reversed representation can occur in other cases too, e.g., when a transparent sample carrier area includes text on its rear side as seen from the overview camera.

[0059] In the illustrated example, the overview image contains text 45 in separate areas (A, B, C, 1-4) indicating a column and row numeration of wells of the sample carrier 7.

[0060] Respective text regions 35 containing text 45 are identified by means of image analysis. The text regions 35 can either match the shape of the text 45 as exactly as possible, i.e. outline the text 45 ideally pixel-exactly, or contain a surrounding area around the text 45 as illustrated. The identified text regions 35 are analyzed by means of image analysis in order to establish geometric properties of the text 45, e.g., an orientation of the text 45, a text size in pixels, a character distortion, or an indication of whether the text 45 is displayed mirror-reversed. In the illustrated example, the image analysis reveals that the text 45 is displayed mirror-reversed in each instance. In step S7, transformed text regions 55 are calculated from the text regions 35 and, in step S8, the text regions 35 are replaced by the transformed text regions 55, whereby an edited overview image 50 is obtained. The established geometric properties, in this example the knowledge that the text is mirror-reversed, are used to calculate the transformed text regions 55. The transformed text regions 55 are thus calculated via a mirroring of the respective text regions 35. As a result, a desired geometrical property, namely the absence of a mirror-reversed representation, is provided in the transformed text region 55.

[0061] As a result of the transformation carried out with the calibration data K, the edited overview image 50 is suitable to act as a navigation map that a user can use to select a sample receptacle or sample area to be analyzed. At the same time, text 45 in the edited overview image 50 is easy to read for a user or software-based OCR.

[0062] A similar method variant is described in the following with reference to the next figure.

FIG. 3

[0063] Processes of a further example embodiment of a method according to the invention are shown in FIG. 3.

[0064] The illustrated overview image 30 is—with the exception of changes in brightness and contrast and the suppression of the 2D barcode—an actual example of an overview image calculated from a raw overview image of an overview camera using calibration data. The overview picture 30 shows a sample carrier (transparent slide) 7 with a sample 32 and a cover slip 31. The sample carrier 7 further comprises a label on which a text 45 is printed, e.g., regarding the sample type and sample preparation.

[0065] In step S3, there occurs a segmentation of the overview image 30 by means of a machine-learned (segmentation) model trained to localize a text region 35 in an overview image 30. An output of the segmentation model is a binary mask or segmentation mask 40 in which one pixel value indicates pixels belonging to the text region 35 while another pixel value designates an image area 42 that is not a text region. In this example, the segmentation model has been trained to identify labels as text regions 35 so that not only the characters per se but also surrounding pixels form part of the identified text region 35. Depending on training data, the segmentation model can be trained to only identify labels that actually contain characters as text regions 35 or to only identify sections of labels as text regions 35. The segmentation model can in particular be an instance segmentation model which distinguishes different text regions 35 from one another even when they touch or overlap. An instance segmentation model can output a plurality of binary masks or a segmentation mask with more than two different pixel values. Instead of a segmentation model, it is also possible to use a detection model to localize or determine text regions 35.

[0066] In step S4, the text region 35 of the overview image 30 localized by means of the segmentation mask 40 is extracted. An analysis and modification of the image brightness and contrast has also already been carried out in order to enhance a contrast of characters vis-à-vis a background. The text region 35 is analyzed with regard to geometric properties of its text 45, a mirror-reversed text representation being determined in this example. In step S7, a mirroring of the text region 35 is carried out in order to form a transformed text region 55. In step S8, the text region 35 is replaced by the transformed text region 55, whereby an edited overview image 50 is obtained.

[0067] In the illustrated example, a shape of the text region 35 is not exactly mirror-symmetrical. It is thus not possible for the transformed text region 55 to replace all image pixels of the text region 35 exactly, i.e. without gaps. If the segmentation is not perfect, it is also possible for edges or frames to occur, for example when an identified text region 35 includes not only the brighter label but also pixels from the darker area surrounding the label. An inpainting is accordingly employed in which a machine-learned model (reconstruction or inpainting model) realistically fills in gaps or flaws in an input image using other content in the image. It is possible to designate for the input image which image areas are to be filled in via inpainting. The input image in question can be the overview image including the transformed text region 55, wherein a boundary area is designated between the transformed text region 55 and the adjacent content of the overview image and modified accordingly by the inpainting model. The inpainting model can thus both fill in gaps and compensate artefacts of an imperfect segmentation.

[0068] The image processing in step S7 only relates to the text region and leaves image content 57 outside the text region (or outside the text regions in cases where a plurality of text regions are established) unchanged. As a result, text regions 55 and other image content 57 are thus transformed differently, in particular rotated, mirrored, distorted and/or rescaled differently relative to each other.

[0069] Further method variants are explained with reference to the following figure.

FIG. 4

[0070] FIG. 4 shows a flowchart of processes of an example embodiment of a method according to the invention.

[0071] In step S1, a raw overview image is obtained, e.g., captured by an overview camera or loaded from a memory. It is optionally also possible for raw data of the overview camera to have been pre-processed in order to generate the raw overview image.

[0072] In step S2, the raw overview image is transformed, which involves an image mirroring in the illustrated example, in order to form an overview image. Additionally or alternatively to the image mirroring, a rotation, a perspective distortion or a mapping onto another plane, e.g. in the form of a homography estimation, can also occur.

[0073] In step S3, the overview image is analyzed in order to localize text regions. This can occur, e.g., with an image segmentation model trained for this purpose. Depending on training data, the model can be trained to segment text pixel-exactly, to always segment text with surrounding pixels, or to segment areas (e.g. labels) that potentially contain text.

[0074] In step S5, geometric properties of the text in the text region or areas are established. This analysis can be carried out based on the entire overview image or based on the text regions only. It can be advantageous to take into account the entire overview image, e.g., when a text orientation can be detected via a sample carrier orientation. This can be the case, for example, with microtiter plates on which the text serves to identify the individual sample receptacles. The establishment of a geometric property can be carried out by means of a model trained for this purpose. The model can be learned, e.g., using training data showing respective image data of text and comprising as a target result an associated (e.g. manually specified) piece of geometric information, e.g. an indication of whether the text is displayed mirror-reversed or not. It is also possible to implement a model trained for OCR.

[0075] In step S6, it is evaluated whether a geometric correction of the text is required. Optionally, it is further evaluated whether further corrections of the text are required. To this end, e.g., the established geometric properties and optionally other established properties (e.g., image brightness, contrast or image definition of the text region) can be compared with predetermined criteria.

[0076] Depending on the outcome of the evaluation in step S6, the text region or areas are changed or transformed accordingly in step S7 so that a desired (geometric) property occurs in the text regions. The changes only relate to text regions so that, for example, text is displayed in a higher definition and a higher contrast while avoiding the risk that other image content appears unrealistic due to, e.g., an increased contrast.

[0077] In step S8, the at least one transformed text region is inserted into the overview image. Optionally, an inpainting can occur as described. In variants, an inpainting is implemented to remove text from the overview image calculated in S2. This is appropriate when the localization of text regions in S3 identifies characters as pixel-exactly as possible, i.e. without surrounding pixels or barely including surrounding pixels. These localized text regions can be replaced by means of inpainting in order to create a text-free background. Inpainting preserves a background structure, for example a pattern or brightness gradient on a label. The transformed text region in this example comprises the characters pixel-exactly and can now be inserted in the overview image modified by inpainting.

[0078] An order of the described steps can vary. For example, the inpainting for removing text can occur before or in parallel with the execution of the steps S5 and S6.

[0079] In an alternative to step S8, the transformed text region can also be displayed next to the overview image as calculated in S2. This likewise allows an improved legibility of the text while simultaneously providing a suitable orientation and perspective of the overview image.

[0080] The variants described in relation to the different figures can be combined with one another. The described example embodiments are purely illustrative and variants of the same are possible within the scope of the attached claims.

LIST OF REFERENCE SIGNS

[0081] 1 Microscope [0082] 2 Stand [0083] 3 Objective revolver [0084] 4 (Microscope) objective [0085] 5 Illumination device [0086] 6 Sample stage [0087] 7 Sample carrier [0088] 8 Sample camera [0089] 9 Overview camera [0090] 9A Field of view of the overview camera [0091] 9B Mirror [0092] 10 Computing device [0093] 11 Computer program [0094] 12 Eyepiece [0095] 20 Raw overview image [0096] 30 Overview image [0097] 31 Cover slip [0098] 32 Sample [0099] 35 Text region of an overview image which contains characters [0100] 40 Segmentation mask for the localization of text regions [0101] 42 Areas of a segmentation mask that do not designate a text region [0102] 45 Text of a text region [0103] 50 Edited overview image containing at least one transformed text region [0104] 55 Transformed text region [0105] 57 Image content outside text regions [0106] 100 Microscopy system [0107] K Calibration data [0108] S1-S8 Steps of example embodiments of methods according to the invention [0109] X, Y Image directions in the overview image [0110] X′, Y′ Image directions in the raw overview image [0111] X1, Y1 (Fixed/given) directions