STAND FOR AN OPTICAL OBSERVATION UNIT, OPTICAL OBSERVATION APPARATUS, METHOD FOR CALIBRATING AN OPTICAL OBSERVATION APPARATUS, AND COMPUTER PROGRAM
20210157112 · 2021-05-27
Inventors
- Andreas Raab (Neuler, DE)
- Jonathan Essig (Neresheim, DE)
- Dominik SCHERER (Aalen, DE)
- Christian Voigt (Abtsgmuend, DE)
- Stefan Saur (Aalen, DE)
Cpc classification
G02B27/32
PHYSICS
International classification
Abstract
To simplify the optical calibration of an optical observation apparatus, a stand for an optical observation unit including a calibration object arranged directly on the stand in a fixed location is specified. Moreover, an optical observation apparatus, which includes such a stand and an optical observation unit connected to the stand, a method for calibrating such an optical observation apparatus, and a computer program are specified.
Claims
1. A stand for an optical observation unit, the stand comprising: a stand body; and, a calibration object arranged directly on said stand body in a fixed location.
2. The stand of claim 1, wherein said calibration object is a two-dimensional calibration pattern
3. The stand of claim 1, wherein said calibration object is a three-dimensional calibration body.
4. The stand of claim 1, wherein said calibration object is part of said stand body.
5. An optical observation apparatus comprising: a stand; a calibration object arranged directly on said stand in a fixed location; and, an optical observation unit connected to said stand.
6. The optical observation apparatus of claim 5 further comprising: a control unit configured to output control signals to at least one of said stand and said optical observation unit in order to carry out a calibration method.
7. A method for calibrating an optical observation apparatus having a stand, a calibration object arranged directly on the stand in a fixed location, and, an optical observation unit connected to the stand and having a camera, the method comprising: making recordings of the calibration object via the camera of the optical observation unit; and, evaluating the recordings.
8. The method of claim 7 further comprising: aligning the optical observation unit with respect to the calibration object.
9. The method of claim 8, wherein said aligning the optical observation unit is performed in an automated manner.
10. The method of claim 7 further comprising: carrying out an adjustment of a diopter setting on an eyepiece of the optical observation unit using the evaluated recordings.
11. The method of claim 7 further comprising: calibrating the camera of the optical observation unit using the evaluated recordings.
12. The method of claim 7 further comprising: performing an intrinsic calibration of the camera of the optical observation unit using the evaluated recordings.
13. The method of claim 7 further comprising: performing a hand/eye calibration of the camera of the optical observation unit using the evaluated recordings.
14. The method of claim 7 further comprising: performing an extrinsic calibration of internal cameras for at least one of topography applications and augmentation applications using the evaluated recordings.
15. The method of claim 7 further comprising: calibrating kinematics of the stand using the evaluated recordings.
16. A computer program comprising program code configured to, when executed by a processor, cause an optical observation apparatus having a stand, a calibration object arranged directly on the stand in a fixed location, and, an optical observation unit connected to the stand to carry out the method of claim 7.
17. The computer program of claim 16, wherein said program code is stored on a non-transitory computer readable medium.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
[0078] The invention will now be described with reference to the drawings wherein:
[0079]
[0080]
[0081]
[0082]
[0083]
[0084]
[0085]
[0086]
[0087]
[0088]
[0089]
DESCRIPTION OF THE PREFERRED EMBODIMENTS OF THE INVENTION
[0090]
[0091] Below, the stand 201 and the degrees of freedom facilitated by the stand for the optical observation unit 102 are described in more detail on the basis of
[0092] As stand links, the actual stand 201 includes a height-adjustable stand column 208, a support arm 209, a spring arm 210 and a mount for the optical observation unit 211, which in turn includes a connection element 213, a swivel arm 215 and a holding arm 214. The degrees of freedom provided by the stand links for positioning the optical observation unit 102 are shown in
[0093] The mount 211 has an axis of rotation D, a swivel axis E and a tilt axis F, about which the optical observation unit 102 can be rotated, swiveled, and tilted, respectively. Using a connection element 213, the mount 211 is fastened at the outer end of the spring arm 210 in a manner rotatable about the axis of rotation D. The axis of rotation D extends along the connection element 213. The connection element 213 is adjoined by a swivel arm 215, with the aid of which the optical observation unit 102, more precisely a holding arm 214 which is attached to the swivel arm 215 and on which holding arm the optical observation unit 102 is fastened via a holder (not illustrated), can be swiveled about the swivel axis E. The swivel axis E extends through the swivel arm 215. The angle between the swivel arm 215 and the connection element 213, that is, the angle between the swivel axis E and the axis of rotation D, can be varied via an adjustment mechanism arranged between the connection part 213 and the swivel arm 215.
[0094] The tilt axis F, which enables tilting of the optical observation unit 102, extends through the holding arm 214 in a manner perpendicular to the plane of the illustration. The optical observation unit 102 is fastened to the holding arm 214 via a holder (not illustrated here).
[0095] The degrees of freedom of the mount 211 and the adjustment options of the optical observation unit 102, for example, focusing, sharpness, magnification factor, et cetera, can be set by way of an actuating device 202, which is illustrated as a foot control panel in the present embodiment. However, the actuating device 202 can also be realized as a hand-operated switching element or as a combination of foot- and hand-operated switching element. Moreover, a remote control can be facilitated.
[0096] A calibration object 300 is arranged in a fixed location directly on the stand 201, to be precise in such a way that it is observable via the observation unit 102 connected to the stand 201. In
[0097] The calibration object 300 can be embodied as a two-dimensional calibration pattern, for example, as a checkerboard pattern, or as a three-dimensional calibration body. Alternatively, there is also the option of the calibration object 300 being integrated into the geometrical structure of the stand 201, that is, forming part of the stand 201.
[0098] Even if the stand 201 has been described on the basis of a specific example, a person skilled in the art will recognize that differently formed stands can also find use.
[0099]
[0100]
[0101]
[0102] As illustrated in
[0103] In the illustration of
[0104] Moreover, unlike what is shown in
[0105] Furthermore, it is possible to configure calibration marks situated further from the center of the 3D calibration body to be larger than calibration marks lying closer to the center of the 3D calibration body. Larger calibration marks can be obtained by virtue of calibration marks situated at the edge being formed by a larger number of adjacent pixels of the displays 301.sub.1 to 301.sub.9 than small calibration marks. What can be achieved as a result of using a pattern with calibration marks of different sizes is that, in the case of different zoom levels, the calibration marks in the images recorded with the different zoom levels substantially have the same size and distances that are suitable for the calibration.
[0106] Only a small section of the image is visible in the case of high zoom levels, for example the center of the 3D calibration body; by contrast, the entire calibration body is visible at low zoom levels. Moreover, high zoom levels show smaller structures than low zoom levels, and so the small, tightly adjacent, small calibration marks in the center of the pattern can find use in a high zoom level and the less densely packed, larger calibration marks at the edge of the pattern can be used at low zoom levels. Here, it is also possible, in principle, for a pattern with a constant spatial frequency, that is, a pattern with constant distances between the calibration marks in the entire calibration body, to be superposed on a pattern of calibration marks with a variable spatial frequency, that is, with distances of the calibration marks from one another, which distances vary over the calibration body.
[0107] As an alternative to the patterns described until now, it is also possible to generate self-similar patterns of calibration marks with the aid of the displays 301.sub.1 to 301.sub.9. By way of example, a checkerboard-like pattern of calibration marks can be formed, wherein the calibration marks can be formed by a square area in which a number of pixels 303 are switched to be non-transparent. Between these calibration marks there are square areas of identical size, in which all pixels 303 are switched to be transparent.
[0108] However, not all pixels 303 in a calibration mark are switched to be non-transparent; instead, the calibration mark itself represents, in turn, a checkerboard-like pattern made of now smaller square calibration marks with transparent square areas of identical size situated therebetween. The smaller calibration marks can also be formed by a display region in which a number of pixels 303 are switched to be non-transparent and a number of pixels 303 are switched to be transparent. In this way, each of the small calibration marks, in turn, still consists of a checkerboard-like pattern, which is formed of even smaller calibration marks with transparently switched pixel regions situated therebetween.
[0109] This can be continued until a checkerboard-like pattern of individual pixels 303 of the displays 301.sub.1 to 301.sub.9, which are alternately switched to be transparent and non-transparent, is present. The pattern can either be a pattern formed within a single display 301.sub.1 to 301.sub.9 or be a pattern formed together by a plurality of displays 301.sub.1 to 301.sub.9, such as the pattern shown in
[0110] The described patterns are not restricted to the use in a 3D calibration body; instead, they can find use in any calibration object, for example also a plane calibration object.
[0111] A further example of a calibration object 300 in the form of a 3D calibration body is illustrated in
[0112] There is light propagation, that is, a propagation of electromagnetic waves of the light, only within the respective light guides 307.sub.1 to 307.sub.5 on account of the total-internal reflection at the interfaces between the layers 308 and 309. By contrast, the electromagnetic fields of the light decrease exponentially in the layers 308.sub.1 to 308.sub.5 with the low refractive index, and so no propagation of the light occurs in these layers. The exponentially decaying electromagnetic fields are also referred to as evanescent fields.
[0113] In order to be able to represent calibration marks in the light guides 307.sub.1 to 307.sub.5 of the 3D calibration body, thin films 311 are applied at certain distances on the layers 309.sub.1 to 309.sub.5. Here, the refractive index of these films 311 is chosen in such a way that the total-internal reflection is suppressed at these points such that there is light propagation into the layers 308.sub.1 to 308.sub.5 with a low refractive index. By way of example, to this end, the refractive index of the films 311 has a value lying between the high refractive index of the layers 309 and the low refractive index of the layers 308.
[0114] If light is now coupled into one of the light guides 307.sub.1 to 307.sub.5, the light is output coupled from the light guide 307.sub.1 to 307.sub.5 at those points at which the films 311 have been applied such that luminous points arise in the volume of the transparent body constructed from the layers 308, 309, the luminous points serving as calibration marks. The luminous points arise at different depths of the transparent body depending on which light source 310.sub.1 to 310.sub.5 is activated. In order to be able to individually choose the depth at which calibration marks should be represented, the individual light sources 310.sub.1 to 310.sub.5 are selectively activatable and deactivatable via a controller 302.
[0115] In the example illustrated in
[0116] In respect of the films 311 that assist with the output coupling, it is also possible to generate the output coupling not by way of the suitable choice of a refractive index but by virtue of the evanescent fields inducing an emission of electromagnetic waves in the thin films, for example by virtue of exciting fluorescence.
[0117]
[0118] The method described with reference to
[0119] In step S2, which can be carried out after or at the same time as step S1, the optical observation unit 102 independently sets its maximum magnification.
[0120] In step S3, the calibration object 300 is illuminated or, should this be a self-luminous calibration object, it is activated. In the case of positioning in the z-direction that is too inaccurate (the Z-direction extends parallel to the optical axis of the surgical microscope), the surgical microscope can be refocused, for example, via a varioscope and image evaluation.
[0121] In step S4, a recording of the calibration object 300, for example, a photographic recording, is produced via the camera 103 of the optical observation unit 102.
[0122] The produced recording is evaluated in step S5 such that the focus can be set in step S6. In step S7, the zoom objective of the surgical microscope is set to the lowest magnification and the user is invited to put the image displayed to them in focus. With this, the method is completed.
[0123] Once the user has adapted the eye spacing, they can peer through the eyepiece and focus both partial images by rotating the diopter adjustment ring on the eyepiece.
[0124] In
[0125] After the start of the method, the optical observation unit 102 is aligned with respect to the calibration object 300 in step S1 such that the calibration object 300 is observable via the optical observation unit 102. By way of example, the alignment can be implemented by driving to a specifiable reference position via the motor-driven stand 201.
[0126] In step S3, the calibration object 300 is illuminated or, should this be a self-luminous calibration object, it is activated. In step S4, recordings of the calibration object 300, for example, photographic recordings, are produced via the camera 103 of the optical observation unit 102. Here, it is possible to produce recordings at different focal planes when necessary.
[0127] The recordings produced are evaluated in step S5. By way of example, it is possible to ascertain the relative position of the optical observation unit 102 with respect to the calibration object 300. Now, various calibration steps can be carried out in step S8; by way of example, a correction factor can be ascertained corresponding to the ascertained relative position and can be included in algorithms, or settings can be adjusted accordingly. With this, the method is completed.
[0128]
[0129] After the start of the method, the optical observation unit 102 is aligned with respect to the calibration object 300 in step S1 such that the calibration object 300 is observable via the surroundings camera 103, that is, is situated in the field of view of the surroundings camera 103. By way of example, the alignment can be implemented by driving to a specifiable reference position via the motor-driven stand 201. In step S3, the calibration object 300 is illuminated or, should this be a self-luminous calibration object, it is activated.
[0130] In step S4, a photographic recording of the calibration object 300 is produced via the surroundings camera 103 of the optical observation unit 102. The recorded image or the photo recording is stored in subsequent step S9, for example, in a memory unit assigned to a data processing unit of the optical observation apparatus 100.
[0131] In step S10, the optical observation unit 102 is moved relative to the calibration object 300. By way of example, the angle of the observation angle can be tilted. Subsequently, the method returns to step S4 and another recording of the calibration object 300 is produced. Steps S4, S9, and S10 are repeated until recordings of the calibration object 300 have been produced and stored in the complete image region of the surroundings camera 103 under different angles. Once this is the case, the method proceeds from step S10 to step S5.
[0132] In step S5, the recordings produced are evaluated; that is, the intrinsic calibration is calculated from the recorded images and stored in the surroundings camera 103 or in the memory unit. A recording log can be produced in the optional step S11, in which, for example, the movements of the optical observation unit 102 are noted such that the same movements can be carried out in each further intrinsic calibration.
[0133] In one variant of the method described in relation to
[0134] Via the hand/eye calibration, it is possible to ascertain how the image plane (chip) of the camera is physically assembled with respect to the TCP. This represents an extrinsic calibration. To this end, recordings of the calibration object are produced, wherein the position of the TCP is also registered for each recording. During the next recording, the observation angle and hence also the TCP are altered such that value pairs arise, which include an estimate of the camera position relative to the calibration object and the respective TCP. As a result, the absolute and fixed position of the image plane with respect to the TCP can be calculated by way of mathematical optimization methods. A calculation process can be gathered from the following publication: “A New Technique for Fully Autonomous and Efficient 3D Robotics Hand/Eye Calibration”, Tsai and Lenz, IEEE Transactions on Robotics and Automation, vol. 5, no. 3, June 1989.
[0135] In
[0136] After the start of the method, the optical observation unit 102 is aligned with respect to the calibration object 300 in step S1 such that the calibration object 300 is observable via the optical observation unit 102. By way of example, the alignment can be implemented by driving to a specifiable reference position via the motor-driven stand 201.
[0137] In optional step S3, the calibration object 300 is illuminated or, should this be a self-luminous calibration object, it is activated. In step S4, recordings of the calibration object 300, for example, photographic recordings, are produced via the internal camera 103 of the optical observation unit 102.
[0138] The recordings produced are evaluated in step S5. By way of example, correction factors can be calculated and updated. With this, the method is completed.
[0139] In
[0140] After the start of the method, the optical observation unit 102 is aligned with respect to the calibration object 300 in step S1 such that the calibration object 300 is observable via the optical observation unit 102. The alignment is brought about via the motor-driven stand 201.
[0141] In optional step S3, the calibration object 300 is illuminated or, should this be a self-luminous calibration object, it is activated. In step S4, a recording of the calibration object 300, for example, a photographic recording, is produced via a camera 103 of the optical observation unit 102.
[0142] In step S5, the recording produced is evaluated and the position of the optical observation unit 102 relative to the calibration object 300 is ascertained. Subsequently, the method can return to step S1 and a new position, not adopted previously, is adopted relative to the calibration object 300; that is, the optical observation unit 102 is aligned anew.
[0143] Once steps S1 to S5 have been carried out a sufficient number of times for the first calibration object 300, that is, once a sufficient number of different orientations between optical observation unit 102 and calibration object 300 have been analyzed, the method proceeds with step S12. In step S12, steps S1 to S5 are repeated, depending on requirements, for each further calibration object should a plurality of calibration objects be used.
[0144] In step S13, the parameters of the kinematics of the stand 201 are ascertained and optimized in such a way that the measured positions relative to the calibration object correspond to the values calculated by the kinematics.
[0145] In one variant, a calibration of the camera 103 and the kinematics can also be carried out simultaneously by virtue of the camera parameters additionally being incorporated in the optimization. The orientations with which the optical observation apparatus 100 aligns with respect to the calibration object or objects can be fixedly specified or can be ascertained online on the basis of previous results. As a result, the influence of still unknown or suboptimal parameters on the result can be increased and hence the number of orientations still necessary can be kept as low as possible.
[0146] The calibration object or objects 300 need not necessarily be arranged at the base of the stand 201. Alternatively, they can also be arranged at any other point on the stand 201. Hence, this only allows a calibration of the kinematic parameters of the stand links between the calibration object 300 and the optical observation unit 102 but, in return, these can be calibrated independently of the other parameters. Naturally, any combination of positions of the calibration objects 300 and/or a multi-stage procedure are also possible in order to be able to optimize the time outlay for calibration of the stand kinematics.
[0147] The present invention has been explained in detail on the basis of exemplary embodiments for explanatory purposes. However, a person skilled in the art will appreciate that they may depart from details of these exemplary embodiments.
[0148] Since it is possible to deviate from the individual described exemplary embodiments in a manner evident to a person skilled in the art, the present invention should not be restricted by the described exemplary embodiments, but merely by the attached claims.
LIST OF REFERENCE SIGNS
[0149] 100 Optical observation apparatus [0150] 102 Optical observation unit [0151] 103 Camera [0152] 104 Eyepiece [0153] 201 Stand [0154] 202 Actuating device [0155] 205 Stand base [0156] 206 Rollers [0157] 207 Foot brake [0158] 208 Stand column [0159] 209 Support arm [0160] 210 Spring arm [0161] 211 Mount for the optical observation unit [0162] 213 Connection element [0163] 214 Holding arm [0164] 215 Swivel arm [0165] 300 Calibration object [0166] 301.sub.1 to 301.sub.9 Transparent display [0167] 302 Controller [0168] 303 Pixel [0169] 303.sub.IT1 to 303.sub.IT4 Non-transparent pixel [0170] 304 Illumination panel [0171] 305 Light source [0172] 306 Diffusor plate [0173] 307.sub.1 to 307.sub.5 Light guides [0174] 308.sub.1 to 308.sub.5 Transparent layer with a low refractive index [0175] 309.sub.1 to 309.sub.5 Transparent layer with a high refractive index [0176] 310.sub.1 bis 310.sub.5 Light source [0177] 311 Film [0178] 401 Control unit [0179] 403 Control signal [0180] 404 Control signal [0181] A Axis of rotation [0182] B Axis of rotation [0183] C Tilt axis [0184] D Axis of rotation [0185] E Swivel axis [0186] F Tilt axis [0187] S1 to S13 Method steps