Method for Imaging Objects Contained in a Droplet
20250085665 ยท 2025-03-13
Inventors
- Yuqian Li (Heverlee, BE)
- Ziduo LIN (Heverlee, BE)
- Murali Jayapala (Kumtich, BE)
- Zhenxiang Luo (Leuven, BE)
- Vasileios Lemonidis (Heverlee, BE)
Cpc classification
G03H2001/0825
PHYSICS
G03H2001/005
PHYSICS
G03H1/0443
PHYSICS
G03H1/0808
PHYSICS
G03H1/22
PHYSICS
G03H1/0866
PHYSICS
G06V20/647
PHYSICS
G03H1/08
PHYSICS
International classification
Abstract
Examples include imaging one or more objects contained inside a droplet. A method includes generating at least one hologram of the one or more objects contained in the droplet by using in-line lens-free imaging. The at least one hologram includes at least one artifact that is caused by the droplet and that affects the at least one characteristic of the one or more objects contained in the droplet. The method includes at least partially removing the at least one artifact or the cause of the at least one artifact. The method further includes generating an image, after or during removing the at least one artifact or the cause of the at least one artifact. The image includes the one or more objects. The method also comprises recognizing the at least one characteristic of the one or more objects based on the image.
Claims
1. A method comprising: generating a hologram of an object contained in a droplet, wherein the hologram is generated using in-line lens-free imaging and includes an artifact that is caused by the droplet and that affects a characteristic of the object contained in the droplet; at least partially removing the artifact or a cause of the artifact from the hologram; thereafter using the hologram to generate an image comprising the object; and determining the characteristic of the object based on the image.
2. The method according to claim 1, wherein the artifact comprises an interference in the hologram of a holographic signal of the droplet.
3. The method according to claim 1, wherein the artifact comprises an interference in the hologram of a holographic signal of the object.
4. The method according to claim 1, wherein the artifact comprises noise in a holographic signal of the droplet.
5. The method according to claim 1, wherein the artifact comprises noise in a holographic signal of the object.
6. The method according to claim 1, wherein at least partially removing the artifact or the cause of the artifact comprises regenerating the hologram of the object, wherein a second illumination wavefront is used for regenerating the hologram that is different from a first illumination wavefront that was used for generating the hologram.
7. The method according to claim 6, wherein the second illumination wavefront comprises a single wavelength.
8. The method according to claim 6, wherein the second illumination wavefront comprises multiple wavelengths.
9. The method according to claim 1, wherein at least partially removing the artifact or the cause of the artifact comprises processing the hologram with a numerical optimization algorithm.
10. The method of claim 9, wherein the numerical optimization algorithm comprises a fast iterative shrinkage-thresholding algorithm.
11. The method of claim 9, wherein the numerical optimization algorithm comprises an Alternating Direction Method of Multipliers algorithm.
12. The method according to claim 1, wherein at least partially removing the artifact or the cause of the artifact comprises processing the hologram with an image processing method.
13. The method according to claim 1, wherein generating the image comprises generating the image by multi-depth reconstruction of the hologram.
14. The method according to claim 1, further comprising: determining a strength of the artifact; and selecting a process for at least partially removing the artifact or the cause of the artifact based on the strength of the artifact.
15. The method according to claim 14, wherein determining the strength of the artifact comprises comparing first parameters of the object in the hologram and second parameters of the object in a reference hologram.
16. The method according to claim 1, wherein the object comprises one or more objects and the characteristic of the one or more objects comprises: a number of the one or more objects contained in the droplet; a density of the one or more objects in the droplet; a respective size and/or shape of each object of the one or more objects in the droplet; a respective position of each object of the one or more objects in the droplet; a motion pattern of the one or more objects in the droplet; or a reflective index of the one or more objects in the droplet.
17. The method of claim 1, wherein at least partially removing the artifact or the cause of the artifact comprises using a computational model to at least partially remove the artifact or the cause of the artifact.
18. The method of claim 17, wherein the computational model comprises a deep learning neural network.
19. A computer program comprising instructions that, when executed by a processor, causes the processor to perform the method according to claim 1.
20. An apparatus comprising: a holographic imager configured to generate a hologram of an object contained in a droplet using in-line lens-free imaging, wherein the hologram includes an artifact that is caused by the droplet and that affects a characteristic of the object in the droplet; and a processor configured to at least partially remove the artifact or a cause of the artifact from the hologram, wherein the processor is further configured to generate an image comprising the object after partially removing the artifact or the cause of the artifact and to determine the characteristic of the object based on the image.
Description
BRIEF DESCRIPTION OF THE FIGURES
[0042] The above, as well as additional, features will be better understood through the following illustrative and non-limiting detailed description of example embodiments, with reference to the appended drawings.
[0043]
[0044]
[0045]
[0046]
[0047]
[0048]
[0049]
[0050]
[0051]
[0052]
[0053]
[0054]
[0055]
[0056]
[0057]
[0058] All the figures are schematic, not necessarily to scale, and generally only show parts which are necessary to elucidate example embodiments, wherein other parts may be omitted or merely suggested.
DETAILED DESCRIPTION
[0059] Example embodiments will now be described more fully hereinafter with reference to the accompanying drawings. That which is encompassed by the claims may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided by way of example. Furthermore, like numbers refer to the same or similar elements or components throughout.
[0060]
[0061] As has been discussed above, artifacts may appear in the hologram 13 of the object 11 contained in the droplet 12, which may make it hard to image and recognize characteristics of the object 11. Therefore,
[0062] The method 20 comprises a step 21 of generating at least one hologram 13 of the objects 11 contained in the droplet 12 using in-line and lens-free holographic imaging. For reasons of simplicity, like in
[0063] The method 20 further comprises a step 22 of removing fully or partially the at least one artifact 14, or the cause of the at least one artifact 14. Different ways to remove the artifact 14 or the cause of the artifact 14 are described below.
[0064] The method 20 further comprises a step 23 of generating an image 15 after or during removing the at least one artifact 14 or the cause of the at least one artifact 14. The image 15 may be generated by reconstruction of the hologram 13 after artifact removal, for example, by reconstruction methods known in the art. The image 15 comprises the object 11 and may comprise the droplet 12 as shown in
[0065] The method 20 then comprises a step 24 of recognizing the at least one characteristic 25 of the object 11 based on the image 15, wherein the determination of the characteristic 25 is not anymore prevented or made difficult by the artifact 14. The at least one characteristic 25 may be directly derived from the image 15, for instance, if the characteristic is a shape, size, location, or the like. Other characteristics 25 could be obtained by post-processing the image 15.
[0066]
[0067] Also, noise in the holographic signal 12a of the droplet 12 and/or noise in the holographic signal 11a of the object 11 may cause an artifact 14. As shown in the middle of
[0068] As shown on the right side of
[0069] A size of the droplet 12.
[0070] A 3D shape of the droplet 12, e.g., round, square, an irregular shape, a contact angle difference, etc.
[0071] A mismatch of an optical property of the material of the droplet 12 and the environment.
[0072] A droplet density in case of multiple droplets 12.
[0073] An object density in a droplet 12, in case of multiple objects 11 inside a droplet 12.
[0074] A homogeneity of multiple droplets 12, in particular, a homogeneity in shape, size, and/or material.
[0075] A homogeneity of objects 11 inside a droplet 12, in particular, a homogeneity in size, shape, and/or type of object 11.
[0076] Foreign objects (not to be analyzed) in the droplet 12 (e.g., inside, outside, variations of the foreign objects in shape, size, materials, etc.).
[0077] Multiple droplets 12 at different heights/depth into the imaging direction.
[0078] Multiple objects 11 located at different heights/depth inside a droplet 12 in the imaging direction.
[0079]
[0080] The pipeline may provide various methods for removing the at least one artifact 14 or the cause of the at least one artifact 14. In particular, the pipeline may offer wavefront engineering 41, deep learning filtering 42x, a FISTA algorithm or a similar algorithm, conventional image and signal processing 44, and multi-depths reconstruction 46 as possible methods to remove the artifact 14.
[0081] One or more of the various methods may be selected, for instance, depending on the type of the at least one artifact 14 and/or based on environmental conditions. In the example of
[0082] If this is not the case, the pipeline further determines whether the at least one artifact 14 and the object 11 are consistent. If yes, then deep learning 42x can be applied. Afterwards, any reconstruction method 45e.g. as described in the literaturecan be applied to reconstruct the image 15 from the hologram 13.
[0083] After applying the wavefront engineering 41, the pipeline may further determine how the artifact 14 caused by the droplet 12 is compared to the object signal. Possibilities are strong or not regular, weak or regular, or no artifacts. If there are no artifacts 14, the conventional reconstruction 45 can be used as described above.
[0084] In the case of stronger or not regular artifacts 14, the FISTA algorithm 43 can be applied. In the case of weaker or regular artifacts 14, conventional image and signal processing 44 may be applied, and may be sufficient to remove such artifacts 14.
[0085] Subsequently, the pipeline may further determine whether there are objects 11 or droplets 12 at different heights/depth of the imaging direction. If yes, then multi-depths reconstruction 46 can be applied, which may be followed by post-processing afterwards, to generate the final image 15. If not, then the post-processing can be applied directly.
[0086]
[0087] In view of
[0088] Notably, the strength of the at least one artifact 14 may be determined by comparing at least one parameter of the object 11 in the hologram 13 (with the droplet 12) and in at least one reference hologram of the object 11 without droplet 12.
[0089] In the following, the various methods for removing the artifact(s) 14 or cause of the artifact(s) 14 are illustrated and explained.
[0090]
[0091] Before applying the FISTA algorithm 43, regular pre-processing steps may be applied to the hologram 13, for instance illumination balancing. After applying the FISTA algorithm 43, a high-quality reconstruction may be performed to obtain the image 15.
[0092]
[0093]
[0094]
[0095]
[0096]
[0097] The method 100 comprises a step 101 of obtaining at least one hologram 13 of the one or more objects 11 contained in the droplet 12. For instance, the at least one hologram 13 may be provided to the computer or processor that implements the method 100, and/or may be obtained from a holographic imager. The at least one hologram 13 has been previously generated using in-line lens-free imaging, for example, by the holographic imager. As in the method 20 of
[0098] The method 100 further comprises a step 102 of processing the at least one hologram 13 using a computational model 42, for instance, a neural network. The computational model 42 removes fully or partially the at least one artifact 14. The hologram 13 with the artifact(s) 14 may be input into the computational model 42, and the hologram without the artifact(s) may be generated as output by the computational model 42. The computational model 42 may be a trained and/or trainable model.
[0099] The method 100 further comprises a step 103 of generating an image 15, after or during the removing step 102 of the at least one artifact 14, wherein the image 15 comprises the one or more objects 11. The image 15 may be generated by reconstruction of the hologram 13. Then, the method 100 comprises a step 104 of recognizing the at least one characteristic 25 of the one or more objects 11 based on the image 15. The steps 103 and 104 may be either or both performed by the computational model 42.
[0100] According to the above, the computational model 42 may remove artifacts 14 caused by droplets 12 on the hologram level, prior to reconstruction.
[0101] A simulated dataset may be used to train the computational model 42, wherein the following parameters may generally be considered in the simulated dataset: experimental conditions; desired artifacts properties; or expected objects properties. Specifically, the dataset may include or be based on the certain parameters e.g. sensor specifications, optical parameters of the setup, the morphological features (like size, shape, etc.) and properties (for example, reflex index, which is related to the materials) of the droplets, objects, properties of the artifacts, or simulated setup configurations.
[0102] During the training of the computational model 42, the input may be a signal that includes both a plurality of possible artifacts 14 and a plurality of objects 11 in any combination with another. The expected output to train the model 42 is a signal comprising the objects 11 alone. During the training, a loss may be a mean squared error (MSE) loss, an ADAM optimizer may be used, a starting learning rate (LR) may be small, and an LR update policy may be employed.
[0103] The main observations of using the computation model 42 are that the result is quite satisfactory on the simulated data, and that the removal of the artifacts 14 from holograms 13 including droplet signal and the object signal works well.
[0104]
[0105] The apparatus 1200, to this end, comprises a holographic imager 1201, which is configured to generate at least one hologram 13 of the one or more objects 11 contained in the droplet 12 using in-line lens-free holographic imaging. The holographic imager 1201 may to this end illuminate the droplet 12 including the object 11, for instance, with a selected wavefront including one or more wavelengths. Wavefront engineering 41 may be applied using the holographic imager 1201. Like in
[0106] The apparatus 1200 further comprises a processor 1202, which is configured to remove fully or partially the at least one artifact 14 or the cause of the at least one artifact 14. The processor 1202 is further configured to generate an image 15, after or during removing the at least one artifact 14 or the cause of the at least one artifact 14, comprising the one or more objects 11, and to recognize the at least one characteristic 25 of the one or more objects 11 based on the image 15. The processor 1202 is accordingly configured to perform artifact removal and reconstruction of the hologram 13, for example, by reconstruction methods known in the art.
[0107] The processor 1202 may comprise processing circuitry (not shown) configured to perform, conduct, or initiate the various operations described. The processing circuitry may comprise hardware and/or the processing circuitry may be controlled by software. The hardware may comprise analog circuitry or digital circuitry, or both analog and digital circuitry. The digital circuitry may comprise components such as application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), digital signal processors (DSPs), or multi-purpose processors. The processor 1202 may further comprise memory circuitry, which stores one or more instruction(s) that can be executed by the processor 1202 or by the processing circuitry, in particular under control of the software. For instance, the memory circuitry may comprise a non-transitory storage medium storing executable software code which, when executed by the processor or the processing circuitry, causes the various operations of the processor 1202 to be performed.
[0108]
[0109] The holographic imager 1201 further comprises a recording medium 1201b, typically a digital sensor, which is configured to record an interference pattern of the respective light wavesi.e., those light waves that have interacted with the droplet/object and those light waves which have not. The holographic imager 1201 is in this way configured to obtain the hologram 13. Since the recording medium 1201b is arranged behind the droplet/object along the light path originating at the light source 1201a, and is thus arranged to record also directly light waves passing through the droplet 12, the holographic imager 1201 has an in-line configuration for in-line holographic imaging. Moreover, no lens is needed in this configuration, because no light beam needs to be deflected, so the holographic imaging performed by the apparatus 1200 is also lens-free. Generally, in-line lens-free holographic imaging captures holograms 13 without using lenses by recording interference patterns directly on the recording medium 1201b, wherein a coherent light source 1201a aligned with the droplet 12 and recording medium 1201b may be used as an example (but not required).
[0110] The processor 1202 of the apparatus 1200 is further able perform the reconstruction of the hologram 13 after or during artifact removal, for example, by reconstruction methods known in the art, to obtain the image 15.
[0111] In sum, the present disclosure provides a method for using in-line lens-free holographic imaging to obtain high-quality images 15 of objects 11 in droplets 12, and to obtain characteristics 25 of the objects 11 based on the images 15.
[0112] In the claims as well as in the description of this disclosure, the word comprising does not exclude other elements or steps and the indefinite article a or an does not exclude a plurality. A single element may fulfill the functions of several entities or items recited in the claims. The mere fact that certain measures are recited in different dependent claims does not indicate that a combination of these measures cannot be used.
[0113] While some embodiments have been illustrated and described in detail in the appended drawings and the foregoing description, such illustration and description are to be considered illustrative and not restrictive. Other variations to the disclosed embodiments can be understood and effected in practicing the claims, from a study of the drawings, the disclosure, and the appended claims. The mere fact that certain measures or features are recited in mutually different dependent claims does not indicate that a combination of these measures or features cannot be used. Any reference signs in the claims should not be construed as limiting the scope.