AUTOMATED SYSTEM FOR ACQUIRING IMAGES OF ONE OR MORE CAPILLARIES IN A CAPILLARY BED

Abstract

An automated system for acquiring images of one or more capillaries in a capillary bed includes a platform for receiving a body portion of a subject, an imaging subsystem having a repositionable field of view and coupled to the platform to acquire images of at least a capillary bed of the body portion and a controller communicably coupled to the imaging subsystem to automatically reposition the field of view of the imaging subsystem to different areas of the capillary bed, and at each field of view within the capillary bed, activate the imaging subsystem to acquire images of one or more capillaries in the capillary bed.

Claims

1. An automated system for acquiring images of one or more capillaries in a capillary bed, the system comprising: a platform for receiving a body portion of a subject; an imaging subsystem having a repositionable field of view and coupled to the platform to acquire images of at least a capillary bed of the body portion; and a controller communicably coupled to the imaging subsystem to: automatically reposition the field of view of the imaging subsystem to different areas of the capillary bed, and at each field of view within the capillary bed, activate the imaging subsystem to acquire images of one or more capillaries in the capillary bed.

2. The system of claim 1 in which the controller is configured to process the images of each said area and assign a location and confidence of one or more high-quality capillaries in each said area.

3. The system of claim 2 in which information from each field of view is stitched together by the controller to produce a distribution of capillary locations, capillary sizes, and confidence levels of capillary existence across a super field of view.

4. The system of claim 3 in which the controller is configured to process said distribution of capillary locations and confidence levels and determine a location and/or size of an optimal field of view for recording a set of images or videos of high-quality capillaries.

5. The system of claim 1 in which the images are stitched together by the controller across a super field of view to produce a single image and the controller detects the capillary locations, sizes and confidence levels in that single image.

6. The system of claim 1 in which the body portion of the subject includes one of: a finger, a toe, a tongue, a lip, a gum, or an earlobe of the subject.

7. The system of claim 6 in which the body portion includes the nailfold of the finger or the toe.

8. The system of claim 4 in which the location of the super field of view capillary distribution plot is acquired from a middle bottom location of the nailfold.

9. The system of claim 1 in which the controller is configured to automatically calibrate the imaging subsystem by setting the imaging subsystem to a predetermined start location.

10. The system of claim 1 in controller is configured to automatically adjust the exposure time of the imaging subsystem.

11. The system of claim 1 in a controller is configured to automatically adjust a gain of the imaging subsystem.

12. The system of claim 1 in controller is configured to automatically adjust the focus of the imaging subsystem.

13. The system of claim 1 in controller is configured to automatically ensure the imaging subsystem is operating at a desired rate.

14. The system of claim 1 in controller is configured to automatically ensure illumination of at least capillary bed is within a predetermined illumination range.

15. The system of claim 10 in controller is configured to automatically ensure the exposure time is within a predetermined exposure time range.

16. The system of claim 10 in controller is configured to automatically ensure the imaging system gain is within a predetermined range.

17. The system of claim 12 in controller is configured to automatically ensure the focusing is within a predetermined focusing range.

18. The system of claim 1 in which the controller is configured to control one or more light sources of the imaging system to emit light at one or more selected wavelengths or wavelength ranges.

19. The system of claim 18 in which the controller is configured to select one or more wavelengths or wavelength ranges based on image quality.

20. The system of claim 1 in which the controller activates or not activates the imaging system based on the quality of the images of the capillary bed.

21. The system of claim 1 in which the controller activates the imaging system to acquire the images including one or more capillaries in the capillary bed for a period of time based on the number of optical adsorption gaps (OAGs) detected in one or more capillaries.

22. The system of claim 1 in which the controller communicably coupled to the imaging subsystem is configured to detect finger movement in the platform by the imaging subsystem.

23. A method for acquiring images of one or more capillaries in a capillary bed, the method comprising: receiving a body portion of a human subject; acquiring images of at least a capillary bed of the body portion with a repositionable field of view; automatically repositioning the field of view to different areas of the capillary bed; at each field of view within the capillary bed, acquiring images of one or more capillaries in the capillary bed.

24. The method of claim 23 further including processing the images of each said area and assigning a location and confidence level of one or more high-quality capillaries in each said area.

25. The method of claim 24 in which information from each field of view is stitched together to produce a distribution of capillary locations, capillary sizes, and confidence levels of capillary existence across a super field of view.

26. The method of claim 25 further including processing said distribution of capillary locations and confidence levels and determining a location and/or size of an optimum field of view for recording a set of images of high-quality capillaries.

27. The method of claim 23 in which images are stitched together by the controller across a super field of view to produce a single image and detecting capillary locations, sizes, and confidence levels in that single image.

28. The method of claim 23 in which the super field of view capillary distribution plot is acquired from a middle bottom location of a nailfold.

29. The method of claim 23 further including automatically calibrating the imaging to a predetermined start location.

30. The method of claim 23 further including automatically adjusting the exposure time.

31. The method of claim 23 further including automatically adjusting imaging system gain.

32. The method of claim 23 further including automatically adjusting the focus.

33. The method of claim 23 further including automatically ensuring the imaging is at a desired rate.

34. The method of claim 23 further including automatically ensuring the illumination of at least the capillary bed is within a predetermined illumination range.

35. The method of claim 23 including automatically ensuring the exposure time is within a predetermined exposure time range.

36. The method of claim 23 including automatically ensuring the imaging system gain is within a predetermined range.

37. The method of claim 23 including automatically ensuring focusing is within a predetermined focusing range then automatically emitting light at one or more selected wavelengths or wavelength ranges.

38. The method of claim 37 in which selecting the one or more wavelengths or wavelength ranges is based on image quality.

39. The method of claim 23 including activating or not activating the imaging based on the quality of images in the capillary bed.

40. The method of claim 23 including acquiring the images of one or more capillaries in the capillary bed for a period of time based on the number of optical absorption gaps detected in the one or more capillaries.

41. The method of claim 23 including detecting movement of a finger during the imaging.

Description

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

[0012] Other objects, features and advantages will occur to those skilled in the art from the following description of a preferred embodiment and the accompanying drawings, in which:

[0013] FIG. 1 is a schematic block diagram showing the primary components of one example of the automated system for acquiring images of one or more capillaries in a capillary bed;

[0014] FIGS. 2A and 2B are schematic views showing in further detail the primary components of the system shown in FIG. 1;

[0015] FIG. 3 shows an example of a capillary bed in a finger;

[0016] FIG. 4 shows one example of the controller shown in one or more of FIGS. 1-2B coupled to an x-axis motor and a y-axis motor coupled to scanning mirrors;

[0017] FIG. 5 shows an example of a repositionable field of view (FOV) repositioned to different areas of a capillary bed;

[0018] FIG. 6 shows an example of information from two FOVs stitched together by controller produce a distribution of capillary locations, capillary sizes, and confidence levels of capillary existence across a super FOV;

[0019] FIG. 7 shows an example of a super FOV, an optimal FOV, distributions of capillary location sizes and confidence levels, and capillaries within the optimal FOV to be analyzed;

[0020] FIG. 8 shows am example of capillaries in multiple FOVs stitched together by the controller across a super FOV;

[0021] FIG. 9 shows an example of a prototype of the system shown in one or more of FIGS. 1-8 with a finger placed in a finger well and images stitched together by the controller across a super FOV to produce a single image to detect the capillary locations, sizes and confidence levels in a single image;

[0022] FIG. 10 is a flow chart showing one example of the method for acquiring images of one or more capillaries in a capillary bed;

[0023] FIG. 11 is a schematic block diagram showing one example an automated wavelength selection for the system and method shown in one or more of FIGS. 1-10; and

[0024] FIG. 12 is a flow chart showing one example of finger movement detection for the system and method shown in one or more of FIGS. 1-11.

DETAILED DESCRIPTION OF THE INVENTION

[0025] Aside from the preferred embodiment or embodiments disclosed below, this invention is capable of other embodiments and of being practiced or being carried out in various ways. Thus, it is to be understood that the invention is not limited in its application to the details of construction and the arrangements of components set forth in the following description or illustrated in the drawings. If only one embodiment is described herein, the claims hereof are not to be limited to that embodiment. Moreover, the claims hereof are not to be read restrictively unless there is clear and convincing evidence manifesting a certain exclusion, restriction, or disclaimer.

[0026] There is shown in FIG. 1, one example of automated system 10 for acquiring images of one or more capillaries in a capillary bed. System 10 includes platform 12 for receiving a body portion of a subject. In one example, platform 12 may be a finger well, e.g., finger well 14, FIGS. 2A and 2B, disclosed in this example in ergonomic hand-holder 16 (discussed in further detail below). The body portion of the subject may include a finger, a toe, a tongue, a lip, a gum, or an earlobe of a human subject. FIG. 1 shows an example of a top view of finger 18 inserted into platform 12. FIG. 3 shows an example of finger 18 with capillary bed 36 to be automatically imaged by system 10.

[0027] System 10, FIGS. 1-2B, also includes imaging subsystem 20 having a repositionable field of view (FOV) with an adjustable focus coupled to platform 12 and controller 22 as shown. Imaging subsystem 20 acquires images of at least a capillary bed of the body portion. Imaging subsystem 20 preferably includes one or more light emitting devices 24, e.g., light emitting diodes (LEDs) or similar type light emitting device, one or more focusing lenses 26, one or more scanning mirrors 28, and optical window 30 as shown.

[0028] System 10 also includes controller 22, communicably coupled to imaging subsystem 20, one or more focusing lenses 26, and one or more scanning mirrors 28. Controller 22 automatically repositions the FOV of imaging subsystem 20 to different areas of the capillary bed. FIG. 4 shows one example of controller 22 coupled to X-axis motor 32 and Y-access motor 34 each preferably coupled to a scanning mirror 28 as shown. In this example, controller 22 automatically repositions repositionable FOV 28 to different areas or positions of a capillary bed, e.g., as shown by FOV.sub.1 and FOV.sub.2. FIG. 5 shows an example of repositionable FOV 28 repositioned to different areas of capillary bed 36, FIG. 3 of finger 18, e.g., FOV.sub.1-28 and FOV.sub.2-28. At each repositionable FOV 28 within capillary bed 18, controller 22, FIGS. 1 and 2A, activates imaging subsystem 20 to acquire images of one or more capillaries in capillary bed 36. In this example, repositionable FOV 28 is preferably repositioned by controller 22 to different areas of capillary bed 36, FIG. 3, of finger 18. In other examples, repositionable FOV 28 may be repositioned by controller 22 to different areas of capillary bed of a toe, a tongue, a lip, a gum, or an earlobe of a human subject.

[0029] Controller 22 may be a processor, one or more processors, an application-specific integrated circuit (ASIC), firmware, hardware, and/or software (including firmware, resident software, micro-code, and the like) or a combination of both hardware and software. Controller 22 preferably includes one or more programs stored in a memory which are configured to be executed by the one or more processors. Computer program code for the programs for carrying out the instructions or operation of controller 22 may be written in any combination of one or more programming languages, including but not limited to an object-oriented programming language, e.g., C++, Smalltalk, Java, and the like, or conventional procedural programming languages, such as the “C’ programming language, Assembly language or similar programming languages.

[0030] Preferably, controller 22, FIGS. 1, 2A and 4, processes the images in each area of the capillary bed and assigns a location and confidence level to indicate the presence of one or more high-quality capillaries in each area. As disclosed herein, one or more high-quality capillaries preferably meets the requirements of one or more desired features, e.g., a high-quality capillary contains one or more optical absorption gaps (OAGs), are substantially parallel to the imaging plane, have a similar width to a white blood cell, have a sufficient length to allow for the passage of OAGs, have a high-quality contrast, are in focus, are well illuminated, and blood flow is preferably not restricted. Thus, high-quality capillaries are defined herein are preferably amiable for analysis to produce an accurate blood measurement including a white blood cell count, and/or neutropenia classification, and/or the subtype of any white blood cells and/or other blood parameters.

[0031] In one design, information from each FOV discussed above is preferably stitched together by controller 22 to produce a distribution of capillary locations, capillary sizes, and confidence levels of capillary existence across a super FOV. As defined herein, a super FOV is a geometric area across which the imaging FOV is repositioned.

[0032] FIG. 6 show an example of FOV.sub.1-28 to FOV.sub.N-28 where information from each FOV.sub.1-28 and FOV.sub.N-28 is stitched together by controller 22 to produce a distribution of capillary locations, capillary sizes, and confidence levels of capillary existence across super FOV-40. In this example, the distribution of capillary locations, capillary sizes, and confidence levels of capillary existence for FOV.sub.1-28 is shown in bounding boxes 42, 44, and 46, where x indicates the X-coordinate, y indicates the Y-coordinate, S indicates the capillary size (width and height), and C indicates the confidence level of the existence of a high-quality capillary. Similarly, for FOV.sub.N-28, the distribution of capillary locations, capillary sizes, and confidence levels of capillary existence for FOV.sub.N-28 is shown in bounding boxes 48, 50, 52, and 54, where x indicates the X-coordinate, y indicates the Y-coordinate, S indicates the capillary size (width and height), and C indicates the confidence level of the existence of a high-quality capillary.

[0033] As shown in example in FIGS. 4 and 6, the information from FOV.sub.1-28 and FOV.sub.N-28 is preferably stitched together by controller 22 to produce a distribution of capillary locations, capillary sizes, and confidence levels of capillary existence across super FOV 40.

[0034] Controller 22, FIGS. 1, 2A and 4, preferably processes the distribution of capillary locations, sizes, and confidence levels discussed above and determines a location and/or size of an optimal field of view for recording a set of images or videos of high-quality capillaries. FIG. 7 shows an example of super FOV 40 discussed above with reference to FIGS. 4 and 6 and examples of optimal field of view 60 for recording a set of images or videos of high-quality capillaries 62 to be analyzed to determine WBC count and/or neutropenia and/or white blood cell type and/or other blood parameters. The distribution of capillary locations, capillary sizes, and confidence levels of capillary existence across super FOV 40 in the bounding boxes discussed above with reference to FIG. 6 are also indicated generally at 64, FIG. 7.

[0035] In one example, optimal FOV 60, FIG. 7, for recording a set of images or videos of high-quality capillaries 62 may be determined by calculating the x-y coordinates of optimal FOV 60 and the width and height of optimal FOV 60 (x_opt, y_opt, w_opt, h_opt). To determine x_opt and y_opt, x_target and y_target are determined, where x_target is the weighted mean of x coordinates of detected capillaries weighted by their confidence levels and y_target is the weighted 95% percentile of the y coordinates of detected capillaries, where the weights are the confidences of each corresponding capillary. Then, x_opt equals x_target and y_opt equals y_target+0.25*h_FOV. In this example, w_opt equals w_FOV and h_opt equals h_fov, where w_FOV and h_FOV and the width and height of the fixed field of view of the imaging system.

[0036] In one design, images may be stitched together by controller 22, FIGS. 1, 2A and 4, across a super FOV to produce a single image and controller 22 preferably detects the capillary locations, sizes, and confidence levels in the single image.

[0037] FIG. 8 shows one example of a depiction of images of one or more capillaries 70 for FOV.sub.1-28 and a depiction of images of one or more capillaries 72 for FOV.sub.N-28 which are stitched together by controller 22 across super FOV-80 as shown. Controller 22 preferably detects capillary locations, sizes, and confidence levels in the single image. The optimal FOV is determined as discussed above with reference to one or more FIGS. 1-7.

[0038] FIG. 9 shows an example of a prototype of system 10 where a user has placed a finger in finger well 14 of platform 12 and images are stitched together by controller 22, FIGS. 1, 2A and 4, across a super FOV 80 to produce a single image 82 where controller 22 preferably detects the capillary locations, sizes, and confidence levels in the single image. An example of the optimal FOV is indicated at 84.

[0039] In one example, the super FOV of the capillary distribution plot discussed above with reference to one or more of FIGS. 1-9 may be acquired from the middle location of the nailfold, e.g., indicated generally at 100, FIG. 5.

[0040] FIGS. 2A and 2B show examples of a prototype of system 10 configured as ergonomic hand-holder 16. System 10 preferably includes base section 110 including convex hand support 112 with centrally located downwardly oriented removable finger well or tube 14 for receiving a finger of a human subject as shown or other body portion of the subject including one of: a finger, a toe, a tongue, a lip, a gum, or an earlobe of the subject. System 10 also preferably includes angled section 118 extending upwardly over disposable finger well or tube insert 14 terminating in display screen 114, FIG. 2B. Angled section 118 also preferably includes imaging subsystem 20, shown in on or more of FIGS. 1-2B. In this example, base section 110, also preferably includes controller 22, power board 130, FIG. 2B and input/output board 132.

[0041] In one example, convex hand support 112 slopes upward from base section 110 and extends downwardly to intersection 120 between base section 110 and angled section 118. In one example, the intersection is radiused as shown. In one example, convex hand support 112 may include replaceable pads to accommodate various hand sizes. Additional details of finger well 14 shown in at least FIGS. 2A and 2B are disclosed in U.S. Pat. No. 11,160,492, by one of the inventors hereof, incorporated by reference herein.

[0042] Imaging subsystem 20 preferably acquires and processes images and/or videos of one or more high quality capillaries of a capillary bed of a finger in removal finger well or tube insert 14 to determine a blood measurement including a white blood cell (WBC) count and/or neutropenia classification and/or WBC type and/or other blood parameters.

[0043] One example of the method for acquiring images of one or more capillaries in a capillary bed includes receiving a body portion of a human subject, step 90, FIG. 10. The method also includes acquiring images of at least a capillary bed of the body portion with a repositionable field of view FOV, step 92. The method also includes automatically repositioning the FOV to different areas of the capillary bed, step 94. The method also includes at each FOV within the capillary bed, acquiring images of one or more capillaries in the capillary bed.

[0044] In one design, controller 22, FIGS. 1, 2A and 4, preferably automatically calibrates imaging subsystem 20 by setting imaging subsystem 20 to a predetermined start location. Controller 22 may also automatically adjust the exposure time of the imaging subsystem 20, may automatically adjust the focus of the imaging subsystem 20, and/or may ensure imaging subsystem 20 is operating at a desired rate, e.g., frames per second. Controller 22 may also automatically ensure the illumination of the one or more capillary bed is within a predetermined illumination range, automatically ensure the exposure time is within a predetermined exposure time, and/or automatically ensure the focusing is within a predetermined focusing range. Controller 22 also preferably controls the one or more light sources 24, FIG. 1, of imaging subsystem 20 to emit light at one or more selected wavelengths or wavelength ranges. Preferably, controller 22 selects one or more wavelengths or wavelength ranges based on image quality.

[0045] System 10 and the method thereof discussed above with reference to one or more of FIGS. 1-10 preferably includes software or computer code that provides for wavelength selection to select the optimal wavelength for imaging quality preferably on a per-measurement basis. Wavelength selection is preferably determined during the setup phase before each measurement. Wavelength selection preferably determines the optimal wavelength by evaluating an imaging quality metric using each available wavelength. The wavelength with the greater metric value is then selected.

[0046] Wavelength selection preferably maximizes acquisition success over a wide range of skin characteristics, including, inter alia, pigmentation levels across the Fitzpatrick skin phototype scale, and local pigmentations such as freckles. Without an automatic wavelength selection procedure by system 10, wavelength selection would need to be performed manually by a trained technician. Minimizing cost and maximizing ease of use of system 10 for both the patient and the care team are important objectives.

[0047] In one example, controller 22 is preferably configured to automatically selects the appropriate wavelength. Such an automated procedure additionally ensures that the correct methodology is preferably followed. In one example, the wavelength selection performed by controller 22 preferably includes running a setup procedure using illumination with spectral content in the approximate range, e.g., about 540-580 nm illumination (wavelength range 1). At the final target FOV and after auto-exposure and auto-gain have settled, one frame is captured using wavelength range 1, step 100, FIG. 11. Illumination is preferably switched to in the approximate range, e.g., about 405-430 nm illumination (wavelength range 2). After auto-exposure and auto-gain have settled, one frame is captured using wavelength range 2. In each frame, capillaries are detected using an object detection routine to identify capillaries, create bounding boxes around the capillaries, and attribute a capillary confidence value, step 102. Capillaries above a given classification confidence threshold are then evaluated for imaging contrast by running a contrast algorithm on the image regions identified by the capillary bounding boxes. A single metric for each capillary is made by taking the product of the capillary confidence and the capillary contrast. A single score for each wavelength's FOV is preferably then taken by taking the sum of the capillary scores, step 104. The range that has the highest metric value is identified, and switched to the associated illumination regime, step 106. After auto-exposure and auto-gain have settled, finish the setup procedure. See FIG. 11.

[0048] Controller 22 may activate or not activate imaging subsystem 20 based on the quality of images in the capillary bed. Controller 22 may activate imaging subsystem 20 to acquire images of one or more capillaries in the capillary bed for a period of time based on the number of OAGs detected in the one or more capillaries.

[0049] Users of system 10 need to keep their finger still in finger well 14, FIGS. 2A-2B, while a measurement is taken. Some users may experience difficulty following this instruction and may either remove their finger prematurely or move their finger in finger well 14 during the measurement. If the finger movement is excessive, the analysis performed by system 10 may be suboptimal so it is recommended that the user repeat the measurement without any movement.

[0050] To address this problem, controller 22 communicably coupled to imaging system 20 preferably detects movement of a finger within finger well 14 using imaging subsystem 20. Controller 22 preferably includes software or computer code configured as finger movement detector module 150, FIG. 1, capable of detecting problematic finger movement. Finger movement detection is preferably used at the end of every measurement to determine if the pre measurement needs to be repeated due to excessive finger movement. Finger movement detector module 150 preferably operates by assessing the movement between pairs of video frames from the measurement. An error is raised if the measured movement exceeds the allowable threshold.

[0051] One example finger movement detector module 150 includes the steps of: 1) estimating high-frequency movement between all contiguous video frames in the measurement video, 2) estimating low-frequency movement between video frames spaced one second apart in the measurement video, 3) raising an error if any movement estimates exceed a threshold, and 4) estimating movement using Robinson's motion estimation algorithm. See e.g., Robinson, D., Milanfar, P., Fast Local and Global Projection-Based Methods for Affine Motion Estimation, Journal of Mathematical Imaging and Vision 18, 35-54 (2003), https://doi.org/10.1023/A:1021841127282, incorporated by reference herein. See FIG. 12.

[0052] Although specific features of the invention are shown in some drawings and not in others, this is for convenience only as each feature may be combined with any or all of the other features in accordance with the invention. The words “including”, “comprising”, “having”, and “with” as used herein are to be interpreted broadly and comprehensively and are not limited to any physical interconnection. Moreover, any embodiments disclosed in the subject application are not to be taken as the only possible embodiments. Other embodiments will occur to those skilled in the art and are within the following claims.

[0053] In addition, any amendment presented during the prosecution of the patent application for this patent is not a disclaimer of any claim element presented in the application as filed: those skilled in the art cannot reasonably be expected to draft a claim that would literally encompass all possible equivalents, many equivalents will be unforeseeable at the time of the amendment and are beyond a fair interpretation of what is to be surrendered (if anything), the rationale underlying the amendment may bear no more than a tangential relation to many equivalents, and/or there are many other reasons the applicant cannot be expected to describe certain insubstantial substitutes for any claim element amended.