AUTOMATED HARDWARE AND SOFTWARE FOR MOBILE MICROSCOPY
20170285320 · 2017-10-05
Assignee
Inventors
- Daniel A. Fletcher (Berkeley, CA, US)
- Michael V. D'Ambrosio (Berkeley, CA, US)
- Arunan Skandarajah (Berkeley, CA, US)
- Frank B. Myers, III (Oakland, CA, US)
- Clay D. Reber (Berkeley, CA, US)
Cpc classification
G02B21/365
PHYSICS
G02B21/0008
PHYSICS
International classification
G02B21/36
PHYSICS
Abstract
An automated slide scanning system, comprising one or more optical elements, including an objective, having an optical path configured to be disposed within view of a camera of a portable device. An automated stage disposed within the optical path, the automated stage comprising a platform configured for receiving a slide containing a biological sample and having a drive mechanism for translating the stage in at least one direction with respect to said objective. A communications interface coupled to the automated stage, and is configured for receiving a command from the portable device to control operation of the mechanical stage.
Claims
1. An automated slide scanning system, comprising; one or more optical elements having an optical path configured to be disposed within view of a camera of a portable device; the one or more optical elements comprising an objective; an automated stage disposed within the optical path; the automated stage comprising a platform configured for receiving a slide containing a biological sample; the automated stage comprising a drive mechanism for translating the stage in at least one direction with respect to said objective; and a communications interface coupled to the automated stage; wherein the communication interface is configured for receiving a command from the portable device to control operation of the mechanical stage.
2. A system as recited in claim 1, the portable device comprising a cellular device having a processor and memory, the system further comprising: application programming configured to be stored as instructions in non-transitory computer-readable memory on the portable device; and wherein the application programming is executable on the processor for executing the command.
3. A system as recited in claim 2: wherein the automated stage comprises an x-y stage comprising independent drive mechanisms for x-axis translation and y-axis translation of the slide with respect to the objective; and wherein the application programming is configured for controlling motion of the x-y stage via one or more commands initiated from the portable device.
4. A system as recited in claim 3: wherein the automated stage comprises a z-axis stage comprising an independent drive mechanism for z-axis translation of the slide with respect to the objective; and wherein the application programming is configured for controlling motion of the stage via one or more commands initiated from the portable device to focus imaging of the sample on the slide.
5. A system as recited in claim 2, further comprising: an illumination source coupled to the automated stage, the illumination source configured to direct light at the slide; and wherein the programming is further configured to control operation of the illumination source via one or more commands initiated at the portable device.
6. A system as recited in claim 1, wherein the objective comprises an infinity objective, and the optical elements further comprise a plurality of mirrors folding the light path into a compact light path in line with an achromatic tube lens and the infinity objective.
7. A system as recited in claim 2: wherein the portable device comprises a touch screen; and wherein the application programmming comprises a gesture recognition module configured to control motion of the automated stage in response to one or more hand gestures applied to the touch screen.
8. A system as recited in claim 7: wherein the communications interface comprises a wireless communication interface, and wherein motion of the automatic stage is controlled wirelessly from one or more hand gestures initiated at the touch screen.
9. A system as recited in claim 7, wherein translation of the automatic stage is in proportion to the one or more gestures.
10. A system as recited in claim 7, wherein translation of the automatic stage is affected as an automated sequence of motions in response to the one or more gestures.
11. A system as recited in claim 5, wherein the programming is further configured to control one or more of bright field, dark field, phase, or intensity of the light directed by the illumination source.
12. A system as recited in claim 5: wherein the illumination source comprises an LED array; and wherein the application programming is further configured to vary the orientation and numerical aperture of the illumination light directed by the illumination source or initiate the acquisition of a series of illumination states with respect to the light directed by the illumination source.
13. A system as recited in claim 2, wherein the application programming is configured to vary one or more of the numerical aperture, resolution, and field of view of the objective.
14. A system as recited in claim 2, wherein the application programming is configured to vary a digital magnification of a region of interest of the sample and display a corresponding size calibration corresponding to the magnification.
15. A system as recited in claim 5, wherein where illumination source and application programming are configured for fluorescence imaging of the sample.
16. A system as recited in claim 2, further comprising: a calibration sample positioned within the optical path; wherein the application programming is configured to perform a calibration routine to provide calibration images of the sample of one or more of: focus, field flatness, illumination intensity, illumination uniformity, exposure, contrast, optical resolution, or stage translation/backlash.
17. A system as recited in claim 2, wherein the application programming is configured to apply one or more shape metrics to classify features within an image of the sample.
18. A system as recited in claim 2, wherein the application programming is configured to process an image of the sample during motion of the automatic stage to determine a distance traveled and providing feedback for pixel-scale positioning of the automatic stage.
19. A system as recited in claim 2, further comprising: one or more proximity sensors coupled to the automatic stage; and wherein the one or more proximity sensors provide position data of the automatic stage for establishing one or more of an origin point or limit for translation of the automatic stage.
20. A system as recited in claim 1, wherein the drive mechanism comprises a stepper motor coupled to a rack and pinion.
21. A system for automatically scanning and imaging a slide containing a biological sample, comprising: (a) one or more optical elements having an optical path configured to be disposed within view of a camera of a portable device; (b) the one or more optical elements comprising an objective; (c) an automated stage disposed within the optical path, the automated stage comprising a platform configured for receiving a slide containing a biological sample and a drive mechanism for translating the stage in at least one direction with respect to said objective; (d) a communications interface coupled to the automated stage; (e) a computer processor; and (f) a non-transitory computer-readable memory storing instructions executable by the computer processor; (g) wherein said instructions, when executed by the computer processor, perform steps comprising: for receiving a command from the portable device; (ii) sending the command via the communication interface to the automatic stage; and (iii) translating the mechanical stage in response to said command.
22. A system as recited in claim 21: wherein the automated stage comprises an x-y stage comprising independent drive mechanisms for x-axis translation and y-axis translation of the slide with respect to the objective; and wherein said instructions when executed by the computer processor are further configured to control motion of the x-y stage via one or more commands initiated from the portable device.
23. A system as recited in claim 22: wherein the automated stage comprises a z-axis stage comprising an independent drive mechanism for z-axis translation of the slide with respect to the objective; and wherein said instructions when executed by the computer processor are further configured to control motion of the stage via one or more commands initiated from the portable device to focus imaging of the sample on the slide.
24. A system as recited in claim 21, further comprising: an illumination source coupled to the automated stage, the illumination source configured to direct light at the slide; and wherein said instructions when executed by the computer processor are further configured to control operation of the illumination source via one or more commands initiated at the portable device.
25. A system as recited in claim 21, wherein the objective comprises an infinity objective, and the optical elements further comprise a plurality of mirrors folding the light path into a compact light path in line with an achromatic tube lens and the infinity objective.
26. A system as recited in claim 21: wherein the portable device comprises a touch screen; and wherein said instructions when executed by the computer processor are further configured to control motion of the automated stage in response to one or more hand gestures applied to the touch screen.
27. A system as recited in claim 26: wherein the communications interface comprises a wireless communication interface; and wherein motion of the automatic stage is controlled wirelessly from one or more hand gestures initiated at the touch screen.
28. A system as recited in claim 26, wherein translation of the automatic stage is in proportion to the one or more gestures.
29. A system as recited in claim 26, wherein translation of the automatic stage is affected as an automated sequence of motions in response to the one or more gestures.
30. A system as recited in claim 24, wherein said instructions when executed by the computer processor are further configured to control one or more of bright field, dark field, phase, or intensity of the light directed by the illumination source.
31. A system as recited in claim 24: wherein the illumination source comprises an LED array; and wherein said instructions when executed by the computer processor are further configured to vary the orientation and numerical aperture of the illumination light directed by the illumination source or initiate the acquisition of a series of illumination states with respect to the light directed by the illumination source.
32. A system as recited in claim 21, wherein said instructions when executed by the computer processor are further configured to vary one or more of the numerical aperture, resolution, and field of view of the objective.
33. A system as recited in claim 21, wherein said instructions when executed by the computer processor are further configured to vary a digital magnification of a region of interest of the sample and display a corresponding size calibration corresponding to the magnification.
34. A system as recited in claim 24, wherein where illumination source is configured for fluorescence imaging of the sample.
35. A system as recited in claim 21, further comprising: a calibration sample positioned within the optical path; wherein said instructions when executed by the computer processor are further configured to calibrate images of the sample of one or more of: focus, field flatness, illumination intensity, illumination uniformity, exposure, contrast, optical resolution, or stage translation/backlash.
36. A system as recited in claim 21, wherein said instructions when executed by the computer processor are further configured to apply one or more shape metrics to classify features within an image of the sample.
37. A system as recited in claim 21, wherein said instructions when executed by the computer processor are further configured to process an image of the sample during motion of the automatic stage to determine a distance traveled and providing feedback for pixel-scale positioning of the automatic stage.
38. A system as recited in claim 21, further comprising: one or more proximity sensors coupled to the automatic stage; and wherein the one or more proximity sensors provide position data of the automatic stage for establishing one or more of an origin point or limit for translation of the automatic stage.
39. A method for automatically scanning and imaging a slide containing a biological sample, comprising: providing an automated stage having an optical path and an objective disposed within the optical path, the automated stage comprising a platform configured for receiving a slide containing a biological sample and a drive mechanism for translating the stage in at least one direction with respect to said objective; positioning a camera of a portable device in view of the optical path; sending a command from the portable device to the automatic stage; translating the mechanical stage in response to said command; and imaging the biological sample with the camera.
40. A method as recited in claim 39, wherein the portable device comprises a touch screen, the method further comprising: controlling motion of the automated stage in response to one or more hand gestures applied to the touch screen.
41. A method as recited in claim 39, the method further comprising: controlling motion of the stage via one or more commands initiated from the portable device to focus imaging of the sample on the slide.
Description
BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING(S)
[0020] The technology described herein will be more fully understood by reference to the following drawings which are for illustrative purposes only:
[0021]
[0022]
[0023]
[0024]
[0025]
[0026]
[0027]
[0028]
[0029]
[0030]
DETAILED DESCRIPTION
[0031]
[0032]
[0033]
[0034]
[0035] System 10 may also include vibration-isolation feet attached to the device housing or stages 50, 70 of the device.
[0036] While not required, stepper motors are preferably used to provide a precise, inexpensive, simple automated translation mechanism. Stepper motors allow for acceptable accuracy and precision in open-loop operation, and high torque, low speed movement without gearing. In general, the smallest stepper motors that are able to provide an adequate amount of torque at reasonable power consumption are preferred (smaller stepper motors are often notably less efficient than larger ones).
[0037] In one embodiment, system 10 measures 155 mm×225 mm×135 mm. In order to achieve such a small form factor, several design decisions were made to minimize size. First, the light path 30 is folded by four mirrors 24, allowing a 200 mm optical path length (the minimum required to achieve acceptable magnification at the imaging sensor, e.g. of an iPad, or similar tablet, or portable cellular device) to be fit into the system 10 with the sample 40 centered. In order to allow for a compact focusing mechanism that involves only moving the objective 28, an infinity system with an achromatic lens 26 is used, with an eyepiece 22 to couple optics to the portable device camera 14. Second, tall components (the objective 28 and the stepper motors 52, 62 and 74) share vertical space to minimize the device's height. Third, elements of the focusing mechanism 70 are in parallel vertical space to the objective 28, occupying otherwise empty space instead of contributing to the device's height.
[0038]
[0039] If the step requires an image processing step (like calculating the focus metric via an autofocus module 154), the user interface 110 hands this task to the image processor module 104.
[0040] In order to enable the system to intelligently scan a slide, a number of software control techniques may be implemented in application software 100. One obstacle to scanning a large area is maintaining focus. To overcome this obstacle, an auto focusing system may be used based around the standard deviation of the laplacian of the image (e.g. focus metric). Numerous other implementations of the focus metric can be implemented. By moving the z-axis a set amount (via stage 70), and recalculating and comparing the focus metric, proper focus can be converged upon.
[0041] In a preferred embodiment, autofocus module 154 sweeps the objective through 5 positions and calculates the focus at each. If a satisfactory peak is not found, the sweep is repeated with a larger distance between each position. In order to limit the possibility of focus being incorrectly moved at position on the slide 40 where there is little or sample present, the autofocus module 154 calculates a content score, which is simply the mean of the image, and will only attempt to focus if it is above a user-set threshold. Note that this safe-guard generally works when the system is already near focus, such as if it is in the middle of scanning a large area where if found focus at a previous position.
[0042] The autofocus module 154 may also be configured to analyze the relationship of the focus metric scores to determine how likely it is that focus is achieved, and optionally increases the search area. As the content score can be calculated rapidly, it also enables a mode of acquisition where the device rapidly scans an area at low resolution, only slowing to produce full-resolution images where there is sufficient content. Alternate implementations of focusing can include capturing focus information from both bright field and fluorescence channels and using some combination as a focus metric.
[0043] In order to make the system 10 user intuitive, a gesture recognition—based control module 146 may be implemented for control of the stages 50, 70. The gesture recognition module 146 is configured to move the stages 50, 70 corresponding to dragging or swiping gestures on the screen of the portable device (e.g. iPad). The speed and positional movement (in pixels) of these gestures may be used to calculate the rate and distance that the stage 50 or 70 will move. The rate and distance is then relayed to the stage via wireless communication (e.g. Bluetooth LE). Gesture controls may also be used to allow zooming and panning, both on live and previously taken images, display the magnified images, as well as provide an interface for control data collection, analysis, and transmission. Gesture recognition—based control module 146 may be configured to operate with hand gestures either in contact with the screen of device 12 or detected by a non-contact method (e.g. front-facing camera) to control microscope, stage, and data functions. Gesture recognition—based control module 146 may also control stage movement through a wired or wireless (e.g. Bluetooth) connection to the motors that control sample movement and focus through specific sequences (e.g. double taps, long taps, etc).
[0044] Application software 100 may also be configured to enhance the accuracy of the stages 50, 70. When scanning large objects of which only a piece fits in one field of view, it is desirable to reconstruct these pieces to be able to interact with the entire object. With micron-scale resolution, as is preferred for tuberculosis diagnosis, this would require sub-micron scale movement of the sample 40 if done purely mechanically. Instead, an image registration routine 152 may be included within processing module 104 and or hardware control module 108 that is configured for stitching together images taken with some positional ambiguity based on the content of the image. Image registration module 152 may also be used to provide closed-loop control to the z-stage motor 74 and x-y stage motors 52 and 62 in near real-time. By tracking the motion of features on the screen, it can be determined how far the sample 40 has actually translated with sub-micron accuracy. Furthermore, when attempting to move the sample 40 back to a previous location, image registration module 152 can be used to reposition the stage with enhanced accuracy if sufficient optical content is present.
[0045] Image processing module 104 may also include algorithms 150 for the automated detection of features within an image, such as bacteria, that are relevant for diagnosis. These features can be presented to a trained user for evaluation or can be automatically evaluated using a computational diagnosis algorithm.
[0046] It is appreciated that software 100 may be configured for implementing a number of various automated algorithms for calibration, analysis, and diagnosis. By using a known reference sample (not shown) embedded within the instrument, automated self-testing and calibration can be performed on a regular basis.
[0047] For example, a calibration sample may be automatically positioned under the collection optics during a “self test” routine in order to provide calibration images for focus, field flatness, illumination intensity, illumination uniformity, exposure, contrast, optical resolution, or translation/backlash measurements, among other features.
[0048] Furthermore, the results from the self test routine may be compared with thresholds set for performance along dimensions of calibration images for focus, field flatness, illumination intensity, illumination uniformity, exposure, contrast, monitoring resolution, signal-to-noise ratio, and stage translation/backlash to alert the user for maintenance and automatically order replacement parts to be delivered to the location of the system 10.
[0049] An embodiment of software 100 for device control is provided in Appendix A.
[0050]
[0051] In one embodiment, one or more wireless communication devices 126 may be coupled to the system hardware (e.g. x-y stage 50, z-stage 70, LED 44) for transmission of commands or data between the portable device 12, individual components, or other external device. In one embodiment, communication device 126 may comprise a Bluetooth LE board, which communicates via SPI to an Arduino, and instructs downstream electronics driving the motors (e.g. a stepper motor controller featuring an Allegra A4983 or an on-board servo control hardware (both not shown)). Bluetooth LE is an ideal protocol for battery-powered applications such as system 10 where high communication bandwidth is not required.
[0052] As seen in
[0053] It is appreciated that the components shown in system 10 of
[0054] For example,
[0055] A particular implementation of this first embodiment is for imaging of stained cells from brush biopsies of suspected oral cancer lesions. Early diagnosis of oral cancer dramatically improves the outcome of clinical interventions. In parts of the world where oral cancer is most prevalent, there is often also insufficient infrastructure to detect patients at early stages. Patients are unable to reach centralized facilities to reach qualified staff and the necessary instrumentation because of cultural and economic barriers. There are local providers of healthcare, but they are not equipped to make a determination and ineffective at encouraging patients to pursue further care. A distributed system for microscopic analysis, if paired with appropriate sample collection and preparation, could enable broader screening of at-risk members of the population at this local level. This first embodiment preferably includes transmission and remote analysis of images collected from the system 10, although diagnosis or risk-stratification may also be implemented on system 10 through on-board approaches similar to those described in the other embodiments described below.
[0056] A second embodiment includes the features of the second embodiment, and alternatively incorporates an image processing module 104 that includes modules for fluorescence imaging 156, resolution calibration, and on-board image analysis 158 for the detection and quantification of particles in the image.
[0057] One implementation of this second embodiment includes quantification and/or detection of Mycobacterium tuberculosis bacilli in human sputum smears (sample 40) collected from patients in various healthcare settings. In this second configuration, the smears are stained with Auramine-O, a fluorescent dye commonly used for TB diagnosis. The smear slide 40 is then inserted into the system 10, which automatically focuses and images (via application software 100) several hundred fields across the smear. A image analysis module 158 may include a support vector machine (SVM) algorithm that has been trained to detect the rod-shaped bacilli, analyze each image, and then return a quantitative estimate of the number of bacilli present in the sample. This modified system enables the automated diagnosis of tuberculosis from a symptomatic patient in under an hour. The second configuration is preferably portable, battery powered, and requires minimal user training. As such, it is ideally positioned to address the estimated three million TB cases that go undiagnosed each year due to inadequate diagnostic resources in remote and low income settings.
[0058] In a third embodiment, the system alternatively uses a one-dimensional scanning system (e.g. stage 50 of
[0059] In this third configuration, application software 100 may contain programming or a user interface 110 that gives simple instructions to the user on loading the device, and automatically acquires 5-second videos of 5 fields of view. When the acquisition is complete, the application software 100 runs an image-processing algorithm within module 104 on the videos to quantify Loa mf/ml. To start a test, the operator simply loads blood from a finger prick into a capillary pre-fitted into a sample holder 40, inserts the holder into the system, and presses a single button on the portable device (e.g. iPhone or the like).
[0060] A fourth embodiment of system 10 alternatively combines the compact inverted lens imaging system of the third embodiment with three-axis motion control. This fourth embodiment is distinct from the first embodiment in its use of simplified optics, and a simple, less accurate drive mechanism that is nonetheless sufficient for imaging at the lower magnification of the simplified optics. One implementation of this device is for analysis of blood smears to quantify malaria load.
[0061] The fourth embodiment preferably runs an application 100 that allows gesture recognition 146 based slide control with optical feedback, and can use multiple autofocus algorithms 154. This four the embodiment is preferably configured to exchange images and data with a remote server via wireless communication module 126 (see
[0062] In the simplified third and fourth configurations above, where less precision is required due to lower magnification and resolution, hobby servo motors with built-in feedback are preferably used in place of stepper motors. Servo motors provide higher max torque at lower power, but suffer from torque limitations when approaching the set point, leading to less reproducibility and inability to move small amounts. In order to enable gross reproducibility when scanning multiple slides, one absolute position is encoded using a pair of hall-effect sensors 130 on the system housing (one for x and one for y) which correspondingly include a pair of magnets on the x-y stage. While other sensor types may be employed, contactless hall-effect sensors were selected because of their long lifetimes and high precision.
[0063] With respect to compatibility of system 10 with differing portable devices 12, and particularly mobile devices such as iPhones or iPads, a significant advantage of using an iPad or iPhone in the above implementation is connectivity. Because these devices 12 are capable of connecting to the internet and accessing remote servers, images and data can be sent to and from the system 10. This allows for remote diagnosis of images, remote quality control, offloading complex computational tasks from the device, and pushing of software and algorithmic updates to the system 10. In addition, the GPS sensor in the iPad or iPhone 12 can be used to geotag diagnoses, enabling disease monitoring and epidemiological studies.
[0064] Finally, thanks to the rich, intuitive user interface of the iPad or iPhone 12, training materials may be directly embedded into the application software 100. For example, language-specific videos can be used to demonstrate sample collection, sample preparation, slide loading, and diagnosis interpretation, further simplifying user training.
[0065] When implemented with system 10 of the present description, these devices also provide advantageous optics/imaging qualities.
[0066] All of the above embodiments are preferably complemented by an intuitive user interface 110 for gesture-based control 146 and on-board video tutorials for instruction. The system may also operate in an automatic mode that includes slide loading, auto focusing, and taking pictures over a large sample area, often from a slide.
[0067] In accordance with the above embodiments, application software 100 may be configured to include gesture control to move one or more stages in proportion to the gesture and/or initiate an automated sequence of motions., control imaging features, including focus, exposure, color balance, contrast, etc., control sample illumination mode (bright field, dark field, phase, etc) or intensity, control an arrayed LED illumination source to vary the orientation and numerical aperture of the illumination or initiate the acquisition of a series of illumination states, vary the digital magnification of the region of interest and display a corresponding size calibration (the dynamic scale bar), switch the light collecting optic (i.e. the objective 28) to vary the numerical aperture, resolution, and field of view, etc.
[0068] Furthermore, application software 100 may be configured to include modules for analyzing the image captured and additionally process the image during stage motion to determine the distance traveled (e.g. by autocorrelation), providing feedback on motor control for pixel-scale positioning accuracy. The application software 100 may be configured to discern the amount of relevant material in the field of view and automatically capture fields above a user-defined threshold while passing over the remainder, and additionally process the image to determine a reference point based on feature recognition of a reference point in the sample.
[0069] Application software 100 may also incorporate electronic medical record keeping, and GPS geotagging of acquired images, where geotagging is used for disease epidemiology, outbreak monitoring, or patient follow-up planning. Functionality for SMS text-message or voice calls may be implemented, which relay information about the data collected by the device to stakeholders (users, supervisors, doctors, patients, etc.). The application software 100 may be configured to transmit images, videos, analysis results, device usage data, and medical record data to a cloud server for analysis, storage, or dissemination, as well as analyze collected images or videos and provide an automated clinical diagnosis.
[0070] Further aspects of application software 100 include functionality for detection of bacteria in stained specimens, provide sub-images of interest to the user for evaluation, include a support-vector machine algorithm which uses shape metrics to classify features within an image, automatically align and stitch neighboring fields, creating a larger image mosaic, acquire a mosaic of the slide area at a lower capture resolution and present it to the user for selection of the region of interest to be captured at a higher resolution, and detect a calibration target and automatically adjusts camera, stage, or analysis settings depending on the features of said target.
[0071] As a result of this novel combination of hardware and software features, the system 10 represents a platform for microscopic data acquisition in low-resource or non-traditional locations where infrastructure and user skill may be insufficient for existing technologies.
[0072] Embodiments of the present technology may be described herein with reference to methods and systems according to embodiments of the technology, and/or procedures, algorithms, steps, operations, formulae, or other computational depictions, including flowchart illustrations thereof, which may also be implemented as computer program products. In this regard, each block or step of a flowchart, and combinations of blocks (and/or steps) in a flowchart, as well as any procedure, algorithm, step, operation, formula, or computational depiction can be implemented by various means, such as hardware, firmware, and/or software including one or more computer program instructions embodied in computer-readable program code. As will be appreciated, any such computer program instructions may be executed by one or more computer processors, including without limitation a general purpose computer or special purpose computer, or other programmable processing apparatus to produce a machine, such that the computer program instructions which execute on the computer processor(s) or other programmable processing apparatus create means for implementing the function(s) specified.
[0073] Accordingly, blocks of the flowcharts, and procedures, algorithms, steps, operations, formulae, or computational depictions described herein support combinations of means for performing the specified function(s), combinations of steps for performing the specified function(s), and computer program instructions, such as embodied in computer-readable program code logic means, for performing the specified function(s). It will also be understood that each block of the flowchart illustrations, as well as any procedures, algorithms, steps, operations, formulae, or computational depictions and combinations thereof described herein, can be implemented by special purpose hardware-based computer systems which perform the specified function(s) or step(s), or combinations of special purpose hardware and computer-readable program code.
[0074] Furthermore, these computer program instructions, such as embodied in computer-readable program code, may also be stored in one or more computer-readable memory or memory devices that can direct a computer processor or other programmable processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory or memory devices produce an article of manufacture including instruction means which implement the function specified in the block(s) of the flowchart(s). The computer program instructions may also be executed by a computer processor or other programmable processing apparatus to cause a series of operational steps to be performed on the computer processor or other programmable processing apparatus to produce a computer-implemented process such that the instructions which execute on the computer processor or other programmable processing apparatus provide steps for implementing the functions specified in the block(s) of the flowchart(s), procedure (s) algorithm(s), step(s), operation(s), formula(e), or computational depiction(s).
[0075] It will further be appreciated that the terms “programming” or “program executable” as used herein refer to one or more instructions that can be executed by one or more computer processors to perform one or more functions as described herein. The instructions can be embodied in software, in firmware, or in a combination of software and firmware. The instructions can be stored local to the device in non-transitory media, or can be stored remotely such as on a server, or all or a portion of the instructions can be stored locally and remotely. Instructions stored remotely can be downloaded (pushed) to the device by user initiation, or automatically based on one or more factors.
[0076] It will further be appreciated that as used herein, that the terms processor, computer processor, central processing unit (CPU), and computer are used synonymously to denote a device capable of executing the instructions and communicating with input/output interfaces and/or peripheral devices, and that the terms processor, computer processor, CPU, and computer are intended to encompass single or multiple devices, single core and multicore devices, and variations thereof.
[0077] From the description herein, it will be appreciated that that the present disclosure encompasses multiple embodiments which include, but are not limited to, the following:
[0078] 1. An apparatus comprising a mobile device (phone, tablet, etc) with a touch screen and camera, integrated into a microscope with an automated stage that moves a sample below an objective.
[0079] 2. The apparatus of any preceding embodiment, with an internal rechargeable battery or a switchable external battery.
[0080] 3. The apparatus of any preceding embodiment, where the touch screen is used to display the magnified images, control the stage movement through hand gestures, and/or control data collection, analysis, and transmission.
[0081] 4. The apparatus of any preceding embodiment, where hand gestures either in contact with the screen or detected by a non-contact method (e.g. front-facing camera) control microscope, stage, and data functions.
[0082] 5. The apparatus of any preceding embodiment, where gestures control stage movement through a wired or wireless (e.g. Bluetooth) connection to motors that control sample movement and focus through specific sequences (e.g. double taps, long taps, etc).
[0083] 6. The apparatus of any preceding embodiment, where the motors move the sample and/or the optical system (in the case of an infinity optics system) to collect data.
[0084] 7. The apparatus of any preceding embodiment, where software on the mobile device under gesture control is used to move the stage in proportion to the gesture and/or initiate an automated sequence of motions.
[0085] 8. The apparatus of any preceding embodiment, where software on the mobile device under gesture control is used to control imaging features, including focus, exposure, color balance, contrast, etc.
[0086] 9. The apparatus of any preceding embodiment, where software on the mobile device under gesture control is used to control sample illumination mode (bright field, dark field, phase, etc) or intensity.
[0087] 10. The apparatus of any preceding embodiment, where software on the mobile device under gesture control is used to control an arrayed LED illumination source to vary the orientation and numerical aperture of the illumination or initiate the acquisition of a series of illumination states.
[0088] 11. The apparatus of any preceding embodiment, where software on the mobile device under gesture control is used to vary the digital magnification of the region of interest and display a corresponding size calibration (the dynamic scale bar).
[0089] 12. The apparatus of any preceding embodiment, where software on the mobile device under gesture control is used to switch the light collecting optic (the objective) to vary the numerical aperture, resolution, and field of view.
[0090] 13. The apparatus of any preceding embodiment, where the illumination and/or collection optics utilize a compact folded optical path.
[0091] 14. The apparatus of any preceding embodiment, where the camera is a separate from the mobile device and controlled wirelessly.
[0092] 15. The apparatus of any preceding embodiment, where activation of proximity sensors (e.g., Hall Effect or limit switches) on each axis are used to represent an origin point for the sample position in the microscope.
[0093] 16. The apparatus of any preceding embodiment, where software for analyzing the image captured can additionally process the image during stage motion to determine the distance traveled (e.g. by autocorrelation), providing feedback on motor control for pixel-scale positioning accuracy.
[0094] 17. The apparatus of any preceding embodiment, where software on-board can discern the amount of relevant material in the field of view and automatically capture fields above a user-defined threshold while passing over the remainder.
[0095] 18. The apparatus of any preceding embodiment, where onboard software incorporates tutorials to guide the user of the instrument.
[0096] 19. The apparatus of any preceding embodiment, where software for analyzing the image captured can additionally process the image to determine a reference point based on feature recognition of a reference point in the sample.
[0097] 20. The apparatus of any preceding embodiment, where illumination optics and interference filters are used for fluorescence imaging.
[0098] 21. The apparatus of any preceding embodiment, where the illumination optics and filters are changed for multi-channel fluorescence imaging.
[0099] 22. The apparatus of any preceding embodiment, incorporating a calibration sample which is automatically positioned under the collection optics during a “self test” routine in order to provide calibration images for focus, field flatness, illumination intensity, illumination uniformity, exposure, contrast, optical resolution, or stage translation/backlash.
[0100] 23. The apparatus of any preceding embodiment, where results from the self test routine are compared with thresholds set for performance along dimensions of calibration images for focus, field flatness, illumination intensity, illumination uniformity, exposure, contrast, optical resolution, and stage translation/backlash to alert the user for maintenance and automatically order replacement parts to be delivered to the location of the instrument.
[0101] 24. The apparatus of any preceding embodiment, with vibration-isolation feet attached to the base.
[0102] 25. The apparatus of any preceding embodiment, incorporating electronic medical record keeping.
[0103] 26. The apparatus of any preceding embodiment, incorporating GPS geotagging of acquired images.
[0104] 27. The device and GPS geotagging feature above, in which geotagging is used for disease epidemiology, outbreak monitoring, or patient follow-up planning.
[0105] 28. The apparatus of any preceding embodiment, incorporating software for SMS text-message or voice calls which relay information about the data collected by the device to stakeholders (users, supervisors, doctors, patients, etc.).
[0106] 29. The apparatus of any preceding embodiment, incorporating software to transmit images, videos, analysis results, device usage data, and medical record data to a cloud server for analysis, storage, or dissemination.
[0107] 30. The apparatus of any preceding embodiment, where the mobile device incorporates analysis software to analyze collected images or videos.
[0108] 31. The apparatus of any preceding embodiment, where the software provides an automated clinical diagnosis.
[0109] 32. The apparatus of any preceding embodiment, where the software is a support-vector machine algorithm which uses shape metrics to classify features within an image.
[0110] 33. The apparatus of any preceding embodiment, where the software detects bacteria in stained specimens.
[0111] 34. The apparatus of any preceding embodiment, where the software presents sub-images of interest to the user for evaluation.
[0112] 35. The apparatus of any preceding embodiment, where the software automatically aligns and stitches neighboring fields, creating a larger image mosaic.
[0113] 36. The apparatus of any preceding embodiment, where the software can acquire a mosaic of the slide area at a lower capture resolution and present it to the user for selection of the region of interest to be captured at a higher resolution.
[0114] 37. The apparatus of any preceding embodiment, where the software detects a calibration target and automatically adjusts camera, stage, or analysis settings depending on the features of said target.
[0115] 38. The apparatus of any preceding embodiment, where low-cost rack and pinion parts are fabricated and meshed utilizing the complementary laser cutter kerfs to ensure precise meshing.
[0116] 39. An automated slide scanning system, comprising; one or more optical elements having an optical path configured to be disposed within view of a camera of a portable device; the one or more optical elements comprising an objective; an automated stage disposed within the optical path; the automated stage comprising a platform configured for receiving a slide containing a biological sample; the automated stage comprising a drive mechanism for translating the stage in at least one direction with respect to said objective; and a communications interface coupled to the automated stage; wherein the communication interface is configured for receiving a command from the portable device to control operation of the mechanical stage.
[0117] 40. The system of any preceding embodiment, the portable device comprising a cellular device having a processor and memory, the system further comprising: application programming configured to be stored as instructions in non-transitory computer-readable memory on the portable device; and wherein the application programming is executable on the processor for executing the command.
[0118] 41. The system of any preceding embodiment: wherein the automated stage comprises an x-y stage comprising independent drive mechanisms for x-axis translation and y-axis translation of the slide with respect to the objective; and wherein the application programming is configured for controlling motion of the x-y stage via one or more commands initiated from the portable device.
[0119] 42. The system of any preceding embodiment: wherein the automated stage comprises a z-axis stage comprising an independent drive mechanism for z-axis translation of the slide with respect to the objective; and wherein the application programming is configured for controlling motion of the stage via one or more commands initiated from the portable device to focus imaging of the sample on the slide.
[0120] 43. The system of any preceding embodiment, further comprising: an illumination source coupled to the automated stage, the illumination source configured to direct light at the slide; and wherein the programming is further configured to control operation of the illumination source via one or more commands initiated at the portable device.
[0121] 44. The system of any preceding embodiment, wherein the objective comprises an infinity objective, and the optical elements further comprise a plurality of mirrors folding the light path into a compact light path in line with an achromatic tube lens and the infinity objective.
[0122] 45. The system of any preceding embodiment: wherein the portable device comprises a touch screen; and wherein the application software comprises a gesture recognition module configured to control motion of the automated stage in response to one or more hand gestures applied to the touch screen.
[0123] 46. The system of any preceding embodiment: wherein the communications interface comprises a wireless communication interface, and wherein motion of the automatic stage is controlled wirelessly from one or more hand gestures initiated at the touch screen.
[0124] 47. The system of any preceding embodiment, wherein translation of the automatic stage is in proportion to the one or more gestures.
[0125] 48. The system of any preceding embodiment, wherein translation of the automatic stage is affected as an automated sequence of motions in response to the one or more gestures.
[0126] 49. The system of any preceding embodiment, wherein the programming is further configured to control one or more of bright field, dark field, phase, or intensity of the light directed by the illumination source.
[0127] 50. The system of any preceding embodiment: wherein the illumination source comprises an LED array; and wherein the application programming is further configured to vary the orientation and numerical aperture of the illumination light directed by the illumination source or initiate the acquisition of a series of illumination states with respect to the light directed by the illumination source.
[0128] 52. The system of any preceding embodiment, wherein the application programming is configured to vary one or more of the numerical aperture, resolution, and field of view of the objective.
[0129] 53. The system of any preceding embodiment wherein the application programming is configured to vary a digital magnification of a region of interest of the sample and display a corresponding size calibration corresponding to the magnification.
[0130] 54. The system of any preceding embodiment, wherein where illumination source and application programming are configured for fluorescence imaging of the sample.
[0131] 55. The system of any preceding embodiment, further comprising: a calibration sample positioned within the optical path; wherein the application software is configured to perform a calibration routine to provide calibration images of the sample of one or more of: focus, field flatness, illumination intensity, illumination uniformity, exposure, contrast, optical resolution, or stage translation/backlash.
[0132] 56. The system of any preceding embodiment, wherein the application programming is configured to apply one or more shape metrics to classify features within an image of the sample.
[0133] 57. The system of any preceding embodiment, wherein the application programming is configured to process an image of the sample during motion of the automatic stage to determine a distance traveled and providing feedback for pixel-scale positioning of the automatic stage.
[0134] 58. The system of any preceding embodiment, further comprising: one or more proximity sensors coupled to the automatic stage; and wherein the one or more proximity sensors provide position data of the automatic stage for establishing one or more of an origin point or limit for translation of the automatic stage.
[0135] 59. The system of any preceding embodiment, wherein the drive mechanism comprises a stepper motor coupled to a rack and pinion.
[0136] 60. An apparatus for automatically scanning and imaging a slide containing a biological sample, comprising: (a) one or more optical elements having an optical path configured to be disposed within view of a camera of a portable device; (b) the one or more optical elements comprising an objective; (c) an automated stage disposed within the optical path, the automated stage comprising a platform configured for receiving a slide containing a biological sample and a drive mechanism for translating the stage in at least one direction with respect to said objective; (d) a communications interface coupled to the automated stage; (e) a computer processor; and (f)a non-transitory computer-readable memory storing instructions executable by the computer processor; (g) wherein said instructions, when executed by the computer processor, perform steps comprising: (i) for receiving a command from the portable device; (ii) sending the command via the communication interface to the automatic stage; and (iii) translating the mechanical stage in response to said command.
[0137] 61. The system of any preceding embodiment: wherein the automated stage comprises an x-y stage comprising independent drive mechanisms for x-axis translation and y-axis translation of the slide with respect to the objective; and wherein said instructions when executed by the computer processor are further configured to control motion of the x-y stage via one or more commands initiated from the portable device.
[0138] 62. The system of any preceding embodiment: wherein the automated stage comprises a z-axis stage comprising an independent drive mechanism for z-axis translation of the slide with respect to the objective; and wherein said instructions when executed by the computer processor are further configured to control motion of the stage via one or more commands initiated from the portable device to focus imaging of the sample on the slide.
[0139] 63. The system of any preceding embodiment, further comprising: an illumination source coupled to the automated stage, the illumination source configured to direct light at the slide; and wherein said instructions when executed by the computer processor are further configured to control operation of the illumination source via one or more commands initiated at the portable device.
[0140] 64. The system of any preceding embodiment, wherein the objective comprises an infinity objective, and the optical elements further comprise a plurality of mirrors folding the light path into a compact light path in line with an achromatic tube lens and the infinity objective.
[0141] 65. The system of any preceding embodiment: wherein the portable device comprises a touch screen; and wherein said instructions when executed by the computer processor are further configured to control motion of the automated stage in response to one or more hand gestures applied to the touch screen.
[0142] 66. The system of any preceding embodiment: wherein the communications interface comprises a wireless communication interface; and wherein motion of the automatic stage is controlled wirelessly from one or more hand gestures initiated at the touch screen.
[0143] 67. The system of any preceding embodiment, wherein translation of the automatic stage is in proportion to the one or more gestures.
[0144] 68. The system of any preceding embodiment, wherein translation of the automatic stage is affected as an automated sequence of motions in response to the one or more gestures.
[0145] 69. The system of any preceding embodiment, wherein said instructions when executed by the computer processor are further configured to control one or more of bright field, dark field, phase, or intensity of the light directed by the illumination source.
[0146] 70. The system of any preceding embodiment: wherein the illumination source comprises an LED array; and wherein said instructions when executed by the computer processor are further configured to vary the orientation and numerical aperture of the illumination light directed by the illumination source or initiate the acquisition of a series of illumination states with respect to the light directed by the illumination source.
[0147] 71. The system of any preceding embodiment, wherein said instructions when executed by the computer processor are further configured to vary one or more of the numerical aperture, resolution, and field of view of the objective.
[0148] 72. The system of any preceding embodiment, wherein said instructions when executed by the computer processor are further configured to vary a digital magnification of a region of interest of the sample and display a corresponding size calibration corresponding to the magnification.
[0149] 73. The system of any preceding embodiment, wherein where illumination source is configured for fluorescence imaging of the sample.
[0150] 74. The system of any preceding embodiment, further comprising: a calibration sample positioned within the optical path; wherein said instructions when executed by the computer processor are further configured to calibrate images of the sample of one or more of: focus, field flatness, illumination intensity, illumination uniformity, exposure, contrast, optical resolution, or stage translation/backlash.
[0151] 75. The system of any preceding embodiment, wherein said instructions when executed by the computer processor are further configured to apply one or more shape metrics to classify features within an image of the sample.
[0152] 76. The system of any preceding embodiment wherein said instructions when executed by the computer processor are further configured to process an image of the sample during motion of the automatic stage to determine a distance traveled and providing feedback for pixel-scale positioning of the automatic stage.
[0153] 77. The system of any preceding embodiment, further comprising:
[0154] one or more proximity sensors coupled to the automatic stage; wherein the one or more proximity sensors provide position data of the automatic stage for establishing one or more of an origin point or limit for translation of the automatic stage.
[0155] 78. A method for automatically scanning and imaging a slide containing a biological sample, comprising: providing an automated stage having an optical path and an objective disposed within the optical path, the automated stage comprising a platform configured for receiving a slide containing a biological sample and a drive mechanism for translating the stage in at least one direction with respect to said objective; positioning a camera of a portable device in view of the optical path; sending a command from the portable device to the automatic stage; translating the mechanical stage in response to said command; and imaging the biological sample with the camera.
[0156] 79. The method of any preceding embodiment, wherein the portable device comprises a touch screen, the method further comprising: controlling motion of the automated stage in response to one or more hand gestures applied to the touch screen.
[0157] 80. The method of any preceding embodiment, the method further comprising: controlling motion of the stage via one or more commands initiated from the portable device to focus imaging of the sample on the slide.
[0158] Although the description herein contains many details, these should not be construed as limiting the scope of the disclosure but as merely providing illustrations of some of the presently preferred embodiments. Therefore, it will be appreciated that the scope of the disclosure fully encompasses other embodiments which may become obvious to those skilled in the art.
[0159] In the claims, reference to an element in the singular is not intended to mean “one and only one” unless explicitly so stated, but rather “one or more.” All structural, chemical, and functional equivalents to the elements of the disclosed embodiments that are known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the present claims. Furthermore, no element, component, or method step in the present disclosure is intended to be dedicated to the public regardless of whether the element, component, or method step is explicitly recited in the claims. No claim element herein is to be construed as a “means plus function” element unless the element is expressly recited using the phrase “means for”. No claim element herein is to be construed as a “step plus function” element unless the element is expressly recited using the phrase “step for”.