Instrument parameter determination based on Sample Tube Identification

Abstract

A system and method for reducing the responsibility of the user significantly by applying an optical system that can identify container like sample tubes with respect to their characteristics, e.g., shapes and inner dimensions, from their visual properties by capturing images from a rack comprising container and processing said images for reliably identifying a container tyle.

Claims

1. A method for determining characteristics of a sample container in an automated testing system, comprising the steps of: identifying the presence of a rack based on information received from a sensor; capturing at least one image of the rack; determining whether the rack comprises at least one container; illuminating each of the at least one container from the side opposite the side of the at least one container where the camera is arranged; determining a region of interest (ROI) in the at least one captured image by identifying reference points; determining the upper end of the of the rack in the ROI; determining edges of an upper end of the at least one container present in the rack within the ROI; determining the at least one container's width; determining the container's height; determining a false bottom of the at least one container by comparing within the ROI the position of the container's lower end with the position of a rack's bottom end or a rack's insert bottom end; determining the presence of a cap; and determining the type or class of a container by aggregating the results from the group comprising rack type, diameter, height, cap presence, false bottom presence determined in the at least one image's ROI with data of container stored in a database.

2. The method of claim 1, wherein height and width of each the at least one container is measured at different heights in each image and mean values with standard deviation are calculated for determining a container's dimensions.

3. The method according to claim 1, wherein the database comprises a set of instrument parameters assigned to a container type or container class.

4. The method according to claim 1, wherein the presence of the at least one container is determined by determining intersection points of the rack top and a background illumination by identifying a 2D pixel intensity matrix in different sections in the at least one captured image where the background illumination is present, followed by a first-order differentializing for identifying a slope in the intensity profile.

5. The method according to claim 4, wherein the 2D pixel intensity matrix is convolved to reduce the noise in the image.

6. The method of claim 5, wherein the 2D matrix is converted to a 1D matrix by taking an average along each row, wherein an intensity plot and variance of the 1D is used for determining the presence of the test-tube.

7. The method according to claim 1, wherein the width for illumination of each of the at least one container from the side opposite the side of each of the at least one container where the camera is arranged is in a range between 15 to 35 mm.

8. The method according to claim 1, wherein the LEDs are arranged in two opposite arranged LED stripes.

9. The method of claim 1, wherein the at least one container images are classified into one of two classes by a convolutional neural network (CNN).

10. A method for determining hematocrit, comprising the steps of: capturing at least one image of a rack containing at least one container; determining the type or class of the at least one container; determining boundaries of separated layers of a material located in the at least one container; determining hematocrit based on one or more of layers, liquid levels or liquid volumes percentage of the first layer and the second layer from the bottom in the at least one container.

11. The method of claim 10, wherein the first layer from the bottom comprises red blood cells and the second layer comprises plasma.

12. The method of claim 10, wherein determining the type or class of the at least one container comprises determining a false bottom of the at least one container by comparing the position of the container's lower end with the position of a rack's bottom end or a rack's insert bottom end.

13. The method of claim 10, comprising the step of capturing multiple images of the at least one container during its rotation in front of the camera and forming a segmented picture from the multiple images.

14. The method of claim 13, comprising the step of applying the segmented image to a convolutional neural network (CNN) for determining the upper and the lower boundary of the plasma layer in the segmented picture for generating a bounding box enclosing the plasma layer in all segments of the segmented image.

15. The method of claim 14, comprising the step of rearranging the segments of the segmented image prior to determining again the upper and the lower boundary of the plasma layer in the newly arranged segmented picture for generating a bounding box enclosing the plasma layer in all segments of the segmented image.

Description

BRIEF DESCRIPTION OF THE FIGURES

[0048] The invention will be described based on figures. It will be understood that the embodiments and aspects of the invention described in the figures are only examples and do not limit the protective scope of the claims in any way. The invention is defined by the claims and their equivalents. It will be understood that features of one aspect or embodiment of the invention can be combined with a feature of a different aspect or aspects of other embodiments of the invention, in which:

[0049] FIG. 1 shows several examples of false bottom tubes.

[0050] FIG. 2 shows in a flow chart the steps of a method for identifying a container type or class.

[0051] FIG. 3A shows an example where different tube types can be clearly separated from another.

[0052] FIG. 3B shows an example where tube types can be merged into a class of tubes with common properties.

[0053] FIG. 3C shows an example for false-bottom tubes.

[0054] FIG. 4A shows a camera front illumination system according to the present disclosure.

[0055] FIG. 4B shows a back illumination system according to the present disclosure.

[0056] FIGS. 4A-6 show embodiments of a front illumination system according to the present disclosure.

[0057] FIGS. 7A, 7B and 7C show embodiments of the back illumination system with different widths of 35 mm, 25 mm and 15 mm.

[0058] FIG. 8A shows a perspective view of an embodiment with 13 LEDs.

[0059] FIG. 8B shows a side view of an embodiment with 13 LEDs.

[0060] FIG. 9A shows a possible camera setup with a camera, a rack with rack inserts and container which are illuminated from the back with a back illumination system.

[0061] FIG. 9B shows an example for identifying the pixel where the background illumination is present in the image.

[0062] FIG. 10 shows an example for identifying a false bottom tube.

[0063] FIG. 11 shows also an example of a false bottom detection pipeline for a situation where the tube is not a false bottom tube.

[0064] FIG. 12 shows an approach collecting a dataset of labeled images for training a classifier.

[0065] FIG. 13 shows a training process.

[0066] FIG. 14 shows training of a CNN.

[0067] FIG. 15 shows the architecture made of bottleneck and bottleneck residual blocks.

[0068] FIGS. 16A and 16B show a curve with a negative slope of the intensity values decreasing from a higher value to a lower value.

[0069] FIGS. 17 and 18 show an example for determining the plasma portion in a test specimen.

[0070] FIG. 19 shows the network output with a line below the image indicating the plasma layer.

[0071] FIG. 20 illustrated a cell offset from the top left corner of the image by (cx, cy) and the bounding box prior has width pw in accordance with an embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

[0072] The technical problem is solved by the independent claims. The dependent claims cover further specific embodiments of the invention.

[0073] A container according to the present disclosure can be a tube, vial, a vessel, a pipette tip, a plate with one or multiple recesses or wells, a microfluidic device or a bottle.

[0074] The invention relates to a system for reducing the responsibility of the user significantly by applying an optical system that can identify container like sample tubes with respect to their characteristics, e.g., shapes and inner dimensions, from their visual properties by capturing images from a rack comprising container and processing said images for reliably identifying a container tyle.

[0075] FIG. 1 shows a rack 10 configured to receive different container 12 types in the same rack 10. Positions 1, 3 and 5 contain so called false bottom tubes. The term false bottom within the meaning of the present disclosure relates to a tube's bottom wall thickness being >3 mm).

[0076] The present disclosure relates in a first aspect to a method for identifying a container type or class, like a sample tube for instance. In a further aspect, the present disclosure relates to a method for identifying level of separated layers in a container, which can be used for the Hematocrit assessment.

[0077] The method may comprises the following steps which are also shown in FIG. 2: [0078] 1. Providing a tube in front of a camera; [0079] 2. Capturing an image; [0080] 3. Determining reference points in the image that define relative positions of regions of interest (ROI); [0081] 4. Determining the presence of a tube; [0082] 5. In parallel: [0083] 5.1. Determining dimensions of the tube, comprising at least height, width, and shape; [0084] 5.2. optionally detecting level of separated liquid layers; [0085] 5.3. detecting the presence of a cap; [0086] 6. detecting false bottom properties if images from all angles of the tube are available.

[0087] Finally, all results are aggregated, and the container type or class is determined by comparing the obtained data with data stored in a database. During rotation of the tube in front of a camera, multiple images can be captured, and the results are generated for individual images or for all images. Results from individual images are consolidated in a single result. For example, the height of each container is measured in each applicable image and averaged with standard deviation. The width of each container at different heights is also measured in each applicable image and averaged with standard deviation. Cap or no-cap properties are determined. False-Bottom properties are determined. Also, if centrifuged blood is present in the tube, the method can be applied to the establishment of the different levels of the centrifuged blood applying it to the hematocrit assessment.

[0088] The overall goal of the methods and systems according to the present disclosure includes finding instrument parameters to safely and accurately pipet liquids from a container like a tube, comprising pipetting only a specific layer in a container or tube. The overall goal can also include providing medical measurements such as hematocrit, from a centrifuged-blood tube automatically, without further manual sample preparation or taking a tube holding the sample off a track or other ways of interfering with the tube's transfer on the track. Data processing and algorithms are applied such that the containers are accurately characterized, and the medical measurements are accurately made.

[0089] A set of instrument parameters can be applied to a single container type of a group or to a class of container types that share similar dimensions. FIG. 3A shows an example, where different tube types can be clearly separated from another and FIG. 3B shows an example where tube types can be merged into a class of tubes with common properties.

[0090] Some tube types have the same or similar outer dimensions but differ in their inner dimensions. One aspect in this regard are the so-called false-bottom tubes (FIG. 3C). An incorrect determination may result in a probe crash that makes the instrument inoperable until repaired. False bottom container may be difficult to detect due to the bottom being covered by a label. Another problem related to false bottom tube can be caused from an incorrect knowledge of container volumes which will lead to a wrong calculation of a hematocrit boundary. To circumvent this situation a false-bottom indication allows to distinguish between those specific cases.

[0091] The methods and systems according to the present disclosure can comprise information about container types or classes, in a database for instance, which are to be queried during determining a container type or container class. The information includes a mapping between one or more characteristics of a container, e.g., instrument parameters such as height or width, along with their tolerance levels, to a type or class of the container. The aggregated results of the in-process container characteristics determination can be used to query the database to identify the type of the container. Only distinctive characteristics (not all) need to be measured to know all characteristics of a container. Mapping in a first step only selectively measured parameters like height, width, etc. allows the system to retrieve all geometric information about the container from the database in a second step. The information is accurate, more accurate than measured ones, such that operations on the container or medical measurements can be accurately performed. This two-step process is advantageous because the system does not need to measure all tube parameters, because they are available from the database which can be extended to provide additional parameters which will not have to be measured in the first step.

[0092] The database can be extended with information about new container features and applicable instrument parameters to provide the system with information regarding new container types or container classes used in the field. Additionally, an installation and operational qualification (IQ/OQ) can be performed to ensure a correct differentiation of expected container types or classes.

[0093] The optical hardware subsystem for the container identification can perform at least the following tasks: [0094] Illumination of the container (e.g., sample tube) from the front; [0095] (optional) Illumination of the sample tube from the back, which means that the container are illuminated from the side opposite the side of the container where the camera is arranged; [0096] Capturing images of the sample tube with a quality sufficient to perform dimensional measurements and to decode bar- or matrix codes attached to a tube; [0097] In order to meet these requirements, an example optical hardware subsystem can include the following components: [0098] Front illumination system; [0099] (optional)Back Illumination system; [0100] Imaging objective lens; and [0101] Imaging sensor circuit board.

[0102] A possible embodiment of a front illumination system according to the present disclosure is shown in FIG. 4A to FIG. 6. The embodiment in FIGS. 4A and 4B show a camera front illumination system 2 and a back illumination system 3. A container 12 is shown in front of the camera and front illumination system 5, 2. The container 12 is located (e.g., on a track that automatically moves the container) next to the back illumination system 3. A mirror 7 is placed opposite the back illumination system 3 so that the container 12 is located between back illumination system 3 and mirror 7.

[0103] In some implementations, a monochrome CMOS sensor can be used. The sensor can have a resolution of 1200×1600 pixels with a pixel size of 4.5 μm can be used in combination with an objective lens with a focal length of 16 mm, although sensors with other suitable, e.g., higher, resolutions can be used. The working distance of the imaging system is roughly 300 mm resulting in a field of view of about 130×100 mm.sup.2.

[0104] The pixel size in the sample plane which is the imaginary plane at a distance of about 300 mm representing the working distance to the camera can be about 80 μm or less. Such a hardware configuration can allow to differentiate items on the tube like a barcodes with a bar width of about 7.5 mil or larger. Those who are familiar with the art will appreciate that also other combinations of sensor area, pixel sizes and focal lengths are possible, depending on the respective requirements of the application. In particular, the use of a color sensor may be advantageous, in case different colors will have to be detected or distinguished in a container's content or on a container label.

[0105] Using a color sensor usually reduces the pixel resolution of the sensor, as a Bayer pattern for realization of the RGB channels is imposed on the pixel array. This reduction of resolution may be detrimental for barcode reading. This can be overcome by accessing each single pixel on the hardware level before the pixel interpolation in the color channels is done. In order to avoid artefacts caused by the Bayer pattern, it is necessary to calibrate the response of each color channel in combination with the used Illumination. As the relative responses of the single-color channels depend on the spectral composition of the illumination it is important to use the same illumination system during calibration than in the later application. The objective lens used in such an embodiment can be a standard catalogue S-Mount objective lens. The method according to the present disclosure is not limited to a hardware configuration of standard objective lenses or S-Mount lenses. Custom designed lenses may have advantages for various applications.

[0106] The front Illumination system 2 may comprise four single dye LEDs 40 in combination with standard reflectors 45 (FIG. 6), which are lined up along the long side of the field of view (FOV) and which are symmetrically arranged about the camera subsystem in the embodiment shown in FIG. 6. As indicated in FIG. 5, each pair of reflectors is tilted with respect to the optical axis of the image sensor 20 and image objective lens 25 by around 7 degrees. This results in a rather homogeneous irradiance in the plane of the sample tube. Of course, the angle is dependent on the size of the field of view, the working distance and the collimation of light emerging from the TIR-reflectors (TIR=total internal reflection) and may need adjustment for each application. Those who are familiar with the art will appreciate, that the setup is not limited to the use of standard TIR-reflectors. Custom designs may be advantageous for different application scenarios. The setup is not limited to any color of illumination. Single color or broadband spectral LEDs may be used depending on the application. Broadband LEDs may be advantageously combined with the use of a Color imaging sensor.

[0107] FIG. 5 and FIG. 6 do not show polarizers in front of the illumination system in combination with a crossed analyzer in front of the imaging objective lens 25 which are required for reading barcodes. If these polarizers are not included in the setup of a system for performing the method according to the present disclosure, a bright stripe in the middle of the tube can appear due to direct specular back reflection. This bright stripe can reduce the contrast of the barcode in the image significantly. Furthermore, spectral filters in front of the camera, matching the illumination spectrum could be used to suppress disturbances by ambient light.

[0108] The back illumination, meaning the illumination of a container from the side opposite the side of a container where a camera is arranged, is used to facilitate the determination of tube characteristics such as the tube diameter. Using the front illumination only in combination with a white reflective surface behind the tube may lead to shadows smearing out the edges of the tub and will thus reduce the precision and accuracy of the measurement. A further important parameter is the width of the back illumination. Example images of the back illumination system with different widths of 35 mm, 25 mm and 15 mm are shown in FIGS. 7A, 7B and 7C.

[0109] Aperture within the meaning of the present disclosure refers to the width of the back illumination. The smaller the width of the aperture the stronger becomes the contrast of the tube edges in the images and the more accurate can the tube diameter be measured (left to right in FIGS. 7A-7C). In the present embodiment of the system the aperture can have a width in the range of 25 mm to 35 mm, e.g., 30 mm or less. The selection of an aperture width can be based on multiple factors, e.g., among others, the application, the tube diameter, and the distance between the tube and the back Illumination.

[0110] The back Illumination system comprises two LED stripe circuit boards and a control or supply circuit board, each of them equipped with a set of LEDS. In the example shown in the figure, 13 LEDs (FIGS. 8A and 8B) are used. The subject-matter of the present disclosure is not limited to this exact number of LEDs. Both LED strips can be arranged opposite to each other on the two long sides of a doped Perspex panel. The doping in the Perspex bulk material leads to scattering of light in the material and outcoupling of light from the panel on the front and back side. Without doping, the light may be enter by internal reflection inside the panel and would leave it on the opposite side at the place of the second LED stripe.

[0111] PLEXIGLAS 0E010SM by Evonik has been used in an embodiment of the system. The setup is not limited to the use of this material, other materials with similar properties allowing to scatter the LED light for obtaining a homogenous back illumination may also be used. Defined structuring of the plastic panel on the back side, that disturbs internal reflection in a controlled way within the meaning of scattering the light may further improve the light efficiency in terms of emitted lights by the LEDs of the Illumination system. As light can emerge from, both front and back sides of the panel, a white reflective film (KIMOTO Ref White RW188) is used on the back side of the panel. This white film reflects light emerging from the back side back towards the front aperture of the illumination system and enhances the efficiency of the Illumination system. In order to further homogenize the luminance of the system, a diffusing film is put on the front of the panel. Optionally, an additional protection glass can be put on the front of the panel, which is not shown in the figure. The high density of LEDs, the homogeneous doping of the Perspex material in combination with the diffusing film led to a very homogeneous luminance over the output aperture of this back Illumination system. The spectra of the LEDs are not limited to any special color. Depending on the application, single color or spectral broadband LEDs can be used. In the present example white LEDs are used.

[0112] As the camera has a fix-focus and due to mechanical and optical tolerances, the system has to be calibrated during production, which means a magnification factor is determined for each system by the use of a known target. This magnification factor is then stored as parameter in the Software and used for all image analysis/calculation steps.

[0113] The test tube presence detection is part of a sample tube identification method. A test tube presence detection module decides if a step can be skipped because a sample tube is not present. FIG. 9A shows a possible camera setup with a camera 5, a rack 10 with rack inserts 14 and container 12 which are illuminated from the back with a back illumination system 3. The camera can be tilted by 90° clockwise which causes all images of tubes being tilted as well. The back illumination illuminates the top area of a rack.

[0114] The method comprises basically the step of scanning a rack with a camera, wherein the rack provides for instance five slots for receiving container or test tubes. Each possible position for a tube is indicated with position markers, e.g. a 2d barcode. A region of interest (ROI) is defined in an area where back illumination is detected and a tube has to be visible if present.

[0115] The method for the test tube presence detection comprises in more detail the following steps: [0116] 1) Identifying pixel with background illumination (FIG. 9B): [0117] Choosing a column of pixels in one half of the image; and [0118] Performing a first-order differential for identifying a first slope in an intensity profile of grey values in an image, wherein a first slope indicates a left or right boundary of the background illumination; [0119] 2) Identifying a top corner of the rack: [0120] Choosing a row of pixels on the left or right side of the identified boundary of the background illuminated area; and [0121] Performing a first-order differential for identifying a second slope in the intensity profile of grey values in an image, wherein the second slope indicates transition from rack to illuminated background thus indicting the top of the rack; [0122] 3) Defining the intersection between the line of the identified left or right boundary and the rack top, wherein the intersection represents a corner point which is required for defining a ROI; [0123] 4) Defining the ROI starting from corner point, wherein the size of the ROI depends from known mechanical boundaries of the rack; [0124] 5) Convolving the 2D pixel intensity with a Gaussian kernel of 5×5 matrix for reducing the noise to signal ratio in the image; [0125] 6) Converting the 2D matrix into 1D array by taking an average value along each row; and [0126] 7) Determining the presence of a tube from an intensity plot of the 1D array by determining intensity changes in the 1D array.

[0127] In some embodiments, the measurement of tube width and height comprises one or more the following steps: [0128] Detection of the region of interest (ROI) as the top area of the image (area above the top rack); this area contains the top part of the tube; [0129] Detection of the edges on the ROI using the Canny algorithm; [0130] Detection of the rack top; [0131] Detection of left profile, right profile and top profile of the tube; [0132] Measurement of tube width at predefined heights above the top of the rack; and [0133] Measurement of the tube height upwards above the rack up to the highest point of a tube's top profile.

[0134] The method can be applied to colored and monochrome images.

[0135] The present disclosure refers also to the detection of false bottom properties of a container using convolutional neural networks (CNN). Deep learning, when implemented as a supervised learning task, learns a function that maps an input to an output based on input-output examples provided by the user. The function, also called a deep neural network, learns low-level and high-level features from the image, such as edges, corners or shapes and object parts. A classification method classifies the image into a set of predefined classes based on the learned features.

[0136] Training a deep learning model is a highly iterative process that includes one or more of the following steps: creation of a high-quality training dataset, design of the ConvNet architecture, training the model, evaluation of the model on the validation dataset. This process can be repeated until the desired accuracy is achieved (e.g., 100%).

[0137] Sample tube identification includes detection of false bottom properties. A tube can have a secondary elevated bottom to raise sample level in order to facilitate a more optimal automated analysis. The secondary bottom is also denoted false bottom. The false bottom tubes vary in appearance and false bottom size, defined as the distance between the true bottom and the false bottom. FIG. 1 shows several examples of false bottom tubes with bottom height ranging from 7 mm to 43 mm (position 1, 3, and 5). The presence or absence of a false bottom is detected by processing images of the tubes.

[0138] The tube stops at a designated position in front of the camera. The tube is turned 360 degrees and a fixed camera captures multiple, e.g., 10 of the rotating tube. The images are processed, and the tube will be classified as NO FALSE BOTTOM (=0), indicating the absence of the false bottom, or FALSE BOTTOM (=1), indicating the presence of the false bottom. Image classification may be solved using supervised deep learning algorithms. A classification algorithm classifies the image into a set of predefined classes based on the learned features, in our case NO FALSE BOTTOM (=0) and FALSE BOTTOM (=1).

[0139] The tube images are classified into one of the 2 classes by a convolutional neural network (CNN), a class of deep neural network algorithms. The CNN receives a batch of, e.g., ten images of the tube. The images are grayscale and have a size of 10 (images)×H×W, where for example H=1536 pixels, W=2048 pixels. The images are cropped relative to the position code with the region of interest (ROI) of size H′×W′, which contains the bottom half of the tube. The ten ROIs are concatenated along the channel dimension. The input to the CNN network is the batch of B (batch)×10×H′×W′ images, where B=1 during inference and B>=1 during training. CNNs perform image feature extraction by applying convolutional operations with filters, whose parameters are learned in the training process. A ConvNet stacks a variable number of convolutional layers and activation functions such as ReLU and learns basic features such as corners, edges at initial layers, while complex high-level features are learned at deeper layers. A classification algorithm such as SoftMax takes as input the extracted features and outputs a normalized probability distribution of the predefined classes. The class with the highest probability is chosen as the class of the tube.

[0140] FIG. 10 shows an example of a false bottom detection pipeline when a container like a tube is a so-called false bottom tube. A batch of ten images has been processed by the CNN and the CNN outputs a normalized probability distribution of the classes. The tube is classified as false bottom since the probability for the false bottom class is the largest.

[0141] FIG. 11 shows also an example of a false bottom detection pipeline for a situation where the tube is not a false bottom tube. A batch of ten images has been processed by the CNN and outputs a normalized probability distribution of the classes. The tube is classified as no false bottom since the probability of then no false bottom class is the largest.

[0142] Each image can be annotated with one of the labels: NO FALSE BOTTOM (=0), FALSE BOTTOM (=1). Training may be done on as many images as necessary to achieve a good accuracy. Using the available dataset, a training and a validation dataset are to be created. Around 80% of the data goes to the training set and 20% of the data to the validation set. In the training phase, the network sees examples only from the training set and learns the weights based on the training image-label samples. To evaluate the performance of the network, the accuracy on the validation set has been computed, which contains unseen examples. If the accuracy on the validation set is not satisfactory, which should be close to 100%, more samples can be added to the training set and the network can be re-trained. Then, the network will be evaluated again on the validation set. The training process can be stopped when the desired accuracy in determining the correct label of an image is reached.

[0143] In the training process, one or more batches of multiple, e.g., 10, images with their associated labels are randomly sampled from the training set. A Region of Interest (ROI) with the bottom of the tube is extracted from each image. The image ROIs are passed through the network and the loss function is calculated, which measures how close the predictions are compared to the training labels.

[0144] The training process can be done in any suitable deep learning frameworks: TensorFlow, PyTorch, or MXNet.

[0145] A CNN network architecture according to the present disclosure is made of residual blocks, pooling operations and fully connected layers. Residual blocks learn residual functions with respect to the input layer and facilitate convergence especially in the case of deeper networks. First, the network is comprised of two Convolutions, Batch Normalization, ReLU activation, that computes features at half input resolution using 32 filters. The convolution filters have a small receptive field of 3×3 and the stride is set to 2 for the first convolution to perform down sampling and 1 to the second convolution. The spatial padding has been set to 1 for 3×3 convolutions to preserve the spatial resolution after convolution. Spatial pooling followed using maximal pooling layers with a filter size of 3×3 and stride 2. The residual blocks are comprised of two [3×3] convolutions, batch normalization layer and ReLU activation. A shortcut connection between the input and output of the block has been introduced, which are identity shortcuts when the input and output have the same size and a projection shortcut with a 1×1 convolution to match the dimension when sizes differ. At each layer following the spatial pooling two residual blocks have been added. Layers [4-6] perform down sampling with the first convolution in the first residual block having a stride of 2. In order to ensure the low complexity of the model, the number of learned filters is small and increases from 32 to 128 in the last layer of the network. Finally, an average pooling and a fully connected layer are applied that yield the logits for the 2 classes. Softmax is the last layer of the network, which assigns a probability to each class and represents the likelihood of the image to belong to that class. The reported class for an image is the class with the highest probability.

TABLE-US-00001 TABLE 1 Summary of the architecture of the Convolutional Neural Network. Input and output size format: Batch Size × Height × Width. Filters: [filter height × filter width, number of filters, stride]. Layer Input size Output size Filters 1 10 × 75 × 10 × 38 × [3 × 3, 32, 2], Batch 255 128 normalization, ReLU [3 × 3, 32, 1], Batch normalization, ReLU 2 10 × 38 × 10 × 19 × [3 × 3, —, 2] MaxPooling 128 64 3 10 × 19 × 10 × 19 × Residual block: [3 × 3, 32, 64 64 1], BN, ReLU, [3 × 3, 32, 1], BN, ReLU Residual block: [3 × 3, 32, 1], BN, ReLU, [3 × 3, 32, 1], BN, ReLU 4 10 × 19 × 10 × 10 × Residual block: [3 × 3, 64, 64 32 2], BN, ReLU, [3 × 3, 64, 1], BN, ReLU Residual block: [3 × 3, 64, 1], BN, ReLU, [3 × 3, 64, 1], BN, ReLU 5 10 × 10 × 10 × 5 × Residual block: [3 × 3, 128, 32 16 2], BN, ReLU, [3 × 3, 128, 1], BN, ReLU Residual block: [3 × 3, 128, 1], BN, ReLU, [3 × 3, 128, 1], BN, ReLU 6 10 × 5 × 10 × 3 × Residual block: [3 × 3, 128, 16 8 2], BN, ReLU, [3 × 3, 128, 1], BN, ReLU Residual block: [3 × 3, 128, 1], BN, ReLU, [3 × 3, 128, 1], BN, ReLU 7 average pool, fully connected layer, softmax

[0146] The network is trained end-to-end with one optimization step. The implementation has been tested on the false bottom dataset. In all experiments, the network with Kaiming Uniform method has been initialized. The network is trained with a mini-batch of samples and Adam optimizer. The weighted cross entropy loss is minimized during training. In order to solve class imbalance, the loss is weighted by the inverse class frequency.

[0147] Detecting the presence of caps for sample tubes is important for the control system. This information is used for aiding the pipetting process. Tubes inserted into the system may have cap, no cap or a special type of cap called “false cap”. False caps are a particular type of caps which are hollow on the inside but given the perspective from which the tube is observed, the hollowness is not visible. Tubes with such caps are considered to be without cap and the system should identify them appropriately. For each container, like a sample tube, one or 10 images (360 degrees turning) may be acquired. The acquired image(s) are processed and will be classified as: NO CAP (=0), CAP (=1), FALSE CAP (=2). For solving this task a data driven approach may be used, such as machine learning. This approach requires collecting a dataset of labeled images, which is used for training a classifier which can operate with high accuracy on images it has never seen before (FIG. 12)

[0148] For detecting the presence of caps, a region of interest (ROI) containing the upper part of the tube shall be extracted from the full-sized image. The ROI is then processed through a normalizing procedure, which involves rescaling the pixel values to be in the interval [0-1] followed by centering the values to have zero mean and unit variance. The obtained image can then be classified using support vector machine (SVM), tree-based classifiers (e.g. decision trees), adaptive boosting classifiers (e.g. Adaboost) or artificial neural networks (e.g. Convolutional Neural Networks (CNN)). The approach taken in this implementation is based on CNNs, the obtained image is inserted into a CNN which classifies the input image. The output of the CNN is a probability distribution over all the possible pre-defined classes, which are cap, no cap, false cap. In the training process, the network is adjusting the connection strengths between its layers such as the probability assigned to the correct class will be maximized, while the probabilities assigned to the other classes will be minimized. (FIG. 13)

[0149] For training the CNN, a dataset containing multiple reference images for each supported container type is constructed. According to each container specific characteristics, reference images are acquired for the container with cap, without cap and with false cap (if false cap is a characteristic of the tube type). All the images are labeled with one of the pre-defined classes and each image is added inside the appropriate class directory. The dataset is then split again in two parts, the training set (e.g., 80%) and the test set (e.g., 20%). The images in the training set are used for fitting the model in the training process (adjusting the weights of the neural network). The images in the test set help providing an unbiased evaluation of how good of a fit the model achieved after training on the training set (FIG. 14).

[0150] The dimensions of the initial acquired images are H×W×1, with H=1536 pixels and W=2048 pixels. For each image the ROI is extracted, which has the dimensions 350 pixels×675 pixels×1 which will be the input for the neural network. The output of the classification algorithm is a probability distribution, which can be achieved using a function that output normalized probability distributions, such as Softmax. To achieve a CNN that is robust to variations of the images (e.g., brightness, contrast, noise), data augmentation is used in the training process. During training, before feeding an image as input to the neural network, random flipping, random brightness and/or contrast variations are applied. The training process is continued until a sufficiently high accuracy is obtained (nearly 100%). The network is trained end-to-end with a mini batch of 64 samples from the Cap_NoCap_FalseCap dataset.

[0151] The architecture can be made of multiple, e.g., 10 or more, 15 or more, or 17, bottleneck and bottleneck residual blocks followed by one or more global average pooling layers and one or more, e.g., 2, fully connected layers, the last fully connected layer being the classification layer or keeping the amount of computation and memory footprint low, the first layers in the neural network architecture are 2 blocks of “Convolution-Batch Normalization-Relu6”, the convolutions being performed with a stride of 2, thus reducing the dimensionality of the input by a factor of 4. Another optimization for efficiency that was used consisted of setting the expansion factor in the first 6 bottleneck residual blocks to equal 4, while for the last two bottleneck layers it equals 6. The total number of feature maps in the last bottleneck residual block is kept to 192. The total number of parameters of the neural network model is 20.243, out of which 18.435 are trainable parameters (FIG. 15).

[0152] The detailed CNN architecture used for detecting the presence of Cap_NoCap_FalseCap is presented in Table 2. It can be implemented using any suitable computing framework or toolbox, such as Tensorflow, Pytorch, or Keras.

TABLE-US-00002 TABLE 2 CNN architecture used for detecting the presence of Cap_NoCap_FalseCa Input Output Output Expansion Type size size Channels Factor Input block 350 × 675 × 1 175 × 338 × 8  8 — (Zero Padding, Conv2D with 3 × 3 and stride 2, Batch Normalization, Relu6) Downsampling block 175 × 338 × 8 87 × 168 × 8 8 — (Conv2D with 3 × 3 and stride 2, Batch Normalization, Relu6) Bottleneck residual block 1  87 × 168 × 8 87 × 168 × 8 8 1 Bottleneck block 1  87 × 168 × 8 44 × 84 × 8 8 4 Bottleneck residual block 2 44 × 84 × 8 44 × 84 × 8 8 4 Bottleneck block 2 44 × 84 × 8 22 × 42 × 8 8 4 Bottleneck residual block 3 22 × 42 × 8 22 × 42 × 8 8 4 Bottleneck block 3 22 × 42 × 8 11 × 21 × 8 8 4 Bottleneck residual block 4 11 × 21 × 8  6 × 11 × 16 16 6 Bottleneck block 4  6 × 11 × 16  6 × 11 × 16 16 6 Conv Output block  6 × 11 × 16   6 × 11 × 192 192 — (Conv2D with 1 × 1 and stride 1, Batch Normalization, Relu6) Global Average Pooling   6 × 11 × 192 192 — — Fully Connected Layer 1 (with 192 16 — — Dropout(0.4) and with Relu activation) Fully Connected Layer 2 (with 16 8 — — Dropout(0.4) and with Relu activation) Fully Connected Layer 3 (with 8 3 — — softmax) activation

[0153] For validating that the proposed architecture and the training procedure can generate models that work at a sufficiently high accuracy when new container types or classes are introduced in the system, k-fold cross validation is used. It provides a robust estimate of the performance of the model on unseen data. In k-fold cross validation the dataset is partitioned into k equal-sized subsets. Out of the k subsets, one subset is retained as the validation data for testing the model, and the remaining k-1 subsets are used as training data. This process is repeated k times, with each of the k subsets being used exactly once as the validation data, thus generating k models. The performance measures (e.g. accuracy) obtained in each iteration are then averaged across all created models and standard deviation is computed, providing a less biased estimate of the performance achieved using a specific architecture and training procedure.

[0154] The container from the container database (which contains all the container containers supported by the system) are split into k subsets, where k is the number of supported container, each container type being used as the validation data once. An even more robust estimate is obtained when more container types are held out in one iteration for validation (e.g. out of the k container types, two or more container types can be held out for validation in each iteration). The model is accepted if the average accuracy of the generated models is above a target accuracy (e.g. 98%) and if the standard deviation is smaller than a target sigma value (e.g. 1).

[0155] The obtained model is then evaluated on the test set to see if it operates at a sufficiently high level of confidence (e.g., greater than 98%). For each prediction, it is first evaluated if the confidence value (out of the three output probabilities of the network, the one with the highest value) is greater than a defined threshold (e.g., 0.5). If the confidence value is below this threshold, then the prediction of the network is considered to be uncertain. The number of predictions with small confidence on the test set is then counted. The lower the number of predictions with small confidence, the better fit the model has on the task. The predicted class is selected as the output probability with the highest value. If the predicted class is different from the expected correct class, then the number of errors on the test set is incremented. For better assessing the confidence and the quality of the predictions a new metric called confidence margin will be evaluated. For each output of the network, the difference between the highest probability value and the second highest probability value is computed. This metric helps at assessing if the features found for an input image are representative enough to discriminate it from the features specific to other classes. A low confidence margin means that the network is uncertain in its prediction and cannot discriminate between classes, so further images of that kind shall be acquired and the model trained on them. The minimum, maximum and average confidence margin is assessed on the test set for the correct predictions, the incorrect predictions and for the predictions with small confidence. For the correct predictions the minimum confidence margin should be as high as possible, the average confidence margin should be above a pre-defined threshold (e.g. 0.5) and the maximum confidence margin should be close to 1.0. For the incorrect predictions, the minimum confidence margin should as small as possible, the average confidence margin should be as small as possible, the maximum confidence margin should be as low as possible, conditions if satisfied then such incorrect predictions are discarded. Predictions with small confidence value are discarded as well.

[0156] Liquid level detection, based on which hematocrit can be determined for samples in a container, is also an object of the present invention. The input is a container like a sample tube comprising a centrifuged sample. The container can be rotated in front of a camera to generate a number of images (images/rotation). The images can be used to determine the tube characteristics as described above, and can also be used for detecting the liquid levels within the tube, e.g., the total liquid level, the interface between the plasma -and the red blood cells level and the bottom of the red blood cells level

[0157] First, images showing container which do not have a barcode or a label are further processed to detect if the test-container is empty or not.

[0158] For empty container detection, the images are processed to check whether the test container is empty or not; if the test container is empty, the sample container feature extraction steps can be skipped. A sample container with a blood sample has a higher variance in intensity than the sample container that is empty. Statistical parameters such as variance and CV (coefficient of variation, is defined as the ratio of the standard deviation to the mean) are used to define the flatness of the intensity profile within the sample-tube to identify if the sample-tube is empty or not.

[0159] Real centrifuged blood samples with different % Hct (hematocrit) values and blood challenges were used to establish a clear line of separation between red blood cells and plasma. White illumination was used as a front illumination to make the red blood cells to be dark and the plasma to be bright in the image.

[0160] In an initial step, an ROI-smoothed image is generated with a Gaussian filter of 5 pixels by 5 pixels which is convolved to eliminate noise in the taken image. Based on the kernel size and the filter type applied the results can deviate to [by?] one or two pixels. [0161] Next, one or more the following steps can be performed:dimension reduction: The ROI-image which is a 2d (two-dimensional) array of intensity values is reduced to a 1d (one-dimensional) array, e.g., by summing all the column or row values together so that they become a single row and multiple, e.g., over 700, over 750, or 787, columns or a single column and multiple rows. [0162] Normalization: the 1D array is normalized, e.g., with the maximum value, from the array; [0163] Differentiation: A first-order differential of the normalized array is computed, where the maxima and minima define the different liquid levels in the container, e.g., the bottom liquid level or blood level and the too liquid level or plasma level, respectively. In particular, [0164] i. Bottom level is a liquid level at which a transition from the bottom liquid component, e.g., blood cells, to the top liquid component, e.g., plasma, takes place. Such a transition is represented by a positive slope of the intensity with a pixel value between 0-255, because the image intensity value increases from a lower value to a higher value. [0165] ii. Top level is a liquid level at which a transition from top liquid component, e.g., plasma, to the empty space, e.g., air, in the test-tube takes a place. Such a transition is represented by a negative slope of the intensity with greyscale values between 0-255 because of the concave nature of the top of the liquid component (due to building a meniscus which is concave) as the image intensity value decreases from a higher value to a lower value (FIGS. 16A and 16B). [0166] iii. Accordingly, the maxima and minima of the differentiated array provide information about the geometric locations the bottom liquid level and top liquid level within the sample container.

[0167] There are two potential types of interferences: [0168] A) Interferences that can impact the correct assessment of the levels essential to calculate the Hematocrit: [0169] A1) Bottom level of the RBC due to deficient establishment of the real RBC bottom level (e.g.: not distinguishing the color of the RBC from the bottom of the tube/rack shelf); [0170] A2) Interface level between the RBC and White cells/plasma due to not distinguishing the difference in color because interferents in the plasma produce color similar to RBC (e.g.: high concentration of lipids, or/and hemoglobin, or/and bilirubin); and [0171] A3) Top level of plasma, this level can be difficult to identify if its color is not clearly differentiated form the color of the tube.

[0172] For these three cases, real centrifuged blood samples, with the above-mentioned interferents, and different Hematocrit concentrations are used to challenge the algorithm to find out and improve its reliability to identify the described levels. [0173] B) Interferences that can prevent the detection of the above-explained levels due to physical impediments, including but not limited to labels covering the whole surface of the tube, printed graphic/lettering on the tube, and others. If the tube does not have a minimum sliver of room to reliably detect the above-mentioned levels; then, the Hematocrit will not be possible to assess and the result will be given as undetectable.

TABLE-US-00003 TABLE 3 Summary of possible interference from illumination. Illumi- nation Pros Cons Blue — 1. Not suitable for animal blood Yellow 1. Distinguishes red blood 1. No clear separation between cells (RBC) and plasma RBC and white blood cells (WBC) 2. Easy to identify the 2. Requires long exposure time meniscus of the plasma as 500 microseconds it is thin Amber 1. Distinguishes RBC and 1. Often penetrates through the plasma label on the test tube and 2. Clear separation between increases the complications RBC and WBC 2. Hard to find meniscus of the 3. Std-dev of the signal plasma, because a thick layer of seems to be lesser than the the meniscus is seen yellow illumination IR 1. Lesser exposure time 1. No clear separation between 2. Less current RBC and WBC consumption

[0174] In some implementations, a first-order derivative of the reduced, normalized array as described above may not be appropriate when there is no clear separation between the liquid components, e.g., RBC, vs WBC/, and the plasma. For example, each region of the image may seem to fall in a certain category of intensity level, because they have the same intensity level. If the WBC or the buffy region is too thick or if a clear boundary between buffy region to the plasma is not clearly visible, then in the image we see a gradual transition in the intensity. Such a situation makes it hard to define a clear level detection of plasma. In some implementations, a histogram can be constructed to show the distribution of intensity.

[0175] For determining hematocrit, the container stops at a designated position in front of the camera. The container is turned 360 degrees and a fixed camera captures multiple, e.g., ten images of the rotating tube. In an embodiment, from each image, a region of interest (ROI) can be generated relative to the position code. Generally, for detecting the hematocrit levels, a custom image is generated by vertically stacking the ten images (FIG. 17). To determine the hematocrit, the tube's geometric information, e.g., shape, dimensions, etc., and liquid levels can be determined whether by a user input or by determining the container type as described above, so that for the geometric information, in one embodiment, the above-described systems and methods for determining container parameters can be used. For the total liquid level and the plasma/red blood cell separation liquid level, the above-described liquid level determination methods can be used. From the tube geometric information and the liquid levels, the volume of the liquid portion that contains the red blood cells (Volume_red) and the total volume (Volume_total) of the liquid in the tube can be calculated. The ratio of Volume_red and Volume_total is the hematocrit level of the liquid in the tube.

[0176] FIG. 17 and FIG. 18 show an example for determining the plasma portion in a test specimen. As already mentioned, multiple pictures of the container are captured, wherein the container is rotated so that the multiple pictures are taken from different perspectives. Each horizontal line in FIG. 17 and FIG. 18 represents a single picture as shown in FIGS. 16A and 16B. The container has been rotated ten times in front of the camera so that ten pictures have been capture in the example shown in FIG. 17 and FIG. 18 forming the single picture of FIG. 17 and FIG. 18 comprising a kind of ten horizontal stacked segments. It has to be noted that each image shows the container rotated by 90°.

[0177] The top and bottom edges of the plasma comprising layer is marked, which is indicated in FIG. 18 with a square in the middle of the picture. Plasma portion detection can be viewed as a specific case of the object detection problem which has been described above—bounding box regression and classification which describes the spatial location of an object. The bounding box is rectangular, which is determined by the xx and yy coordinates of the upper-left corner of the rectangle and the coordinates of the lower-right corner of the rectangle. The left and right boundary of the bounding box indicate and correspond to the upper and lower ends of the plasma layer due to the 90° rotated image. With respect to the present disclosure, only the width of the box is detected while the height is fixed and is equal to the height of the image.

[0178] For determining the type or class of the container during hematocrit determination, the above described methods for determining characteristics of a sample container in an automated testing system are also applicable.

[0179] More specifically, a convolutional neural network (CNN) is created that takes as input an image like that in FIG. 17 and detects if plasma is present in the image and if so, it generates a localized bounding box around the plasma portion. The cost function used for training the CNN is modified to ignore error computation for the Y axis coordinate and for the height of the bounding box. Thus, in the present disclosure, only the beginning of the bounding box (left side) and the width of the bounding box are considered, while the Y axis coordinate and the height of the bounding box predicted by the CNN are ignored.

[0180] For enhancing the CNN model robustness and generalization capabilities, a novel data augmentation technique is used, in which each original input image (such as the one in FIG. 17) is split into 10 full width and equally spaced vertical components and then the extracted components are shifted and restacked vertically together).

[0181] FIG. 19 shows the network output with a line below the image indicating the plasma layer. The output is the left and right boundary of the plasma layer corresponding to the upper and lower end of the plasma layer in the container.

[0182] The network predicts two coordinates for each cell, tx and tw. While in the original implementation, the network also predicts ty and th, used for computing the bounding box height and center, we do not need these predictions, as the height of the bounding box is fixed and spans the entire height of the image.

[0183] If the cell is offset from the top left corner of the image by (cx, cy) and the bounding box prior has width pw, then the predictions correspond to (FIG. 20):

bx=σ(tx)+cx

bw=p.sub.we.sup.tw

[0184] During training, anchors are used for predefining boxes. p.sub.w is the anchor width. For each location of the 3-level FPN, two anchors are distributed as follow: the lower level FPN with anchors (119,160) and (154,160), the mid-level FPN (40,160) and (57,160) and high-level FPN (78,160), (97,160), where the first coordinate is the anchor width and the second is the anchor height. For learning tx and tw, the sum of squared error loss is used in training.

[0185] The network predicts an objectness score for each bounding box using logistic regression. This should be 1 if the bounding box prior overlaps a ground truth object by more than any other bounding box prior. Lastly, the network predicts a class probability. In our case, we currently use only one class, which is plasma. Binary cross entropy loss is used for training the objectness and class probability.

[0186] During inference, bx and bw are determined for each location. Then, by objectness is thresholded, if the objectness is lower than a threshold (0.5), we remove the box. Finally, the box with the lower class confidence is removed if two bounding boxes of the same class having intersection over union larger than a threshold are present.

[0187] The advantages of a method according to the present disclosure relate in a first aspect to a reduction of human failure as the user does not have to learn and follow special rules to sort container into specific racks that might just differ by their barcodes and not in any physical properties.

[0188] In a further aspect, the advantages of a method according to the present disclosure relate to a reduction of rack types and a better usability as the system can determine the container type or class in a generalized way so that no classical discriminators like rack barcodes are required. Only container with significant difference in their outer dimensions will have to be sorted into specific racks that are able to accommodate them.

[0189] An important advantage of a method according to the present disclosure is, that the system can measure dimensions and to determine cap and false bottom in a generalized way by CNN. It is not required to retrain the network in order to introduce new container types or classes.

[0190] The foregoing description of the preferred embodiment of the invention has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed, and modifications and variations are possible in light of the above teachings or may be acquired from practice of the invention. The embodiment was chosen and described in order to explain the principles of the invention and its practical application to enable one skilled in the art to utilize the invention in various embodiments as are suited to the particular use contemplated. It is intended that the scope of the invention be defined by the claims appended hereto, and their equivalents. The entirety of each of the aforementioned documents is incorporated by reference herein.

REFERENCE NUMERALS

[0191] 2 front illumination system [0192] 3 back illumination system [0193] 5 camera [0194] 7 mirror [0195] 10 rack [0196] 12 container [0197] 14 rack insert [0198] 15 background light [0199] d distance between camera and rack [0200] 20 image sensor [0201] 25 imaging objective lens [0202] 30 TIR element [0203] 40 LED stripes [0204] 42 doped Perspex [0205] 44 reflective film [0206] 45 diffusing film