DETERMINING INTERACTIONS BETWEEN CELLS BASED ON FORCE SPECTROSCOPY

Abstract

Methods and systems for determining interaction between cells are described wherein the method includes determining or receiving a sequence of images representing manipulating first cells, in a holding space, the holding space including a functionalized wall comprising second cells, the manipulating including settling of the first cells onto the functionalized wall and applying a force on the settled first cells; detecting groups of pixels representing first cells in first images representing the settling of the first cells onto the functionalized wall; tracking locations of detected first cells in the first images; and, determining settling events, a settling event being determined if a cell in a first image is not distinguishable from background of the first image, the location in the image at which a cell settling event is detected defining a cell settling location; detecting groups of pixels representing cells in second images captured during the application of the force and tracking locations of detected cells, wherein tracked locations of a detected cell in the second images form a tracking path, the first location of the tracking path defining a pop-up event, the location in a second image at which a pop-up event is detected defining a pop-up location; and, determining detachment events based on the settling locations and based on the pop-up locations, a detachment event defining a first cell being detached from a second cell due to application of the force on the first cell, and determining information about the interaction between first and second cells based on the force applied to the first cells.

Claims

1. A method for determining interaction between cells comprising: determining or receiving a sequence of images representing manipulating first cells, in a holding space, the holding space including a functionalized wall comprising second cells, the manipulating including settling of the first cells onto the functionalized wall and applying a force on the settled first cells; detecting groups of pixels representing the first cells in first images representing the settling of the first cells onto the functionalized wall; and, tracking locations of the detected first cells in the first images; and, determining settling events, a settling event being determined if a cell is no longer distinguishable from the background, the location in the first image at which a cell settling event is detected defining a cell settling location; detecting groups of pixels representing first cells in second images captured during the application of the force and tracking locations of the detected first cells, wherein tracked locations of the detected first cells in the second images form a tracking path, the first location of the tracking path defining a pop-up event, the location in a second image at which a pop-up event is detected defining a pop-up location; and, determining detachment events based on the settling locations and based on the pop-up locations, a detachment event defining a first cell being detached from a second cell due to application of the force on the first cell, and determining information about the interaction between the first and the second cells based on the force applied to the first cells.

2. The method according to claim 1 wherein the determining of the detachment events includes: determining a distance between one said settling location and one said pop-up location and determining one said detachment event based on a threshold value.

3. The method according to claim 1, wherein classification into a trackable or non-trackable cell is based on at least one of: intensity values of pixels of the group of pixels representing a cell; a shape, texture and/or dimensions of the group of pixels representing a cell; and/or, a contrast ratio between pixel values of group of the pixels representing a cell and pixel values representing the background of an image in which the group of pixels is tracked.

4. The method according to claim 1, wherein the determining if a cell is no longer distinguishable from the background is based on at least one of: one or more changes in pixel values of a group of pixels representing a cell, a shape, texture and/or, a contrast ratio between pixel values of the group of pixels representing a cell and pixel values representing the background of an image in which the group of pixels is tracked.

5. The method according to claim 1, wherein the determining of the images includes: determining or receiving one or more background images of the functionalized wall comprising the second cells; and removing the background from the images representing the manipulation of the first cells by using the one or more background images.

6. The method according to claim 1, wherein the determining of die detachment events includes: determining or receiving locations of one or more non-functional areas in the images of the functionalized wall surface, a non-functional area defining an area of the functionalized wall surface from which the second cells are absent; and disregarding one said pop-up location in the determining of detachment events if the pop-up location is located in or within a predetermined distance of the one or more non-functional areas.

7. The method according to claim 1, wherein the determining of the detachment events includes: determining or receiving one or more cluster locations in the images captured during the application of the force, a cluster defining an aggregation of cells which are not bound to the functionalized cell surface in the images when the force is applied to the first cells; and disregarding one said pop-up location in the determining of detachment events if the pop-up location is detected within one of the one or more cluster locations.

8. The method according to claim 1, wherein the determining of the detachment events includes: disregarding one said pop-up location in the determining of detachment events if the pop-up location is located within a predetermined distance of the edges of the images.

9. The method according to claim 1, wherein tracking the location of each detected cell includes: linking positions of detected cells in subsequent images using a minimization technique.

10. The method according to claim 1, wherein the method further includes: determining an avidity curve based on the detachment events and the force associated with each of the detachment events

11. The method according to claim 1, wherein the first cells are effector cells and the second cells are target cells.

12. The method according to claim 1, wherein first cells include at least one of lymphocytes, monocytic cells, granulocytes, T cells, natural killer cells, B-Cells, CAR-T cells, dendritic cells, Jurkat cells, bacterial cells, red blood cells, macrophages, TCR Tg T-cells, OT-I/OT-II cells, splenocytes, thymocytes, BM derived hematopoietic stem cells, TILs, tissue derived macrophages, and innate lymphoid cells; and/or, the second cells include at least one of: tumor cells, stem cells, epithelial cells, B16 melanoma, fibroblasts, endothelial cells, HEK293, HeLa, 3T3, MEFs, HuVECs, microglia, and neuronal cells.

13. A module for analyzing images of cells being manipulated in a holding space, the module comprising a non-transitory computer readable storage medium having computer readable program code embodied therewith, and a processor coupled to the computer readable storage medium, wherein responsive to executing the computer readable program code, the processor is configured to perform executable operations comprising: determining or receiving a sequence of images representing manipulating first cells, in the holding space, the holding space including a functionalized wall comprising second cells, the manipulating including settling of the first cells onto the functionalized wall and applying a force on the settled first cells; detecting groups of pixels representing the first cells in first images representing the settling of the first cells onto the functionalized wall; and, tracking locations of detected first cells in the first images; and, determining settling events, a settling event being determined if a cell is no longer distinguishable from the background, the location in the first image at which one said cell settling event is detected defining a cell settling location; detecting groups of pixels representing the cells in second images captured during the application of the force and tracking locations of detected cells, wherein tracked locations of detected cells in the second images form a tracking path, the first location of the tracking path defining a pop-up event, the location in a second image at which a pop-up event is detected defining a pop-up location; and, determining detachment events based on the settling locations and based on the pop-up locations, a detachment event defining one said first cell being detached from one said second cell due to application of the force on the first cell, and determining information about the interaction between the first and the second cells based on the force applied to the first cells.

14. A system for determining interaction between cells comprising: a sample holder comprising a holding space for cells; a force generator for applying a force to the cells; an imaging system capturing images of the cells in the holding space; a controller module for controlling the force generator and the imaging system; a non-transitory computer readable storage medium having computer readable program code embodied therewith, and a processor coupled to the computer readable storage medium, wherein responsive to executing the computer readable program code, the processor is configured to perform executable operations comprising: determining or receiving a sequence of images representing manipulating first cells, in the holding space, the holding space including a functionalized wall comprising second cells, the manipulating including settling of the first cells onto the functionalized wall and applying a force on the settled first cells; detecting groups of pixels representing the first cells in first images representing the settling of the first cells onto the functionalized wall; tracking locations of detected first cells in the first images; and, determining settling events, a settling event being determined if one said cell is no longer distinguishable from background of the first image, the location in the image at which one said cell settling event is detected defining a cell settling location; detecting groups of pixels representing the cells in second images captured during the application of the force and tracking locations of detected cells, wherein the tracked locations of detected cells in the second images form a tracking path, the first location of the tracking path defining a pop-up event, the location in a second image at which the pop-up event is detected defining a pop-up location; and, determining detachment events based on the settling locations and based on the pop-up locations, a detachment event defining one said first cell being detached from one said second cell due to application of the force on the first cell, and determining information about the interaction between first and second cells based on the force applied to the first cells.

15. A computer program or suite of computer programs comprising at least one software code portion or a computer program product storing at least one software code portion, the software code portion, when run on a computer system, being configured for executing the method steps according to claim 1.

16. The method of claim 2, wherein the detachment event is determined if the distance between the settling location and the pop-up location is smaller than the threshold value.

17. The method of claim 4, wherein the changes in the pixel values include changes of intensity values.

18. The method according to claim 1, wherein the determining of the detachment events includes: determining or receiving one or more cluster locations in the images captured during the application of the force, a cluster defining an aggregation of cells which are not bound to the functionalized cell surface in the images when the force is applied to the first cells; and disregarding one said pop-up location in the determining of detachment events if the pop-up location is detected within a predetermined distance of one of the one or more cluster locations.

19. The method according to claim 1, wherein the second cells are effector cells and the first cells are target cells.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0054] FIG. 1 depicts a schematic of a force spectroscopy system according to an embodiment of the invention;

[0055] FIGS. 2A and 2B schematically depict a flow cell for a force spectroscopy system according to an embodiment of the invention;

[0056] FIG. 3 depicts a schematic of a typical experiment in a force spectroscopy system according to an embodiment of the invention;

[0057] FIG. 4 depicts two cell avidity curves determined using a force spectroscopy system according to an embodiment of the invention;

[0058] FIG. 5A-5E depicts a schematic of a cell selection, tracking and classification process according to an embodiment of the invention;

[0059] FIG. 6 depicts a video image of cells which is analyzed using an image processing algorithm according to an embodiment of the invention;

[0060] FIG. 7 depicts a video image of cells which is analyzed using an image processing algorithm according to an embodiment of the invention:

[0061] FIG. 8 depicts a flow diagram of a method for determining interaction between cells according to an embodiment of the invention;

[0062] FIG. 9 depicts a background correction method according to an embodiment of the invention;

[0063] FIG. 10 depicts a cell detection method according to an embodiment of the invention;

[0064] FIG. 11 depicts a method for detecting and tracking cells in images according to an embodiment of the invention;

[0065] FIG. 12 depicts a schematic of a force spectroscopy system according to another embodiment of the invention;

[0066] FIG. 13 is a block diagram illustrating an exemplary data processing system that may be used for executing methods and software products described in this application.

DETAILED DESCRIPTION

[0067] FIG. 1 schematically depicts a force spectroscopy system according to an embodiment of the invention. The force spectroscopy system 100 may comprise a sample holder 102 comprising a holding space 104 for holding a sample 106 for holding cellular bodies in a fluid medium, e.g. a liquid or a gel. The holding space may be part of a flow cell (also referred to as a microfluidic cell). The system may further comprise a force field generator 108, e.g. an acoustic wave generator based on a piezo element, connected to the sample holder 102 for generating a bulk acoustic wave in the holding space so that a force is exerted on cellular bodies that may be present in the holding space. The force field generator may be connected to a controller 110 so that the force exerted on the cellular bodies can be controlled.

[0068] The system of FIG. 1 may further comprise an imaging system for imaging the processes in the holding space. The imaging system may include a microscope 112 including optics, e.g. adjustable objective 114, and a camera 116 for capturing pictures, e.g. video frames, of the processes in the holding space. The imaging system may be connected to a computer 118 comprising a processor connected a memory comprising one or more software programs, which when executed, allow control of the different elements of the system.

[0069] The system may further comprise a light source 120 for illuminating the sample using any suitable optics (not shown) to provide a desired illumination intensity and intensity pattern, e.g. plane wave illumination, Köhler illumination, etc., known per se. Here, the light 122 emitted from the light source may be directed through the force field generator 108 to (the sample in) the sample holder 102 and sample light 124 from the sample is transmitted through the objective 114 and through an optional tube lens 126 and/or further optics (not shown) to the camera 116. The objective and the camera may be integrated. In an embodiment, two or more optical detection tools, e.g. with different magnifications, may be used simultaneously for detection of sample light, e.g. using a beam splitter.

[0070] In another embodiment, not shown but discussed in detail in WO2014/200341, the system may comprise a partially reflective reflector and light emitted from the light source is directed via the reflector through the objective and through the sample, and light from the sample is reflected back into the objective, passing through the partially reflective reflector and directed into a camera via optional intervening optics. Further embodiments may be apparent to the reader.

[0071] The sample light may comprise light affected by the sample (e.g. scattered and/or absorbed) and/or light emitted by one or more portions of the sample itself e.g. by chromophores/fluorophores attached to the cellular bodies.

[0072] Some optical elements in the system may be at least one of partly reflective, dichroic (having a wavelength specific reflectivity, e.g. having a high reflectivity for one wavelength and high transmissivity for another wavelength), polarisation selective and otherwise suitable for the shown setup. Further optical elements e.g. lenses, prisms, polarizers, diaphragms, reflectors etc. may be provided, e.g. to configure the system 100 for specific types of microscopy.

[0073] The sample holder 102 may be formed by a single piece of material with a channel inside, e.g. glass, injection moulded polymer, etc. (not shown) or by fixing different layers of suitable materials together more or less permanently, e.g. by welding, glass bond, gluing, taping, clamping, etc., such that a holding space 106 is formed in which the fluid sample is contained, at least during the duration of an experiment. While, the force spectrometry system of FIG. 1 includes an acoustic force generator, other ways of applying a force on the cells may be used as well. For example, a force spectrometry system that uses a centrifugal force generator for applying a force on the cells is described in more detail with reference to FIG. 12.

[0074] FIGS. 2A and 2B schematically depict cross-sectional views of a flow cell for a force spectroscopy system according to an embodiment of the invention. The sample holder 212 may comprise a first base part 206.sub.1 that has a recess being, at least locally, U-shaped in cross section and a cover part 206.sub.2 to cover and close (the recess in) the U-shaped part providing an enclosed holding space in cross section.

[0075] Further, the sample holder 212 may be connected to a fluid flow system 214 for introducing fluid and unbound cells into the holding space of the sample holder and/or removing fluid from the holding space, e.g. for flowing fluid through the holding space (see arrows in FIG. 2A depicting the flow direction). The fluid flow system may be comprised in or part of a manipulation and/or control system including one or more of reservoirs 216, pumps, valves, and conduits 218,219 for introducing and/or removing one or more fluids, sequentially and/or simultaneously. The sample holder and the fluid flow system may include connectors, which may be arranged on any suitable location on the sample holder, for coupling/decoupling. The sample holder may further include a force field generator 222, e.g. an acoustic wave generator which may be implemented based on a (at least partially transparent) piezoelectric element connected to a controller 224.

[0076] FIG. 2B schematically depicts a cross-section of part of the sample holder including objective 232 that is positioned underneath part of a sample holder (a chip) wherein the sample holder may comprise a capping layer 212.sub.1, a matching layer 212.sub.2, a fluid medium 230 contained in the holding space formed by the capping and the matching layer, and part of a force generator 222, e.g. a piezo element. An immersion liquid 234 between the objective and the capping layer may be used to improve the optical numerical aperture (NA) of the imaging system but the data shown in FIGS. 6 and 7 was obtained using an embodiment without immersion liquid. Application of an AC voltage to the piezo element at an appropriate frequency will generate a resonant bulk acoustic standing wave 236 in the sample holder. The standing wave may have a nodal plane 215 in the fluid layer at a certain height above the functionalized wall comprising target cells 226 attached to the wall of the sample holder and effector cells 228 that are bound to the target cells. The standing wave may also have lateral nodes 220.sub.1,2. Cells that have a positive acoustic contrast factor with respect to the fluid medium will experience a force towards to the nodes.

[0077] One or more software programs that run on the computer 118 of the force spectroscopy system may be configured to control the camera, the force field generator and the flow cell to conduct different experiments. In a typical experiment, cells, e.g. effector cells, may be flushed into the holding space of the flow cell and may interact, e.g. bind, with the target cells. This interaction can be probed by analysing the response of cells that are bound to target cells as a function of the force applied. Typically, the response of the cells is determined by analysing video frames that are captured by the camera. To that end, the computer may include an image processing module 128 comprising one or more image processing algorithms for analysing the response of the cells when they are manipulated in the flow cell using the force field generator. The image analysis of the video frames is described hereunder in greater detail.

[0078] FIG. 3A-3D depict schematics of manipulating cells in a holding space of a microfluidic cell comprising a functionalised wall surface to which target cells are attached. The microfluidic cell may be part of a force spectroscopy system as described with reference to FIGS. 1 and 2. The manipulation of the cells may be imaged from below or from the top using an imaging system as described with reference to FIG. 1. As depicted in FIG. 3A, the process may start with flushing cells 306, e.g. effector cells, into the holding space of the microfluidic chip, comprising a functionalized wall 302 including target cells 304. The introduction of the cells into the holding space may take a predetermined period of time, e.g. between 1 and 5 seconds. After flushing, the cells are allowed to settle onto the functionalized wall comprising the target cells (FIG. 3B). When the effector cells reach the functionalized wall, the cells may move around over the functionalized surface until they find a suitable target cell to bind to (surveillance) thus forming a bound effector—target cell pair 310 (FIG. 3C). The steps of effector cells settling onto the functionalized wall and binding to target cells may be referred to as the incubation phase. In a typical experiment, incubation may take up to 1-15 minutes or longer.

[0079] As will be described hereunder in more detail, the incubation phase may be imaged and when the cells are introduced into the holding space and move towards the functionalized wall, groups of pixels in the captured images may be detected and tracked. However, as effector cells approach the functionalized wall, move other the wall surface and bind to target cells, the contrast between pixels representing effector cells and pixels representing the functionalized wall including the target cells may become very low so that if the contrast drops below a certain level cells can no longer be reliably detected and tracked.

[0080] After the incubation phase, a force may be applied to the effector cells that are bound to the target cells. The force may have a direction away from the functionalized wall surface. Typically, a force ramp will be applied to the effector cells, so that if the force becomes larger than the binding force, effector cells will detach from the target cells and move away in the direction of the force (FIG. 3D). As the target cells are bound firmly to the functionalized wall, the force will “pull” the effector cells from the target cells and move the effector cells away from the functionalized wall surface. Pulling or pushing the effector cells from the target cells will be visible in the images as groups of pixels representing cells that “pop up” in the foreground against a background image of the functionalized wall. Such an event may be referred to as a pop-up event.

[0081] When the force is larger than the binding force, the effector cell will detach from the target cell and move in a direction that depends on the applied force, which may have a component perpendicular to the functionalized wall (e.g. the z-direction) and two components in the plane of the functionalized wall (e.g. the x and y direction). The location in the image in which a pop-up event is detected (i.e. the contrast between groups of pixels representing cell and pixels representing the background is above a certain level) and the point in time at which the pop-up event occurred can be determined on the basis of the images (video frames) which are captured during the experiment. The time at which cells detach may determine the force that is exerted on the effector cells. In a typical experiment, the force ramp may take between 2-10 minutes, but it can also be shorter or longer.

[0082] Based on a measurement scheme as described with reference to FIG. 3, various parameters of the effector cells can be determined. For example, FIG. 4 depicts two cell avidity curves which may be determined by applying a force ramp to the functionalized wall surface and determining the number of attached cells as a function of the applied force. Based on the type of effector and target cells, cells may exhibit a low cell avidity curve 402 (weak binding forces between effector and target cells) or a high cell avidity curve 404 (strong binding forces between effector and target cells).

[0083] While FIGS. 2A,2B and 3 describe experiments wherein unbound effector cells are introduced in the holding space and target cells are attached to the functionalized wall other configuration are also possible. For example, in certain embodiments, unbound target cells may be introduced into the holding space, while effector cells are attached to the functionalized wall.

[0084] FIG. 5A-5E depict schematics of images of cells in a microfluidic cell which are manipulated using a force spectroscopy system. These images are processed based on an image processing algorithm according to an embodiment of the invention. As already described with reference to FIG. 1, the force spectroscopy system comprises an imaging system, e.g. a video camera, for capturing images of the process in the microfluidic cell. Depending on the system configuration, the images may represent a top view or bottom view of the functionalized wall comprising the target cells. During a typical force spectroscopy experiment, many video frames of processes in the flow cell will be captured. The images of FIG. 5A-5E are schematics of representative images of the flow cell during the different phases of a force spectroscopy experiment as e.g. described with reference to FIG. 3. The images of a force spectroscopy experiment may be associated with a time of a clock, e.g. time-stamped, so that information derivable from the images can be determined as a function of time and the force corresponding to the time-stamps may be correlated with the images.

[0085] FIG. 5A depicts an image 502 representing a top view of the microfluidic cell including a functionalized wall 503 comprising target cells 504. This image may represent an exemplary situation at a start of a force spectroscopy experiment, wherein unbound, unlabelled cells are flushed into the microfluidic cell. The cells are visible in the image against a background representing the functionalized wall comprising the target cells. Known image processing algorithms may be used to detect and locate effector cells in each image and to track the locations of the detected cells in subsequent images. Such image processing algorithm may for example detect groups of pixels of a certain intensity and use classification rules that include e.g. size, geometry, contrast and changes in size, geometry and contrast, to determine if such group of pixels that is detected in subsequent images may be classified as a trackable cell.

[0086] A dotted circle 505 in the image may indicate that the algorithm has detected a group of pixels 506 within the circular area which is classified as a trackable cell. The cells that are visible in the images are depicted using a white color. This detection and classification process may be applied to the captured images. The location of detected groups of pixels that are classified as trackable, unbound cells (which may move both parallel and perpendicular to the functionalized wall) may be determined so that the movement of the detected cell as a function of time can determined. The locations of a cell that is detected and tracked may form a so-called tracking path (not shown). Each cell that is detected and tracked in subsequent images may be linked with a unique identifier so that the locations of a tracked cell (the tracking path) and other information can be stored on a storage medium of the computer that executes the image processing algorithm.

[0087] During the incubation phase, effector cells that are introduced into the microfluidic cell will gradually descent by e.g. gravitational forces towards the functionalized wall. Further, the cells may move over the surface of the functionalized wall until they encounter suitable target cells to bind to. When cells are descending towards the functionalized wall, the change in contrast, shape and/or dimensions of the group of pixels representing a cell may cause the image processing algorithm to classify the group of pixels as no longer trackable. Thus, during the incubation phase, the classification of a group of pixels that is classified as a trackable cell may change into non-trackable (i.e. a cell in a first image is no longer distinguishable from background of the first image) when it moves towards the wall surface and binds to a target cell. Detected and tracked groups of pixels that are classified as cells may “disappear” into the background of pixels representing the functionalized wall surface. In FIG. 5B non-trackable cells 508.sub.1-3 that are settled onto the wall surface have a grey color.

[0088] During the tracking of detected cells in subsequent images, the image processing algorithm may determine a so-called cell settling event in an image if the classification of a group of pixels is changed from a trackable cell into a non-trackable cell. The settling event may occur at a location in the image as schematically depicted in FIG. 5B by dotted squares 510.sub.1-3. The location in the image at which a cell settling event is determined defines a cell settling location.

[0089] A cell settling event may further be associated with a time instance indicating at which time (or in which image of the video) the cell settling event was determined. Such time instance may for example be determined based on a clock or a time stamp of the image in which the event was detected. As shown in FIG. 5B, the location of a cell settling event 510.sub.1-3 may deviate from the actual location at which the cells 508.sub.1-3 bind to target cells. These actual locations cannot be determined by the algorithm as effector cells are no longer trackable if they are very close to the functionalized wall (the bound cells 508.sub.1-3 are gray colored and are therefore not visible in the image or at least will very difficult to detect in the image). The process of incubation may continue until for each (or most of) the cells a cell settling event 510.sub.1-9 is determined so that that (most of) the cells may be bound to target cells or the functionalized wall as depicted in FIG. 5C. In that case, the contrast between groups of pixels representing cells and pixels representing the background (the functionalized wall) may be below a certain level so that the algorithm may classify the cells as no longer trackable. The settling locations of the settled cells may be stored in a memory of a computer that executes the image processing algorithm.

[0090] Thereafter, a force generator of the force spectroscopy system, e.g. an acoustic force generator as explained with reference to FIGS. 1 and 2, may be switched on and images may be captured when a force is applied onto the cells that are bound to the target cells. FIG. 5D depicts an image illustrating this situation. The direction of the force generated by the force generator is pointing away from the functionalized wall surface. Hence, when a certain force is applied, cells may detach from the target cells and move away from the functionalized wall. Typically, the force may be increased as a function of time.

[0091] As shown in FIG. 5D, due to the exerted force, certain cells may detach which may become visible in the image as groups of pixels which the algorithm detects as cells 512.sub.1-3. Thus, groups of pixels classified as cells may “pop-up” out of the background formed by pixels representing the functionalized wall. Cells appearing in the image during the force ramp may be detected by the image processing algorithm in a similar way as described with reference to FIG. 5A. Groups of pixels that are detected by the image processing algorithm as cells are depicted by the dotted circles 512.sub.1-3 in FIG. 5D.

[0092] Further, the location at which a cell is detected (for the first time) as a cell may define a cell pop-up location. The location of detected cells in subsequent images may be tracked, wherein the tracked cells may not only move away from the functionalized wall but also sideward within the plane of the functionalized wall. For example, as illustrated in FIG. 2B, a bulk acoustic standing wave in the microfluidic cell may create an axial nodal plane substantially parallel to the functionalized wall. The axial nodal plane may be positioned at a certain height above the functionalized wall surface. Also, lateral acoustic nodes may develop in the holding space, which will cause lateral perturbations of the axial nodal plane. So, when cells detach from the functionalized wall, the unbound cells may move towards the nodal plane which is substantially parallel to the functionalized wall surface. Simultaneously they may move also laterally to settle in the superposition of the axial and lateral acoustic nodes.

[0093] FIG. 5E depicts an image at a time instance when the force has increased over a period of time. As shown in this figure, groups of pixels that were classified as trackable cells in FIG. 5D moved towards a node of the force field. These tracked cells and other detected cells may form a cluster of cells 516 that are kept in position at the node. As shown in the figure, some unbound, detected cells, for example cell 518, move towards the cluster. Based on the locations of tracked cells in subsequent images, the image processing algorithm may determine tracking paths 522.sub.1-9 for each detected cell. In FIG. 5E tracking paths are represented by dotted lines. A tracking path represents the path along which a group of pixels which is classified as a trackable cell has moved to the cluster area. The image processing algorithm may determine the start of a tracking path as a pop-up event, i.e. a position in an image in which—during the force exertion—a group of pixels was detected for the first time. The position of the start of a tracking path may be referred to as a pop-up location. Pop-up events in the image of FIG. 5E are marked by the grey color squares 520.sub.1-7.

[0094] The tracking paths, the pop-up events and settling events may be used by the image processing algorithm to distinguish detachment events, i.e. pop-up events associated with detachment of a cell that was bound to a target cell, from other events, i.e. pop-up events that were recognized by the image processing algorithm but relate to other processes. As shown in FIG. 5E such “false positives” may include for example target cells 528.sub.1,2 that were detached during the force ramp up and detected and tracked as effector cells associated with a pop-up location 520.sub.3,5. The image processing algorithm may also classify a group of pixels representing such target cell as a trackable cell. Such false positives may be especially relevant in case the distinction (in terms of shape, dimensions, color, etc.) between effector cells and target cells is small.

[0095] Further, detached cells or debris that enter the field of view of the imaging system may be detected, classified as trackable cells and tracked until the cells accumulate at the cluster area. For example, tracking paths 522.sub.4,5,9 relate to detected cells or detected debris that is recognized by the algorithm as a cell. Such cells may come from an area outside the field of view and enter the field of view when the acoustic force attracts these cells to move towards the node. Events such as detached target cells or cells or debris that enter the field of view may be filtered out (disregarded) by using a distance correlation between the settling and pop-up events. This correlation is based on the observation that settled cells may move a certain limited distance before it finds a suitable target cell to bind to. Thus, the location of a settling event and a pop-up event of a detached cell (a detachment event) should be within a certain distance. When using the distance correlation to filter out the relevant pop-up events, the image processing algorithm may determine five detachment events 524.sub.1-5 in FIG. 5E.

[0096] FIG. 6 depicts a video image taken by a camera of a force spectroscopy system which is analyzed using an image processing algorithm according to an embodiment of the invention. As shown in this video image 600, events and tracking paths are identified in the video image. The tracking paths are determined by tracking detected cells in multiple images. The paths show that cells are accumulated in positions of acoustic nodes that exists when the force generator of the AFS is turned on. FIG. 7 is an expanded picture of the area 602 of the video image of FIG. 6. FIG. 7 depicts a background of the functionalized wall including target cells 710, settling events 702.sub.1,2, pop-up events 704.sub.1-4 and tracking paths 708.sub.1-4 of detected cells during the force ramp that were accumulated in a cluster of cells 712 at an acoustic node. Correlating the settling events with the pop-up events allows identification of pop-up events associated with cells that were bound to a target cell. For example, a rule may be used that a pop-up event should be located within a predetermined distance from a settling event (but a more general classification for detachment events using other or more information than the location of settling and pop-up events may also be used). This distance may for example be expressed in a number of pixels or another suitable unit. Hence, pop-up events 704.sub.1,2 were marked as detachment events 706.sub.1,2 related to the detachment of an effector cell that was bound to a target cell. The parameters of these events, e.g. the force that was applied at the moment a detachment event was imaged, may be used for determining the interaction, e.g. the cell avidity or the like.

[0097] FIG. 8 depicts a flow diagram of a method for determining interaction between cells, e.g. the cell avidity, according to an embodiment of the invention.

[0098] The method may start with a step 802 determining or receiving a sequence of images of manipulating first cells, e.g. effector cells, in a holding space, wherein the holding space may include a functionalized wall comprising second cells, e.g. target cells. Here, the target cells may be connected to the wall of the flow cell so that they will not detach when a force is applied to the functionalized wall surface. The manipulating of the cell may include settling of the first cells onto the functionalized wall. The settling of the cells allows the cells to move around over the wall in order to find a suitable target cell it can bind to. The process of settling and binding may be referred to the incubation phase. Thereafter, a force may be applied to the settled first cells.

[0099] In step 804, groups of pixels may be detected in in first images representing the settling of the first cells onto the functionalized wall. These detected groups of pixels may represent first cells. Further, tracking locations of detected first cells in the first images may be determined, wherein during tracking a cell may be classified as no longer trackable. In that case, a cell that is tracked in consecutive images becomes no longer distinguishable from background of the first image, because the cell is close to the functionalized wall surface. Such event may be referred to as a settling event. The location in the image at which a cell settling event is detected defines a cell settling location.

[0100] In a further step 806, groups of pixels may be detected in second images captured during the application of the force. These detected groups of pixels may represent first cells and tracking locations of detected first cells, wherein tracked locations of a detected first cell in the second images may form a tracking path, the first location of the tracking path defining a pop-up event, the location in a second image at which a pop-up event is detected defining a pop-up location.

[0101] Thereafter, detachment events may be determined (step 808), wherein detachment events are related to first cells being detached from the second cells due to application of the force on the first cells. These events may be determined based on the settling locations and the pop-up locations. The detachment events are then used to determine information about the interaction between first and second cells. In particular, the information about the interaction between the cells may be based on the force that was applied to the first cells when the detachment events occurred.

[0102] In an embodiment, a background correction method may be applied to the captured video images before the images are processed for cell detection and tracking. In such method, the background of the captured images (video frames), i.e. the functionalized wall comprising the target cells, may be removed based one or more captured background images. This way, the foreground information, i.e. the unbound cells, may be more clearly visible so that subsequent image processing, e.g. detection and tracking can be improved. Thus, the background correction method may be used as a pre-processing step for improving the accuracy of the detection and tracking algorithm. This background subtraction may be based on a pre-acquired image or series of images (before flush in of the first cells) and may be static (unchanging). It may however also involve a background model which is dynamically updated based on images taken during the experiment and may involve an advanced background model describing e.g. the dynamic behavior of the target cells.

[0103] FIG. 9 depicts a background correction method according to an embodiment of the invention. In order to accurately remove the background information in the captured images, the background patterns of the background image should be accurately aligned with the background pattern of the captured images. To that end, a first region of interest (ROI) may be cropped out at a predetermined position in a captured image and at the same position in a background image (e.g. an image of the functionalized wall surface comprising target cells but no other (effector) cells) a second ROI may be cropped out (steps 902,904). A pattern matching algorithm, preferably a geometric pattern matching algorithm, or another suitable image registration method may be used to align the patterns in the first and second ROIs. The alignment information may include information about rotation and/or translation parameters that allow accurate matching of the patterns of the first and second ROI (step 906). The determined alignment information is subsequently applied to the background image (step 908) so that the aligned background image can be used to remove the background of the current image (step 910) or at least a substantial part thereof.

[0104] Instead of using a static background picture, a rolling median background correction algorithm may be used or any other suitable algorithm for dynamically correcting the background. In case if a rolling median background correction, a background image may be formed by taking the median intensity value over a predetermine number of images, e.g. 10 images, for each pixel separately or for groups of pixels. This background image may then used for removing the background of the current image (or at least a substantial part thereof).

[0105] FIG. 10 depicts a cell detection method according to an embodiment of the invention. As shown in this figure, the method may start with background correction method (step 1002) that allows removal of the background, i.e. the functionalized wall including the target cells, from the captured images so that foreground objects can better distinguished from the background. Exemplary background correction methods that may be used as described with reference to FIG. 9. One or more filters, e.g. a Laplacian or Gaussian filter, may be applied to the background corrected images (step 1004) for blurring high frequency information in the image. The intensity of the pixels may be cropped to improve the black and white saturation (step 1006). This step may be applied to ensure that the blobs in the processed image have sufficient contrast with respect to the background so that the blobs can be detected using a segmentation method (step 1008). In an embodiment, a known thresholding operation, preferably a local thresholding operation, may be used to create a binary image, wherein blobs are formed by pixels having a first value and the background is formed by pixels having a second value. Blobs that contain “holes” due to irregularities or noise may be processed so that these holes are removed by filling each of these holes with pixel values (step 1010). Finally, the location of the blobs in the image may be determined (step 1012). Optionally, the blobs may be classified based on proximity, size, intensity, area, symmetry, etc.

[0106] FIG. 11 depicts a method for detecting and tracking cells in images according to an embodiment of the invention. In particular, this figure provides a flow diagram describing a method of detecting and tracking cells during a force ramp up. This process may start with cell detection and localization (step 1102) using a method as described with reference to FIG. 10. Further, locations of detected cells in subsequent images may be linked using a minimization algorithm, preferably a global minimization algorithm (step 1106). For example, to efficiently link large numbers of detected cells in subsequent images a minimization of a cost function may be used as described in the article by Jaquaman et al, Robust single-particle tracking in live-cell time-lapse sequences, Nature Methods, Vol. 8, no. 8, August 2008, pp. 295-702, which is hereby incorporated by reference in this application. The linking of the cell locations will generate tracking paths as described with reference to FIG. 5-8, which are used to determine pop-up events and which can be used to classify cells that are detected in the images into detached cells and other cells (step 1206). Further information derived from the images, e.g. non-functional areas in the functionalized wall, cluster locations and edges, may be used to further filter out pop-up events are not related to cells being detached from target cells due to the application of the force (step 1108).

[0107] FIG. 12 schematically depicts a further force spectroscopy system configured to capture images of a force spectroscopy experiment. This system includes a rotary arm 1202 that is rotatable mounted onto a rotor system that includes a rotor 1214 and a rotary joint 1216. A holding space 1204, e.g. a microfluidic cell, comprising cells (e.g. effector cells and target cells attached to a functionalized wall of the holding space) may be mounted onto the rotary arm. Further, a light source 1206, an adjustable objective 1208, a tube lens 1210 and a camera 1212 may form an imaging system for capturing images of effector cells interacting with target cells. A rotor controller 1218 may be used to control the rotor system to spin the rotary arm in a circular motion so that a centrifugal force is exerted onto the effector cells. The system may be connected to a computer 1220 that includes a processing module 1222 that is configured to control the rotor controller and the imaging system. Further, when in use, the computer may receive captured images 1224 of the holding space while a force ramp is exerted onto effector cells. Further, the processing module is configured to process the captures images in a similar way as described with reference to FIG. 3-11 above. Exemplary embodiments of such force spectroscopy systems are described in WO2017/147398 and U.S. Pat. No. 8,795,143 which hereby incorporated by reference into this application.

[0108] FIG. 13 is a block diagram illustrating exemplary data processing systems described in this disclosure. Data processing system 1300 may include at least one processor 1302 coupled to memory elements 1304 through a system bus 1306. As such, the data processing system may store program code within memory elements 1304. Further, processor 1302 may execute the program code accessed from memory elements 1304 via system bus 1306. In one aspect, data processing system may be implemented as a computer that is suitable for storing and/or executing program code. It should be appreciated, however, that data processing system 1300 may be implemented in the form of any system including a processor and memory that is capable of performing the functions described within this specification.

[0109] Memory elements 1304 may include one or more physical memory devices such as, for example, local memory 1308 and one or more bulk storage devices 1310. Local memory may refer to random access memory or other non-persistent memory device(s) generally used during actual execution of the program code. A bulk storage device may be implemented as a hard drive or other persistent data storage device. The processing system 1300 may also include one or more cache memories (not shown) that provide temporary storage of at least some program code in order to reduce the number of times program code must be retrieved from bulk storage device 1310 during execution.

[0110] Input/output (I/O) devices depicted as input device 1312 and output device 1314 optionally can be coupled to the data processing system. Examples of input device may include, but are not limited to, for example, a keyboard, a pointing device such as a mouse, or the like. Examples of output device may include, but are not limited to, for example, a monitor or display, speakers, or the like. Input device and/or output device may be coupled to data processing system either directly or through intervening I/O controllers. A network adapter 1316 may also be coupled to data processing system to enable it to become coupled to other systems, computer systems, remote network devices, and/or remote storage devices through intervening private or public networks. The network adapter may comprise a data receiver for receiving data that is transmitted by said systems, devices and/or networks to said data and a data transmitter for transmitting data to said systems, devices and/or networks. Modems, cable modems, and Ethernet cards are examples of different types of network adapter that may be used with data processing system 1300.

[0111] As pictured in FIG. 13, memory elements 1304 may store an application 1318. It should be appreciated that data processing system 1300 may further execute an operating system (not shown) that can facilitate execution of the application. Application, being implemented in the form of executable program code, can be executed by data processing system 1300, e.g., by processor 1302. Responsive to executing application, data processing system may be configured to perform one or more operations to be described herein in further detail.

[0112] In one aspect, for example, data processing system 1300 may represent a client data processing system. In that case, application 1318 may represent a client application that, when executed, configures data processing system 1300 to perform the various functions described herein with reference to a “client”. Examples of a client can include, but are not limited to, a personal computer, a portable computer, a mobile phone, or the like.

[0113] In another aspect, data processing system may represent a server. For example, data processing system may represent an (HTTP) server in which case application 1318, when executed, may configure data processing system to perform (HTTP) server operations. In another aspect, data processing system may represent a module, unit or function as referred to in this specification.

[0114] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

[0115] The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention. The embodiment was chosen and described in order to best explain the principles of the invention and the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.

[0116] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

[0117] The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention. The embodiment was chosen and described in order to best explain the principles of the invention and the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.