SYSTEM AND METHOD FOR MACHINE-LEARNING ENABLED MICRO-ASSEMBLY CONTROL WITH THE AID OF A DIGITAL COMPUTER

Abstract

System and method that allow utilize machine learning algorithms to move a micro-object to a desired position are described. A sensor such as a high speed camera or capacitive sensing, tracks the locations of the objects. A dynamic potential energy landscape for manipulating objects is generated by controlling each of the electrodes in an array of electrodes. One or more computing devices are used to: estimate an initial position of a micro-object using the sensor; generate a continuous representation of a dynamic model for movement of the micro-object due to electrode potentials generated by at least some of the electrodes and use automatic differentiation and Gauss quadrature rules on the dynamic model to derive optimum potentials to be generated by the electrodes to move the micro-object to the desired position; and map the calculated optimized electrode potentials to the array to activate the electrodes.

Claims

1. A system for machine-learning enabled micro-assembly control with the aid of a digital computer, comprising: one or more processors implementing one or more machine learning algorithms, the one or more processors configured to: obtain one or more parameters of a system for positioning a micro-object, the system comprising a plurality of electrodes, each of the electrodes are programmable, the electrodes configured to induce a movement of the micro-object when the micro-object is proximate to the electrodes upon a generation of one or more electric potentials by one or more of the electrodes; model capacitance between the micro-object and the electrodes using the parameters; estimate a position of the micro-object with a sensor; receive further position of the micro-object; define using the capacitance as a continuous representation of a dynamic model for movement of the micro-object due to electrode potentials generated by at least some of the electrodes, wherein the dynamic model is associated with constraints expressed as expectations; apply one of the machine learning algorithms to at least a portion of the continuous representation of the dynamic model to perform an optimization of electrode potentials to be generated by at least some of the electrodes to move the micro-object from the estimated position to the further position; and use the optimized electrode potentials to actuate at least some of the electrodes to induce the movement of the micro-object from the estimated position to the further position.

2. A system according to claim 1, wherein one of the constraints comprises a potential energy of the micro-object and the potential energy is approximated during the optimization using Gauss quadrature rules.

3. A system according to claim 2, the one or more processors further configured to: represent a function that determines an allocation of the optimized electrode potentials as a neural network associated with weights and biases.

4. A system according to claim 3, wherein the one machine learning algorithm is automatic differentiation, the one or more processors further configured to: use the automatic differentiation to compute partial derivatives of the potential energy as a function of the weights and biases associated with the neural network.

5. A system according to claim 3, wherein the neural network comprises two hidden layer and uses tanh as an activation function.

6. A system according to claim 1, wherein the further position is separated by a distance in a direction from the estimated position, the one or more processors further configured to: receive an additional position that is separated by the distance from the estimated position in a direction opposite to the direction in which the further position is separated from the estimated position; and change a sign of the function to optimize the potentials for moving the micro-object from the estimated position to the additional position.

7. A system according to claim 1, wherein the movement from the estimated position to the further position is in one dimension, the one or more processors further configured to: receive an additional position that is separated from the estimated position in two dimensions; and apply a rotational transformation to the optimized electrode potentials to derive the electrode potentials to be generated by the electrodes for moving the micro-object from the estimated position to the additional position.

8. A system according to claim 1, the one or more processors further configured to: perform a plurality of simulations of the capacitance between the electrodes and the micro-object; and define a function describing the capacitance between the micro-object and each of the electrodes as a function of a distance between the micro-object and that electrode.

9. A system according to claim 1, one or more of the processors comprise one or more of at least one CPUs and at least one GPUs.

10. A system according to claim 1, wherein at least some of the electrodes are rectangular.

11. A method for machine-learning enabled micro-assembly control with the aid of a digital computer, comprising: obtaining, by one or more processors implementing one or more machine learning algorithms, one or more parameters of a system for positioning a micro-object, the system comprising a plurality of programmable electrodes, the electrodes configured to induce a movement of the micro-objects when the micro-objects are proximate to the electrodes upon a generation of one or more electric potentials by one or more of the electrodes; modeling by one or more of the processors capacitance between the micro-object and the electrodes using the parameters; estimating by one or more of the processors a position of the micro-object based on at least one sensor measurement; receive by one or more of the processors further position of the micro-object; define by one or more of the processors using the capacitance a continuous representation of a dynamic model for movement of the micro-object due to electrode potentials generated by at least some of the electrodes, wherein the dynamic model is associated with constraints expressed as expectations; apply by one or more of the processors one of the machine learning algorithms to at least a portion of the continuous representation of the dynamic model to perform an optimization of electrode potentials to be generated by at least some of the electrodes to move the micro-object from the estimated position to the further position; using the optimized electrode potentials to actuate at least some of the electrodes to induce the movement of the micro-object from the estimated position to the further position.

12. A method according to claim 11, wherein one of the constraints comprises a potential energy of the micro-object and the potential energy is approximated during the optimization using Gauss quadrature rules.

13. A method according to claim 12, further comprising: representing a function that determines an allocation of the optimized electrode potentials as a neural network associated with weights and biases.

14. A method according to claim 13, wherein the one machine learning algorithm is automatic differentiation, further comprising: using the automatic differentiation to compute partial derivatives of the potential energy as a function of the weights and biases associated with the neural network.

15. A method according to claim 13, wherein the neural network comprises two hidden layers and uses tanh as an activation function.

16. A method according to claim 11, wherein the further position is separated by a distance in a direction from the estimated position, further comprising: receiving an additional position that is separated by the distance from the estimated position in a direction opposite to the direction in which the further position is separated from the estimated position; and changing a sign of the function to optimize the potentials for moving the micro-object from the micro-object from the estimated position to the additional position.

17. A method according to claim 11, wherein the movement from the estimated position to the further position is in one dimension, the one or more processors further configured to: receiving an additional position that is separated from the estimated position in two dimensions; and applying a rotational transformation to the optimized electrode potentials to derive the electrode potentials to be generated by the electrodes for moving the micro-object from the estimated position to the additional position.

18. A method according to claim 11, further comprising: performing a plurality of simulations of the capacitance between the electrodes and the micro-object; and defining a function describing the capacitance between the micro-object and each of the electrodes as a function of a distance between the micro-object and that electrode.

19. A method according to claim 11, one or more of the processors comprise one or more of at least one CPUs and at least one GPUs.

20. A method according to claim 11, wherein at least some of the electrodes are rectangular.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0018] FIG. 1 is a block diagram showing a system for machine-learning enabled micro-assembly control with the aid of a digital computer

[0019] FIG. 2 is a diagram illustrating interactions between the video projector and the photo-transistor array of FIG. 1 in accordance with one embodiment.

[0020] FIGS. 3A-3C show several views of an example of a photo-transistor array of FIG. 1 in accordance with one embodiment.

[0021] FIGS. 4(A)-4(B) are examples of micro-object patterns realized using the system of FIG. 1.

[0022] FIG. 5 is diagram illustrating capacitive-based interactions between one micro-object and five electrodes of the system of FIG. 1 in accordance with one embodiment.

[0023] FIG. 6 is a diagram showing the COMSOL model of two conductors (spherical micro-object and an electrode) for capacitance computation in accordance with one embodiment.

[0024] FIG. 7 shows the capacitance function fitted with training data generated by COMSOL simulations in accordance with one embodiment.

[0025] FIG. 8 is a diagram showing 2D capacitance function determined from the 1D model using the symmetry property in accordance with one embodiment.

[0026] FIG. 9 is a diagram showing a control policy obtained using by solving equation (2) in accordance with one embodiment.

[0027] FIG. 10 is a diagram illustrating a control policy for moving a micro-object to a desired position in two dimensions in accordance with one embodiment.

[0028] FIG. 11 is a flow diagram showing a method for machine-learning enabled micro-assembly control with the aid of a digital computer.

[0029] FIG. 12 is a flow diagram showing a routine for modeling micro-object motion based on capacitance for use in the method of FIG. 11 in accordance to one embodiment.

[0030] FIG. 13 is a diagram illustrating the bi-modal control policy in accordance with one embodiment, where x is the micro-object position, and the voltage polarity changes at x+b.

[0031] FIG. 14 is a diagram showing force distribution as a function of b for Vmax=1 and σ=0.1 FIG. 15 is a diagram showing a bi-modal control policy in accordance with one embodiment, where x is the micro-object position, and the potential becomes non-zero starting at x+b.

[0032] FIG. 16 is a diagram showing force distribution as a function of b for V_max=1 and σ=0.1 using the control policy of equation (6).

DETAILED DESCRIPTION

[0033] The system and method described below provide an efficient, scalable way to find control policies, electrode potential assignments that move a micro-object towards a desired position. The analysis involves transforming a micro-model (the dynamic model with distributed parameters) to a macro-model (the dynamic model with a continuous distribution) and using machine-learning based optimization to find the control policies. Transforming the micro-model to a macro-model significantly increases the efficiency of the computations. Further, the control policies can be in one or two dimensions, with rotational transformation allowing to use the 1-dimensional control policies to derive the optimal 2-dimensional control policy.

[0034] FIG. 1 is a block diagram showing a system 10 for machine-learning enabled micro-assembly control with the aid of a digital computer. The system 10 allows for coordinated assembly involving multiple objects 11, such as micro-objects. The size of the objects 11 vary between being on the nano scale (<1 μm), and micro-scale (as much as 100s of μm), though other sizes are possible. The objects 11 can be spherical, though other shapes are also possible, such as a rectangular shape, though still other shapes are also possible. In one embodiment, the diameter of the spherical objects 11 can be 20 μm to 50 μm, though other diameters are also possible. In one embodiment, the dimension of the rectangular objects 11 can be 200 μm×300 μm×100 μm, though other dimensions are also possible. The objects 11 are immersed in a dielectric fluid (not shown) contained within an enclosure (not shown), with a layer of film 38 being below the dielectric fluid contained within the enclosure. In one embodiment, the dielectric fluid is Isopar® M, manufactured by ExxonMobil Chemical Company of Spring, Tex., though other dielectric fluids 17 are also possible. The dielectric fluid can include one or more additives, such as with di-2-ethylhexylsulfosuccinate (AOT) charge director molecules, though other additives are also possible. The objects 11 can be made out of aluminium oxide (AlOx), though other materials are possible. Each of the objects 11 can be a semiconductor chip, an integrated circuit, a particle, a nanodevice or structure, or any object which an array of electrodes can physically manipulate. In the description below, the objects 11 are referred to as micro-objects 11, though other ways to name the objects 11 are possible. In one embodiment, the film can be a 50 μm thick perfluroalkoxy (PFA) film, though other kinds of film of other thicknesses are possible.

[0035] Below the suspended micro-objects 11 are a plurality of programmable electrodes 12 forming an array 35, the electrodes 12 configured to generate a dynamic potential energy landscape for manipulating objects with both dielectrophoretic (“DEP”) and electrophoretic (“EP”) forces. The film is laminated on the electrodes 12. In one embodiment, the electrodes can be square shapes and made off copper, though other shapes and materials are also possible. In one embodiment, the dimensions of a square electrode 12 can be a 16 μm width and 100 nm thickness, though other dimensions are also possible in a further embodiment. The array 35 can include multiple rows of the electrodes 12, with each row including multiple electrodes 12.

[0036] The electric potentials generated by the electrodes 12 are controlled by array 13 of photo-transistors, the array including an active-matrix photo-transistor backplane that is set on glass. The multiple photo-transistors on the backplane form the array 13, with each of the photo-transistors in the array 13 being in control of the electric potentials generated by a single electrode 12. In particular, each of the photo-transistors in the array 13 is attached to one electrode 12. The array 13 of phototransistors can have additional characteristics, such as those described in Rupp et al., “Chiplet micro-assembly printer,” 2019 IEEE 69th Electronic Components and Technology Conference (ECTC), pages 1312-1315, May 2019, the disclosure of which is incorporated by reference.

[0037] The array 13 is optically addressed by a video projector 14 to enable dynamic control of the electrostatic energy potential and manipulation of the positions of multiple micro-objects 11 at the same time. In particular, the video projector 14 is used to address each photo-transistor controlled electrode 12, as illustrated by FIG. 2, allowing for facile zoom factor changes and stitching into larger arrays. FIG. 2 is a diagram illustrating interactions between the video projector 14 and the photo-transistor array 13 of FIG. 1 in accordance with one embodiment. The video projector 14 actuates the electrodes 12 by projecting pre-defined patterns: images 37 that are generated based on control inputs: the electric potentials to be generated by the electrodes to achieve desired motion of the micro-objects 11, as further described below. The pixelated light projected by the projector 14 that makes up the images charges each individual phototransistor 13 in the array to the degree necessary so that the electrode 12 charged by that phototransistor produces the desired electric potential.

[0038] FIGS. 3(A)-3(C) show several views of an example of a photo-transistor array 13 of FIG. 1 in accordance with one embodiment. The photo-transistor array 13 shown with reference to FIGS. 3(A)-3(B) was built using a fabrication techniques similar to those used for building display transistor backplanes, making the fabrication techniques thus compatible with large areas (measuring meters). Voltages of up to +/−200V are controlled by these photo-transistors to display a dynamically varying complicated electric potential landscape. A single voltage signal is synchronized with optical addressing to charge storage capacitors on each pixel. Dynamic electric field patterns are generated by refreshing the projected image pattern (control input). The array 13 with 50 μm, 10 μm and even 3 μm pitch has been built, with pitch being distance between centers of the electrodes to which the phototransistors are attached. Dielectric fluid 17 media (isopar M) and various additives are used to control the charge.

[0039] Returning to FIG. 1, the system 10 further includes a high speed camera 15 that tracks locations of the micro-objects 11 being moved. Both the video projector 14 and the camera 15 are interfaced to one or more computing devices 16, which can control the electrodes 12 through the projector 14 to induce a formation of a desired micro-object 11 pattern. FIGS. 4(A)-4(B) are examples of micro-object 11 patterns realized using the system 10 of FIG. 1. Other micro-object 11 patterns are also possible. In a further embodiment, another kind of sensor (such as utilizing capacitive sensing) could be used to measure and track a position of a micro-object. In a still further embodiment, electrode array 35 could have an integrated active matrix addressing electronics, the potential on each electrode of the array are updated electronically upon the electrode potentials that are necessary for desired micro-object movement having been generated.

[0040] The connections between the one or more computing devices 16 can be direct, such as a via wires or a wireless connection to physically proximate one or more computing devices 16, or the connection can be via an Internetwork to physically remote one or more computing devices 16, such as via the Internet or a cellular network. The one or more computing devices 16 include one or more processors. In one embodiment, the computer processors can be central processing units (CPUs), graphics processing unit (GPU), or a mixture of CPUs and GPU, though still other kinds of processors or a mixture of processors are possible. In one embodiment, two or more of the processors can perform processing in parallel, as further described in U.S. patent application Ser. No. 17/098,816, filed Nov. 16, 2020, entitled System and Method For Multi-Object Micro-Assembly Control With The Aid of A Digital Computer, to Matei et al., the disclosure of which is incorporated by reference. Additional efficiency and improved scalability can be obtained through parallel computation capabilities for tracking multiple particles and for voltage pattern generation.

[0041] While the one or more computing devices 16 are shown as a server, other types of computer devices are possible. The computing devices 16 can include one or more modules for carrying out the embodiments disclosed herein. The modules can be implemented as a computer program or procedure written as source code in a conventional programming language and is presented for execution by the processors as object or byte code. Alternatively, the modules could also be implemented in hardware, either as integrated circuitry or burned into read-only memory components, and each of the computing devices 16 can act as a specialized computer. For instance, when the modules are implemented as hardware, that particular hardware is specialized to perform the computations and communication described above and other computers cannot be used. Additionally, when the modules are burned into read-only memory components, the computer storing the read-only memory becomes specialized to perform the operations described above that other computers cannot. The various implementations of the source code and object and byte codes can be held on a computer-readable storage medium, such as a floppy disk, hard drive, digital video disk (DVD), random access memory (RAM), read-only memory (ROM) and similar storage mediums. Other types of modules and module functions are possible, as well as other physical hardware components. For example, the computing device 16 can include other components found in programmable computing devices, such as input/output ports, network interfaces, and non-volatile storage, although other components are possible. In the embodiment where the computing devices 16 are servers, the server can also be cloud-based or be dedicated servers.

[0042] The one or more computing devices 16 are interfaced to a storage 17 and execute a dynamic modeler 18 that obtains parameters 19 of the system 10, stores the parameters 19 in the storage 17, and models capacitance between the electrodes 12 and the micro-objects. The parameters 19 can include the diameter of the micro-objects 11, the dimensions of the electrodes 12, the dielectric fluid constant (e.g. as ε=2), the fixed positions of the electrodes 12, material of the micro-objects 11 and electrodes 12, and the vertical distance between the micro-objects 11 and the electrodes 12 (“height” of the micro-objects 11). Other parameters 19 are still possible.

[0043] The design of the control policy is based on the dynamical model of the micro-object motion, under the action of the electric field generated by the electrode array. The result of applying electric potentials to the electrodes 12 is the generation of dielectrophoretic and electrophoretic forces that act on the micro-objects 11. A viscous drag force proportional to the velocity opposes the micro-object's motion (with the drag force being proportional to the velocity in non-turbulent flows, that is, when the Reynolds number is small). Due to the negligible mass of the micro-object, acceleration is ignored. Accordingly, the dynamic model 20 for the micro-object motion can be described by:

μ{dot over (x)}=Fx(x,y)

μ{dot over (y)}=Fy(x,y),

[0044] where (x, y) denotes the micro-object 11 position measured at the chilplet's center of mass, μ is the fluid dependent viscous coefficient, and F.sub.x(x, y) and F.sub.y(x, y) are projections on the x and y axis, respectively of the force induced by the potential field. The forces F.sub.x and F.sub.y are expressed as functions of the potential energy of the micro-object. The potential energy is computed by using a capacitive-based electrical circuit that lumps the interaction between the electrodes and the micro-object. FIG. 5 is diagram illustrating capacitive-based interactions between one micro-object 11 and five electrodes of the system of FIG. 1 in accordance with one embodiment. The micro-object and the electrodes act as metal plates; hence the capacitances of these capacitors are dependent on the micro-object position. Let U(x, y) denote the micro-object's potential energy, The forces F.sub.x and F.sub.y can be formally expressed as

[00001]$F_x\left(x,y\right)=\frac{\partial U}{\partial x}\left(x,y\right)\mathrm{and}{F}_y\left(x,y\right)=\frac{\partial U}{\partial y}\left(x,y\right).$

The potential energy can be expressed in terms of the circuit capacitances:

[00002]$U\left(x,y\right)=\frac{1}{2}\underset{i=1}{\overset{N}{.Math.}}{C_i\left(x,y\right)\left[V_i-V\left(x,y\right)\right]}^2,$

[0045] where C.sub.i (x, y) is the capacitance between the micro-object and electrode i, V.sub.i is the electric potential of electrode i, V(x, y) is the electric potential of the micro-object assumed uniformly distributed across the micro-object. In integer N is the number of actuated electrodes. The micro-object potential can be explicitly computed in terms of the electrode potentials, by solving for the voltages and currents in the electrical circuit show with reference to FIG. 5. In particular the steady state micro-object potential is given by

[00003]$V\left(x,y\right)=\frac{1}{{.Math.}_{i=1}^N{C}_i\left(x,y\right)}\underset{i=1}{\overset{N}{.Math.}}{C}_i\left(x,y\right)V_i,$

[0046] Feedback control design requires explicit expressions for the capacitances between the micro-object and electrodes. The capacitance modeler 18 creates the model 20 using high fidelity simulations using the COMSOL Multiphysics® software developed by COMSOL, Inc. of Burlington, Mass. (“COMSOL”), though other kinds of simulations using other kinds of software are also possible. For symmetric micro-objects (e.g., beads) assuming the electrodes are symmetric also, the capacitances are estimated by simulating a 2-dimensional electrostatic COMSOL model. This implies that the capacitance function is of the form C.sub.i(η)=C(∥η−η.sub.i∥), where η.sup.T=[x, y] denotes the micro-object position, and η.sub.i.sup.T=[x.sub.i, y.sub.i] is the fixed position of electrode i. FIG. 6 is a diagram showing the COMSOL model of two conductors (spherical micro-object and an electrode) for capacitance computation in accordance with one embodiment. With reference to FIG. 5, the electrode's position is (0,0). As shown with reference to FIG. 6, in the COMSOL model, a 16 μm width and 100 nm thickness copper plate, and a 10 μm diameter, aluminium oxide (AlOx) spherical object are surrounded by a dielectric with the same properties as the isopar-M solution. The capacitance matrix entries are computed from the charges that result on each conductor when an electric potential is applied to one of them and the other is set to ground. The COMSOL simulations do reflect the field distortion when the micro-object approaches the electrode. The COMSOL electrostatic model has as parameters, the diameter of the sphere, the electrode dimensions, the dielectric fluid constant (ε=2) and the positions and material of the sphere and electrode. The micro-object height is fixed (z=5 μm) and the position of the micro-object is varied on the x-axis over the interval [−1 mm, 1 mm]. Due to the size of the micro-object versus the size of the electrodes, fringe effects (electric field distortions at the edges) are significant. The capacitance function is parametrized using error functions:

[00004]$C\left(\xi \right)={.Math.}_{i=1}^m{a}_i\left[\Phi \left(\frac{\xi +\delta }{c_t}\right)-\Phi \left(\frac{\xi -\delta }{c_t}\right)\right],\mathrm{where}$$\Phi \left(\xi \right)=\frac{1}{\sqrt{\pi }}{\int }_{-\xi }^{\xi }{e}^{-t^2}dt$

is the error function, is the distance between the center of the sphere and the electrode center assumed at the origin, α.sub.i and c.sub.i>0 are positive scalars, and δ is half of the electrode pitch, i.e., 10 μm in this example.

[0047] FIG. 7 shows the capacitance function fitted with training data generated by COMSOL simulations in accordance with one embodiment. FIG. 7 depicts C(ξ), the capacitance between the micro-object and the electrode as a function of the micro-object horizontal position, where the numerical values were fitted on the error function parameterization. The 1D model can be mapped to a 2D model by using the transformation ξ.fwdarw.√{square root over (x.sup.2+y.sup.2)}, which results in a capacitance function

[00005]$C\left(x,y\right)={\Sigma }_{i=1}^m{a}_i\left[\Phi \left(\frac{\sqrt{x^2+y^2}+\delta }{c_i}\right)-\Phi \left(\frac{\sqrt{x^2+y^2}-\delta }{c_i}\right)\right].$

For the sphere shape chiplet, considering only one terms in the parameterization of the capacitance function, the resulting 2D shape is depicted in FIG. 8. FIG. 8 is a diagram showing 2D capacitance function determined from the 1D model using the symmetry property in accordance with one embodiment.

[0048] The one or more computing devices 16 further execute a position tracker 36 that analyzes the images 21 captured by the high-speed camera 15 (or another kind of sensor) to determine the positions 22 of the micro-objects 11. The positions 22 and the dynamic capacitance model 20 are used to determine a control policy 24, which includes the control inputs 27 (the electric potentials) that need to be generated by the electrodes 12 induce the electrodes to the desired positions.

[0049] The determination of the control policy 24 is performed using electrode potential modeler 25 executed by the one or more computing devices 16. As mentioned above, the determination of the control policy 24 involves transforming a micro-model (the dynamic model 20 with distributed parameters) to a macro-model (the dynamic model 20 with a continuous distribution) and using machine-learning based optimization to find the control policies. Transforming the micro-model to a macro-model significantly increases the efficiency of the computations. Further, the control policies 24 can be in one or two dimensions, with rotational transformation allowing to use the 1-dimensional control policies to derive the optimal 2-dimensional control policy. To enable tractability, a mean field-like approximation of the dynamic model 20 is used, where instead of considering a discrete distribution of the array potentials, a continuous distribution is assumed. Under this assumption, a bi-modal control policy, where the voltage distribution shifts polarity just in front of the micro-object does indeed induce micro-object motion to a desired direction. The description below starts with the one-dimensional (1D) case and show how the approach is extended to the 2-dimensional (2D) case. In addition, as described below, the policy is derived by solving a regression problem in terms of the electrode 12 voltages, where the a machine learning algorithm, automatic differentiation in particular, is used to compute the gradients of the micro-object potential function.

[0050] The intuition behind going from a discrete representation to a continuous one can be seen in the expression of the micro-object potential

[00006]$V\left(x\right)=\frac{1}{{\Sigma }_{t=1}^N{C}_i\left(x\right)}{\Sigma }_{i=1}^N{C}_i\left(x\right)V_i.$

In particular V(x) can be interpreted as the expected value of a discrete random function V(Y) over a discrete distribution p.sub.i(x)=C.sub.i(x)/Σ.sub.i=1.sup.N C.sub.i(x)={tilde over (C)}.sub.i(x). The mass function p.sub.i(x) can be interpreted as a conditional probability p.sub.i (x)=Pr(Y=y.sub.i|X=x), and hence the micro-object potential can be expressed as V(x)=E[V(Y)|X=x], where V(y) is a function that reflects the potential at each point y. The discrete probability distribution can be seen as a discretization of a continuous probability distribution, i.e., p.sub.i(x)=∫.sub.y.sub.i.sub.−δ.sup.y.sup.i.sup.+δf(y|x)dy, where y.sub.i is the position of the i.sup.th electrode and δ is half of the electrode pitch. The parameterization of the capacitance function in terms of the error functions tells us that the conditional probability density function (pdf) is a mixture of Gaussian functions. For the sphere-shaped micro-object 11 example, the mixture has only one term, and hence the capacitance is expressed as

[00007]$C_i\left(x\right)=a\left[\Phi \left(\frac{x-y_i+\delta }{c}\right)-\Phi \left(\frac{x-y_i-\delta }{c}\right)\right]=2af\left(y|x\right),\mathrm{where}f\left(y|x\right)=\frac{1}{\sqrt{2\pi {\sigma }^2}}{e}^{-\frac{{\left(y-x\right)}^2}{2{\sigma }^2}},\mathrm{where}\sigma =\frac{c}{\sqrt{2}}.$

[0051] Therefore, the micro-object potential in the continuous representation can be expressed as v(x)=E[V(Y)|X=x], where Y is a continuous random variable with a Gaussian distribution, i.e., Y˜(X, σ), and V(Y) is a function which assigns an electric potential to each point y. The potential energy can now be represented as U(x)=αE[(V(Y)−v(X)).sup.2|X=x], and thus follows that the 1D micro-object dynamics is given by the following partial differential equation

[00008]$\frac{\partial x}{\partial t}=\frac{\partial U\left(x\right)}{\partial x}.$

[0052] where U(x)=αE[(V(Y)−v(X)).sup.2|X=x], and v(x)=E[V(Y)|X=x].

[0053] The electrode potential modeler 25 models the function that determines the allocation of the potential as a neural network (NN) with two hidden layers of size 60 and tanh as activation function, that is V (y)=V(y; β), where β is the vector of parameters encompassing the weights and the biases of the NN. The following optimization problem can be formulated:

[00009]$\begin{array}{cc}\underset{\beta }{\max }\frac{\partial U}{\partial x}\left(x\right)s.t.:U\left(x;\beta \right)=E\left[{\left(V\left(Y;\beta \right)-\stackrel{\_}{v}\left(X;\beta \right)\right)}^2|X=x\right]\stackrel{\_}{v}\left(X;\beta \right)=E\left[V\left(Y;\beta \right)|X\right]& \left(1\right)\end{array}$

[0054] The electrode potential modeler 25 optimal solution of (1) maximizes the micro-object solution in the positive direction. For the negative direction, the sign of the cost function describing the partial derivative of the potential energy U can be changed. The electrode potential modeler makes use of Gauss quadrature rules often found in the theory of generalized chaos polynomials, as described in A. O'Hagan, “Polynomial Chaos: A Tutorial and Critique from a Statistician's Perspective,” University of Sheffield, UK, May 2013; N. Wiener, “The Homogeneous Chaos,” Am. J. Math., vol. 60, no. 4, pp. 897-936, 1938; D. Xiu, D. Lucor, C.-H. Su, and G. Em Karniadakis, “Performance Evaluation of Generalized Polynomial Chaos,” in Computational Science—ICCS 2003: International Conference, Melbourne, Australia and St. Petersburg, Russia, Jun. 2-4, 2003 Proceedings, Part IV, Berlin, Heidelberg, 2003, pp. 346-354, the disclosures of which are incorporated by reference. Gauss quadrature rules provide the tool to efficiently evaluate the conditional expectations. Thus, the expectation of a function of a random variable with a Gaussian distribution, can be accurately approximated using Gauss-Hermite quadrature. Accordingly, the expectation of V(Y; β) is given by

[00010]$\stackrel{\_}{v}\left(x;\beta \right)=E\left[V\left(Y;\beta \right)|X=x\right]\approx \frac{1}{\sqrt{\pi }}{.Math.}_{i=1}^n{w}_{i}V\left(\sqrt{2}\sigma y_i+x;\beta \right),$

[0055] where n is the number of sample points, y.sub.i are the roots of the physicists' version of the Hermite polynomial H.sub.n(y) and w.sub.i are associated weights given by

[00011]$w_i=\frac{2^{n-1}n!\sqrt{\pi }}{{n^2\left[H_{n-1}\left(yt\right)\right]}^2}.$

Similarly, the variance of V(Y; β) can be approximated as

[00012]$U\left(x;\beta \right)=E\left[{\left(V\left(Y;\beta \right)-\stackrel{\_}{v}\left(X;\beta \right)\right)}^2|X=x\right]\approx \frac{1}{\sqrt{\pi }}{\Sigma }_{i=1}^n{w_i\left[V\left(\sqrt{2}\sigma y_i+x;\beta \right)-\stackrel{\_}{v}\left(x;\beta \right)\right]}^2.$

[0056] Hence, the optimization problem (1) becomes

[00013]$\begin{array}{cc}\underset{\beta }{\max }\frac{\partial \hat{U}}{\partial x}\left(x\right)s.t.:\hat{U}\left(x;\beta \right)=\frac{1}{\sqrt{\pi }}\underset{i=1}{\overset{n}{.Math.}}{w_i\left[V\left(\sqrt{2}\sigma y_i+x;\beta \right)-\hat{v}\left(x;\beta \right)\right]}^2\hat{v}\left(X;\beta \right)=\frac{1}{\sqrt{\pi }}\underset{i=1}{\overset{n}{.Math.}}{w}_{i}V\left(\sqrt{2}\sigma y_i+x;\beta \right)& \left(2\right)\end{array}$

[0057] The electrode potential modeler 25 uses a machine learning algorithm, automatic differentiation to perform the optimization according to equation (2). In particular, the electrode potential modeler 25 can leverage the automatic differentiation feature of Autograd Module of PyTorch® software developed by Facebook, Inc. of Menlo Park, Calif. and described by Maclaurin et al., “Autograd: Effortless gradients in numpy,” ICML 2015 AutoML Workshop, 2015, the disclosure of which is incorporated by reference, to evaluate the cost function which requires the computation of the partial derivative of .Math.(x) as a function of the NN weights and biases, though other implementations of automatic differentiation are possible. To solve the optimization of equation (2), the electrode potential modeler 25 runs Adam gradient based optimization algorithm that utilizes results of automatic differentiation, such as described in D. P. Kingma and J. Ba, Adam: A Method for Stochastic Optimization. 2014 (though other implementations are possible), for 1500 iteration with a stepsize α=0.001 (though in a further embodiment, other numbers of iterations and other stepsizes are possible).

[0058] As also described below, experimentally, the optimization problem was solved for x=0, and a sequence of the number of sample points n∈{10,20,30,40,50}. They all resulted in the same type of control policy 24, as shown in with reference to FIG. 9 for n=30. FIG. 9 is a diagram showing a control policy obtained using by solving equation (2) in accordance with one embodiment. The resulting control policy is a bi-modal control policy that shows the sudden change in polarite just in front of the micro-object. Due to the translational symmetry of the micro-object dynamics, sufficient to solve the optimization problem for x=0 only.

[0059] Due to the physical limitations of the addressability of the electrode array, the special case of a bi-modal control policy is considered by the electrode potential modeler 25:

[00014]$V\left(y\right)=\{\begin{array}{cc}0& y\le x+b\\ V_{m\mathrm{ax}}& y>x+b^{\prime }\end{array}$

[0060] The optimal parameters of the policy are determined by the electrode potential modeler 25. In particular, for b=0.4307σ the micro-object will move at maximum speed in the positive direction (i.e., {dot over (x)}>0) and for b=−0.4307σ the will move the micro-object at maximum speed in the negative direction (i.e., {dot over (x)}<0), where σ is the variance of the Gaussian random variable Y.

[0061] Note that activating all of the electrodes 12 does not make sense, but only the ones in the interval [x+b, 4σ], hence cutting the electrode activations after 4σ does not affect the generated force. Hence, this interval imposes a particular length of the image pattern. The discretized control policy is given by V.sub.i=V.sub.max for y.sub.i∈[x+b, 4σ], and unassigned otherwise, where y.sub.i is the position of electrode i, which is exactly the control scheme used experimentally. Therefore, the discretized control policy 24 generated for moving a micro-object in one dimension is given by V.sub.i=V.sub.max for y.sub.i∈[x+b, 4σ], and unassigned (0) otherwise, where y.sub.i is the position of electrode i.

[0062] The results given above can also be extended to moving a micro-object 11 in two dimensions. The micro-object 11 position is denoted by x=(x.sub.1, x.sub.2) and by y.sup.(i)=(y.sub.1.sup.(i), y.sub.2.sup.(i)) is denoted the position of electrode i. The micro-object 11 dynamics becomes

μ{dot over (x)}=VU(x),

[0063] where μ is the viscous coefficient, and U(x) denotes the micro-object's potential energy. As is in the 1-dimensional case, the potential energy is given by

[00015]$U\left(x\right)=\frac{1}{2}\underset{i=1}{\overset{N}{.Math.}}{C_i\left(x\right)\left[V_i-V\left(x\right)\right]}^2,$

[0064] where V(x) denotes the micro-object electric potential, C.sub.i(x)=C(∥x−y.sup.(i)∥) represents the capacitance between the micro-object and electrode i a position y.sup.(i), and V.sub.i represents the potential of electrode i. Similar to the 1-dimensional case, an assumption is made that the capacitance C.sub.i can be represented as the un-normalized discretization of a multivariable Gaussian pdf, that is:

[00016]$C_i\left(x\right)\sim {\int }_{y_1^{\left(i\right)}-\delta /2}^{y_1^{\left(i\right)}+\delta /2}{\int }_{y_2^{\left(i\right)}-\delta /2}^{y_2^{\left(i\right)}+\delta /2}f\left(y|x\right)dy,$

[0065] where Y˜(X, σ.sup.2I). The control objective is to determine a distribution for the electrode potential that moves the micro-object towards a position x.sup.(d). FIG. 10 is a diagram illustrating a control policy 24 for moving a micro-object to a desired position in two dimensions in accordance with one embodiment. The scenario shown with reference to FIG. 10 is where the motion is parallel to the y.sub.1 axis. This is sufficient since the potential distribution for other scenarios are derived by performing a rotational transformation as the micro-object dynamics is endowed with translational and rotational symmetries. The corresponding 2D control policy is

[00017]$V\left(y\right)=V\left(y_1,y_2\right)=\{\begin{array}{cc}{V}_{m\mathrm{ax}}& y_1\le x_1+b,\forall y_2\\ -V_{m\mathrm{ax}}& y_1>x_1+b,\forall y_2\end{array}$

[0066] All results determined for the 1D case concerning the choice of b to generate the maximum force hold for the 2D case, as well. Note that discretization of the ideal control policy will introduce some errors since the micro-object may not be exactly on top of an electrode. That is why, the experimental policy is based on a pattern that includes neighboring electrodes for cases when electrodes are not exactly aligned with the direction towards the desired position.

[0067] By using the rotational symmetries of the model, the electrode potential modeler 25 determines the 2D control policy for any direction of motion. Let x.sup.(0) be the current micro-object position and let x.sup.(d) be the desired micro-object position. Then the control policy 24 is given by:

[00018]$V\left(y\right)=V\left(y_1,y_2\right)=\{\begin{array}{cc}{V}_{m\mathrm{ax}}& y_2\le x_2^{\left(b\right)}+\frac{1}{s}\left(y_1-x_1^{\left(b\right)}\right)\\ -V_{m\mathrm{ax}}& y_2>x_2^{\left(b\right)}+\frac{1}{s}\left(y_1-x_1^{\left(b\right)}\right),\end{array}\mathrm{where}{x}^{\left(b\right)}=x^{\left(0\right)}+b\frac{x^{\left(d\right)}-x^{\left(0\right)}}{.Math.x^{\left(d\right)}-x^{\left(0\right)}.Math.}\mathrm{and}s=\frac{x_2^{\left(d\right)}-x_2^{\left(0\right)}}{x_1^{\left(d\right)}-x_1^{\left(0\right)}}$

ensure micro-object motion towards the desired position x.sup.(d) for positive b and away from x.sup.(d) for negative b.

[0068] Thus, by taking as input the dynamic model 20, the initial micro-object position 22, and the desired micro-object position 23, the electrode potential modeler 25 can derive the control policy for moving a micro-object 11 from an initial position 22 to a desired micro-object position 23. The control policy 24 is designed for moving a single micro-object position at a time, as the micro-objects in this design do not come close enough to each other to cause a significant micro-object-to-micro-object interaction (with the exact distance at which one micro-object 11 needs to stay away from another micro-object to avoid causing interactions that need to be accounted for being determined in accordance with the geometry and electric potential of the micro-objects—generally, to cause micro-object-to-micro-object interactions, the distance between two micro-objects has to be on the same order of magnitude as the distance from micro-objects 11 to electrodes 12). The control policy 24 generated by the electrode potential modeler is used by a projector controller 28 executed by the one or more computing devices 16 to map the control policy to the images 37 that are provided to the video projector 14, and which in turn projects the images 37 to the photo-transistors controlling the electrodes 11, thus actuating the movement of the micro-objects 11. The mapping can be done as described in U.S. Patent Application Publication No. US20200207617, entitled “Micro-assembler system for manipulating micro-objects,” to Plochowietz et al., published Jul. 2, 2020, the disclosure of which is incorporated by reference. In a further embodiment, if the electrode array 35 has an integrated active matrix addressing electronics, the potential on each electrode of the array are updated electronically upon the control policy 24 having been generated.

[0069] Transforming the dynamic model from a discrete formulation to a continuous formulation allows to rewrite the constraints of the model as expectations. In turn, this allows to rediscritize the model in a much more efficient way so that allows to derive more efficiently the optimal electrode potentials using automatic differentiation. FIG. 11 is a flow diagram showing a method 50 for multi-object micro-assembly control with the aid of a digital computer. The method 50 can be performed using the system of FIG. 1. Parameters of the system are obtained, such as diameter of the micro-objects, the dimensions of the electrodes, the dielectric fluid constant, the initial positions and material of one or more micro-objects that need to be moved and of the electrodes (which can be obtained using the high-speed camera or another kind of sensor), and the height of the micro-objects, though other parameters are possible (step 51). A capacitance-based dynamic model of the motion of the micro-objects is created using the parameters (step 52), as further described below with reference to FIG. 10 and as described above with reference to FIG. 1. A desired position of at least one of the micro-objects is obtained (step 53). A control policy is determined using the initial positions of one of the micro-objects and a desired position of that micro-object as described above with reference to FIG. 1 (step 54). Briefly, the dynamic model with distributed parameters is transformed into a continuous representation of the dynamic model and automatic differentiation together with Gauss quadrature rules are used to solve an optimization problem to derive electrode potentials necessary to move the micro-object to a desired position, either in one or two dimensions. The derived electrode potentials are mapped to images for display by the video projector (step 54) and electrodes in the electrode array are controlled to induce the movement of the micro-object to desired positions (step 55). If additional movement is necessary (step 57), such as if other micro-objects need to be moved or if the micro-object that was moved did not reach the desired position (determined using the high speed camera or another kind of sensor), the method returns to step 54. If no additional movement is necessary (step 57), the method 50 ends.

[0070] In a further embodiment, if the electrode array 35 has an integrated active matrix addressing electronics, the potential on each electrode of the array are updated electronically upon the control policy 24 having been generated instead through images projected by the projector in step 54.

[0071] As the system and method use only one micro-object at a time and the micro-objects do not come close enough to generate micro-object-micro-object interactions, the dynamic model needs to only account for interactions between one micro-object and the electrodes. The capacitance-based model of motion of the micro-objects accounts both for capacitance between a micro-object and an electrode and a capacitance between two different micro-objects. FIG. 12 is a flow diagram showing a routine 60 for modeling micro-object motion based on capacitance for use in the method 50 of FIG. 11 in accordance to one embodiment. Simulations, such as COMSOL simulations, are performed for capacitance between an electrode and a micro-object at multiple distances as described above with reference to FIG. 1 (step 61). A function describing the dependence between the capacitance between an electrode and a micro-object is defined based on the simulations regarding the capacitance between the micro-object and the electrodes, as described above with reference to FIG. 1 (step 62), ending the routine 60.

[0072] The system and method described above leverage a bi-modal control policy that is derived using a machine learning algorithm and Gauss quadrature rules to derive the necessary electrode potentials. Additional mathematical reasoning and experimental details behind the derivation and application of the control policy is given below. The reasoning and details are provided for purposes of illustration and not limitation.

[0073] Control Design: 1D Case

[0074] As described above, the intuition behind going from a discrete representation to a continuous one can be seen in the expression of the micro-object potential

[00019]$V\left(x\right)=\frac{1}{{\Sigma }_{t=1}^N{C}_i\left(x\right)}{\Sigma }_{i=1}^N{C}_i\left(x\right)V_i.$

In particular V(x) can be interpreted as the expected value of a discrete random function V(Y) over a discrete distribution p.sub.i(x)=C.sub.i(x)/Σ.sub.i=1.sup.N C.sub.i(x)={tilde over (C)}.sub.i(x). The mass function p.sub.i(x) can be interpreted as a conditional probability p.sub.i(x)=Pr(Y=y.sub.i|X=x), and hence the micro-object potential can be expressed as V(x)=E[V(Y)|X=x], where V(y) is a function that reflects the potential at each point y. The discrete probability distribution can be seen as a discretization of a continuous probability distribution, i.e., p.sub.i(x)=∫.sub.y.sub.i.sub.−δ.sup.y.sup.i.sup.+δf(y|x)dy, where y.sub.i is the position of the i.sup.th electrode and δ is half of the electrode pitch. The parameterization of the capacitance function in terms of the error functions tells us that the conditional probability density function (pdf) is a mixture of Gaussian functions. For the sphere shaped example, the mixture has only one term, and hence the capacitance is expressed as

[00020]$C_i\left(x\right)=a\left[\Phi \left(\frac{x-y_i+\delta }{c}\right)-\Phi \left(\frac{x-y_i-\delta }{c}\right)\right]=2\mathrm{af}\left(y|x\right),\mathrm{where}$$f\left(y|x\right)=\frac{1}{\sqrt{2\pi {\sigma }^2}}{e}^{-\frac{{\left(y-x\right)}^2}{2{\sigma }^2}},\mathrm{where}\sigma =\frac{c}{\sqrt{2}}.$

[0075] Therefore, the micro-object potential in the continuous representation can be expressed as v(x)=E[V(Y)|X=x], where Y is a continuous random variable with a Gaussian distribution, i.e., Y˜(X, σ), and V(Y) is a function which assigns an electric potential to each point y. The potential energy can now be represented as U(x)=αE[(V(Y)−v(X)).sup.2|X=x], and it follows that the 1D micro-object dynamics is given by the following partial differential equation

[00021]$\begin{array}{cc}\frac{\partial x}{\partial t}=\frac{\partial U\left(x\right)}{\partial x}.\mathrm{where}U\left(x\right)=aE\left[{\left(V\left(Y\right)-\stackrel{\_}{v}\left(X\right)\right)}^2|X=x\right],\mathrm{and}\stackrel{\_}{v}\left(x\right)=E\left[V\left(Y\right)|X=x\right].& \left(3\right)\end{array}$

[0076] The following result shows that a bi-modal potential distribution is sufficient to move the micro-object towards the desired position.

[0077] Proposition 1 Let the function that describes the electric potential allocation be chosen as

[00022]$\begin{array}{cc}V\left(y\right)=\{\begin{array}{cc}{V}_{m\mathrm{ax}}& y\le x+b\\ -V_{m\mathrm{ax}}& y>x+b,\end{array}& \left(4\right)\end{array}$

[0078] where V.sub.max is the maximum absolute value of the electric potential. Then for b=0.87696 the micro-object will move at maximum speed in the positive direction (i.e., {dot over (x)}>0) and for b=−0.8769σ will move the micro-object at maximum speed in the negative direction (i.e., {dot over (x)}<0), where a is the variance of the Gaussian random variable Y.

[0079] Proof. The graph of the function V(y), as defined by equation (4) can be seen with reference to FIG. 13. FIG. 13 is a diagram illustrating the bi-modal control policy in accordance with one embodiment, where x is the micro-object position, and the voltage polarity changes at x+b. The Gaussian distribution of Y enables to calculate the closed form expressions for v(x) and the potential energy. In particular:

[00023]$\begin{array}{c}\stackrel{\_}{v}\left(x\right)={\int }_{-\infty }^{x+b}{V}_{\mathrm{ma}x}f\left(y|x\right)\mathrm{dy}-{\int }_{x+b}^{\infty }{V}_{\mathrm{ma}x}f\left(y|x\right)\mathrm{dy}\\ =V_{\mathrm{ma}x}\Phi \left(\frac{b\sqrt{2}}{2\sigma }\right).\end{array}$

[0080] To evaluate the partial derivative of the potential energy, the energy function can be evaluated at points x and x+ε, followed by the computation of the

[00024]$\mathrm{limit}\frac{\partial U\left(x\right)}{\partial x}={\lim }_{\mathrm{.Math.}.fwdarw.0}\frac{U\left(x+\mathrm{.Math.}\right)-U\left(x\right)}{\mathrm{.Math.}}.$

Thus,

[0081] [00025]$U\left(x\right)=E\left[{\left(V\left(Y\right)-\stackrel{\_}{v}\left(x\right)\right)}^2|X=x\right]=-V_{m\mathrm{ax}}^2{\Phi \left(\frac{b\sqrt{2}}{2\sigma }\right)}^2+V_{m\mathrm{ax}}^2.$

[0082] The perturbed mean voltage is given by:

[00026]$\stackrel{\_}{v}\left(x+\mathrm{.Math.}\right)=E\left[V\left(Y\right)|X=x+\mathrm{.Math.}\right]=-V_{m\mathrm{ax}}\Phi \left(\frac{\left(\mathrm{.Math.}-b\right)\sqrt{2}}{2\sigma }\right).$

[0083] which allows to compute the potential energy at x+ε, i.e.,

[00027]$\begin{array}{c}U\left(x+\mathrm{.Math.}\right)=E\left[{\left(V\left(Y\right)-\stackrel{\_}{v}\left(x+\mathrm{.Math.}\right)\right)}^2|X=x+\mathrm{.Math.}\right]\\ =-V_{m\mathrm{ax}}^2{\Phi \left(\frac{\left(\mathrm{.Math.}-b\right)\sqrt{2}}{2\sigma }\right)}^2+V_{\mathrm{ma}x}^2.\end{array}$

[0084] Finally,

[00028]$\begin{array}{cc}\frac{\partial U\left(x\right)}{\partial x}=\underset{\mathrm{.Math.}.fwdarw.0}{\lim }\frac{U\left(x+\mathrm{.Math.}\right)-U\left(x\right)}{\mathrm{.Math.}}=\frac{4V_{m\mathrm{ax}}^2\Phi \left(\frac{b\sqrt{2}}{2\sigma }\right)e^{-\frac{b^2}{2{\sigma }^2}}}{\sqrt{2\pi {\sigma }^2}}.& \left(5\right)\end{array}$

[0085] Expression (5) shows that {tilde over (V)}(y)=−V(y) is a valid control policy resulting in the same energy function and induced force. Hence the maximum velocity is dictated by the maximum value of

[00029]$\Phi \left(\frac{b\sqrt{2}}{2\sigma }\right)e^{-\frac{b^2}{2{\sigma }^2}}.$

The force dependence on b is depicted with reference to FIG. 14. FIG. 14 is a diagram showing force distribution as a function of b for V.sub.max=1 and σ=0.1 The force derivative in terms of b as two critical points that are the solution of the equation

[00030]$-\sqrt{2}\mathrm{\pi \Phi }\left(\frac{b}{\sigma \sqrt{2}}\right)b+2\sigma \sqrt{\pi }{e}^{-\frac{b^2}{2{\sigma }^2}}=0,$

[0086] whose solution is evaluated numerically. In particular, a sequence of values is generated for sigma and the critical points corresponding to them are computed. Thus, the critical points having a linear dependence on a, namely b*=±0.8769σ were discovered.

[0087] The previous result shows that a bi-modal voltage distribution across the electrode 11 does indeed ensure that the micro-object moves in the desired direction. More importantly, and not unexpectedly, the result shows that only a limited number of electrodes are relevant, namely the electrodes whose positions are in the interval [−4σ, 4σ]. The control policy in the continuous representation of the electrodes is practically implemented by discretizing the control policy, resulting in

[00031]$V_i=\{\begin{array}{cc}{V}_{m\mathrm{ax}}& y\le x+b\\ -V_{m\mathrm{ax}}& y>x+b,\end{array},$

[0088] where y.sub.i is the position of electrode i and x is the micro-object position.

[0089] Control Policy

[0090] As described above, an experimentally designed control policy based on projecting an image containing particular patterns is used to push the micro-objects in a desired direction. The description that follows shows analytically why the control policy actually works, and formally re-discovers the pattern shapes. As in the previous section, the 1D case is considered and subsequently extended to the 2D case. Consider a control policy given by:

[00032]$\begin{array}{cc}V\left(y\right)=\{\begin{array}{cc}0& y\le x+b\\ V_{m\mathrm{ax}}& y>x+b^{\prime }\end{array}& \left(6\right)\end{array}$

The control policy is graphically depicted with reference to FIG. 15. FIG. 15 is a diagram showing a bi-modal control policy in accordance with one embodiment, where x is the micro-object position, and the potential becomes non-zero starting at x+b.

[0091] In the experimentally designed control policy, the voltage is not held constant for all points y≥x+b, and for y<x+b the electrode potential is free floating. An assumption is made here that the frequency at which the images are projected on the array is small enough so that the capacitors associated with electrode photo-diodes get completely discharged when they are not activated (i.e., black color projected on them). The following results summarizes the effect of this new policy on the micro-object motion.

[0092] Proposition 2. Let the function that describes the electric potential allocation be chosen as in (6), where V.sub.max is the maximum absolute value of the electric potential. Then for b=0.4307a the micro-object will move at maximum speed in the positive direction (i.e., {dot over (x)}>0) and for b=−0.4307σ it will move the micro-object at maximum speed in the negative direction (i.e., {dot over (x)}<0), where a is the variance of the Gaussian random variable Y.

[0093] Proof. The proof follows the same steps as in the proof of Proposition 1. The main difference is that now integration over the interval [x+b, ∞) only is performed. As before, the Gaussian distribution of Y allows to calculate closed form expressions for the potential energy. In particular, the potential energy at points U(x) and U(x+E) are given by

[00033]$U\left(x\right)=\frac{1}{8}\Phi \left(\frac{b}{\sigma \sqrt{2}}\right)V_{m\mathrm{ax}}^2-\frac{1}{8}{\Phi \left(\frac{b}{\sigma \sqrt{2}}\right)}^2{V}_{\mathrm{ma}x}^2-\frac{1}{8}{\Phi \left(\frac{b}{\sigma \sqrt{2}}\right)}^3{V}_{m\mathrm{ax}}^2+\frac{1}{8}{V}_{\mathrm{ma}x}^2,\mathrm{and}$$U\left(x+\mathrm{.Math.}\right)=-\frac{1}{8}\Phi \left(\frac{-b+\mathrm{.Math.}}{\sigma \sqrt{2}}\right)V_{\mathrm{ma}x}^2-\frac{1}{8}{\Phi \left(\frac{-b+\mathrm{.Math.}}{\sigma \sqrt{2}}\right)}^2{V}_{\mathrm{ma}x}^2+\frac{1}{8}{\Phi \left(\frac{-b+\mathrm{.Math.}}{\sigma \sqrt{2}}\right)}^3{V}_{\mathrm{ma}x}^2+\frac{1}{8}{V}_{\mathrm{ma}x}^2,$

[0094] respectively. Taking £ to zero, we get the force expression:

[00034]$\begin{array}{c}F\left(x\right)=\frac{\partial U\left(x\right)}{\partial x}\\ =\frac{1}{4\sqrt{2{\mathrm{\pi \sigma }}^2}}{e}^{-\frac{b^2}{2{\sigma }^2}}\left[3{\Phi \left(\frac{b\sqrt{2}}{2\sigma }\right)}^2+2\Phi \left(\frac{b\sqrt{2}}{2\sigma }\right)-1\right].\end{array}$

[0095] The force as a function of the variable b is shown with reference to FIG. 16. FIG. 16 is a diagram showing force distribution as a function of b for V_max=1 and σ=0.1 using the control policy of equation (6). FIG. 16 depicts as well two critical points that maximize velocity in the desired direction. There are two important differences, as compared to the policy introduced in equation (4). The critical points are given by b*=+0.43076, where the points were estimated numerically, and the magnitude of the force is significantly reduced (more than %80 reduction).

[0096] Note activating all electrodes does not make sense, but only the ones in the interval [x+b, 4σ], hence cutting the electrode activations after 4σ does not affect the generated force. The discretized control policy is given by V.sub.i=V.sub.max for y.sub.i∈[x+b, 4σ], and unassigned otherwise, where y.sub.i is the position of electrode i, which is exactly the control scheme used experimentally. The experimentally designed control policy was implemented by considering some physical limitations of the actuation mechanism, which does not allow simultaneously setting positive and negative polarities for the electrode potentials.

[0097] Machine Learning Enabled Optimal Control Design

[0098] Above a formal proof that a bi-modal control policy is sufficient to induce a micro-object motion in a desired direction was provided. An interesting question is if such a policy is optimal. Although the description below does not analytically prove optimality, leveraging the deep-learning optimization framework (as described Dougal Maclaurin, David Duvenaud, and Ryan P Adams. Autograd: Effortless gradients in numpy. In ICML 2015 AutoML Workshop, 2015; and Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017, the disclosures of which are incorporated by reference) to solve a regression problem whose result is the array's electric potential distribution is provided. As seen in what follows, the optimization results strengthen the conjecture that the control policy derived through equation (4) is indeed optimal.

[0099] The function that determines the allocation of the potential was modeled as a neural network (NN) with two hidden layers of size 60 and tanh as activation function, that is V (y)=V (y; β), where β is the vector of parameters encompassing the weights and the biases of the NN. The following optimization problem was formulated, previously provided in equation (1).

[00035]$\begin{array}{cc}\underset{\beta }{\max }\frac{\partial U}{\partial x}\left(x\right)s.t.:U\left(x;\beta \right)=E\left[{\left(V\left(Y;\beta \right)-\stackrel{\_}{v}\left(X;\beta \right)\right)}^2|X=x\right]\stackrel{\_}{v}\left(X;\beta \right)=E\left[V\left(Y_{;}\beta \right)|X\right]& \left(1\right)\end{array}$

[0100] The optimal solution of (1) maximizes the micro-object solution in the positive direction. For the negative direction, the sign of the cost function describing the partial derivative of the potential energy U can be changed. Note that unlike the case when policy of equation (4) is used, the conditional expectations can no longer be analytically evaluated. The address this challenge, Gauss quadrature rules are used, which, as mentioned above, are often found in the theory of generalized chaos polynomials as described by Anthony O'Hagan. Polynomial chaos: A tutorial and critique from a statistician's perspective. Technical report, University of Sheffield, UK, May 2013; Ralph C. Smith. Uncertainty Quantification: Theory, Implementation, and Applications. Society for Industrial and Applied Mathematics, Philadelphia, Pa., USA, 2013; Norbert Wiener. The Homogeneous Chaos. American Journal of Mathematics, 60(4):897-936, 1938; Dongbin Xiu, Didier Lucor, C.-H. Su, and George Em Karniadakis. Performance evaluation of generalized polynomial chaos. In Computational Science—ICCS 2003: International Conference, Melbourne, Australia and St. Petersburg, Russia, Jun. 2-4, 2003 Proceedings, Part IV, pages 346-354, Berlin, Heidelberg, 2003. Springer Berlin Heidelberg, the disclosures of which are incorporated by reference. Gauss quadrature rules provide the tool to efficiently evaluate the conditional expectations. Thus, the expectation of a function of a random variable with a Gaussian distribution, can be accurately approximated using Gauss-Hermite quadrature. Accordingly, the expectation of V(Y; β) is given by

[00036]$\stackrel{\_}{v}\left(x;\beta \right)=E\left[V\left(Y;\beta \right)|X=x\right]\approx \frac{1}{\sqrt{\pi }}{.Math.}_{i=1}^n{w}_i\mathrm{fV}\left(\sqrt{2}\sigma y_i+x;\beta \right),$

[0101] where n is the number of sample points, y.sub.i are the roots of the physicists' version of the Hermite polynomial H.sub.n(y) and w.sub.i are associated weights given by

[00037]$w_i=\frac{2^{n-1}n!\sqrt{\pi }}{{n^2\left[H_{n-1}\left(y_i\right)\right]}^2}.$

Similarly, the variance of V(Y; β) can be approximated as

[00038]$U\left(x;\beta \right)=E\left[{\left(V\left(Y;\beta \right)-\stackrel{\_}{v}\left(X;\beta \right)\right)}^2|X=x\right]\approx \frac{1}{\sqrt{\pi }}{\Sigma }_{i=1}^n{w_i\left[V\left(\sqrt{2}\sigma y_i+x;\beta \right)-\overline{v}\left(x;\beta \right)\right]}^2.$

[0102] Hence, the optimization problem (1) becomes as previously given in equation (2):

[00039]$\begin{array}{cc}\underset{\beta }{\max }\frac{\partial \hat{U}}{\partial x}\left(x\right)s.t.:\hat{U}\left(x;\beta \right)=\frac{1}{\sqrt{\pi }}{\Sigma }_{i=1}^n{w_i\left[V\left(\sqrt{2}\sigma y_i+x;\beta \right)-\hat{v}\left(x;\beta \right)\right]}^2\hat{v}\left(X;\beta \right)=\frac{1}{\sqrt{\pi }}{\Sigma }_{i=1}^n{w}_{i}V\left(\sqrt{2}\sigma y_i+x;\beta \right)& \left(2\right)\end{array}$

[0103] Automatic differentiation feature of Autograd Module of PyTorch® software developed by Facebook, Inc. of Menlo Park, Calif. (and described by Maclaurin et al., “Autograd: Effortless gradients in numpy,” ICML 2015 AutoML Workshop, 2015) was used to evaluate the cost function which requires the computation of the partial derivative of U(x) as a function of the NN weights and biases. To solve equation 2, Adam gradient based optimization algorithm was run for 1500 iteration with a stepsize α=0.001. The optimization problem was solved for x=0, and a sequence of the number of sample points n∈{10,20,30,40,50}. They all resulted in the same type of control policy, as shown with reference to FIG. 9 for n=30. We recover the bi-modal control policy introduced in equation (4) that shows the sudden change in polarite just in front of the micro-object. Due to the translational symmetry of the micro-object dynamics, solve the optimization problem for x=0 only is sufficient.

[0104] Extension to 2D Control Policy

[0105] In this section is demonstrated how the previous results can be extended to the 2D case. Denoted by x=(x.sub.1, x.sub.2) is the micro-object position and by y.sup.(i)=(y.sub.1.sup.(i), y.sub.2.sup.(i)) the position of electrode i. The micro-object dynamics becomes

μ{dot over (x)}=VU(x),

[0106] where μ is the viscous coefficient, and U(x) denotes the micro-object's potential energy. As is the 1D case, the potential energy is given by

U(x)=½Σ.sub.i=1.sup.NC.sub.i(x)[V.sub.i−V(x)].sup.2,

[0107] where V(x) denotes the micro-object electric potential, C.sub.i(x)=C(∥x−y.sup.(i)∥) represents the capacitance between the micro-object and electrode i a position y.sup.(i), and V.sub.i represents the potential of electrode i. Similar to the 1D case, an assumption that the capacitance C.sub.i can be represented as the un-normalized discretization of a multivariable Gaussian pdf, that is:

[00040]$C_i\left(x\right)\sim {\int }_{y_1^{\left(i\right)}-\delta /2}^{y_1^{\left(i\right)}+\delta /2}{\int }_{y_2^{\left(i\right)}-\delta /2}^{y_2^{\left(i\right)}+\delta /2}f\left(y|x\right)\mathrm{dy},$

[0108] where Y˜(X, σ.sup.2I). The control objective is to determine a distribution for the electrode potential that moves the micro-object towards a position x.sup.(d). A scenario shown with reference to FIG. 10, where the motion is parallel to the y.sub.i axis. This is sufficient since the potential distribution for other scenarios are derived by performing a rotational transformation as the micro-object dynamics is endowed with translational and rotational symmetries.

[0109] We use the following bi-modal control policy:

[00041]$\begin{array}{cc}V\left(y\right)=V\left(y_1,y_2\right)=\{\begin{array}{cc}{V}_{m\mathrm{ax}}& y_1\le x_1+b,\forall y_2\\ -V_{m\mathrm{ax}}& y_1>x_1+b,\forall y_2\end{array}& \left(7\right)\end{array}$

[0110] The micro-object potential is computed as the mean of the potential function V(y), that is

[00042]$\stackrel{\_}{v}\left(x\right)=E\left[V\left(Y\right)|X=x\right]=V_{m\mathrm{ax}}\Phi \left(\frac{b\sqrt{2}}{2\sigma }\right).$

[0111] The potential energy can be computed in closed form, and is given by

[00043]$\begin{array}{c}U\left(x\right)=E\left[{\left(V\left(Y\right)-\stackrel{\_}{v}\left(x\right)\right)}^2|X=x\right]\\ =-V_{m\mathrm{ax}}^2{\Phi \left(\frac{b\sqrt{2}}{2\sigma }\right)}^2+V_{m\mathrm{ax}}^2\end{array}$

[0112] To evaluate the gradient of the potential energy, the partial derivatives of U(x) are computed. Thus,

[00044]$\begin{array}{c}U\left(x_1+\mathrm{.Math.},x_2\right)=E\left[{\left(V\left(Y\right)-\stackrel{\_}{v}\left(x_1+\mathrm{.Math.},x_2\right)\right)}^2|X=\left(x_1+\mathrm{.Math.},x_2\right)\right]\\ =-V_{\mathrm{ma}x}^2{\Phi \left(\frac{\left(\mathrm{.Math.}-b\right)\sqrt{2}}{2\sigma }\right)}^2+V_{\mathrm{ma}x}^2,\end{array}$$\mathrm{where}$$\begin{array}{c}\stackrel{\_}{v}\left(x_1+\mathrm{.Math.},x_2\right)=E\left[V\left(Y\right)|X=\left(x_1+\mathrm{.Math.},x_2\right)\right]\\ =-V_{m\mathrm{ax}}\Phi \left(\frac{\left(\mathrm{.Math.}-b\right)\sqrt{2}}{2\sigma }\right).\end{array}$

[0113] Accordingly,

[00045]$\begin{array}{c}\frac{\partial U}{\partial x_1}\left(x_1,x_2\right)=\underset{\delta .fwdarw.0}{\lim }\frac{U\left(x_1+\mathrm{.Math.},x_2\right)-U\left(x_1,x_2\right)}{\mathrm{.Math.}}\\ =\frac{4V_{\mathrm{ma}x}^2\Phi \left(\frac{b\sqrt{2}}{2\sigma }\right)e^{-\frac{b^2}{2{\sigma }^2}}}{\sqrt{2\pi {\sigma }^2}}\end{array}$

[0114] Similarly, the partial derivatives of U(x.sub.1, x.sub.2) are computed with respect to x.sub.2, which turns out to be zero, that is

[00046]$\frac{\partial U}{\partial x_2}\left(x_1,x_2\right)=0.$

[0115] Note that the control policy generates a force on the y.sub.1 axis only, and the control policy has the exact expression as the force generated in the 1D case. All results determined for the 1D case concerning the choice of b to generate the maximum force hold for the 2D case, as well. Note that discretization of the ideal control policy will introduce some errors since the micro-object may not be exactly on top of an electrode. That is why, the experimental policy is based on a pattern that includes neighboring electrodes for cases when electrodes are not exactly aligned with the direction towards the desired position.

[0116] The general 2D control policy can be summarized in the following result.

[0117] Proposition 3 Let x.sup.(0) be the current micro-object position and let x.sup.(d) be the desired micro-object position. Then the control policy

[00047]$\begin{array}{cc}V\left(y\right)=V\left(y_{1\prime }{y}_2\right)=\{\begin{array}{cc}{V}_{\mathrm{ma}x}& y_2\le x_2^{\left(b\right)}+\frac{1}{s}\left(y_1-x_1^{\left(b\right)}\right)\\ -V_{\mathrm{ma}x}& y_2>x_2^{\left(b\right)}+\frac{1}{s}\left(y_1-x_1^{\left(b\right)}\right),\end{array}\mathrm{where}{x}^{\left(b\right)}=x^{\left(0\right)}+b\frac{x^{\left(d\right)}-x^{\left(0\right)}}{.Math.x^{\left(d\right)}-x^{\left(0\right)}.Math.}\mathrm{and}s=\frac{x_2^{\left(d\right)}-x_2^{\left(0\right)}}{x_1^{\left(d\right)}-x_1^{\left(0\right)}}& \left(9\right)\end{array}$

ensure micro-object motion towards the desired position x.sup.(d) for positive b and away from x.sup.(d) for negative b.

[0118] Proof. The proof follows by applying a rotational transformation so that {circumflex over (x)}.sup.(0)=R(θ)x.sup.(0) and {circumflex over (x)}.sup.(d)=R(θ)x.sup.(d) are such that {circumflex over (x)}.sub.2.sup.(0)={circumflex over (x)}.sub.2.sup.(d)=0 and applying the policy (11). We also note that

[00048]$y_2=x_2^{\left(b\right)}+\frac{1}{s}\left(y_1-x_1^{\left(b\right)}\right)$

is orthogonal on the line that connects x.sup.(0) and x.sup.(d) and passes through a point x.sup.(b) at a distance b from x.sup.(0), on the same line.

[0119] While the invention has been particularly shown and described as referenced to the embodiments thereof, those skilled in the art will understand that the foregoing and other changes in form and detail may be made therein without departing from the spirit and scope of the invention.