Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for three-dimensionally stacked neural network accelerators. In one aspect, a method includes obtaining data specifying that a tile from a plurality of tiles in a three-dimensionally stacked neural network accelerator is a faulty tile. The three-dimensionally stacked neural network accelerator includes a plurality of neural network dies, each neural network die including a respective plurality of tiles, each tile has input and output connections. The three-dimensionally stacked neural network accelerator is configured to process inputs by routing the input through each of the plurality of tiles according to a dataflow configuration and modifying the dataflow configuration to route an output of a tile before the faulty tile in the dataflow configuration to an input connection of a tile that is positioned above or below the faulty tile on a different neural network die than the faulty tile.

Some embodiments provide a three-dimensional (3D) circuit structure that has two or more vertically stacked bonded layers with a machine-trained network on at least one bonded layer. As described above, each bonded layer can be an IC die or an IC wafer in some embodiments with different embodiments encompassing different combinations of wafers and dies for the different bonded layers. The machine-trained network in some embodiments includes several stages of machine-trained processing nodes with routing fabric that supplies the outputs of earlier stage nodes to drive the inputs of later stage nodes. In some embodiments, the machine-trained network is a neural network and the processing nodes are neurons of the neural network. In some embodiments, one or more parameters associated with each processing node (e.g., each neuron) is defined through machine-trained processes that define the values of these parameters in order to allow the machine-trained network (e.g., neural network) to perform particular operations (e.g., face recognition, voice recognition, etc.). For example, in some embodiments, the machine-trained parameters are weight values that are used to aggregate (e.g., to sum) several output values of several earlier stage processing nodes to produce an input value for a later stage processing node.

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for three-dimensionally stacked neural network accelerators. In one aspect, a method includes obtaining data specifying that a tile from a plurality of tiles in a three-dimensionally stacked neural network accelerator is a faulty tile. The three-dimensionally stacked neural network accelerator includes a plurality of neural network dies, each neural network die including a respective plurality of tiles, each tile has input and output connections. The three-dimensionally stacked neural network accelerator is configured to process inputs by routing the input through each of the plurality of tiles according to a dataflow configuration and modifying the dataflow configuration to route an output of a tile before the faulty tile in the dataflow configuration to an input connection of a tile that is positioned above or below the faulty tile on a different neural network die than the faulty tile.

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for three-dimensionally stacked neural network accelerators. In one aspect, a method includes obtaining data specifying that a tile from a plurality of tiles in a three-dimensionally stacked neural network accelerator is a faulty tile. The three-dimensionally stacked neural network accelerator includes a plurality of neural network dies, each neural network die including a respective plurality of tiles, each tile has input and output connections. The three-dimensionally stacked neural network accelerator is configured to process inputs by routing the input through each of the plurality of tiles according to a dataflow configuration and modifying the dataflow configuration to route an output of a tile before the faulty tile in the dataflow configuration to an input connection of a tile that is positioned above or below the faulty tile on a different neural network die than the faulty tile.

Some embodiments provide a three-dimensional (3D) circuit structure that has two or more vertically stacked bonded layers with a machine-trained network on at least one bonded layer. As described above, each bonded layer can be an IC die or an IC wafer in some embodiments with different embodiments encompassing different combinations of wafers and dies for the different bonded layers. The machine-trained network in some embodiments includes several stages of machine-trained processing nodes with routing fabric that supplies the outputs of earlier stage nodes to drive the inputs of later stage nodes. In some embodiments, the machine-trained network is a neural network and the processing nodes are neurons of the neural network. In some embodiments, one or more parameters associated with each processing node (e.g., each neuron) is defined through machine-trained processes that define the values of these parameters in order to allow the machine-trained network (e.g., neural network) to perform particular operations (e.g., face recognition, voice recognition, etc.). For example, in some embodiments, the machine-trained parameters are weight values that are used to aggregate (e.g., to sum) several output values of several earlier stage processing nodes to produce an input value for a later stage processing node.

Some embodiments of the invention provide an integrated circuit (IC) with a defect-tolerant neural network. The neural network has one or more redundant neurons in some embodiments. After the IC is manufactured, a defective neuron in the neural network can be detected through a test procedure and then replaced by a redundant neuron (i.e., the redundant neuron can be assigned the operation of the defective neuron). The routing fabric of the neural network can be reconfigured so that it re-routes signals around the discarded, defective neuron. In some embodiments, the reconfigured routing fabric does not provide any signal to or forward any signal from the discarded, defective neuron, and instead provides signals to and forwards signals from the redundant neuron that takes the defective neuron's position in the neural network. In some embodiments that implement a neural network by re-purposing (i.e., reconfiguring) one or more individual neurons to implement neurons of multiple stages of the neural network, the IC discards a defective neuron by removing it from the pool of neurons that it configures to perform the operation(s) of neurons in one or more stages of neurons, and assigning this defective neuron's configuration(s) (i.e., its machine-trained parameter set(s)) to a redundant neuron. In some of these embodiments, the IC would re-route around the defective neuron and route to the redundant neuron, by (1) supplying machine-trained parameters and input signals (e.g., previous stage neuron outputs) to the redundant neuron instead of supplying these parameters and signals to the defective neuron, and (2) storing the output(s) of the redundant neuron instead of storing the output(s) of the defective neuron.

Some embodiments provide a three-dimensional (3D) circuit structure that has two or more vertically stacked bonded layers with a machine-trained network on at least one bonded layer. As described above, each bonded layer can be an IC die or an IC wafer in some embodiments with different embodiments encompassing different combinations of wafers and dies for the different bonded layers. The machine-trained network in some embodiments includes several stages of machine-trained processing nodes with routing fabric that supplies the outputs of earlier stage nodes to drive the inputs of later stage nodes. In some embodiments, the machine-trained network is a neural network and the processing nodes are neurons of the neural network. In some embodiments, one or more parameters associated with each processing node (e.g., each neuron) is defined through machine-trained processes that define the values of these parameters in order to allow the machine-trained network (e.g., neural network) to perform particular operations (e.g., face recognition, voice recognition, etc.). For example, in some embodiments, the machine-trained parameters are weight values that are used to aggregate (e.g., to sum) several output values of several earlier stage processing nodes to produce an input value for a later stage processing node.

Some embodiments of the invention provide a three-dimensional (3D) circuit structure that uses latches to transfer signals between two bonded circuit layers. In some embodiments, this structure includes a first circuit partition on a first bonded layer and a second circuit partition on a second bonded layer. It also includes at least one latch to transfer signals between the first circuit partition on the first bonded layer and the second circuit partition on the second bonded layer. In some embodiments, the latch operates in (1) an open first mode that allows a signal to pass from the first circuit partition to the second circuit partition and (2) a closed second mode that maintains the signal passed through during the prior open first mode. By allowing the signal to pass through the first circuit partition to the second circuit partition during its open mode, the latch allows the signal to borrow time from a first portion of a clock cycle of the second circuit partition for a second portion of the clock cycle of the second circuit partition.

Techniques are disclosed for processor synchronization within a reconfigurable computing environment for processor array redundancy. Processing elements are configured within a reconfigurable fabric to implement two or more redundant processors, where the two or more redundant processors are enabled for coincident operation. An agent is loaded on each of the two or more redundant processors, where the agent performs a function requiring data validation. The agent is fired on each of the two or more redundant processors to commence coincident operation. The coincident operation can include a lockstep operation. An output data result from each of the two or more redundant processors is compared to enable a data validation result. The data validation result is propagated. The propagating the data validation result can be based on comparing valid output data or can be based on comparing invalid output data.

A reconfiguration circuit (352) generates a control value for controlling an output value of a control target device based on an input value, and when a fault occurs in an information processing circuit (200) that outputs the control value generated to the control target device, performs preparation for substituting for the information processing circuit (200). Upon completion of the preparation, a reconfiguration target circuit (510) generates an output plan of a control value such that a difference between an actual output value of the control target device and an output value in a scheduled temporal transition gradually decreases, based on the scheduled temporal transition that is a temporal transition of an output value of the control target device, which is scheduled before occurrence of the fault in the information processing circuit (200), a difference between an actual output value of the control target device upon completion of the preparation and an output value in the scheduled temporal transition, and an input value and a control value before occurrence of the fault in the information processing circuit (200), and outputs a control value to the control target device instead of the information processing circuit (200) according to the output plan generated.