COLLECTIVE COMMUNICATION METHOD AND SYSTEM, AND COMPUTER DEVICE

Abstract

This application discloses a collective communication method and system, and a computer device. In one example, a collective communication system includes at least a first computing chip and a second computing chip. The first computing chip compresses first data and sends compressed first data to the second computing chip through a communication channel. The second computing chip performs an operation based on the compressed first data.

Claims

1. A collective communication method, wherein method comprises: compressing, by a first computing chip, first data; sending, by the first computing chip, compressed first data to a second computing chip through a communication channel; and performing, by the second computing chip, an operation based on the compressed first data.

2. The collective communication method according to claim 1, wherein the second computing chip is a root node of the communication channel, and the performing, by the second computing chip, an operation based on the compressed first data comprises: aggregating, by the second computing chip, the first data with compressed second data, wherein the second data is to-be-communicated data on the second computing chip; and sending, by the second computing chip, an aggregation result to the first computing chip.

3. The collective communication method according to claim 1, wherein the second computing chip is a root node of the communication channel, and the performing, by the second computing chip, an operation based on the compressed first data comprises: combining, by the second computing chip, the first data with compressed second data, wherein the second data is to-be-communicated data on the second computing chip; and sending, by the second computing chip, a combination result to the first computing chip.

4. The collective communication method according to claim 1, wherein the first computing chip communicates with the second computing chip through a plurality of communication channels, the plurality of communication channels comprise a first communication channel, and the sending, by the first computing chip, compressed first data to the second computing chip through the communication channel comprises: sending, by the first computing chip, a first part of the first data to the second computing chip through the first communication channel, wherein the second computing chip is a root node of the first communication channel; and the performing, by the second computing chip, an operation based on the compressed first data specifically comprises: aggregating, by the second computing chip, a part of the compressed first data with a part of second data, wherein the second data is to-be-communicated data on the second computing chip.

5. The collective communication method according to claim 1, wherein the first computing chip and the second computing chip is included in a collective communication system, the collective communication system is configured to recommend merchandise to a user by using a recommendation model with reference to a user feature and a merchandise feature, and before the compressing, by the first computing chip, first data, the method comprises: converting, by the first computing chip, the user feature and the merchandise feature into the first data based on an embedding table; and the method further comprises: inputting, by the second computing chip into the recommendation model, an operation result obtained by the second computing chip by performing an operation based on the compressed first data, to obtain an updated value of the embedding table and an updated value of the recommendation model; updating, by the second computing chip, the recommendation model based on the updated value of the recommendation model; and updating, by the second computing chip, the embedding table based on the updated value of the embedding table.

6. The collective communication method according to claim 1, wherein the first computing chip and the second computing chip is included in a collective communication system, the collective communication system is configured to recommend merchandise to a user by using a recommendation model with reference to a user feature and a merchandise feature, and before the compressing, by the first computing chip, first data, the method comprises: converting, by the first computing chip, the user feature and the merchandise feature into fourth data based on an embedding table; and inputting, by the second computing chip, the fourth data into the recommendation model, to obtain the first data and an updated value of the recommendation model; and the method further comprises: updating, by the second computing chip, the recommendation model based on the updated value of the recommendation model; and updating, by the second computing chip, the embedding table based on an operation result obtained by the second computing chip by performing an operation based on the compressed first data.

7. The collective communication method according to claim 1, wherein an amount of data whose value is 0 is greater than an amount of data whose value is not 0 in the first data.

8. The collective communication method according to claim 1, wherein the first computing chip or the second computing chip comprises one or more of a graphics processor, a tensor processor, a neural network processor, or a deep learning processor.

9. A collective communication system, wherein the system comprises at least a first computing chip and a second computing chip; the first computing chip is configured to compress first data, and is further configured to send compressed first data to the second computing chip through a communication channel; and the second computing chip is configured to perform an operation based on the compressed first data.

10. The collective communication system according to claim 9, wherein the second computing chip is a root node of the communication channel, and the second computing chip is further configured to: aggregate the first data with second data, wherein the second data is to-be-communicated data on the second computing chip; and send an aggregation result to the first computing chip.

11. The collective communication system according to claim 9, wherein the second computing chip is a root node of the communication channel, and the second computing chip is further configured to: combine the first data with second data, wherein the second data is to-be-communicated data on the second computing chip; and send a combination result to the first computing chip.

12. The collective communication system according to claim 9, wherein the first computing chip communicates with the second computing chip through a plurality of communication channels, the plurality of communication channels comprise a first communication channel, and the first computing chip is further configured to: send a first part of the first data to the second computing chip through the first communication channel, wherein the second computing chip is a root node of the first communication channel; and the second computing chip is further configured to aggregate a part of the compressed first data with a part of second data, wherein the second data is to-be-communicated data on the second computing chip.

13. The collective communication system according to claim 9, wherein the collective communication system is configured to recommend merchandise to a user by using a recommendation model with reference to a user feature and a merchandise feature, and the first computing chip is further configured to: before the first computing chip compresses the first data, convert the user feature and the merchandise feature into the first data based on an embedding table; and the second computing chip is further configured to input, into the recommendation model, an operation result obtained by the second computing chip by performing an operation based on the compressed first data, to obtain an updated value of the embedding table and an updated value of the recommendation model; update the recommendation model based on the updated value of the recommendation model; and update the embedding table based on the updated value of the embedding table.

14. The collective communication system according to claim 9, wherein the collective communication system is configured to recommend merchandise to a user by using a recommendation model with reference to a user feature and a merchandise feature, and the first computing chip is further configured to: before the first computing chip compresses the first data, convert the user feature and the merchandise feature into fourth data based on an embedding table; before the first computing chip compresses the first data, the second computing chip is further configured to input the fourth data into the recommendation model, to obtain the first data and an updated value of the recommendation model; and the second computing chip is further configured to update the recommendation model based on the updated value of the recommendation model; and update the embedding table based on an operation result obtained by the second computing chip by performing an operation based on the compressed first data.

15. The collective communication system according to claim 9, wherein an amount of data whose value is 0 is greater than an amount of data whose value is not 0 in the first data.

16. The collective communication system according to claim 9, wherein the first computing chip or the second computing chip comprises one or more of a graphics processor, a tensor processor, a neural network processor, or a deep learning processor.

17. A computer device, comprising a processor, a memory, a first computing chip, and a second computing chip, wherein the memory is configured to store computer-executable instructions, and the processor executes the computer-executable instructions in the memory, the computer-executable instructions, when executed, cause the first computing chip and the second computing chip perform following operations: compressing, by the first computing chip, first data; sending, by the first computing chip, compressed first data to the second computing chip through a communication channel; and performing, by the second computing chip, an operation based on the compressed first data.

18. The computer device according to claim 17, wherein the second computing chip is a root node of the communication channel, and the operations comprise: aggregating, by the second computing chip, the first data with compressed second data, wherein the second data is to-be-communicated data on the second computing chip; and sending, by the second computing chip, an aggregation result to the first computing chip.

19. The computer device according to claim 17, wherein the second computing chip is a root node of the communication channel, and the operations comprise: combining, by the second computing chip, the first data with compressed second data, wherein the second data is to-be-communicated data on the second computing chip; and sending, by the second computing chip, a combination result to the first computing chip.

20. The computer device according to claim 17, wherein the first computing chip communicates with the second computing chip through a plurality of communication channels, the plurality of communication channels comprise a first communication channel, and the operations comprise: sending, by the first computing chip, a first part of the first data to the second computing chip through the first communication channel, wherein the second computing chip is a root node of the first communication channel; and aggregating, by the second computing chip, a part of the compressed first data with a part of second data, wherein the second data is to-be-communicated data on the second computing chip.

Description

BRIEF DESCRIPTION OF DRAWINGS

[0020] FIG. 1 is a schematic diagram of a structure of a collective communication system 100 according to an embodiment of this application;

[0021] FIG. 2 is a schematic diagram of a data flow of a data migration operation according to this application;

[0022] FIG. 3 is a schematic diagram of a data flow of a collective operation according to this application;

[0023] FIG. 4 is a schematic flowchart of a collective communication method according to this application;

[0024] FIG. 5 is a schematic flowchart of another collective communication method according to this application;

[0025] FIG. 6 is a schematic flowchart of another collective communication method according to this application;

[0026] FIG. 7 is a schematic flowchart of another collective communication method according to this application;

[0027] FIG. 8 is a schematic flowchart of another collective communication method according to this application; and

[0028] FIG. 9 is a schematic flowchart of another collective communication method according to this application.

DESCRIPTION OF EMBODIMENTS

[0029] The following describes technical solutions in embodiments of this application with reference to the accompanying drawings.

[0030] FIG. 1 is a schematic diagram of a structure of a collective communication system 100 according to an embodiment of this application. As shown in the figure, the system 100 includes a device 110. The device 110 may be a device (for example, a server) with a computing function, and is configured to independently complete a computing task involved in deep learning. The device 110 includes a processor 111, a memory 112, a communication interface 114, and at least two computing chips, for example, a computing chip 1131 and a computing chip 1132.

[0031] In the device 110, the processor 111, the memory 112, the communication interface 114, and all GPUs are connected through a bus, for example, peripheral component interconnect express (PCIe). The bus may also be another type of bus that implements connection between components in the device. In addition, in addition to a data bus, the bus may further include a power bus, a control bus, a status signal bus, and the like. Computing chips may also be interconnected through NVLink? bus proposed by NVIDIA?.

[0032] The processor 111 is configured to execute computer-executable instructions stored in the memory 112, to implement a function of the device 110. For example, the processor 111 may be a CPU, or may be another general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or another programmable logic devices, a discrete gate or a transistor logic device, a discrete hardware component, or the like. The general-purpose processor may be a microprocessor, any conventional processor, or the like.

[0033] The memory 112 may include a read-only memory and a random access memory, and provide instructions and data to the processor 111. The memory 112 may further include a non-volatile random access memory.

[0034] The memory 112 may also be a volatile memory or a non-volatile memory, or may include both a volatile memory and a non-volatile memory. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. Through illustrative but not limited description, many forms of RAMs may be used, for example, a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDR SDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchlink dynamic random access memory (SLDRAM), and a direct rambus random access memory (DR RAM).

[0035] The computing chip is a processor suitable for performing a deep learning algorithm, for example, a graphics processing unit (GPU), a tensor processing unit (TPU), a neural network processing unit (NPU), or a deep learning processing unit (DPU). It should be noted that the device 110 may include at least two computing chips of a same type or different types. For example, as shown in FIG. 1, the device 110 may include two GPUs (a graphics processing unit 1131 and a graphics processing unit 1132), or may include two NPUs, or may include one GPU and one NPU. A plurality of computing chips may concurrently perform a deep learning related algorithm, for example, training and inference of a neural network, and data transmission may be completed between computing chips through a bus by using a collective communication technology.

[0036] Optionally, the system 100 may further include a plurality of devices, such as the device 110 and the device 120. Another device has a similar structure to the device 110. Different devices of the system 100 communicate with each other over a network. The network includes a wired or wireless transmission manner. The wired transmission manner includes use of an Ethernet, an optical fiber, or the like of forms for data transmission, and the wireless transmission manner includes a transmission manner such as a mobile hotspot (Wi-Fi), Bluetooth, and infrared. In a specific implementation process, one or more switches and/or routers may be used to implement communication processing among a plurality of nodes.

[0037] When the system 100 includes a plurality of devices, the device 110 may collaborate with another device to complete a computing task involved in deep learning. In this case, the device 110 may include the processor 111, the memory 112, the communication interface 114, and at least one computing chip. Computing chips of different devices may concurrently perform a deep learning-related algorithm. Data transmission is completed between computing chips in a same device through the bus by using the collective communication technology. Data transmission is completed between computing chips of different devices through the network by using the collective communication technology.

[0038] It should be noted that a management system architecture of a computing resource shown in FIG. 1 is merely an example of a system architecture provided in a management method of the computing resource provided in this application, and does not constitute a limitation on this embodiment of this application.

[0039] Next, based on the system shown in FIG. 1, the collective communication method provided in this application is further described in detail with reference to FIG. 2 to FIG. 9. There are three types of operations for collective communication: a barrier, data migration, and a collective operation.

[0040] The barrier operation is used for synchronizing all processes in a communicator. Execution of processes in barrier operation can be continued only after the barrier operation of all the processes are completed.

[0041] The data migration operation is used for sending data in a process to another process in the communicator, and further includes broadcast, gather, all-gather, and scatter. FIG. 2 is a schematic diagram of a data flow of a data migration operation according to this application. As shown in the figure, the first section from the left is broadcast. Data 1 of node 1 may be sent to the node 1 and the node 2 respectively. The second section from the left is the gather. All the data 1 of the node 1, data 2 of the node 2, and data 3 of node 3 may be sent to the node 1. The third section from the left is all-gather. The data 1 of the node 1 may be sent to the node 2 and the node 3 respectively, the data 2 of the node 2 is sent to the node 1 and the node 3 respectively, and the data 3 of the node 3 is sent to the node 1 and the node 2 respectively. Finally, each of the nodes 1 to 3 has the data 1 to 3. The fourth section from the left on the left is scatter. The data 2 of the node 1 may be sent to the node 2, and the data 3 may be sent to the node 3.

[0042] The collective operation is used for implementing a data arithmetic operation, such as solving minimum and maximum values, summing, a logical AND operation, and another user-defined calculation algorithm. The collective operation includes reduce, all-reduce, and reduce-scatter. FIG. 3 is a schematic diagram of a data flow of a collective operation according to this application. That an arithmetic operation is a summation is used as an example. As shown in the figure, the first section from the left is reduce. Data of node 1, node 2, and node 3 is respectively added and stored in the node 1. The second section from the left is all-reduce, which can be considered as reduce plus broadcast, and a reduce result of the node 1 is sent to all other nodes. The third section from the left is reduce-scatter, which may be considered as reduce plus scatter. Finally, the data of the node 1 is a sum of a first piece of data of the node 1 to the node 3, the data of the node 2 is a sum of second data of the node 1 to the node 3, and the data of the node 3 is a sum of third piece of data of the node 1 to the node 3.

[0043] In a recommendation algorithm, the following collective communication methods are widely used: reduce-scatter, all-reduce, and all-gather. Therefore, this application separately provides a reduce-scatter method, an all-reduce method, and an all-gather method that are applicable to a sparse matrix collective communication method, and combines these methods with the recommendation algorithm. It should be noted that the collective communication method provided in this application is also applicable to a scenario in which sparse data collective communication is performed in another algorithm.

[0044] FIG. 4 is a schematic flowchart of a collective communication method according to this application. Specifically, a reduce-scatter operation can be completed. The method may be performed by the device 110 shown in FIG. 1, or may be jointly performed by the device 110 shown in FIG. 1 and another device. The following describes a procedure of the method by using an example in which the device 110 independently performs the method. As shown in FIG. 4, a specific method includes the following steps.

[0045] S401. A computing chip to be communicated with compresses a to-be-communicated matrix.

[0046] A processor 111 on the device 110 delivers instructions to indicate the computing chip to be communicated with to compress the to-be-communicated matrix on the to-be-communicated computing chip. A compression manner may be a matrix compression manner mastered by a person skilled in the art, for example, any one of the following compression manners may be used.

[0047] Manner 1: Row Compression.

[0048] In the row compression, an original matrix is compressed, to obtain a compression matrix and a compression vector (row offset). The compression vector records a row number of a non-zero row and a total row number of a matrix. The compression matrix records data of a non-zero row corresponding to a row number. For example, for a matrix

[00001]$A=\left(\begin{array}{ccc}1& 2& 3\\ 0& 0& 0\\ 0& 0& 0\\ 7& 6& 5\\ 0& 0& 0\\ 5& 5& 5\\ 0& 0& 0\end{array}\right),$

a compression matrix obtained through the row compression is

[00002]$B=\left(\begin{array}{ccc}1& 2& 3\\ 7& 6& 5\\ 5& 5& 5\end{array}\right)$

and the compression vector is (0 3 5 7). 0 represents a row number corresponding to a row 1 of data (1 2 3) in a vector matrix. Similarly, 3 and 5 respectively represent row numbers corresponding to a row 2 and a row 3 of data in the vector matrix, and 7 represents that a total row number of the original matrix is 7. The row compression manner is lossless compression. Compressed data may be stored consecutively or separately.

[0049] Manner 2: Coordinate List (COO) Compression.

[0050] In the COO compression manner, a non-zero element in each original matrix is represented by a triplet. The triplet includes three vectors: a row number vector, a column number vector, and a numeric vector. The row number vector stores a row number of the non-zero element, the column number vector stores a column number of the non-zero element, and the numeric vector stores a numerical value of the non-zero element. Data locations in the three vectors are in a one-to-one correspondence. For example, for a matrix

[00003]$C=\left(\begin{array}{ccc}0& 0& 1\\ 2& 0& 0\\ 0& 3& 4\end{array}\right),$

there are four non-zero elements in total. The row number vector is (0 1 2 2), the column number vector is (2 0 1 2), and the numeric vector is (1 2 3 4).

[0051] Manner 3: Compressed Sparse Row Format (CSR) Compression.

[0052] In the CSR compression manner, a non-zero element in each original matrix is represented by three types of data, namely, a numeric vector, a column number vector, and a row offset vector. The numeric vector and the column number vector are consistent with those in the COO compression method. A first piece of data in the row offset vector represents a location of a first element in a row 1 of all non-zero elements. For example, for a matrix C, non-zero data in the row 1 is 1, which is the first piece of data in a non-zero element. If the location is 0, the first piece of data in the row offset vector is 0. A second piece of data in the row offset vector represents a location of the first element in a row 2 in all non-zero elements. For example, for the matrix C, the non-zero data of the row 2 is 2, which is the second piece of data in the non-zero element. If the location is 1, the second piece of data in the row offset vector is 1. By analogy, a last piece of data in the row offset vector is a total number of all non-zero elements. For the matrix C, the row offset vector is (0 1 2 4).

[0053] For example, in Example 1, it is assumed that the device 110 includes four to-be-communicated GPUs, which are respectively a GPU 1, a GPU 2, a GPU 3, and a GPU 4, and to-be-communicated matrices on each GPU are as follows:

TABLE-US-00001 GPU 1: GPU 2: GPU 3: GPU 4: {1, 1, 1, 1}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {3, 3, 3, 3}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {4, 4, 4, 4}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {3, 3, 3, 3}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {6, 6, 6, 6}, {0, 0, 0, 0}, {1, 1, 1, 1}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {4, 4, 4, 4}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {7, 7, 7, 7}, {0, 0, 0, 0}, {0, 0, 0, 0}, {5, 5, 5, 5}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {8, 8, 8, 8}, {0, 0, 0, 0}, {1, 1, 1, 1}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {6, 6, 6, 6}, {0, 0, 0, 0}, {1, 1, 1, 1}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {3, 3, 3, 3}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {9, 9, 9, 9}, {0, 0, 0, 0} {0, 0, 0, 0} {8, 8, 8, 8} {0, 0, 0, 0}

[0054] After the row compression is used, to-be-communicated matrices on the GPU 1 to the GPU 4 are respectively as follows:

TABLE-US-00002 GPU 1: GPU 2: GPU 3: GPU 4: {1, 1, 1, 1}, {3, 3, 3, 3}, {6, 6, 6, 6}, {9, 9, 9, 9}, {1, 1, 1, 1}, {4, 4, 4, 4}, {7, 7, 7, 7}, {1, 1, 1, 1}, {3, 3, 3, 3}, {8, 8, 8, 8}, {1, 1, 1, 1} {4, 4, 4, 4}, {6, 6, 6, 6}, {5, 5, 5, 5}, {8, 8, 8, 8} {3, 3, 3, 3},

[0055] A communication vector of each GPU is as follows:

TABLE-US-00003 GPU 1: GPU 2: GPU 3: GPU 4: {0, 5, 10, 11, 16} {1, 2, 3, 6, 8, 12, 16} {4, 7, 9, 11, 15, 16} {14, 16}

[0056] S402. Establish a communication channel between to-be-communicated computing chips.

[0057] The communication channel is established based on a quantity of to-be-communicated computing chips, so that a quantity of communication channels is equal to the quantity of to-be-communicated computing chips. Each communication channel transmits some data of the to-be-communicated matrix. Data of the to-be-communicated matrix before compression may be evenly allocated to each communication channel based on a quantity of rows. Optionally, a quantity of rows of the to-be-communicated matrix transmitted by each communication channel may be dynamically planned based on actual data transmissions in the communication channel.

[0058] For Example 1, four communication channels may be established in an even allocation manner. A first communication channel transmits data from row 0 to row 3 of the to-be-communicated matrix before compression, and a second communication channel transmits data from row 4 to row 7 of the to-be-communicated matrix before compression, and the rest can be deduced by analogy.

[0059] S403. The processor 111 determines a root node from the to-be-communicated computing chip.

[0060] For a reduce-scatter operation, each communication channel has a root node, and is for receiving and sending data of another computing chip in the communication channel and complete an arithmetic operation. The root node may be specified by a user, or may be selected by the processor 111 based on performance of each computing chip, for example, a quantity of cores, a core frequency, a storage speed, a bit width of a video random access memory, and a capacity. A selection method is not limited in this application.

[0061] In Example 1, the GPU 1 may be selected as the root node of a communication channel 1, the GPU 2 may be selected as the root node of a communication channel 2, and the rest can be deduced by analogy.

[0062] S404. The root node receives data of another to-be-communicated computing chip, and obtains a collective operation result.

[0063] For each communication channel, a computing chip to be communicated with of a non-root node scans a compressed matrix, and sends data and a row number that belong to the communication channel in the matrix to a root node of the communication channel.

[0064] For example, in Example 1, in the communication channel 1, the communication channel 1 transmits data from row 0 to row 3 of the to-be-communicated matrix before compression, and the GPU 1 is a root node of the communication channel 1. In this case, after scanning the compression vector, the GPU 2 can learn that in first four rows of data of the original matrix, there is to-be-communicated data in the rows 1, 2, and 3. The GPU 2 sends the data {3,3,3,3}, {4,4,4,4}, and {3,3,3,3} that are corresponding to the rows 1, 2, and 3 and the row numbers 1, 2, and 3 in the compression matrix to the GPU 1. However, after scanning the compression vector, the GPU 3 and the GPU 4 can learn that there is no to-be-communicated data in the first four rows of data in the original matrix. Therefore, the GPU 3 and the GPU 4 do not need to send the data. A sending method of other communication channels may be deduced by analogy.

[0065] The root node may create, a new matrix area in storage space to receive data sent by a computing chip of the non-root node. A quantity of rows of a new matrix is equal to a quantity of rows transmitted by a communication channel. Then, the root node aggregates the data based on the row number with the data of the row based on the original matrix before compression, to obtain a final collective communication reduce-scatter result.

[0066] In Example 1, after receiving data {3,3,3,3}, {4,4,4,4}, and {3,3,3,3} and the row numbers 1, 2, and 3, the GPU 1 separately adds {3,3,3,3}, {4,4,4,4}, and {3,3,3,3} to data in rows 0, 1, 2, and 3 of the original matrix. After receiving data {3,3,3,3}, {6,6,6,6}, and {7,7,7,7} and the row numbers 5, 4, and 7, the GPU 2 separately adds the data to data in rows 4, 5, 6, and 7 of the original matrix. Operations of other GPUs may also be deduced by analogy, and the final collective communication reduce-scatter result is as follows:

TABLE-US-00004 GPU 1: GPU 2: GPU 3: GPU 4: {1, 1, 1, 1}, {6, 6, 6, 6}, {5, 5, 5, 5}, {1, 1, 1, 1}, {3, 3, 3, 3}, {1, 1, 1, 1}, {8, 8, 8, 8}, {3, 3, 3, 3}, {4, 4, 4, 4}, {4, 4, 4, 4}, {1, 1, 1, 1}, {9, 9, 9, 9}, {3, 3, 3, 3}, {7, 7, 7, 7}, {6, 6, 6, 6}, {8, 8, 8, 8}

[0067] Optionally, the root node may directly calculate received data with data of the row corresponding to the original matrix before compression based on the row number, without creating a new storage area. Because the root node can simultaneously receive and calculate data sent by a plurality of computing chips, the data needs to be locked, to avoid performing a plurality of calculations on one piece of data at the same time. The root node may receive one piece of data at a time, and may receive new data only after each piece of data is calculated. Optionally, the root node may also receive a row of data at the same time, and may receive new data only after each row of data is calculated.

[0068] According to the foregoing method, in the collective communication method in this application, the to-be-communicated matrix can be compressed before data is sent, so that invalid transmission of 0 on each channel in the reduce-scatter operation is reduced, and efficiency of the reduce-scatter operation is improved.

[0069] The following describes an all-reduce operation method for collective communication according to an embodiment of this application. FIG. 5 is a schematic flowchart of another collective communication method according to this application. The method may be performed by the device 110 shown in FIG. 1, or may be jointly performed by the device 110 shown in FIG. 1 and another device. The following describes a procedure of the method by using an example in which the device 110 independently performs the method. As shown in FIG. 5, a specific method includes the following steps.

[0070] S501. A computing chip to be communicated with compresses a to-be-communicated matrix, which is similar to S401.

[0071] S502. Establish a communication channel between the to-be-communicated computing chips, which is similar to S402.

[0072] S503. The processor 111 determines a root node from the to-be-communicated computing chip.

[0073] For the all-reduce operation, a same root node may be specified for all communication channels, and is configured to receive and send data of another computing chip and complete an arithmetic operation. A manner of determining a root node is similar to a manner of determining the root node in S403.

[0074] In Example 1, the GPU 1 may be selected as the root node.

[0075] S504. The root node receives data of another to-be-communicated computing chip, and obtains a collective operation result.

[0076] Similar to S404, the root node receives data of another computing chip to be communicated with and completes calculation. A difference is that in the all-reduce operation, root nodes of all communication channels are a same computing chip. Therefore, finally, one computing chip receives and aggregates data of all to-be-communicated computing chips. In Example 1, a calculation result of the root node GPU 1 is as follows:

TABLE-US-00005 GPU 1: {1, 1, 1, 1}, {3, 3, 3, 3}, {4, 4, 4, 4}, {3, 3, 3, 3}, {6, 6, 6, 6}, {1, 1, 1, 1}, {4, 4, 4, 4}, (7, 7, 7, 7}, {5, 5, 5, 5}, {8, 8, 8, 8}, {1, 1, 1, 1}, {6, 6, 6, 6}, {1, 1, 1, 1}, {3, 3, 3, 3}, {9, 9, 9, 9}, {8, 8, 8, 8},

[0077] S505. The root node sends the data to another to-be-communicated computing chip.

[0078] The root node sends, in a broadcast form, a matrix obtained in S504 to another to-be-communicated computing chip, to complete the all-reduce operation. In Example 1, an all-reduce result is as follows:

TABLE-US-00006 GPU 1: GPU 2: GPU 3: GPU 4: {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {3, 3, 3, 3}, {3, 3, 3, 3}, {3, 3, 3, 3}, {3, 3, 3, 3}, {4, 4, 4, 4}, {4, 4, 4, 4}, {4, 4, 4, 4}, {4, 4, 4, 4}, {3, 3, 3, 3}, {3, 3, 3, 3}, {3, 3, 3, 3}, {3, 3, 3, 3}, {6, 6, 6, 6}, {6, 6, 6, 6}, {6, 6, 6, 6}, {6, 6, 6, 6}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {4, 4, 4, 4}, {4, 4, 4, 4}, {4, 4, 4, 4}, {4, 4, 4, 4}, {7, 7, 7, 7}, {7, 7, 7, 7}, {7, 7, 7, 7}, {7, 7, 7, 7}, {5, 5, 5, 5}, {5, 5, 5, 5}, {5, 5, 5, 5}, {5, 5, 5, 5}, {8, 8, 8, 8}, {8, 8, 8, 8}, {8, 8, 8, 8}, {8, 8, 8, 8}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {6, 6, 6, 6}, {6, 6, 6, 6}, {6, 6, 6, 6}, {6, 6, 6, 6}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {3, 3, 3, 3}, {3, 3, 3, 3}, {3, 3, 3, 3}, {3, 3, 3, 3}, {9, 9, 9, 9}, {9, 9, 9, 9}, {9, 9, 9, 9}, {9, 9, 9, 9}, {8, 8, 8, 8} {8, 8, 8, 8} {8, 8, 8, 8} {8, 8, 8, 8}

[0079] According to the foregoing method, in the collective communication method in this application, the to-be-communicated matrix can be compressed before data is sent. Invalid transmission of 0 on each channel in the all-reduce operation is reduced, thereby improving efficiency of the all-reduce operation.

[0080] The following describes an all-gather operation method according to an embodiment of this application. FIG. 6 is a schematic flowchart of another collective communication method according to this application. The method may be performed by the device 110 shown in FIG. 1, or may be jointly performed by the device 110 shown in FIG. 1 and another device. The following describes a procedure of the method by using an example in which the device 110 independently performs the method. As shown in FIG. 6, a specific method includes the following steps.

[0081] S601. A computing chip to be communicated with compresses a to-be-communicated matrix, which is similar to S401.

[0082] S602. Establish a communication channel between the to-be-communicated computing chips, which is similar to S402.

[0083] S603. The processor 111 determines a root node from the to-be-communicated computing chip.

[0084] For the all-gather operation, a same root node may be specified for all communication channels, and is configured to receive and send data of another computing chip. A manner of determining a root node is similar to a manner of determining the root node in S403.

[0085] In Example 1, the GPU 1 may be selected as the root node.

[0086] S604. The root node receives data of another to-be-communicated computing chip, and obtains a combination result.

[0087] The another computing chip to be communicated with may send a compression matrix and a compression vector to the root node. In Example 1, the following may be obtained:

TABLE-US-00007 GPU 1: {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {3, 3, 3, 3}, {4, 4, 4, 4}, {3, 3, 3, 3}, {4, 4, 4, 4}, {5, 5, 5, 5}, {3, 3, 3, 3}, {6, 6, 6, 6}, {7, 7, 7, 7}, {8, 8, 8, 8}, {6, 6, 6, 6}, {8, 8, 8, 8}, {9, 9, 9, 9},

[0088] Correspondingly, all compression vectors may also be obtained:

TABLE-US-00008 GPU 1: {0, 5, 10, 11, 16} {1, 2, 3, 6, 8, 12, 16} {4, 7, 9, 11, 15, 16} {14, 16}

[0089] After obtaining the compression matrix and the compression vector, the root node creates a new matrix in storage space of a chip to store a final collective communication result. A quantity of rows of the new matrix is equal to a product of a total quantity of rows of the compression vector and a quantity of communication channels. Then, the root node sequentially fills each non-zero row of the compression matrix into the new matrix based on the quantity of rows of the compression vector. Finally, rows that are not filled with data are filled with 0.

[0090] The GPU 1 in Example 1 is used as an example. The GPU 1 first creates a new matrix with 64 rows and 4 columns in the storage space of the chip, and then sequentially fills non-zero rows in S603 into the new matrix based on the compression vector. For example, if a row number of a row 1 {1,1,1,1} in the compression vector is 0, {1,1,1,1} is filled in a row 0 in a new matrix. If the row number of a row 2 {1,1,1,1} in the compression vector is 5, {1,1,1,1} is filled in a row 5 in the new matrix, and so on. Finally, rows that are not filled with data are filled with 0, and a result of the GPU 1 may be obtained:

TABLE-US-00009 GPU 1: {1, 1, 1, 1}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {1, 1, 1, 1}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {1, 1, 1, 1}, {0, 0, 0, 0}, {1, 1, 1, 1}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0} {0, 0, 0, 0}, {3, 3, 3, 3}, {4, 4, 4, 4}, {3, 3, 3, 3}, {0, 0, 0, 0}, {0, 0, 0, 0}, {4, 4, 4, 4}, {0, 0, 0, 0}, {5, 5, 5, 5}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {3, 3, 3, 3}, {0, 0, 0, 0}, {0, 0, 0, 0} {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {6, 6, 6, 6}, {0, 0, 0, 0}, {0, 0, 0, 0}, {7, 7, 7, 7}, {0, 0, 0, 0}, {8, 8, 8, 8}, {0, 0, 0, 0}, {6, 6, 6, 6}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {8, 8, 8, 8} {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {9, 9, 9, 9}, {0, 0, 0, 0}

[0091] S605. The root node sends the data to another to-be-communicated computing chip.

[0092] Similar to S505, the root node sends, in a broadcast form, a matrix obtained in S604 to another to-be-communicated computing chip, to complete the all-gather operation.

[0093] According to the foregoing method, in the collective communication method in this application, the to-be-communicated matrix can be compressed before data is sent, so that invalid transmission of 0 in the all-gather operation is reduced, and efficiency of the all-gather operation is improved.

[0094] For a GPU collective communication library NCCL? device that has NVIDIA? installed, operations may be further performed based on collective operations provided in the NCCL? communication library. FIG. 7 is a schematic flowchart of another collective communication method according to this application. Specifically, an all-reduce operation may be performed. The method may be performed by the device 110 shown in FIG. 1, or may be jointly performed by the device 110 shown in FIG. 1 and another device. The following describes a procedure of the method by using an example in which the device 110 independently performs the method. As shown in FIG. 7, a specific method includes the following steps.

[0095] S701. A computing chip to be communicated with compresses a to-be-communicated matrix, which is similar to S401.

[0096] S702. A processor 111 obtains a maximum size of a compression matrix.

[0097] The processor 111 may directly invoke an interface of an all-reduce function in an NCCL? communication library to obtain a quantity of rows of a compression matrix on each to-be-communicated computing chip. Optionally, each computing chip to be communicated with may further traverse the compression matrix, and send the quantity of rows to the processor 111. Optionally, each computing chip to be communicated with may further directly read the quantity of rows from a compression vector, and send the quantity of rows to the processor 111.

[0098] The processor 111 fills, based on a maximum value of the quantity of rows, a compression matrix in which the quantity of rows does not reach the maximum value on the computing chip with 0, so that the quantity of rows reaches the maximum value. A compressed matrix in Example 1 is used as an example. The quantity of rows of the GPU 1 to the GPU 4 are respectively 4, 6, 5, and 1. A quantity of rows of the GPU 2 is the largest. Therefore, the remaining GPUs fill the quantity of rows with 6 by using data 0. A result is as follows:

TABLE-US-00010 GPU 1: GPU 2: GPU 3: GPU 4: {1, 1, 1, 1}, {3, 3, 3, 3}, {6, 6, 6, 6}, {9, 9, 9, 9}, {1, 1, 1, 1}, {4, 4, 4, 4}, {7, 7, 7, 7}, {0, 0, 0, 0}, {1, 1, 1, 1}, {3, 3, 3, 3}, {8, 8, 8, 8}, {0, 0, 0, 0}, {1, 1, 1, 1}, {4, 4, 4, 4}, {6, 6, 6, 6}, {0, 0, 0, 0}, {0, 0, 0, 0}, {5, 5, 5, 5}, {8, 8, 8, 8}, {0, 0, 0, 0}, {0, 0, 0, 0} {3, 3, 3, 3}, {0, 0, 0, 0} {0, 0, 0, 0}

[0099] S703. The processor 111 invokes an all-gather interface.

[0100] The processor 111 may invoke the all-gather interface in the NCCL? communication library to send the compression matrix and the compression vector that are on each to-be-communicated GPU to each other to another computing chip, so that each computing chip can obtain all the compression matrices. In Example 1, the following may be obtained:

TABLE-US-00011 GPU 1: GPU 2: GPU 3: GPU 4: {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {3, 3, 3, 3}, {3, 3, 3, 3}, {3, 3, 3, 3}, {3, 3, 3, 3}, {4, 4, 4, 4}, {4, 4, 4, 4}, {4, 4, 4, 4}, {4, 4, 4, 4}, {3, 3, 3, 3}, {3, 3, 3, 3}, {3, 3, 3, 3}, {3, 3, 3, 3}, {4, 4, 4, 4}, {4, 4, 4, 4}, {4, 4, 4, 4}, {4, 4, 4, 4}, {5, 5, 5, 5}, {5, 5, 5, 5}, {5, 5, 5, 5}, {5, 5, 5, 5}, {3, 3, 3, 3}, {3, 3, 3, 3}, {3, 3, 3, 3}, {3, 3, 3, 3}, {6, 6, 6, 6}, {6, 6, 6, 6}, {6, 6, 6, 6}, {6, 6, 6, 6}, {7, 7, 7, 7}, {7, 7, 7, 7}, {7, 7, 7, 7}, {7, 7, 7, 7}, {8, 8, 8, 8}, {8, 8, 8, 8}, {8, 8, 8, 8}, {8, 8, 8, 8}, {6, 6, 6, 6}, {6, 6, 6, 6}, {6, 6, 6, 6}, {6, 6, 6, 6}, {8, 8, 8, 8}, {8, 8, 8, 8}, {8, 8, 8, 8}, {8, 8, 8, 8}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {9, 9, 9, 9}, {9, 9, 9, 9}, {9, 9, 9, 9}, {9, 9, 9, 9}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0} {0, 0, 0, 0} {0, 0, 0, 0} {0, 0, 0, 0}

[0101] Corresponding to each GPU, all compression vectors may also be obtained:

TABLE-US-00012 GPU 1: GPU 2: {0, 5, 10, 11, 16} {0, 5, 10, 11, 16} {1, 2, 3, 6, 8, 12, 16} {1, 2, 3, 6, 8, 12, 16} {4, 7, 9, 11, 15, 16} {4, 7, 9, 11, 15, 16} {14, 16} {14, 16} GPU 3: GPU 4: {0, 5, 10, 11, 16} {0, 5, 10, 11, 16} {1, 2, 3, 6, 8, 12, 16} {1, 2, 3, 6, 8, 12, 16} {4, 7, 9, 11, 15, 16} {4, 7, 9, 11, 15, 16} {14, 16} {14, 16}

[0102] S704. The computing chip to be communicated with performs an operation based on the compression vector.

[0103] After obtaining the compression matrix and the compression vector, the computing chip creates a new matrix in storage space of the chip to store a final collective communication result. A quantity of rows of the new matrix is equal to a total quantity of rows of the compression vector. Then, the computing chip sequentially fills each non-zero row of the compression matrix into the new matrix based on the quantity of rows of the compression vector. If two rows of data in the compression matrix correspond to a same row number, the computing chip fills a computing result of the two rows of data into the new matrix in an operation manner.

[0104] The GPU 1 in Example 1 is used as an example. The GPU 1 first creates a new matrix with 16 rows and 4 columns in the storage space of the chip, and then sequentially fills non-zero rows in S703 into the new matrix based on the compression vector. For example, if a row number of a row 1 {1,1,1,1} in the compression vector is 0, {1,1,1,1} is filled in a row 0 in a new matrix. If the row number of a row 2 {1,1,1,1} in the compression vector is 5, {1,1,1,1} is filled in a row 5 in the new matrix, and so on. Finally, an all-reduce result of all GPUs may be obtained:

TABLE-US-00013 GPU 1: GPU 2: GPU 3: GPU 4: {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {3, 3, 3, 3}, {3, 3, 3, 3}, {3, 3, 3, 3}, {3, 3, 3, 3}, {4, 4, 4, 4}, {4, 4, 4, 4}, {4, 4, 4, 4}, {4, 4, 4, 4}, {3, 3, 3, 3}, {3, 3, 3, 3}, {3, 3, 3, 3}, {3, 3, 3, 3}, {6, 6, 6, 6}, {6, 6, 6, 6}, {6, 6, 6, 6}, {6, 6, 6, 6}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {4, 4, 4, 4}, {4, 4, 4, 4}, {4, 4, 4, 4}, {4, 4, 4, 4}, {7, 7, 7, 7}, {7, 7, 7, 7}, {7, 7, 7, 7}, {7, 7, 7, 7}, {5, 5, 5, 5}, {5, 5, 5, 5}, {5, 5, 5, 5}, {5, 5, 5, 5}, {8, 8, 8, 8}, {8, 8, 8, 8}, {8, 8, 8, 8}, {8, 8, 8, 8}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {6, 6, 6, 6}, {6, 6, 6, 6}, {6, 6, 6, 6}, {6, 6, 6, 6}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {3, 3, 3, 3}, {3, 3, 3, 3}, {3, 3, 3, 3}, {3, 3, 3, 3}, {9, 9, 9, 9}, {9, 9, 9, 9}, {9, 9, 9, 9}, {9, 9, 9, 9}, {8, 8, 8, 8} {8, 8, 8, 8} {8, 8, 8, 8} {8, 8, 8, 8}

[0105] According to the foregoing method, in the collective communication method in this application, an existing communication library can be directly invoked, and an operation is simple. In addition, invalid transmission of 0 on each channel in an all-reduce operation can be reduced by compressing a matrix, thereby improving efficiency of the all-reduce operation.

[0106] In a recommendation system, because data of a user feature and data of a merchandise feature include a large amount of data with a value of 0 and having extremely high sparseness, embedding is a core operation of the recommendation system. The embedding operation is mainly used for using a matrix to convert a sparse vector (when a quantity of elements with a value of 0 in a vector is greater than a quantity of elements with a value of non-zero, the vector is called a sparse vector) into a dense vector. A converted vector is referred to as an embedding vector, and the matrix used for conversion is referred to as an embedding table. For example, in formula 1, two 5-dimensional characteristic vectors are converted into two 3-dimensional vectors:

[00004]$\begin{array}{cc}\left(\begin{array}{ccccc}1& 0& 1& 0& 0\\ 0& 1& 0& 0& 0\end{array}\right)\left(\begin{array}{ccc}{w}_{11}& w_{12}& w_{13}\\ w_{21}& w_{21}& w_{23}\\ w_{31}& w_{32}& w_{33}\\ w_{41}& w_{41}& w_{43}\\ w_{51}& w_{52}& w_{53}\end{array}\right)=\left(\begin{array}{ccc}{w}_{11}+w_{31}& w_{12}+w_{32}& w_{13}+w_{33}\\ w_{21}& w_{21}& w_{23}\end{array}\right)& \mathrm{Formula}1\end{array}$

[0107] In the recommendation system, each iteration of training a recommendation model may include two processes: forward propagation and backpropagation. The forward propagation is used for obtaining a result obtained after conversion of the embedding table and calculation of the recommendation model are performed on input data. Backpropagation is used for obtaining the recommendation model and update of the embedding table based on a difference between a calculated result and an actual value.

[0108] With increasing complexity of a model and an increasing amount of data, a size of the embedding table has reached a level of 100 GB to TB, and a level of 10 TB is coming. Because an amount of data of a complete embedding table is excessively large, the data is generally stored by a plurality of computing chips, and each computing chip stores a part of the data of the embedding table. Therefore, in a forward propagation process, a used embedding vector needs to be queried from embedding tables stored in the plurality of computing chips based on a query vector (batch), and data of a corresponding row needs to be obtained. An embedding vector in this iteration process may be combined through a reduce-scatter or all-reduce operation in collective communication. Finally, the embedding vector is input into the recommendation model for training.

[0109] In addition, because a calculation amount is excessively large, the recommendation model is stored in different computing chips. After each iteration of calculation, each computing chip can obtain only a part of updated values of the embedding vector. Therefore, in a backpropagation process, the updated values of the embedding vector may be summarized and sent to all computing chips through an all-gather operation in the collective communication, to obtain final updated values of the embedding table.

[0110] FIG. 8 is a schematic flowchart of another collective communication method according to this application. The method may be performed by the device 110 shown in FIG. 1, or may be jointly performed by the device 110 shown in FIG. 1 and another device. The following describes a procedure of the method by using an example in which the device 110 independently performs the method. As shown in FIG. 8, a specific method is as follows:

[0111] S801. A computing chip obtains a query vector based on a user feature and a merchandise feature.

[0112] The query vector stores a correspondence between rows of an embedding table and rows of an embedding vector. An amount of data in the query vector is a quantity of rows of the embedding vector, a location of each piece of data is a row of the embedding vector, and a size of each piece of data is a row of the embedding table. For example, when the query vector is {1,2,3,4}, the embedding vector is a 4-row matrix. A first piece of data is 1, indicating that a row 1 of the embedding vector is data in a row 1 of the embedding table. A second piece of data is 2, indicating that a row 2 of the embedding vector is data in a row 2 of the embedding table.

[0113] S802. The computing chip obtains the embedding vector from the embedding table based on the query vector.

[0114] Specifically, this step can be divided into two steps, including:

[0115] S8021. The computing chip obtains a to-be-communicated matrix of each computing chip through the query vector.

[0116] Each computing chip may first create a to-be-communicated matrix in which all data is 0. A quantity of rows in the to-be-communicated matrix is equal to a total amount of data in the query vector. In this case, each computing chip extracts data of a corresponding row from a locally stored embedding table based on the data in the query vector, to form a to-be-communicated matrix.

[0117] For example, in Example 2, it is assumed that the device 110 includes four to-be-communicated GPUs, which are respectively a GPU 1, a GPU 2, a GPU 3, and a GPU 4, and embedding tables stored in each GPU are respectively:

TABLE-US-00014 GPU 1: GPU 2: GPU 3: GPU 4: 1{0, 0, 0, 0}, 4{3, 3, 3, 3}, 7{6, 6, 6, 6}, 10{9, 9, 9, 9}, 2{1, 1, 1, 1}, 5{4, 4, 4, 4}, 8{7, 7, 7, 7}, 11{10, 10, 10, 10}, 3{2, 2, 2, 2}, 6{5, 5, 5, 5}, 9{8, 8, 8, 8}, 12{11, 11, 11, 11},

[0118] A number before each GPU matrix represents a row number of the embedding table. For the GPU 1, rows 1, 2, and 3 of a complete embedding table are stored, and data in each row is {0,0,0,0}, {1,1,1,1}, and {2,2,2,2}.

[0119] It is assumed that the query vector is: {2,4,5,4,7,2,5,8,6,9,2,7,2,4,10,9}, 16 rows in total. The GPU 1 to the GPU 4 first separately create a matrix of 16 rows and 4 columns in which all data is 0. Based on the query vector, if a row 2 of the embedding table is corresponding to rows 1, 6, 11, and 13 of the embedding vector, the GPU 1 extracts data of the row 2 from the locally stored embedding table, and fills the data of the row 2 into the rows 1, 6, 11, and 13 of the matrix respectively. A row 4 of the embedding table corresponds to rows 2, 4, and 14 of the embedding vector. A row 5 of the embedding table corresponds to rows 3 and 7 of the embedding vector. A row 6 of the embedding table corresponds to a row 6 of the embedding vector. In this case, the GPU 2 extracts data of the row 4 from the locally stored embedding table and fills the data into the rows 2, 4, and 14 of the matrix, extracts data of the row 5 and fills the data into the rows 3 and 7 of the matrix, and extracts data of the row 6 and fills the data into the row 6 of the matrix, and so on. Finally, to-be-communicated matrices of each GPU are obtained:

TABLE-US-00015 GPU 1: GPU 2: GPU 3: GPU 4: {1, 1, 1, 1}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {3, 3, 3, 3}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {4, 4, 4, 4}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {3, 3, 3, 3}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {6, 6, 6, 6}, {0, 0, 0, 0}, {1, 1, 1, 1}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {4, 4, 4, 4}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {7, 7, 7, 7}, {0, 0, 0, 0}, {0, 0, 0, 0}, {5, 5, 5, 5}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {8, 8, 8, 8}, {0, 0, 0, 0}, {1, 1, 1, 1}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {6, 6, 6, 6}, {0, 0, 0, 0}, {1, 1, 1, 1}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {3, 3, 3, 3}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {9, 9, 9, 9}, {0, 0, 0, 0} {0, 0, 0, 0} {8, 8, 8, 8} {0, 0, 0, 0}

[0120] S8022. The computing chip obtains the embedding vector by using the reduce-scatter or all-reduce operation provided in this application.

[0121] When a recommendation model is divided into a plurality of parts, and training is separately performed by a plurality of computing chips, the reduce-scatter operation provided in FIG. 4 in this application may be used. Each communicating computing chip may obtain a part of values of the embedding vector, and input the part of values into the recommendation model on the computing chip to perform a next step of calculation.

[0122] When the recommendation model is completely trained by one computing chip or by a plurality of computing chips at the same time, the all-reduce operation provided in FIG. 5 or FIG. 7 in this application may be used for obtaining a complete embedding vector. The embedding vector is input into the recommendation model for next calculation.

[0123] S803. The computing chip inputs the embedding vector into the recommendation model for calculation, to obtain a calculation result.

[0124] S804. Calculate a loss function between the calculation result and a real value.

[0125] S805. Obtain an updated value of the embedding vector through calculation performed through the loss function.

[0126] S806. The computing chip obtains an updated value of the embedding table based on the updated value of the embedding vector.

[0127] Specifically, this step can be divided into two steps, including:

[0128] S8061. The computing chip obtains an updated value of the complete embedding vector by using the all-gather operation provided in this application. When the recommendation model is divided into a plurality of parts, and training is separately performed by a plurality of computing chips, each computing chip can obtain only a part of updated values of the embedding vector. The all-gather operation provided in FIG. 6 in this application may be used, so that each computing chip obtains the updated value of the complete embedding vector.

[0129] Example 2 is used as an example. It is assumed that updated values of the embedding vector obtained on four GPUs are respectively:

TABLE-US-00016 GPU 1: GPU 2: {0.1, 0.1, 0.1, 0.1}, {0.2, 0.2, 0.2, 0.2}, {0.1, 0.1, 0.1, 0.1}, {0.3, 0.3, 0.3, 0.3}, {0.2, 0.2, 0.2, 0.2}, {0, 0, 0, 0}, {0.3, 0.3, 0.3, 0.3}, {0.1, 0.1, 0.1, 0.1}, GPU 3: GPU 4: {0, 0, 0, 0}, {0.1, 0.1, 0.1, 0.1}, {0.8, 0.8, 0.8, 0.8}, {0, 0, 0, 0}, {0.1, 0.1, 0.1, 0.1}, {0.9, 0.9, 0.9, 0.9}, {0.6, 0.6, 0.6, 0.6}, {0.8, 0.8, 0.8, 0.8}

[0130] After the all-gather operation provided in FIG. 6 in this application, the following may be obtained:

TABLE-US-00017 GPU 1: GPU 2: {0.1, 0.1, 0.1, 0.1}, {0.1, 0.1, 0.1, 0.1}, {0.1, 0.1, 0.1, 0.1}, {0.1, 0.1, 0.1, 0.1}, {0.2, 0.2, 0.2, 0.2}, {0.2, 0.2, 0.2, 0.2}, {0.3, 0.3, 0.3, 0.3}, {0.3, 0.3, 0.3, 0.3}, {0.2, 0.2, 0.2, 0.2}, {0.2, 0.2, 0.2, 0.2}, {0.3, 0.3, 0.3, 0.3}, {0.3, 0.3, 0.3, 0.3}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0.1, 0.1, 0.1, 0.1}, {0.1, 0.1, 0.1, 0.1}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0.8, 0.8, 0.8, 0.8}, {0.8, 0.8, 0.8, 0.8}, {0.1, 0.1, 0.1, 0.1}, {0.1, 0.1, 0.1, 0.1}, {0.6, 0.6, 0.6, 0.6}, {0.6, 0.6, 0.6, 0.6}, {0.1, 0.1, 0.1, 0.1}, {0.1, 0.1, 0.1, 0.1}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0.9, 0.9, 0.9, 0.9}, {0.9, 0.9, 0.9, 0.9}, {0.8, 0.8, 0.8, 0.8} {0.8, 0.8, 0.8, 0.8} GPU 3: GPU 4: {0.1, 0.1, 0.1, 0.1}, {0.1, 0.1, 0.1, 0.1}, {0.1, 0.1, 0.1, 0.1}, {0.1, 0.1, 0.1, 0.1}, {0.2, 0.2, 0.2, 0.2}, {0.2, 0.2, 0.2, 0.2}, {0.3, 0.3, 0.3, 0.3}, {0.3, 0.3, 0.3, 0.3}, {0.2, 0.2, 0.2, 0.2}, {0.2, 0.2, 0.2, 0.2}, {0.3, 0.3, 0.3, 0.3}, {0.3, 0.3, 0.3, 0.3}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0.1, 0.1, 0.1, 0.1}, {0.1, 0.1, 0.1, 0.1}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0.8, 0.8, 0.8, 0.8}, {0.8, 0.8, 0.8, 0.8}, {0.1, 0.1, 0.1, 0.1}, {0.1, 0.1, 0.1, 0.1}, {0.6, 0.6, 0.6, 0.6}, {0.6, 0.6, 0.6, 0.6}, {0.1, 0.1, 0.1, 0.1}, {0.1, 0.1, 0.1, 0.1}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0.9, 0.9, 0.9, 0.9}, {0.9, 0.9, 0.9, 0.9}, {0.8, 0.8, 0.8, 0.8} {0.8, 0.8, 0.8, 0.8}

[0131] S8062. The computing chip obtains the updated value of the embedding table based on the query vector.

[0132] Based on a row of the embedding vector that is in the query vector and that is corresponding to a row of the embedding table stored in each computing chip, an updated value of each row of the embedding table is obtained from the updated value of the embedding vector. When the row of the embedding table corresponds to a plurality of rows of the embedding vector in the query vector, data of all rows of the embedding vector is added up and used as the row of the embedding table.

[0133] In Example 2, the query vector is: {2,4,5,4,7,2,5,8,6,9,2,7,2,4,10,9}. A row 2 of the embedding table is corresponding to rows 1, 6, 11, and 13 of the embedding vector. In this case, the GPU 1 extracts data in the rows 1, 6, 11, and 13 from the obtained updated value of the embedding table, {0.1,0.1,0.1,0.1}, {0.3,0.3,0.3,0.3}, {0.8,0.8,0.8,0.8}, and {0.1,0.1,0.1,0.1}, which are added up as an updated value of the row 2 of the embedding table, {1.3,1.3,1.3,1.3}. By analogy, updated values of the embedding table are finally obtained:

TABLE-US-00018 GPU 1: GPU 2: {0, 0, 0, 0}, {0.4, 0.4, 0.4, 0.4}, {1.3, 1.3, 1.3, 1.3}, {0.3, 0.3, 0.3, 0.3}, {0, 0, 0, 0}, {0, 0, 0, 0}, GPU 3: GPU 4: {0.8, 0.8, 0.8, 0.8}, {0.9, 0.9, 0.9, 0.9}, {0.1, 0.1, 0.1, 0.1}, {0, 0, 0, 0}, {1.6, 1.6, 1.6, 1.6}, {0, 0, 0, 0},

[0134] S807. The computing chip updates, based on the updated value of the embedding table, the embedding table stored in the computing chip.

[0135] According to the foregoing method, in the collective communication method in this application, a matrix can be compressed in an embedding vector combination process and an embedding table update process, thereby improving efficiency of collective communication and reducing time used in a training process of the recommendation model.

[0136] FIG. 9 is a schematic flowchart of another collective communication method according to this application. The method may be performed by the device 110 shown in FIG. 1, or may be jointly performed by the device 110 shown in FIG. 1 and another device. The following describes a procedure of the method by using an example in which the device 110 independently performs the method. Compared with the method in FIG. 8, only steps S902 and S906 in the method in FIG. 9 are different from the steps in FIG. 8, and other steps are similar to the steps in FIG. 8. A specific method is as follows:

[0137] S901. A computing chip obtains a query vector based on a user feature and a merchandise feature.

[0138] S902. The computing chip obtains the embedding vector from the embedding table based on the query vector.

[0139] Specifically, this step can be divided into three steps, including:

[0140] S9021. The computing chip compresses the query vector to obtain a compression vector of the query vector and a recovery vector of the query vector. Repeated elements in the query vector are removed, and a location of each piece of data in a compressed query vector in the query vector before compression is recorded through the recovery vector of the query vector.

[0141] As shown in Example 2, the query vector is: {2,4,5,4,7,2,5,8,6,9,2,7,2,4,10,9}. The compressed query vector is {2,4,5,7,8,6,9,10}, and the recovery vector of the query vector is {{1,6,11}, {2,3,14}, {3,7}, {5,12}, {8}, {9}, {10,16}, {15}}, which indicates that 1 appears at locations 1, 6, and 11 of the query vector before compression, 3 appears at locations 2, 4, and 14 of the query vector before compression, and so on.

[0142] S9022. The computing chip obtains a to-be-communicated matrix of each computing chip based on the compressed query vector, which is similar to S8021. In Example 2, to-be-communicated matrices of each GPU may be obtained:

TABLE-US-00019 GPU 1: GPU 2: GPU 3: GPU 4: {1, 1, 1, 1}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {3, 3, 3, 3}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {4, 4, 4, 4}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {6, 6, 6, 6}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {7, 7, 7, 7}, {0, 0, 0, 0}, {0, 0, 0, 0}, {5, 5, 5, 5}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {8, 8, 8, 8}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {9, 9, 9, 9}

[0143] S9023. The computing chip obtains a compressed embedding vector by using the all-reduce operation provided in this application.

[0144] The all-reduce operation provided in FIG. 5 or FIG. 7 in this application may be used for obtaining the compressed embedding vector. In Example 2, the obtained compressed embedding vector is as follows:

TABLE-US-00020 GPU 1: GPU 2: GPU 3: GPU 4: {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {3, 3, 3, 3}, {3, 3, 3, 3}, {3, 3, 3, 3}, {3, 3, 3, 3}, {4, 4, 4, 4}, {4, 4, 4, 4}, {4, 4, 4, 4}, {4, 4, 4, 4}, {6, 6, 6, 6}, {6, 6, 6, 6}, {6, 6, 6, 6}, {6, 6, 6, 6}, {7, 7, 7, 7}, {7, 7, 7, 7}, {7, 7, 7, 7}, {7, 7, 7, 7}, {5, 5, 5, 5}, {5, 5, 5, 5}, {5, 5, 5, 5}, {5, 5, 5, 5}, {8, 8, 8, 8}, {8, 8, 8, 8}, {8, 8, 8, 8}, {8, 8, 8, 8}, {9, 9, 9, 9}, {9, 9, 9, 9}, {9, 9, 9, 9}, {9, 9, 9, 9},

[0145] S9024. The computing chip restores the compressed embedding vector based on the recovery vector of the query vector.

[0146] The computing chip may first create a new matrix to store a final embedding vector. A quantity of rows of the new matrix is equal to a total amount of data of the query vector before compression. Then, the computing chip sequentially determines a location of data of each row in the compressed embedding vector in the recovery vector of the query vector based on a row number, and further determines a location of the data of the row in an original query vector. Finally, the computing chip fills the data of the row into a final matrix of the embedding vector based on the location in the original query vector.

[0147] The GPU 1 in Example 2 is used as an example. The GPU 1 first creates, in storage space of the chip, a matrix with 16 rows and 4 columns of a new embedding vector, and then sequentially determines, based on the compression vector, a location that is in the original query vector and that is corresponding to the data of each row in S9023. For example, if a row number of a row 1 {1,1,1,1} in the compressed embedding vector is 1, the row 1 is a first piece of data in the recovery vector of the query vector. Corresponding data content is {1,6,11}, indicating that data of the row is in rows 1, 6, and 11 in the original query vector. In this case, a computing matrix fills {1,1,1,1} into rows 1, 6, and 11 of the new matrix, and so on. Finally, an all-reduce result of all GPUs may be obtained:

TABLE-US-00021 GPU 1: GPU 2: GPU 3: GPU 4: {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {3, 3, 3, 3}, {3, 3, 3, 3}, {3, 3, 3, 3}, {3, 3, 3, 3}, {4, 4, 4, 4}, {4, 4, 4, 4}, {4, 4, 4, 4}, {4, 4, 4, 4}, {3, 3, 3, 3}, {3, 3, 3, 3}, {3, 3, 3, 3}, {3, 3, 3, 3}, {6, 6, 6, 6}, {6, 6, 6, 6}, {6, 6, 6, 6}, {6, 6, 6, 6}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {4, 4, 4, 4}, {4, 4, 4, 4}, {4, 4, 4, 4}, {4, 4, 4, 4}, {7, 7, 7, 7}, {7, 7, 7, 7}, {7, 7, 7, 7}, {7, 7, 7, 7}, {5, 5, 5, 5}, {5, 5, 5, 5}, {5, 5, 5, 5}, {5, 5, 5, 5}, {8, 8, 8, 8}, {8, 8, 8, 8}, {8, 8, 8, 8}, {8, 8, 8, 8}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {6, 6, 6, 6}, {6, 6, 6, 6}, {6, 6, 6, 6}, {6, 6, 6, 6}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {1, 1, 1, 1}, {3, 3, 3, 3}, {3, 3, 3, 3}, {3, 3, 3, 3}, {3, 3, 3, 3}, {9, 9, 9, 9}, {9, 9, 9, 9}, {9, 9, 9, 9}, {9, 9, 9, 9}, {8, 8, 8, 8} {8, 8, 8, 8} {8, 8, 8, 8} {8, 8, 8, 8}

[0148] S903. The computing chip inputs the embedding vector into a recommendation model for calculation, to obtain a calculation result.

[0149] S904. Calculate a loss function between the calculation result and a real value.

[0150] S905. Obtain an updated value of the embedding vector through calculation performed through the loss function.

[0151] S906. The computing chip obtains an updated value of the embedding table based on the updated value of the embedding vector.

[0152] Specifically, this step can be divided into two steps, including:

[0153] S9061. The computing chip removes duplicate data from the updated value of the embedding vector based on the query vector.

[0154] Similar to S8061, when the recommendation model is divided into a plurality of parts, and training is separately performed by a plurality of computing chips, each computing chip can obtain only a part of updated values of the embedding vector. As described above, a location of each piece of data in the query vector is a row of the embedding vector, and a size of each piece of data is a row of the embedding table. After the computing chip may traverse data corresponding to locations of rows of a part of the embedding vector in the query vector, updated values of rows that are of the embedding vector and that are corresponding to locations with same values of data in the query vector are added up, to be written into any row of rows corresponding to the embedding vector, and other rows are assigned with 0.

[0155] The GPU 1 in Example 2 is used as an example, and updated values of the embedding vector are as follows:

TABLE-US-00022 GPU 1: GPU 2: {0.1, 0.1, 0.1, 0.1}, {0.2, 0.2, 0.2, 0.2}, {0.1, 0.1, 0.1, 0.1}, {0.3, 0.3, 0.3, 0.3}, {0.2, 0.2, 0.2, 0.2}, {0, 0, 0, 0}, {0.3, 0.3, 0.3, 0.3}, {0.1, 0.1, 0.1, 0.1}, GPU 3: GPU 4: {0, 0, 0, 0}, {0.1, 0.1, 0.1, 0.1}, {0.8, 0.8, 0.8, 0.8}, {0, 0, 0, 0}, {0.1, 0.1, 0.1, 0.1}, {0.9, 0.9, 0.9, 0.9}, {0.6, 0.6, 0.6, 0.6}, {0.8, 0.8, 0.8, 0.8}

[0156] The query vector is {2,4,5,4,7,2,5,8,6,9,2,7,2,4,10,9}, a part of embedding vector obtained by the GPU 1 is row 1 to row 4 of a complete embedding vector, and the GPU 1 traverses a first to a fourth piece of data {2,4,5,4} of the query vector. Values of the second piece of data and the fourth piece of data are both 4. Therefore, data in the row 2 and the row 4 of the embedding vector in the GPU 1 may be added up, to be written into to the row 2, and the data of the row 4 is assigned with a value 0. By analogy, the updated values of the embedding vector are obtained:

TABLE-US-00023 GPU 1: GPU 2: {0.1, 0.1, 0.1, 0.1}, {0.2, 0.2, 0.2, 0.2}, {0.4, 0.4, 0.4, 0.4}, {0.3, 0.3, 0.3, 0.3}, {0.2, 0.2, 0.2, 0.2}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0.1, 0.1, 0.1, 0.1}, GPU 3: GPU 4: {0, 0, 0, 0}, {0.1, 0.1, 0.1, 0.1}, {0.8, 0.8, 0.8, 0.8}, {0, 0, 0, 0}, {0.1, 0.1, 0.1, 0.1}, {0.9, 0.9, 0.9, 0.9}, {0.6, 0.6, 0.6, 0.6}, {0.8, 0.8, 0.8, 0.8}

[0157] Because data of a same value in the query vector corresponds to a same row in the embedding table, a plurality of rows of data of the embedding vector corresponding to the query vector may be converted into one row in advance, thereby increasing sparseness of a to-be-communicated matrix in a next step, and improving collective communication efficiency. When more data is duplicated in the query vector, communication efficiency is improved.

[0158] S9062. The computing chip obtains an updated value of a complete embedding vector by using the all-gather operation provided in this application, which is similar to S8062.

[0159] S9063. The computing chip obtains the updated value of the embedding table based on the query vector, which is similar to S8062.

[0160] S907. The computing chip updates, based on the updated value of the embedding table, the embedding table stored in the computing chip.

[0161] According to the foregoing method, in the collective communication method in this application, an embedding vector that needs to be communicated can be compressed to a maximum extent, thereby further reducing an amount of data transmitted in communication, improving efficiency of collective communication, and reducing time used in a training process of the recommendation model.

[0162] It should be noted that, for ease of description, the foregoing method embodiments are described as a series of actions. However, a person skilled in the art should appreciate that this application is not limited to the described sequence of the actions.

[0163] Another appropriate step combination that can be figured out by a person skilled in the art according to the content described above also falls within the protection scope of this application. In addition, a person skilled in the art should also be familiar to embodiments described in this specification all belong to example embodiments, and involved actions are not necessarily mandatory in this application.

[0164] This application further provides a collective communication system, which may be the system 100 shown in FIG. 1. The collective communication system includes at least a first computing chip and a second computing chip. The first computing chip communicates with the second computing chip through at least one communication channel. For example, the first computing chip and the second computing chip may be the computing chip 1131 and the computing chip 1132 shown in FIG. 1, or may be the computing chip 1131 and the computing chip 1231 shown in FIG. 1. The collective communication system is configured to implement the operation steps of the method performed by a corresponding entity in the foregoing collective communication method. According to the foregoing collective communication system, an amount of data between computing chips can be reduced, and collective communication efficiency can be improved. The system may further reduce a computing speed and improve user experience in a scenario of recommending merchandise to a user.

[0165] All or some of the foregoing embodiments may be implemented using software, hardware, firmware, or any combination thereof. When software is used to implement embodiments, the foregoing embodiments may be implemented completely or partially in a form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, all or some of the processes or the functions according to embodiments of this application are generated. The computer may be a general-purpose computer, a dedicated computer, a computer network, or another programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or may be transmitted from a computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website, computer, server, or data center to another website, computer, server, or data center in a wired (for example, a coaxial cable, an optical fiber, or a digital subscriber line (DSL)) or wireless (for example, infrared, radio, and microwave) manner. The computer-readable storage medium may be any usable medium accessible by a computer, or a data storage device, such as a server or a data center, integrating one or more usable media. The usable medium may be a magnetic medium (for example, a floppy disk, a hard disk, or a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium may be a solid-state drive (SSD).

[0166] The foregoing descriptions are merely specific implementations of this application. Any variation or replacement that can be figured out by a person skilled in the art according to the specific implementations provided in this application shall fall within the protection scope of this application.